explicit error terms on Activitysim 1.5.1 #1029
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Fix typos in summarize docstring * Fix log message in util.delete_files * Fix docstring for VehicleAllocationSettings --------- Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
* initial batch of models with pre and post processing added * nonmand sched and transit pass models * first pass at preprocessing and annotate functionality in all models * fixing bugs in jtf and trip purpose * adding persons back in to locals_d in jtc * model name missing in tour scheduling * missing expressions import in tour sched prob * ci unit tests & fixing estimation test error * addressing review comments --------- Co-authored-by: Ali Etezady <58451076+aletzdy@users.noreply.github.com>
* first uv sync * explicitly set np.int64 dtype before calling pd set_index() Pandas 2 got rid of the `Int64Index` class, it was downcasting to int32 for the input land use table * Add all dependencies from Conda environments Consider removing unused ones at a later time * Remove orca This dependency was replaced with “state” for workflow orchestration. * Update release instructions for `uv` * Remove `--no-default-groups` from `uv sync` instructions * Trial updating Github Action to uv instead of conda * Bug fixes * Try downgrading sharrow * change core test to windows runner * use win for all core tests * index type mismatch * formatting * unlock sharrow to 2.14 * correct a typo * Update Model Setup page in user guide * Update Ways to Run the Model page * Remove duplicates * Simplify pyproject.toml for github action and use only this group * Update remainder of user guide docs * Update dev guide [makedocs] * Update lockfile from changes to dependencies * Remove conda from other github actions * Debug doc building [makedocs] * Update install instructions [makedocs] * update installation instructions [makedocs] * notes on uv options [makedocs] * address review comment --------- Co-authored-by: Josie Kressner <josiekressner@gmail.com>
* identify and flag naming conflicts in 2d vs 3d skims * add skims doc * run on workflow_dispatch * add link to docs * write error message in exception also * add error message example * code block formatting * unit tests
* update cat columns in df when new cat exists * blacken * sort two categoricals before union
* minor fixes for SimOR development * blacken * Using PNUM if available in JTFC Required to maintain backwards compatibility in tour indexes in SANDAG models --------- Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
…activitysim version to logfile (ActivitySim#963) * adding activitysim version to log file * fix duplicate columns in parking location choice as described in ActivitySim#633 * making tour mode choice logsum optional in destionation choice models ActivitySim#847 * logsum settings in location choice, trip dest, tour_od * optional logsums in trip destination * blacken --------- Co-authored-by: Ali Etezady <58451076+aletzdy@users.noreply.github.com>
…m#950) * feat: adds initial setting_checker module to abm/models * chore: add .vscode settings to gitignore * feat: adds first test case for prepopulating settings pydantic model (accessibility) * feat: adds initial spec checker - wip * feat: update settings checker to load spec and evaluate coefficients. add second model to validate * fix: adds missing arg to eval_coefficients * feat: adds atworksubtour_frequency to settings checker * feat: moves loading spec and coefficients to independent functions * refactor: renames main settings checker function. moves loading model settings to independent function * refactor: moves load, spec, and coef eval checks to an independent function called from main checker. run formatter * feat: adds load SPEC check for preprocessor settings * chore: run formatter * feat: initial checking for model setting with nested coefficients * feat: adds additional model settings. force try_load_spec to return an empty dataframe, even if no top level spec exists * feat: adds additional model settings to check. adds special handler for disaggregate accessibility (needs more testing) * feat: adds settings check for initalize_landuse * refactor: reorders checking for joint tour composition settings * feat: adds setting check for joint tour destination * feat: adds settings check for joint tour frequency composition * feat: adds settings check for joint tour frequency * feat: adds settings check for joint tour participation * feat: adds setting check for joint tour scheduling * feat: adds settings checks for workplace location and school location * feat: adds setting check for mandatory tour frequency * feat: adds settings check for non mandatory tour destination * feat: adds settings check for parking location choice * feat: adds settings check for school escorting * wip: adds stop_frequencies to components dict, but requires a separate flow for nested specs * feat: adds setting checks for summarize * feat: adds settings check for telecommute frequency * feat: adds setting check for tour mode choice * feat: adds setting check for tour od choice * feat: adds setting check for tour scheduling probabilistic * feat: adds setting check for transit pass ownership * feat: adds setting check for transit pass subsidy * feat: adds settings check for trip departure choice * feat: adds setting check for trip destination * feat: adds setting check for write trip matrices * feat: adds setting check for trip mode choice * feat: adds settings checks for trip purpose and destination * feat: adds setting check for trip purpose * feat: adds setting check for vehicle allocation * feat: adds setting check for vehicle type choice * feat: adds setting check for work from home * docs: remove one comment * refactor: removes spurious debug point around cdap spec load * refactor: refactors setting checking to separate read/evaluate. checks now return result of tuple (Dataframe, Exception | None) for collection of errors. * feat: adds setting check for intialize households * feat: adds setting check for mandatory and non-mandatory tour scheduling * feat: adds partial setting check for non-mandatory tour frequency (not fully implemented due to NESTED_SPEC) * refactor: raise RunTimeError instead of custom * feat: adds module-specific file handler to logging * docs: remove one comment * refactor: add more specific return type * refactor: renames try_load_and_check to try_load_and_check_spec_coefs and directly injects model settings, separing check for reading settings yaml files. useful for checking nested model settings * feat: adds initial handling for SPEC_NEST configs. includes some prequisite refactors to error collection * refactor: fix logging * docs: update one comment * refactor: remove commented raise statements used for debugging * chore: run black formatter * refactor: only ever evaluate spec/coef if both are available * chore: better logging in model settings load * feat: make setting checker optional * feat: adds setting check for input checker * feat: adds setting check for shadow pricing yaml * chore: remove some outdated comments * feat: adds setting checks for intialize los * refactor: changes return type from empty df -> None when spec/coefs are unavaialable * feat: adds logic for checking settings with templated coefficients * chore: run formatter * chore: fix alphabetization of components registry * feat: check settings for write_data_dictionary * feat: allow for detailed checks for SPEC_SEGMENTS with PTYPE against main model spec * feat: allow for arbitrary loading of spec files from any subsettings in model settings * feat: allow custom checks for unusual spec/coefficient pairs * feat: adds custom SettingCheckerError exception for improved logging * refactor: better checks and warnings when SPEC or COEFFICENT fields are missing values. because the fields in the settings classes are inherited, it is not usually possible for the settings checker to determine if a path to an external file is actually required for a particular model component. the workaround is to issue a specific warning that a path *may* be expected and users should check the YAML file. the ultimate solution would be to define more robust Pydantic data models that ensure that fields are marked as required when appropriate. * chore: fix spelling. * refactor: renames COMPONENTS_TO_SETTINGS -> CHECKER_SETTINGS and feeds to checker entrypoint as argument. * refactor: single entry for settings checker. import extension check settings as dict. * chore: run black formatter * refactor: remove commented code * fix: skip errors for write_data_dictionary if optional yaml file is not loadable * chore: run black formatter * fix(tests): remove unused COEFFICIENTS from MWCOG test model * fix(tests): re-add deleted test_mtc.py test script * fix: make model registry for checking a copy * fix(test): remove unused general SPEC in non_mandatory_tour_scheduling.yaml from MWCOG example model --------- Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
* expression profiling * two reports * make simple report sortable * change default to grid * configurable expression_profile_cutoff * works with MP * documentation * expression_profile_style * performance_log can be bool * fix for pandas 2 * test expr profiling * test expression profiling on mtc and semcog examples * set encoding to utf8 because windows * add missing import * add warning about path length in error message * oops, .log not .csv * add timestamping to expression profile logs * fix file naming, update tests * all subprocesses to same timestamp * add note in docs about runtime cost
…Sim#977) * Expression profiling (legacy mode) (ActivitySim#936) * expression profiling * two reports * make simple report sortable * change default to grid * configurable expression_profile_cutoff * works with MP * documentation * expression_profile_style * performance_log can be bool * fix for pandas 2 * test expr profiling * test expression profiling on mtc and semcog examples * set encoding to utf8 because windows * add missing import * add warning about path length in error message * oops, .log not .csv * add timestamping to expression profile logs * fix file naming, update tests * all subprocesses to same timestamp * add note in docs about runtime cost * manual rebuild of docs * add permissions
* add explicit telecommute model * Update explicit_telecommute.py * Revert "Update explicit_telecommute.py" This reverts commit 54bc4cc. * get work and home zone variable from workplace location model * run explicit telecommute before cdap * set workplace zone to -1 for person who is telecommuting * keep workplace location unchanged * format code with Black * documentation * change the model name to telecommute status * update doc * update doc * Update docs/dev-guide/components/telecommute_status.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update docs/dev-guide/components/telecommute_status.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * alphabetical order * address review comments --------- Co-authored-by: Yue Shuai <48269801+yueshuaing@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* initial batch of models with pre and post processing added * nonmand sched and transit pass models * first pass at preprocessing and annotate functionality in all models * fixing bugs in jtf and trip purpose * adding persons back in to locals_d in jtc * model name missing in tour scheduling * missing expressions import in tour sched prob * ci unit tests & fixing estimation test error * addressing review comments * fixing typo in location alts preprocessor names --------- Co-authored-by: Ali Etezady <58451076+aletzdy@users.noreply.github.com>
Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.5.1 to 1.13.0. - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.5.1...v1.13.0) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.13.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix branch on sandag external test * locked * group not only-group * test in dir * rootdir for pytest * test pointer * move uv * one layer off
* multiprocess initial commit * blacken * parquet format for EDBs * adding pkl, fixing edb concat and write * fixing double naming of coefficient files * blacken * fixing missing cdap coefficients file, write pickle function * combact edb writing, index duplication, parquet datatypes * sorting dest choice bundles * adding coalesce edbs as its own step * CI testing initial commit * infer.py CI testing * estimation sampling for non-mandatory and joint tours * adding survey choice to choices_df in interaction_sample * adding option to delete the mp edb subdirs * changes supporting sandag abm3 estimation mode * running test sandag example through trip dest sample * Estimation Pydantic (ActivitySim#2) * pydantic for estimation settings * allow df as type in config * fix table_info * repair for Pydantic * df is attribute * Estimation settings pydantic update * new compact formatting * handling multiple columns for parquet write * dropping duplicate columns * actually removing duplicate columns * dfs with correct indexes and correct mp sorting * ignore index on sort for mp coalesce edbs * updating estimation checks to allow for non-zero household_sample_size * Re-estimation (ActivitySim#3) * pydantic for estimation settings * allow df as type in config * fix table_info * auto ownership * repair for pydantic * update for ruff * updated for simple models * repair for Pydantic * simple simulate and location choice * df is attribute * scheduling * stop freq * test locations * cdap * nonmand_and_joint_tour_dest_choice * nonmand_tour_freq * fix ci to stop using mamba * test updates * use larch6 from pip * use numba for stop freq * fix for pandas 1.5 * fix stop freq test for numba * Sharrow Cache Dir Setting (ActivitySim#893) * setting necessary filesystem changes from settings file * set for multiprocessing * repair github actions * github action updates (ActivitySim#903) * script to make data * unified script for making data * remove older * bug * doc note * load from parquet if available * add original alt ids to EDB output when using compact * fix MP race * script arg to skip to EDB * clean up CDAP and blacken * refactor model_estimation_table_types change to estimation_table_types, to avoid pydantic namespace clash * repair drop_dupes * blacken * location choice with compact * choice_def for compact * spec changes for simple-simulate * re-estimation demo for auto ownership * clean up status messages * change name to stop pydantic warnings * edit configs * default estimation sample size is same as regular sample size * allow location alts not in cv format * dummy zones for location choice * update scheduling model estimation * various cleanup * stop freq * tidy build script * update 02 school location for larger example * update notebook 04 * editable model re-estimation for location choice * fix test names * update notebooks * cdap print filenames as loading * notebook 07 * tests thru 07 * notebooks 08 09 * build the data first * runnable script * change larch version dependency * keep pandas<2 * notebooks 10 11 * notebook 12 * remove odd print * add matplotlib * notebook 13 14 * test all the notebooks * add xlsxwriter to tests * notebook 15 * CDAP revise model spec demo * notebook 16 * notebook 17 * longer timeout * notebook 18 * notebook 19 * notebook 20 * smaller notebook 15 * configurable est mode setup * notebook 21 * notebook 22 * config sample size in GA * notebook 23 * updates for larch and graphviz * change default to compact * compare model 03 * test updates * rename test targets * repair_av_zq * move doctor up * add another repair * oops --------- Co-authored-by: David Hensle <51132108+dhensle@users.noreply.github.com> * Removing estimation.yaml settings that are no longer needed * fixing unit tests, setting parquet edb default * one more missed estimation.yaml * using df.items for pandas 2 compatibility * tidy doc * updating edb file name for NMTF * updating numba and pandas in the conda env files * Improve test stability (ActivitySim#4) * handle dev versions of Larch * test stability * pin multimethod < 2.0 * add availability_expression * starting est docs * Resolve package version conflicts (ActivitySim#923) * limit multimethod version to 2.0 and earlier * add multimethod version to other settings * [makedocs] update installer download link * [makedocs] update branch docs * GitHub Actions updates (ActivitySim#926) * use libmamba solver * add permissions [makedocs] * add write permission for dev docs [makedocs] * conda-solver: classic * trace proto tables if available, otherwise synthetic population (ActivitySim#901) Co-authored-by: Jeffrey Newman <jeff@driftless.xyz> * release instructions (ActivitySim#927) * use libmamba solver * add permissions [makedocs] * add write permission for dev docs [makedocs] * conda-solver: classic * include workflow dispatch option for tests * update release instructions * add installer build to instructions * Pin mamba for now, per conda-incubator/setup-miniconda#392 * conda-remove-defaults * when no unavailability parameters are included * some general estimation docs * Use pandas 2 for docbuild environment (ActivitySim#928) * fix link * allow failure to import larch * workflow * blacken * try some pins * speed up docbuild * use pandas 2 for docs * oops wrong file * restore foundation * Update HOW_TO_RELEASE.md * refactor(shadow_pricing.py): remove a duplicated `default_segment_to_name_dict` (ActivitySim#930) * fix typo * fixing disaggregate accessibility bug in zone sampler * Revert "fixing disaggregate accessibility bug in zone sampler" This reverts commit be5d093. * notes on size terms * clean up docbuild * fix version check * add some doc * tidy * estimation docs * more on alternative avail * model evaluation * add doc on component_model * documentation enhancements * larch6 is now larch>6 * branch docs on workflow_dispatch * missing doc section on model respec --------- Co-authored-by: Yue Shuai <48269801+yueshuaing@users.noreply.github.com> Co-authored-by: David Hensle <51132108+dhensle@users.noreply.github.com> Co-authored-by: amarin <17020181+asiripanich@users.noreply.github.com> Co-authored-by: Ali Etezady <58451076+aletzdy@users.noreply.github.com> Co-authored-by: Sijia Wang <wangsijia0628@gmail.com> * handling missing data or availability conditions * add docs on locking size terms * include constants in CDAP * bump larch requirement * require larch 6.0.40 * add xlsxwriter to envs * require larch 6.0.41 * add links * fix typos and formatting * cdap hh and per parquet read match csv * add missing x_validator for mode choice and nonmand tour freq * add tour mode choice edit example * add to docs * union not addition on sets * restore nb kernel * blacken * replacing conda with uv in estimation tests * add requests to github-action dependencies * running with created virtual env instead * Fix estimation notebook tests (ActivitySim#8) * Update scheduling.py --------- Co-authored-by: Jeffrey Newman <jeff@driftless.xyz> Co-authored-by: Yue Shuai <48269801+yueshuaing@users.noreply.github.com> Co-authored-by: amarin <17020181+asiripanich@users.noreply.github.com> Co-authored-by: Ali Etezady <58451076+aletzdy@users.noreply.github.com> Co-authored-by: Sijia Wang <wangsijia0628@gmail.com>
…ctivitySim#979) * Initial plan * Wrap mem.consolidate_logs in try-except to prevent model run crashes Co-authored-by: jpn-- <1036626+jpn--@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jpn-- <1036626+jpn--@users.noreply.github.com> Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
* Initial plan * Add CITATION.cff file for academic referencing Co-authored-by: jpn-- <1036626+jpn--@users.noreply.github.com> * Change copyright holder in LICENSE file Updated copyright holder from Association of Metropolitan Planning Organizations Research Foundation to Zephyr Foundation. * Update README to remove status badges Removed Build and Coverage Status badges from README, and add link to Zephyr page. * Update CITATION.cff per review feedback Co-authored-by: jpn-- <1036626+jpn--@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jpn-- <1036626+jpn--@users.noreply.github.com> Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
* update uv lock file * strip github-action group, only use dev * update coveralls * update uv lock to larch 6.0.42 * begin to remove placeholder_sandag example
* fix for num_choosers < chunk_size * update assert logic
* Downgraded logging of which expression is being evaluated from "info" level to "debug" level * Downgraded column list before and after dropping unused columns from "INFO" to "DEBUG" * Downgraded reporting of model component results from "INFO" to "DEBUG" * Downgraded a few log messages in settings checker from "INFO" to "DEBUG" * Changed default value of `level` argument in flow.TimeLogger.summary() from 20 to 10 so that it's a DEBUG message by default * Downgraded logging statements when merging the choosers and alternatives from "INFO" to "DEBUG" * Downgraded runtimes for flow evaluations from "INFO" to "DEBUG" * Downgraded "sharrow_enabled is True/False" from "INFO" to "DEBUG" as it's repetitive and will likely only be of concern during development/debugging * Downgraded utility data types reporting from "INFO" to "DEBUG" * Downgraded all instances of logger.info to logger.debug in activitysim.core.chunk * Downgraded line reporting how many rows are being run for eval_interaction_utilities * Downgraded several (primarily repetitive) statements in activitysim.abm.models.util * Downgraded messages reportng how long each of the sampling, logsums, and simulation in trip destination choice from "INFO" to "DEBUG" * Downgraded more detailed messages from "INFO" to "DEBUG" in trip destination * Reclassified some logging statements * Swapped logging statement classifications to get desired statement printed * Downgraded statement from "INFO" to "DEBUG" that wasn't providing much information * blacken * Added documentation on logging levels --------- Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
* Added new exceptions based on RuntimeError inventory results * Changed RuntimeErrors in abm\models * Corrected name of SystemConfigurationError * Updated RuntimeErrors in abm\tables * Added SubprocessError * Updated RuntimeErrors in core directory * Renamed `TableSliceError` to `TableSlicingError` * Classified RuntimeErrors in input and settings checkers as ModelConfigurationErrors * Created `SegmentedSpecificationError` for when there's an issue with a spec table that's created for a specific segment * Added TableIndexError * Classified RuntimeError in initialize_tours as InputPopulationError * Created EstimationDataError * Improved clarity of a couple error messages * Blacken code * Updated code to reflect older version of Black (22.12.0) * Updated error messages to look for in activitysim.core tests * Updated error to look for in abm.text * Changed SystemConfigurationError to TableSlicingError in test * Reclasified error when a step is run more than once from TableSlicingError to DuplicateWorkflowNameError * Renamed InputPopulationError to InputTableError for clarity as it can be raised if tours are being input --------- Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
…m#1011) * Initial plan * Replace deprecated pkg_resources with importlib.resources Co-authored-by: jpn-- <1036626+jpn--@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jpn-- <1036626+jpn--@users.noreply.github.com>
* Expand UV installation and usage instructions * disable auto upgrade * Update modelsetup.rst * address comments
…ations (ActivitySim#1012) * Update front page of docs [makedocs] * formating [makedocs] * re-format [makedocs] * clean up [makedocs] * clarify language [makedocs] * add oregon
* descriptive error * allow partial success in setting up skims - if there is no time period we should still be able to get time-agnostic values * comments for better context on why * require sharrow 2.15 * update lock file * blacken * update docstring * address whitespace
…imation mode in joint tour participation (ActivitySim#1008) * fix(jtp): Correct the participant_num assignment logic for estimation mode in joint tour participation * fix(jtf): Handle missing PNUM column by selecting the first person in the household * style: blacken
…ss (ActivitySim#1005) * single process run with * ensured schedule order when generating alternatives * trip scheduling choice test fix * scheduling choice fix with linting * trip departture choice test fix * deleted extra outputs folder * restoring deleted arc test files * deleting extra added files * more file cleanup * removing arc from CI test list * improved testing (ActivitySim#10) * improved testing * blacken * stable sorting * blacken * addressing review comments --------- Co-authored-by: juangacosta <juan.acosta@rsginc.com> Co-authored-by: Jeffrey Newman <jeff@driftless.xyz>
* Implemented hacky attempt at getting subprocesses to have the same run_id as the whole model run * Reverted previous commit * Moved generation of Run ID to cli/run.py, initialized states with None for RunID, and set the RunID after the state is initialized * pass run_id to subprocesses * Added CI testing for trace ID hashes * Added test_trace_id.py to list of CI tests to run * Renamed trace ID test so that it will actually run * blacken * blacken 'test' directory --------- Co-authored-by: Jeff Newman <jeff@driftless.xyz>
# Conflicts: # activitysim/core/configuration/top.py # activitysim/core/interaction_sample.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
VLC is doing some work with DTP Victoria to test their activitysim model with EET functionality. They've recently rebased to 1.5.1, and so need EET working with the 1.5.1 code for this test. As I understand the consortium is still interested in pursuing developments with EET, I thought we could share this rebasing in case it's a useful starting point.
I had originally intended to keep this on our fork for the moment with the intent of opening a pull request here, but I've inadvertedly created one on the upstream already. Given this is built on work already done by jan across several commits and years, I've opted to merge up to date with main, rather than rebase my way through all the historical conflicts.
Note that the diff here is very large - you'll see a much cleaner diff with the base set to ActivitySim:main instead of ActivitySim:explicit_error_terms, but the intention here was to try and update that branch, rather than supercede it.