Add global option to skip households on simulation failure

[i-am-sijia]

This PR addresses issue #892.

Status: Draft - core functionality implemented, remaining: logging and documentation. Update: logging and documentation are added.

Summary

Adds a new skip_failed_choices setting that allows ActivitySim to skip households that cause choice model failures instead of being masked and forced a choice by the overflow_pretection or crashing the entire simulation. This improves robustness for large-scale production runs where occasional edge cases shouldn't halt the model.

Key Changes

Core Logic (activitysim/core/)

• Modified logit.py to temporarily mark failed choices with -99 and track skipped households in state
• Added functionality in state.py to dynamically drop data that belong to the skipped household. Further updated the logic to only trigger this when new households are being skipped.
• Updated simulate.py and interaction_sample_simulate.py to handle failed choices
• Enhanced report_bad_choices() to log skipped households
• Added code in simulate.py to automatically print out debug tracing for one failed choice per model component, limited to simple simulate models only.

Configuration

• Added skip_failed_choices: bool = True setting in top.py
• Added fraction_of_failed_choices_allowed = 0.1 setting in top.py. Users can optionally specify different values in the settings.yaml to set the threshold of failed/skipped households (relative to sample households) above which the run should terminate.
• Added code in cli/run.py to print out a summary of skipped households by model component at the end of the run.

Model Updates (activitysim/abm/models/)

• Retained household_id in chooser columns across location choice models for tracking
• Modified shadow price calculations to exclude skipped households
• Updated trip matrices to adjust trip matrix expansion factor. The adjustment is based on the remaining households' weight vs the initial sampled households' weight.
• Added null trip mode handling in trip mode choice and school escorting

Testing

• Added new unit test test/skip_failed_choices to deliberately trigger failures and validates households being skipped.
• Added tests with different threshold of skipped households.

Documentation

• Added documentation in docs/users-guide/modelsetup.rst and docs/users-guide/ways_to_run.rst.

Usage

# settings.yaml
skip_failed_choices: true #default
fraction_of_failed_choices_allowed: 0.1 #default

Access skipped household info via:

• state.get("num_skipped_households", 0)
• state.get("skipped_household_ids", dict())

Final count logged at end of simulation in the activitysim.log.

Notes

• Defaults to True for production robustness
• Disables overflow protection when active to surface underlying issues
• Temporary code adds household_id to chooser columns (remove when PR Access to global constants and removing chooser col filtering #1017 merges)

[i-am-sijia]

For components where we intentionally allow some choices to fail without being skipped, the allow_zero_probs option is already set to True, which exempts them from this new feature. Therefore, it may not be necessary to introduce this as a component-level setting. In addition, keeping it as a global setting helps reduce manual errors, avoid confusion, and prevent inconsistencies across the implementation.

[Copilot]

Pull request overview

This PR adds a new skip_failed_choices feature that allows ActivitySim to skip households causing choice model failures rather than crashing the simulation or forcing a choice through overflow protection. This is designed to improve robustness in large-scale production runs where occasional edge cases shouldn't halt the entire model.

Key Changes:

• Core logic modified to mark failed choices with -99, track skipped household IDs, and dynamically remove skipped household data from state tables
• New skip_failed_choices setting added (defaults to True) with overflow protection disabled when active
• Model updates to handle null modes, adjust trip matrices weights, exclude skipped households from shadow pricing, and retain household_id for tracking

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
activitysim/core/logit.py	Adds logic to mark failed choices with -99, track skipped households in state, and conditionally skip vs. raise errors
activitysim/core/workflow/state.py	Implements update_table() to drop rows belonging to skipped households from all state tables
activitysim/core/simulate.py	Updates eval_nl to pass skip_failed_choices flag to report_bad_choices, adds trace_choosers parameter
activitysim/core/interaction_sample_simulate.py	Clips positions to 0 when skip_failed_choices enabled to prevent index errors from -99 values
activitysim/core/util.py	Always includes household_id in columns for tracking skipped households
activitysim/core/configuration/top.py	Adds skip_failed_choices boolean setting (defaults to True)
activitysim/abm/models/location_choice.py	Filters skipped households from shadow price calculations, adds household_id retention
activitysim/abm/models/trip_matrices.py	Adjusts household expansion weights for skipped households, logs final skip count
activitysim/abm/models/trip_mode_choice.py	Handles null trip modes from failed choices, filters skipped households from merged trips
activitysim/abm/models/school_escorting.py	Adds household_id to chooser columns
activitysim/abm/models/util/tour_*.py	Adds household_id to chooser columns in tour scheduling, OD, and destination functions
activitysim/abm/models/util/school_escort_tours_trips.py	Allows null trip modes when skip_failed_choices enabled
test/skip_failed_choices/test_skip_failed_choices.py	New test that deliberately triggers failures and validates skipping behavior
.github/workflows/core_tests.yml	Adds new skip_failed_choices test to CI workflow

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jpn--]

Mostly looks fine. We discussed (possibly while @i-am-sijia was out) that the skipping should not be unlimited; the idea is we want to skip problems when problems are few and probably not seriously consequential. If the problems are many there's little benefit of allowing a simulation to run to completion.

[jpn--]

Rather than just being OK here, code should:

1. emit a warning that N choices have failed, and
2. check if N exceeds configured failure thresholds (e.g. not more than 1% of all choices), in which case an error should be raised.

[i-am-sijia]

Added the warning for item 1. For item 2, I implemented the check in core.workflow.state.update_table(), immediately after the households are being dropped to make sure it raises an error when N (or the weighted N) exceeds the threshold. Implementing the check in the state instead of in model components makes the code base clean, i.e., less code duplication. The global parameter of the threshold is added in core.configuration.top.fraction_of_failed_choices_allowed.

[jpn--]

Should check here how many choices are getting dropped, and

1. emit a warning that N choices have failed, and
2. check if N exceeds configured failure thresholds (e.g. not more than 1% of all households), in which case an error should be raised.

[i-am-sijia]

Both the warning and the check have been implemented in core.workflow.state.update_table(). I implemented them in the state.py instead of in model components to make the code base clean, i.e., less code duplication.

[jpn--]

logging level for this should be higher than "info"; at least warning if not "error" level.

[i-am-sijia]

I've moved the summary of number of skipped households, total and total by model component, to cli.run.run(), which is a better home than trip_matrices.py.

[jpn--]

Should the adjustment to expansion weights be based not on the number of skipped households, but instead on the original expansion weight total and the total expansion weight of the households we dropped? E.g. if we drop a household with a giant expansion weight that's a bigger thing than dropping a household with a smaller one.

[i-am-sijia]

I've updated the adjustment to expansion weights to be based on the weight of households instead of counts. I agree with you in theory that it should be based on weights. However, I don't think it will make a difference in application. Here's why. In simulation, 99% of the time ActivitySim has the same sample rate for all households and the same expansion weight (calculated as 1/sample rate), this is especially true when ActivitySim uses an input synthetic population with weight of 1 for each record implicitly built in. In contrast to simulation, in estimation, the input households are very likely to have different weights, however, we don't use weights when estimating a model, and trip_matrices step is usually skipped in the estimation mode.

[jpn--]

Need additional setting[s] to set thresholds for how many skips are OK and when it's too many and should be an error.

[i-am-sijia]

Added fraction_of_failed_choices_allowed as a global parameter and implemented the checking in core.workflow.state.update_table()

[jpn--]

logger level here should be warn not debug

[i-am-sijia]

Updated to warning for all loggings of skipping households.

[i-am-sijia]

@jpn-- Thank you for your review and comments! I've made further updates. Primary changes include:

• Added a fraction_of_failed_choices_allowed parameter that users can optionally specify in the settings.yaml to set the threshold of failed/skipped households (relative to sample households) above which the run should terminate. I updated the test to incorporate this as well.
• Added code to summarize and print out number of skipped households by model component at the end of the run.
• Added code to update the state tables only when there're new households skipped, improving efficiency.
• Updated code to adjust trip matrix expansion factor using household weight instead of count. For this item, please see my response comment to your original comment.
• As requested by @bhargavasana, I added a feature to automatically trace one failed choice per model component. This turned out to be more complex than I expected. I couldn't find an clean way to do it without significant refactoring of the current tracing mechanism and the utility calculation. I managed to implement a workaround for simple simulate models by dynamically adding trace IDs to the tracing object, and re-simulate a single failed choice to trace the utility calculation with minimal runtime impact runtime. For the interaction simulate models, users can manually add the failed household ID to trace_hh_id for debug tracing.
• Added documentation in the user guide.