[DOCS-155] Add accurate evaluation order documentation#3937
Open
devin-ai-integration[bot] wants to merge 3 commits intomainfrom
Open
[DOCS-155] Add accurate evaluation order documentation#3937devin-ai-integration[bot] wants to merge 3 commits intomainfrom
devin-ai-integration[bot] wants to merge 3 commits intomainfrom
Conversation
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
| 3. **Layer holdouts** — If the experiment is in a layer, layer-level holdout gates are checked | ||
| 4. **Holdout gates** — Experiment-level holdout gates; users in a holdout receive default values | ||
| 5. **Experiment exclusion** — Mutual exclusion segments that prevent users from being in multiple experiments | ||
| 6. **Start status** — If the experiment is not started, users receive default values (with optional non-production environment exceptions) |
Collaborator
There was a problem hiding this comment.
aside: we could get rid of this
or move layer holdout and experiment start status down below to the layers section
| 6. **Start status** — If the experiment is not started, users receive default values (with optional non-production environment exceptions) | ||
| 7. **Layer allocation** — For experiments in a layer, the user's bucket (based on the layer's universe salt) must fall within the experiment's allocated segments. This is checked **before** targeting. | ||
| 8. **Targeting gate** — Users who fail the targeting gate receive default values. This is checked **after** layer allocation. | ||
| 9. **Group assignment** — The user's bucket (based on the experiment salt) determines which group they fall into. Groups are cumulative ranges across 1000 buckets. |
Collaborator
There was a problem hiding this comment.
aside: don't think we need to say Groups are cumulative ranges across 1000 buckets.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a new Evaluation Order section to
sdks/how-evaluation-works.mdxdocumenting the rule evaluation sequence for experiments, layers, holdouts, and gates.Relationship to PR #3933
This PR is intended as a corrected alternative to PR #3933. If this is merged, #3933 should be closed.
Link to Devin run: https://app.devin.ai/sessions/b92306d68a83463d9214e06547d88d49
Requested by: @xhuang-statsig
Best practice checklist
Questions?
Reach out to Brock, T̶o̶r̶e̶,̶ or Logan on Slack!