Speed up removing old orders and quotes#4151
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
| .is_some_and(|data| data.user_valid_to < i64::from(min_valid_to)); | ||
|
|
||
| current_quotes.retain(|uid, _| current_orders.contains_key(uid)); | ||
| if expired { |
There was a problem hiding this comment.
Not sure I am following. Previously the filter was !expired && !invalidated && !onchain_error && !fulfilled for both current and next orders. With this change, this query works only for next orders, and current orders are only checked for expiration. Is that correct? Did I miss something, or do we now ignore the !invalidated && !onchain_error && !fulfilled filter for the current orders?
There was a problem hiding this comment.
The only filter reason that can change without the DB query telling us about it is expired that's why that's the only check we always run on the entire set of orders.
expired, invalidated, and fulfilled can only change if the DB gives us new data about the orders (e.g. fulfilled can only change when we index a new trade event) which is why we only have to run those checks on the small subset of orders we get from the DB.
There was a problem hiding this comment.
Ah, makes sense, thanks!
3 participants
Description
The incremental solvable orders query so far blindly inserted updated orders in to the set of open orders and then ran a filtering step over ALL open orders and quotes separately.
This is pretty wasteful since the new/updated orders are significantly fewer than the set of ALL orders and 3 of 4 things we check we only need to check on the new/updated orders.
Also we don't have to go over ALL quotes if we just remove the quotes together with the orders.
Changes
.retain()on ALL quotes.retain()on the open orders to find expired orders as the incremental orders query will not flag those - while we still scan the whole list twice the most complicated checks could be moved to the part that only runs on the updated orders so this.retain()still is a lot faster than before.How to test
Measured performance with tempo
Retain orders went from 2.5ms to 283µs
and we dropped the 6.6ms from retaining quotes completely
Before
After
