feat: implement background compaction for Lance fragments (#16) #28
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements #16 - Background compaction for Lance fragments with comprehensive features:
compact()methodChanges
Rust Core:
CompactionConfigandCompactionStatstypescompact(),should_compact(),compaction_stats()methodsPython API:
Context.create()Tests:
Usage Example
Manual Compaction:
```python
ctx = Context.create("context.lance")
for i in range(100):
ctx.add("user", f"message {i}")
metrics = ctx.compact()
print(f"Removed {metrics['fragments_removed']} fragments")
```
Background Compaction:
```python
ctx = Context.create(
"context.lance",
enable_background_compaction=True,
compaction_interval_secs=300,
compaction_min_fragments=10,
quiet_hours=[(22, 6)], # 10pm-6am
)
```
Check Status:
```python
stats = ctx.compaction_stats()
print(f"Fragments: {stats['total_fragments']}")
print(f"Last compaction: {stats['last_compaction']}")
```
Test Results
```
10 passed in 5.39s
✅ Manual compaction reduces fragments
✅ Data integrity preserved
✅ Concurrent writes work
✅ Compaction stats accurate
✅ Custom options work
✅ Background compaction triggers
✅ Quiet hours respected
✅ Metrics structure correct
✅ Empty context handled
✅ Multiple compactions work
```
Architecture
Checklist