Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gc_compaction: refactor CompationOptions <-> CompactionTask #10031

Open
Tracked by #9114
skyzh opened this issue Dec 5, 2024 · 0 comments · May be fixed by #10221
Open
Tracked by #9114

gc_compaction: refactor CompationOptions <-> CompactionTask #10031

skyzh opened this issue Dec 5, 2024 · 0 comments · May be fixed by #10221
Assignees
@skyzh
Labels
t/bug Issue Type: Bug

Comments

@skyzh
Copy link
Member

skyzh commented Dec 5, 2024

These two things are used in a mixing way for now. In general, we should:

  • comapct_with_gc (and in the future, the legacy compaction function) should take a CompactionTask which deterministically describe a compaction task (i.e., the LSN range, the L0 files to be included, etc.)
  • the compact iteration task generates such compaction tasks and schedules them.
@skyzh skyzh added the t/bug Issue Type: Bug label Dec 5, 2024
@skyzh skyzh self-assigned this Dec 5, 2024
This was referenced Dec 5, 2024
Merged
Open
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 12, 2024
@skyzh @problame
feat(pageserver): gc-compaction split over LSN (#9900)
5ff4b99 
## Problem

part of #9114, stacked PR
over #9897, partially
refactored to help with
#10031

## Summary of changes

* gc-compaction takes `above_lsn` parameter. We only compact the layers
above this LSN, and all data below the LSN are treated as if they are on
the ancestor branch.
* refactored gc-compaction to take `GcCompactJob` that describes the
rectangular range to be compacted.
* Added unit test for this case.

---------

Signed-off-by: Alex Chi Z <chi@neon.tech>
Co-authored-by: Christian Schwarz <christian@neon.tech>
@skyzh skyzh linked a pull request Dec 20, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@skyzh skyzh

Labels
t/bug Issue Type: Bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(pageserver): add automatic trigger for gc-compaction neondatabase/neon
1 participant
@skyzh