feat(pageserver): support schedule gc-compaction

[skyzh]

Problem

part of #9114

gc-compaction can take a long time. This patch adds support for scheduling a gc-compaction job. The compaction loop will first handle L0->L1 compaction, and then gc compaction. The scheduled jobs are stored in a non-persistent queue within the tenant structure.

This will be the building block for the partial compaction trigger -- if the system determines that we need to do a gc compaction, it will partition the keyspace and schedule several jobs. Each of these jobs will run for a short amount of time (i.e, 1 min). L0 compaction will be prioritized over gc compaction.

Summary of changes

• Add compaction scheduler in tenant.
• Run scheduled compaction in integration tests.
• Change the manual compaction API to allow schedule a compaction instead of immediately doing it.
• Add LSN upper bound as gc-compaction parameter. If we schedule partial compactions, gc_cutoff might move across different runs. Therefore, we need to pass a pre-determined gc_cutoff beforehand. (TODO: support LSN lower bound so that we can compact arbitrary "rectangle" in the layer map)
• Refactor the gc_compaction internal interface.

[github-actions]

7040 tests run: 6731 passed, 0 failed, 309 skipped (full report)

---

Flaky tests (2)

Postgres 16

• test_pageserver_gc_compaction_smoke: release-x86-64

Postgres 15

• test_isolation[None]: release-arm64

Code coverage* (full report)

• functions: 31.4% (8324 of 26499 functions)
• lines: 47.8% (65454 of 137023 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
7a0274c at 2024-12-05T19:49:41.936Z :recycle:}

[problame]

It would be nice to add a management API for clearing the scheduled compactions list.
I guess it can be achieved through tenant reset API, though.

---

The code in fn compaction_iteration is quite messy. I think an abstraction to separate policy (what needs doing next) from the machinery to execute it (calling compact_with_options) would be appropriate.

For example:

• comapction_iteration calls that abstraction
• the abstraction yields aScheduledCompactionTask
• the compaction_iteration invokes compact_with_options to execute

WDYT? Doesn't have to be done in tis PR but I'm unwilling to tolerate the messiness in the longer term.
Feel free to create a new issue that references this proposal.

[skyzh]

the abstraction yields aScheduledCompactionTask

Yes I think we need a separation from compact options from compact task. We already have GcCompactionTask in compaction.rs. I'll do a refactor to separate them out after the patches go through.

[skyzh]

I've created an issue to track that and I'll get it done by the next Wednesday to avoid having tech debt in the code #10031