feat: Modify optimized compaction to cover edge cases #25594
+816
−82
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
devanbenz
force-pushed
the
db/4201/compaction-bugs
branch
from
6e9db1b to
cab638c
Compare
devanbenz
commented
Dec 16, 2024
devanbenz
force-pushed
the
db/4201/compaction-bugs
branch
from
9f5098b to
8c9d7e7
Compare
devanbenz
changed the title
gwossum
reviewed
Dec 16, 2024
gwossum
reviewed
Dec 16, 2024
gwossum
reviewed
Dec 16, 2024
gwossum
reviewed
Dec 16, 2024
gwossum
reviewed
Dec 16, 2024
This PR changes the algorithm for compaction to account for the following cases that were not previously accounted for: - Many generations with a groupsize over 2 GB - Single generation with many files and a groupsize under 2 GB Where groupsize is the total size of the TSM files in said shard directory. closes #25666
devanbenz
force-pushed
the
db/4201/compaction-bugs
branch
from
7de2cf2 to
d631314
Compare
davidby-influx
requested changes
Dec 17, 2024
for shards that may have over a 2 GB group size but many fragmented files (under 2 GB and under aggressive point per block count)
davidby-influx
requested changes
Dec 17, 2024
davidby-influx
requested changes
Dec 18, 2024
| @@ -77,6 +81,9 @@ const ( | |||
| // partition snapshot compactions that can run at one time. | |||
| // A value of 0 results in runtime.GOMAXPROCS(0). | |||
| DefaultSeriesFileMaxConcurrentSnapshotCompactions = 0 | |||
|
|
|||
| // MaxTSMFileSize is the maximum size of TSM files. | |||
|
@davidby-influx I've added checks to verify that I'm not planning PlanOptimize() or FullyCompacted() specific TSM layouts 👍 |
gwossum
reviewed
Dec 18, 2024
gwossum
reviewed
Dec 18, 2024
Comment on lines
+98
to
+103
| // PlanOptimize will return the groups for compaction, the compaction group length, | ||
| // and the amount of generations within the compaction group. | ||
| // generationCount needs to be set to decide how many points per block during compaction. | ||
| // This value is mostly ignored in normal compaction code paths, but, | ||
| // for the edge case where there is a single generation with many | ||
| // files under 2 GB this value is an important indicator. |
var called SingleGenerationReasonText var
Many tsm generations over level 4 compaction single tsm generation under level 4 compaction all in same shard. Group size is over 2 GB for each generation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes the algorithm for compaction to account for the following
cases that were not previously accounted for:
Where groupsize is the total size of the TSM files in said shard directory.