Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add throttling to node prefetchers #1030

Merged
merged 3 commits into from
Sep 19, 2024

Conversation

HerbertJordan
Copy link
Collaborator

@HerbertJordan HerbertJordan commented Sep 18, 2024

This PR introduces two improvements to the parallel visit code:

  • throttling of node pre-fetchers: workers stop prefetching nodes beyond a configurable threshold of prefetched nodes; however, since at the time workers are throttled not all needed nodes may have been fetched, the main thread is extended to contribute to the prefetching as needed, this puts a constant upper limit on the memory usage of the visitAll operation;

  • lockstep prefetching of workers: workers will synchronize their activities to avoid prefetching large shares of far-future nodes and thereby exceeding the throttling limit for a long time. By synchronizing prefetch workers to load nodes in blocks, the focus of what is to be prefetched is forced to remain on the near future window.

For the latter, a barrier utility was introduced facilitating the synchronization of workers.

  • complete a full export run for block 88M

@HerbertJordan HerbertJordan force-pushed the herbert/mpt_visit_parallel_throttle_fetch_if_needed branch from 4610234 to dcf635a Compare September 18, 2024 12:44
@HerbertJordan HerbertJordan marked this pull request as ready for review September 18, 2024 13:27
kjezek
kjezek previously approved these changes Sep 19, 2024
Copy link
Collaborator

@kjezek kjezek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice idea of how to enable the throttling, thanks

go/database/mpt/io/parallel_visit_test.go Show resolved Hide resolved
go/database/mpt/io/parallel_visit_test.go Outdated Show resolved Hide resolved
@kjezek kjezek merged commit 5c737f7 into main Sep 19, 2024
5 checks passed
@kjezek kjezek deleted the herbert/mpt_visit_parallel_throttle_fetch_if_needed branch September 19, 2024 11:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants