Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First memory profile in a process is empty #78

Open
TheJokr opened this issue Oct 18, 2024 · 1 comment
Open

First memory profile in a process is empty #78

TheJokr opened this issue Oct 18, 2024 · 1 comment

Comments

@TheJokr
Copy link

TheJokr commented Oct 18, 2024

Foundations lazily initializes its memory profiling infrastructure on the first request to the telemetry server. This happens here:

pub(super) fn profiler(settings: Arc<TelemetrySettings>) -> Result<MemoryProfiler> {
MemoryProfiler::get_or_init_with(&settings.memory_profiler)?.ok_or_else(|| {
"profiling should be enabled via `_RJEM_MALLOC_CONF=prof:true` env var".into()
})
}
pub(super) async fn heap_profile(settings: Arc<TelemetrySettings>) -> Result<String> {
profiler(settings)?.heap_profile().await
}

MemoryProfiler::get_or_init_with initializes the singleton PROFILER with memory_profiler::init_profiler. Inside the latter function, jemalloc's prof.reset control is used to set the configured sampling rate:

control::write(control::PROF_RESET, settings.sample_interval as u64)
.map_err(|e| BootstrapError::new(e).context("failed to set sample interval"))?;

The problem is that prof.reset also resets the profiling data collected so far. Therefore, the first requested profile is empty instead of exposing data since the process started. As a user, this is highly unexpected especially because jemalloc was already collecting the data. I suggest that we perform this setup as part of the telemetry::init call early in the program startup.

@TheJokr
Copy link
Author

TheJokr commented Oct 18, 2024

Just noticed that this likely already is fixed by 25fd859 in 4.0.0, just need to upgrade the library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant