Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-124742: Add the PYTHON_GC_THRESHOLD environment variable. #124743

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

nascheme
Copy link
Member

@nascheme nascheme commented Sep 29, 2024

Some comments of the approach taken here:

  • I want to keep this change as simple as possible and as robust as possible, in case it is deemed safe enough to merge into 3.13. Getting it into 3.13 could increase the amount of useful feedback from real apps, improving our decision making for the 3.14 release. It is quite likely we change things related to this threshold.
  • In the 'main' branch, I plan to make a corresponding -X option for it and to put it into the config structure, so it can be set for embedding scenarios
  • This is a low level "tuning knob" and I think a high level knob would actually be more useful (e.g. PYTHON_GC_STRATEGY) as proposed on the Ideas forum. However, for 3.13 it seems safest to do the simplest possible thing and that's exposing the threshold0 setting to be set by an env var.
  • I didn't raise an error if the env var value is invalid. For the 'main' branch version, I would put that error in, so that an invalid setting doesn't pass silently.

📚 Documentation preview 📚: https://cpython-previews--124743.org.readthedocs.build/

Comment on lines +175 to +181
if (env == NULL || strcmp(env, "default") == 0) {
return;
}
int threshold = -1;
if (_Py_str_to_int(env, &threshold) < 0) {
return; // parse failed, silently ignore
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, why ignore the error? If a user accidentally sets the variable to something that doesn't work, then they might end up very confused as to why Python is ignoring their request.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hoping this change can be included in the 3.13 RC and so I wanted it as simple and robust as possible. In the 'main' branch, I plan to re-work it so it becomes part of the config structure and I would add an error raise there.

Maybe it is safe enough to raise an error here too. That would be consistent with the principle that "errors should not pass silently".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that making it return an error rather than nothing should still remain fairly simple, return will just become return _PyStatus_ERR("...") and whatnot.

I haven't paid too much attention to the incremental GC problem (#124567), but I'm assuming that this is a possible solution for it? That should help it go into the 3.13 release

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I don't think this would be enough for us to feel safe about including incremental GC into 3.13. In retrospect, the incremental GC should have been opt-in for 3.13 and would become default in 3.14 if people had good experience with it.

The point of this env var is to increase the chances that people will test their programs with a higher threshold with 3.13 and then when 3.14 release nears, we would have a better idea about how "aggressive" the default GC tuning should be.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning an error _PyGC_Init results in this kind of error:

Fatal Python error: _PyGC_Init: Invalid PYTHON_GC_THRESHOLD value.
Python runtime state: preinitialized

Current thread 0x00007fe0505d6740 (most recent call first):
  <no Python frame>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that makes sense. We do the same for other env vars.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hoping this change can be included in the 3.13 RC and so I wanted it as simple and robust as possible.

Since we have already reached the feature-freeze stage and this is a new feature, it needs confirmation from @Yhg1s.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right. It's Thomas's call and I completely understand if he doesn't want it. There has already been a bit too much excitement with the RCs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants