Why not auto-delete "src/PACKAGE_NAME.egg-info/" and "build/"

[VladimirFokow]

Hi, I'm a newbie to building python projects. I've read the setuptools User Guide. I'd like to know why not auto-delete the artifacts.

1. This discussion from the pip repo says:

Is there a way to disable the creation of the egg file inside my src folder or automatically delete it when using
pip install -e '.[dev]'

[Reply there]:
The egg directory is created by the build backend you use (probably setuptools, most backends I know of don’t do that), and you’d want to talk to them about this instead.

---

2. From this discussion, we learn:

The .egg-info directory is a cache artifact.
I had a hard time finding any documentation about this directory at all.

---

3. In this stackoverflow post: Why does "python setup.py sdist" create unwanted "PROJECT-egg.info" in project root directory?

an answer reads:

You may safely delete the directory after your build has completed.

however another upvoted answer right there says:

The -egg.info folder ISN'T ALWAYS a temporary artifact you can delete.

---

4. This old issue suggests writing custom code in setup.py, but I know that using setup.py is an old approach which shouldn't be used anymore. Of course, with hacking anything is possible, but I'm interested in the best-practices and authoritative methods.

---

My observations and questions:

Assuming the src-layout:

• Each of pip install -e ., pip install . and python -m build create "src/PACKAGE_NAME.egg-info/"
  
  If I delete this directory - my package continues to be importable and works fine.
  (maybe I didn't catch where it breaks, but for now it seems that it is completely unnecessary for anything)

• In addition, pip install . also creates "build/"
  
  If I delete this directory - the package continues to work fine.
  As I understand, pip install . copies our package to "site-packages". So why would we need the "dist" with the same contents, and not in the "site-packages"?

Please help me clear things up:

Why are these files created? - what's their purpose, if I can safely delete them? Can / should I?
Why would anyone need to use such "build artifacts"? To me, they seem to only pollute the source code repository.
If they are not needed, why not auto-delete them immediately?

It would be great if these topics are also covered in the excellent User guide!

Or is the only way to NOT create them - use another build backend (instead of setuptools)?
Is setuptools not the recommended build backend? - which alternative could you suggest then? (there are too many of them to learn all)

[webknjaz]

It's likely because it contains structured metadata that may be useful for other tools somebody may use in tandem. Also, editable installs are based on linking the project from site-packages/ and the metadata should still exist somewhere. In editable mode things are not copied over there so the project directory is that place where it remains.

Regardless of whether you delete this metadata, you shouldn't commit it into Git. So a common practice is to exclude it via .gitignore.

[webknjaz]

Reopened. For PyPUG, participate in https://github.com/pypa/packaging.python.org instead.

As for a different back-end, that's up to the end-users. Setuptools contains a lot of compatibility code due to how it grew over the decades, and also because it tries not to break the users who still rely on weird side effects.

The build/ dir is a temporary build location, IIRC. It may end up containing some shared objects for non pure-python modules. The reality is that sometimes downstream packagers rely on what's in build/, even though I'd say it's incorrect to rely on this side effect.

[webknjaz]

This old #1347 suggests writing custom code in setup.py, but I know that using setup.py is an old approach which shouldn't be used it anymore.

This is not really accurate: https://packaging.python.org/en/latest/discussions/setup-py-deprecated/.

[VladimirFokow]

Thanks!
Your answers make sense.
I'd also love to hear from different community members - it would be great to better understrand the reasons behind this behavior, and recommendations how to deal with it, so I'd appreciate if this discussion stays open to new answers for a little while.

Maybe migrate it to issues?.. But I still don't understand if the issue would make sense, or I just have no idea about all the dependencies on this behavior and actually the whole setuptools is bound to live with this issue forever.

---

For example, some info in the User Guide would be nice about this as well:

• if people do use these side effects, - how?
• for building with setup.py, I guess you are suggesting this part from that page:

Custom build steps that for example either overwrite existing steps such as build_py, build_ext, and bdist_wheel or add new build steps are not deprecated. Those will be automatically called as expected.

specifically, "add new build step" = <remove the auto-created "src/PACKAGE_NAME.egg-info/" >
I don't know how this works - would be interesting to look into later..
(Is this the recommended way of dealing with this folder?)

• if this is a "build" step - then I assume - it's for python -m build . Would pip install . work with such setup.py somehow to remove the side effects?..

For now, I'll just look into other libraries.

Cheers!

[webknjaz]

Maybe migrate it to issues?.. But I still don't understand if the issue would make sense, or I just have no idea about all the dependencies on this behavior and actually the whole setuptools is bound to live with this issue forever.

I think it's better to open an issue or a pull request for cleaning this up. It's likely just a historical artifact nobody thought of addressing.

• if people do use these side effects, - how?

That's out of the setuptools' control. See https://www.hyrumslaw.com. Though, it doesn't mean that setuptools recommends relying on this.

• for building with setup.py, I guess you are suggesting this part from that page:

Custom build steps that for example either overwrite existing steps such as build_py, build_ext, and bdist_wheel or add new build steps are not deprecated. Those will be automatically called as expected.

Yes.

specifically, "add new build step" = <remove the auto-created "src/PACKAGE_NAME.egg-info/" > I don't know how this works - would be interesting to look into later.. (Is this the recommended way of dealing with this folder?)

• if this is a "build" step - then I assume - it's for python -m build . Would pip install . work with such setup.py somehow to remove the side effects?..

Perhaps. But typically nobody cares and it just remains on disk. Not committing it anywhere is enough.

For now, I'll just look into other libraries.

It's up to you. setuptools is old but that's not bad or good. It's just how things are. It has a reputation of being complex because of all the legacy. It's simpler to use today, and there's cases for which there's no good alternatives.

That said, we chose to showcase Hatchling as the build backend in the intro tutorial, while mentioning a few others: https://packaging.python.org/en/latest/tutorials/packaging-projects/#choosing-a-build-backend. It's maintained under the PyPA umbrella as well and explores somewhat different configuration mechanisms.

[VladimirFokow]

thanks!

[VladimirFokow]

created an issue: #4198