setuptools is definitely a better place for this than distutils, for various reasons.

One thing I'll note is that we're trying to move away from any user-facing setup.py invocations, so if you're envisioning something like setup.py build_compilation_db or something, that is unlikely to happen. However, I think that, to the extent that this is a standard format, something where either users can opt-in to have this information generated or if it's cheap enough to do and unlikely to cause any problems, it is automatically enabled where appropriate and not explicitly disabled.

The one thing is that I am not entirely convinced as to whether this should go in setuptools proper or if it should be an extension like setuptools_scm. With PEP 518 it's easier than ever to have a modular build system, so we don't need to maintain a monolithic everything-builder to satisfy everyone's needs.

To me the question of whether this should go in setuptools depends on the answers to at least the following questions:

1. How standard is this format? Are there competing formats, or is this the only way anyone includes compilation symbols.
2. How mature is the format? Are we going to have to be constantly keeping track of minor changes, or is it mostly stable?
3. How useful is this to end-users? If this is something that will be very helpful to many end-users, particularly if it's disproportionately useful to end-users, it seems like something we may want to lower the barrier to entry for.
4. How expensive is this? Will it make builds take a lot longer? Will it make wheels considerably larger?

@pganssle

How standard is this format? Are there competing formats, or is this the only way anyone includes compilation symbols.

I do not know of any other formats. It appears to be the only build-system independent format supported by the tools that use it, like meson, cmake, clang-analyzer, cppcheck, neomake, ALE, jetbrains CLion, vscode, etc. I could only find youcompleteme that also supports a custom .ycm_extra_conf.py file.

I'm a bit confused what you mean with symbols. Perhaps I should give an example, to make clear what this format involves. This is a line from a random project of mine:

 [
    {
        "arguments": [
            "gcc",
            "-c",
            "-pthread",
            "-Wno-unused-result",
            "-Wsign-compare",
            "-DDYNAMIC_ANNOTATIONS_ENABLED=1",
            "-DNDEBUG",
            "-O2",
            "-g",
            "-pipe",
            "-Wall",
            "-Werror=format-security",
            "-Wp,-D_FORTIFY_SOURCE=2",
            "-Wp,-D_GLIBCXX_ASSERTIONS",
            "-fexceptions",
            "-fstack-protector-strong",
            "-grecord-gcc-switches",
            "-m64",
            "-mtune=generic",
            "-fasynchronous-unwind-tables",
            "-fstack-clash-protection",
            "-fcf-protection",
            "-D_GNU_SOURCE",
            "-fPIC",
            "-fwrapv",
            "-fPIC",
            "-I./project",
            "-I/usr/include/python3.7m",
            "-o",
            "build/temp.linux-x86_64-3.7/./project/loaders.o",
            "project/loaders.cpp"
        ],
        "directory": "/home/user/prj/project",
        "file": "project/loaders.cpp"
    },

As you can see, these are just the commands that setup.py uses during compilation.

How mature is the format? Are we going to have to be constantly keeping track of minor changes, or is it mostly stable?

With how simple it is, it would be hard for me to imagine any major breaking changes.

How useful is this to end-users? If this is something that will be very helpful to many end-users, particularly if it's disproportionately useful to end-users, it seems like something we may want to lower the barrier to entry for.

This is useful for developers that use any of the tools I've mentioned above and develop cpython extensions. It does not affect users of packages generated by setuptools. It hence does not require declaration in pyproject.toml or similar considerations. If it were to be implemented seperately, it would go in dev_requirements.txt or it's equivalents. However, I feel like most users would not discover such an extension and just be frustrated that their editors don't work properly as they'd expect.

How expensive is this? Will it make builds take a lot longer? Will it make wheels considerably larger?

It only involves dumping the compile commands that would be executed into a json file, so the overhead should be negligible. Meson generates one under build/ by default. Aside from accidental inclusion, it should not affect the size of wheels.

Is there any update on this idea? Also, is there any way that this works now with pyproject.toml? I'd like to be able to lint C/C++ extensions via clang-tidy, but can't figure out how to make a compilation database as a result of pip install ....

It works out of the box when using alternative build backends such as meson-python.

I'm pretty sure that setuptools has not implemented it, though.

For anybody else who ends up here, it's possible to use Bear to generate compile_commands.json.

bear -- python setup.py build_ext --inplace && mv compile_commands.json build