Define a distribution constraint setting

[mboes]

A common trick with Bazel rules is to download prebuilt SDK's and use that as toolchains. However, it is a mistake to assume that keying which SDK to download on the @bazel_tools//platforms:os value will work in every case. SDK's are seldom self-contained and often assume a specific glibc, or filesystem layout. On Linux, not all distributions are alike. On macOS, not all versions are alike. So we need finer grained platform info.

One idea would be to add a generic distribution constraint setting, in addition to the existing os and cpu constraint settings. The distribution setting could be used to choose the SDK format as well. For example, while the default might be a binary distribution as commonly distributed by upstream (official Go SDK's, official GHC binary distributions, etc), alternatives for obtaining a working toolchain might be to use Snappy, AppImage, Nix or Flatpak, depending on what is available or what the user prefers.

Not all toolchain distribution methods work on all platforms. Binary distributions seldom work on NixOS because the dynamic linker is not at a known location. Conversely, using Nix, be it on macOS or Linux, requires first that Nix be installed (same from Snappy etc). One might argue that Flatpak, Nix or Snappy should be an os constraint value. But that's very inconvenient. For example, on macOS dynamic libraries are called libfoo.dylib rather than libfoo.so. Rules want to deal with macOS specifics, while also letting the user choose among several possible sources fo toolchains. To allow that with just one constraint setting, we'd have to define many constraint values, like osx, osx_nix, linux, linux_nix etc.

Nix is available as a distribution method both on Linux and on macOS. Flatpak, Snappy, AppImage and the like are for the moment Linux-specific. But could well likewise be multi-platform in the future. Defining a new constraint_setting would avoid the combinatorial explosion in constraint values that would otherwise be needed if we continue having just @bazel_tools//platforms:os.

[gregestren]

@aiuto for reference (who else should be reading these?)

What's the practical difference of the different distributions? Can it be boiled down to the paths where the tools end up? I wonder if the tool_paths proposal described in Non-Hermetic Tool Paths in Platforms speaks to this?

[mboes]

That proposal looks relevant. But I'm struggling to visualize how it will work in this case. To summarize:

1. Bazel isn't 100% hermetic. Rule sets typically provide macros to download prebuilt toolchains in external repositories. These prebuilt toolchains make arbitrary assumptions about the host environment (like glibc version, location of dynamic linker, etc) that are unaccounted for in the build description. That's a problem (and in my experience daily, not a theoretical one).
2. If we're going to have non-hermetic workspace rules to provision toolchains (like rules_go or rules_nodejs do), we're going to have to offer users some control. If my system has a different glibc than the Go SDK's assume, or a dynamic linker at a different location, I will need a) alternatives to the SDK's and b) a means to select which alternative I want.
3. The alternatives could be "fat" packages (much like "uber jars") that are closures containing everything. Flatpak, AppImage, Nixpkgs are all examples of this. At least these types of packages work on any system without making assumptions like SDK's do. And like downloading SDK's, unpacking these packages does not require any superuser priviledges.
4. Even if we use "fat" packages, no one such fat package type is a one size fits all: there is no de facto standard here and some users prefer Nixpkgs, some other prefer something else, and others want to stick to non-hermetic SDK's rather than bother with any such fat packages that eat up a substantial amount of space on disk.
5. We can provide these alternatives and the means for the user to select among them today. We just need to agree on a constraint_setting name to select between say pulling toolchains from a Nixpkgs, or from a Flatpak package, or ...

So what should that constraint_setting name be, and does it have a place in this repository?

[gregestren]

I'd like to think the right path forward would involve enhancing rule sets' toolchain definitions to auto-configure as necessary. The theoretical tool_paths could fill out the pieces they leave as open variables.

But I understand constraint_settings can have a role to play here too. There are a few discussions / docs floating around trying to figure out how to manage the variety of compilers, compiler versions, runtime versions, etc. installed on diferent machines and how to provide the right flexibility to choose the right ones. I guess a lot of them are assuming the continuation of non-hermetic distributions.

Another example is Versioned Runtime APIs under
Platforms & Toolchains, which also speaks to adding new constraint_settings for platforms, albeit at a more granular level.

[aiuto]

@aragos @katre @agoulti @iirina

[aiuto]

So what should that constraint_setting name be, and does it have a place in this repository?

Those are hard questions. The easy question is "does this idea make sense", to which the answer is yes. I think. My understanding of your proposal is basically

• auto-configuring, hon-hermetic toolchains will eventually download packages from some package manager repository
• the tool owner will have to provide capability to do the download and configure (probably with common tooling for many toolchains)
• the user rather the owner will want the capability to say what kind of package manager they want

The cross product of the 3 things gets a configured tool chain with binaries downloaded from a place of the user's choice.

Before we can figure out the name, we need to know if we want to talk about specific package managers (SaltStack vs. Chocolatey) or do we talk about my OS distribution (Nix vs RedHat). Or both?

Then the question is if this repository is the right place. I think yes if there is going to be common infrastructure for downloading things from different kinds of repositories and the toolchain implementations will share it.

Can you come with a proof of concept that demonstrates the kind of thing you want to do?
Once we figure out which kind of thing makes sense, the naming should fall out of it.

[mboes]

Before we can figure out the name, we need to know if we want to talk about specific package managers (SaltStack vs. Chocolatey) or do we talk about my OS distribution (Nix vs RedHat). Or both?

My hunch is that it should be only the former. There are two use cases under discussion:

1. Non-hermetic builds with non-hermetic toolchains.
2. Hermetic builds, which use toolchains built outside of Bazel that are nonetheless hermetic because they are closures (e.g. a Nix derivation, a Docker container, a snap...).

In use case 1, distro-specific differences are either dealt with via auto-configuration or with constraint settings but these should be more granular than just centos or nixos (which the user is free to define as platforms). So we're left with just use case 2. In this case, we want to choose an appropriate source of toolchain, and this isn't a matter of autoconfiguration: it's as legitimate for a CentOS user, and even a macOS user, to source toolchains using Nix as it is for a NixOS user. It really is up to the user to specify the source of the toolchains.

So I guess distribution isn't a fantastic name. Perhaps package_manager? In very large projects, it's conceivable that one part of the build will use one package manager while using another package manager for other parts because say a Nix package exists but not a Flatpak one for some particular toolchain. This however, should be rare. And hopefully multiplatform builds (see bazelbuild/bazel#6519) will happen someday, at which point it won't be a problem.

[mboes]

How do we make progress on this front?

[mboes]

Another ping.

[mboes]

@gregestren thinking about this more, perhaps a generic package_manager constraint setting is misguided: package managers are not necessarily mutually exclusive. Perhaps it makes more sense to have multiple "feature test" style constraint settings, like nix with a single support_nix constraint value, flatpak with a single support_flatpak, etc.

Constraint settings still have a role, beyond the question of finding the right path for compilers. Because sometimes a specific tool (like the package manager) has to be called to populate a global content addressable store and have paths to point to. But if you agree with the first paragraph above, we can close this issue.

[gregestren]

@mboes I'm trying to get back on track with these themes. Please give me some more time to think through your points more thoroughly before I can offer a reasonable response. This is up-prioritized in my queue.