extending Go forward compatibility

[rsc]

This discussion is about forward compatibility, meaning old versions of Go compiling newer Go code. For the problem of new versions of Go compiling older Go code, see this other discussion about backward compatibility.

It seems strange to me that you can edit go.mod to update all the code your program depends on except Go itself. Suppose go.mod says go 1.17. Why shouldn't changing that to go 1.18 make the build use Go 1.18? I think probably it should. Here is how that might work.

To start, let's get the machinery of an old go version downloading and running another out of the way. Assume that we publish all binary Go distributions as module zip files (in addition to the usual zip files and other installers). Then if the go command notices it needs a different distribution, it can download it from the module proxy and authenticate it using the checksum database. (We don't need to invent a separate authenticated download channel, nor a separate caching mechanism.) After unpacking the distribution somewhere appropriate (perhaps the module cache) and restoring the execute bits on the binaries, the old go command can reinvoke the newly downloaded go command. On a future invocation, the old go version finds the cached copy directly and runs it, no download required.

Having gotten the “is it possible?” out of the way, let's turn to what should cause that to happen. I am thinking about something along these lines:

1. By default, if the go line in go.mod in the current module or workspace lists a newer version of Go than the current one being run, the current one goes and gets the new one and reinvokes it. Supposing the change happens in Go 1.90, if you are running Go 1.90 and change your go.mod line to go 1.92, any go command like go build or go test will download and reinvoke Go 1.92.

   On the other hand, if you change the line to go 1.88, Go 1.90 will keep running and will do its best to emulate Go 1.88. (See also this discussion about backward compatibility.)

2. If you want more fine-grained control, a new go.mod line toolchain, which is only respected in the current module's go.mod (just like replace) sets a specific toolchain to use. For example you could do

   go 1.91
   toolchain go1.92.4
   

   (with Go 1.90 still installed) to stay on Go 1.91 semantics but update to the Go 1.92.4 toolchain. Similarly, you can use toolchain to “downgrade”, as in:

   go 1.70
   toolchain go1.80
   

   (with Go 1.90 still installed).

   The toolchain line overrides the version selection logic in step 1.

   To ensure accurate compilation of code written for newer Go versions, the go command would update the toolchain line when adding a dependency with a newer version of Go to maintain the invariant that the toolchain (or the implied toolchain from the go line) is at least as new as the go version required by any dependent module. Continuing the examples above, if you go get a new or updated dependency that has a go.mod that says go 1.100, then the current module’s go.mod toolchain line would be updated to toolchain go1.100, with a message to standard error about the update.

3. For command-line control, setting GOTOOLCHAIN overrides any go.mod toolchain line, like:

   GOTOOLCHAIN=go1.92.3 go test
   

   Using GOTOOLCHAIN=local would use the local toolchain, whatever its version (Go 1.90 in this running example).

   And GOTOOLCHAIN=auto would be an explicit name for what happens when there's no toolchain line, so that you could have the go.mod above and run

   GOTOOLCHAIN=auto go test
   

   to get Go 1.91.

No matter how the effective toolchain is specified, it would become a build error to attempt to use an old toolchain with new code. (Today the build proceeds and hopes for the best; in case of a compilation error, the go command prints a final note about the version mismatch as a possible explanation of the compilation error.)

This mechanism would make it a lot easier to try out new Go releases, as well as betas and release candidates: just edit your go.mod files and commit them, and everything building your code knows to use the new release. This is analogous to when you edit your go.mod to update a dependency version and everything building your code knows to use that newer version. (In contrast, what happens today with Go toolchain selection is very much like the old days of GOPATH, when each different build machine injects its own local state into the build.)

This mechanism would also help a lot with Go packagers that don't always keep up with Go releases, such as some Linux distributions and cloud providers that I wont name.

If we ever moved toward issuing more frequent ephemeral Go releases (like monthly alpha releases when the tree is open), this mechanism would make those easier to consume.

Finally, I started a separate discussion about a Go toolchain being able to emulate an older one’s runtime semantics based on the go.mod go line. And Go toolchains already emulate an older one’s language semantics based on each package’s go.mod’s go line. The mechanism discussed above is a nice complement to those, allowing a Go toolchain to emulate a newer one based on the go.mod go line.

This is a discussion, not a proposal. I haven't implemented all of this nor even worked out all the implications. I'm curious what people think and what concerns they have. Thanks!

[josharian]

I want this very much.

One possible minor downside is execution cost in transient (CI) environments, where the toolchain download and unpack might happen every time. If a dependency gets updated, and there’s no explicit toolchain line, CI builds could silently get slower and costlier until someone notices.

[willfaught]

Wouldn't this be solved by CI caching?

[josharian]

Yes, except that (at least with GitHub Actions), CI caching of Go stuff doesn’t always provide noticeable improvements. Long story, but this is a pain point.

[rsc]

This is a real downside. It exacerbates an existing problem, which is modules being redownloaded every time. But modules are smaller, of course. I hope this would be handled by CI systems running caching proxies, which would speed up module downloads too. We also have plans to make the Go toolchains smaller by dropping most or all of the .a files.

On the other hand, given the choice between (1) having to wait for a CI system (or a Linux distribution, or a cloud provider) to update the available version of Go and (2) being able to use any Go version at the cost of slightly slower builds, I'd definitely choose (2).

And of course, CI systems that care about never downloading could force GOTOOLCHAIN=local in the environment, and then the build will break if a newer go line slips into go.mod.

[thepudds]

Yes, except that (at least with GitHub Actions), CI caching of Go stuff doesn’t always provide noticeable improvements. Long story, but this is a pain point.

Hi @josharian 👋 , no rush, but do you happen to have a link handy that has some details, or is it possible to boil it down to a ~1-2 sentence description of the area of the problem?

[josharian]

In short: The CI cache is slow. For example, except in pathological cases, it's often faster to recompile than cache build artifacts. People currently have to experiment: Do I cache nothing? The module download cache? The build artifact cache? And then occasionally re-tune it. This would increase that burden.

(I suspect that there are additional issues, like if you do a bunch of work before the cache cleaning mechanism kicks in, the CI cache ends up transferring back and forth a LOT of build artifacts, many/most of which are no longer relevant. Accumulated cache cruft usually doesn't matter all that much on a developer's machine, where the cost is just disk space, but that calculus changes in CI, where the entire cache has to get copied across the network twice--start and end--regardless of how much of it you're going to use. This is a bit OT, and I suppose it should probably should have its own bug to discuss. But I don't want to invest the time right now to chase down the concrete data that would be appropriate when filing said bug. Feel free to file one if you'd like, though!)

[leitzler]

Interesting idea. Can you elaborate a bit more on how this might work with other go invocations, or should it apply to all sub commands? Would go mod tidy for example (that itself modifies go.mod) also be invoked using the version specified by go.mod?

Or would it be exclusive for go build and go test? It would probably be confusing if go help and others that are unrelated to the current module had different results based on where it is invoked.

[rsc]

It would apply to all go commands, even go commands that don't exist in the earlier version of Go. The go command would find the go.mod and reinvoke the newer go command before even parsing the full command line.

Note that go mod tidy already does change behavior based on the go version: Go 1.17 go mod tidy knows how to emulate Go 1.16 go mod tidy for modules that say 'go 1.16'. That's the backward compatibility half. The forward half doesn't work; this would make it work.

[prattmic]

To ensure accurate compilation of code written for newer Go versions, the go command would update the toolchain line when adding a dependency with a newer version of Go to maintain the invariant that the toolchain (or the implied toolchain from the go line) is at least as new as the go version required by any dependent module.

Creating a new module defaults the go line to the version of Go in use, declaring it to "require" that language version. In practice, I'd assume that the vast majority of new modules do not actually require the specified language version; they probably work fine with older versions (but are untested).

Combined with the new rule above, I can imagine this causing more churn than some users would like, particularly if they are using toolchain to select an older version. Whenever they add new dependencies, getting a toolchain update is likely, even though things probably work fine on the original toolchain version.

All that said, I think this has a lot of parallels with modules, and ultimately isn't a big deal. Users regularly update the versions they "require" of their dependencies, but only as a periodic update, not because they really need something from the new version. This new "requirement" propagates to dependents, which now must update the newer versions.

[rsc]

Yes, exactly. This is making the go version work very analogously to module dependency versions. Originally we thought that was too onerous because it required manual updating to a new Go version to resolve a problem. The automatic updating should make it more reasonable.

[willfaught]

to stay on Go 1.91 semantics but update to the Go 1.92.4 toolchain

Some more background would be helpful. Does this mean that a particular Go version's toolchain can already expand, contract, or change the available features of the Go it builds to target a specific Go version? I'm unclear on why the language semantics and toolchain versions are being distinguished here. Doesn't Go 1.19 build a module targeting Go 1.17 as a Go 1.19 binary?

After unpacking the distribution somewhere appropriate (perhaps the module cache) and restoring the execute bits on the binaries

Could there be a mismatch of supported architectures or operating systems? If so, how would those be handled?

the old go command can reinvoke the newly downloaded go command

What if an explicit command-line option given by the user to the installed version isn't supported by the future version, or the meaning of it changed? The Go 1 compat promise doesn't seem to apply to the toolchain. How would differences there be accounted for, or do they even need to be?

Supposing the change happens in Go 1.90, if you are running Go 1.90 and change your go.mod line to go 1.92, any go command like go build or go test will download and reinvoke Go 1.92.

The idea here doesn't seem to want to break the Go 1 compat promise, so why not instead fail the build, and force the user to upgrade their Go version? That seems simpler than essentially re-inventing a command-line package manager just for Go versions. It seems surprising (and perhaps wrong) to have Go 1.20 installed, but get a Go 1.21 build. What if, say, the range of times representable by time.Time in Go 1.21 narrows drastically, and you want to build with Go 1.20 instead?

I do sympathize with users who use OSes that don't bump the Go version right away, but perhaps that's an intended stability or security feature. If users really want the latest and greatest, they can always download and install a standalone binary from https://go.dev/dl/, or define their own package in their OS's package manager.

I can't think of a scenario in which you're building your own main package, in your own module, where you control the go version directive, and you'd want to make that version higher than the one installed on your machine. Can you provide an example? Maybe I'm mixing up the go and toolchain directives here in terms of controlling the overall build.

[rsc]

I agree that it's a good thing to have, but Go modules don't specify the exact version of their deps, just the minimum version.

This is not true. The current module's go.mod (the one that's in your working directory, or somewhere above your working directory in the file tree) lists the exact versions of the modules that are being used in the build. It is true that for a dependency incorporated into a larger program, the larger program may use a newer version of what that dependency's go.mod lists. But at the top level of the build, the go.mod has an explicit list, providing a reproducible build.

[willfaught]

The current module's go.mod (the one that's in your working directory, or somewhere above your working directory in the file tree) lists the exact versions of the modules that are being used in the build.

Sorry, I don't understand. If main module M requires A v1.0.0 and B v1.0.0, and B v1.0.0 requires A v1.1.0 because it provides a new generic function, main module M has to be built with A v1.1.0, otherwise B won't compile. From the mod reference:

MVS starts at the main modules (special vertices in the graph that have no version) and traverses the graph, tracking the highest required version of each module. At the end of the traversal, the highest required versions comprise the build list: they are the minimum versions that satisfy all requirements.

It's my understanding that the main module can constrain versions with replace and exclude directives, but it can't just say "use only version V for this module, no matter what other modules require." From the mod reference:

The graph may be modified by exclude and replace directives in the go.mod file(s) of the main module(s) and by replace directives in the go.work file.

I must be missing something.

[ChrisHines]

Sorry, I don't understand. If main module M requires A v1.0.0 and B v1.0.0, and B v1.0.0 requires A v1.1.0 because it provides a new generic function, main module M has to be built with A v1.1.0, otherwise B won't compile. From the mod reference:

Your understanding is accurate up to Go 1.16. The story changed with Go 1.17. See https://go.dev/doc/go1.17#go-command, https://go.dev/ref/mod#go-mod-file-go, https://go.dev/ref/mod#build-commands, and https://go.dev/ref/mod#go-mod-file-updates.

In summary:

• If the go.mod file has a go directive for Go 1.17+ then "[t]he go.mod file includes an explicit require directive for each module that provides any package transitively imported by a package or test in the main module."
• Most [go tool] commands report an error if go.mod is missing information or doesn’t accurately reflect reality. The go get and go mod tidy commands may be used to fix most of these problems. Additionally, the -mod=mod flag may be used with most module-aware commands (go build, go test, and so on) to instruct the go command to fix problems in go.mod and go.sum automatically.

When these two behaviors are combined (for Go 1.17+ main modules built by a Go 1.17+ toolchain) then I believe @rsc assertion holds. Either the go.mod specifies all the versions required for the build or the build will report an error about an inconsistent go.mod file and push you to fix it.

The only exceptions I can think of are the presence of local replacements in the main go.mod (which specify a local path without a version) and a go.work file in play (which could introduce newer versions of dependencies if other modules in the workspace require them).

[willfaught]

When these two behaviors are combined (for Go 1.17+ main modules built by a Go 1.17+ toolchain) then I believe @rsc assertion holds. Either the go.mod specifies all the versions required for the build or the build will report an error about an inconsistent go.mod file and push you to fix it.

OK, but that's just putting all the transitive requirements into the module, not overriding/controlling them. In my M/A/B example, M still couldn't force the use of both A v1.0.0 and B v1.0.0. It simply can't build in that case, regardless of the module-fu we try (the new function wouldn't exist). It looks like if you do try to do that, the go command will report an error before it ever gets to the point of trying to build it. The only control I see is the ability to bump the minor version higher, but then again, so does every module, not just the main module.

[willfaught]

I think I understand now. If I understand correctly, @rsc was saying that the main module's requirements are complete and explicit about the specific version of each dependency. If the main module requires a lower version than what MVS concludes is required, the go command will report an error, and suggest that the mod deps be updated.

To go back to the original context:

The situation with the Go toolchain today is exactly like what GOPATH did for dependencies before modules. [...] In fact you could argue the Go toolchain may be the most critical dependency any build has, yet it's the only one that we don't provide these kinds of versioning guarantees for.

I agree that it's a good thing to have, but Go modules don't specify the exact version of their deps, just the minimum version. Doesn't the go directive in modules accomplish the same thing for the language version? In your Docker/Kube example, the MVS conclusion would be that v1.20.4 should be used for both. Doesn't that provide a reproducible build?

If you take the go directive to be the exact Go version "required," then it seems to me that there's still a parallel with required module versions. If main module M only needs A v1.0.0, but has to use A v1.1.0 due to another module's requirement, then there's risk there that something will break. If main module M only needs Go v1.18, but has to use Go v1.19 due to another module's requirement, then there's risk there that something will break. In either case, you upgrade at your own risk.

[urandom2]

since go versions do not follow semver (see #32450), this may confuse people: (and I see why you used goX.Y to line up with git tags)

go 1.70          # I am on the go 1.70 release line
toolchain go1.80 # I want the 1.80(.0) version, not 1.80.4

do we have a way of saying I want the latest 1.80 toolchain or at least easily hiding this complexity from developers?

[rsc]

do we have a way of saying I want the latest 1.80 toolchain or at least easily hiding this complexity from developers?

No, for the same reasons we don't have a way of saying I want the latest version of a given module: that changes day by day and you lose reproducible builds: the same source can produce different compiled binaries and possibly even different behavior from one day to the next. Instead the build toolchain version in the go.mod is completely precise and users control when it changes.

Of course, we should make it easy to change. Maybe something like 'go get -u toolchain' or 'go get -p toolchain'.

[josharian]

It's just one data point, but I've been living in something like this world for a while. I have a zsh chpwd script that examines the go.mod that is relevant to my current directory and select my Go toolchain accordingly. It only handles major versions, but it is already a quality of life improvement.

[josharian]

I can imagine that Tailscale might like the option to have toolchain local in the go.mod (not just an env var), to explicitly punt the toolchain choice back to the local environment.

[rsc]

Sure, that seems reasonable. It is almost indistinguishable from not having a toolchain line. The only difference would be that if the toolchain is too old, the build breaks instead of automatically grabbing a new enough one.

[hyangah]

Seeing users wanting toolchain local to opt out, I wonder why other approaches like nvm or rustup that provides an easy way for users to switch between versions when they find a different version of go is not a good solution.

[mvdan]

@hyangah from personal experience, it's a people problem. With a large enough team, it's hard to get everyone to use the same versioning tool consistently. People are used to running go build and changes are hard and painful. I think part of why modules succeeded as a transition is because, largely, the build and test commands remained the same.

[hyangah]

In our organization, tooling updates are done in org-level and they should be security verified. I think showing the warning that suggests actionable messages (e.g. "hey you need this version for best functionality, run go up 1.900.1") seem better than that. Another issue to consider is go is not the only tool developers depend on but some tools are sensitive to what version of go command is being used. For example, dlv needs to be a certain version to work with a certain version of go. gopls is also planning to implement version support window policy. By automatically picking the tool versions, that means other tools or ides have to implement similar mechanism to pick the right versions of tools.

[rsc]

Thanks for mentioning nvm and rustup. I had not looked closely at either before your comment. Both are analogous to what this discussion is about the go command doing. I don't see why it should be 'goup' instead of 'go' that does it. Why use two tools when one tool will do? We can do a much better job inside the 'go' command than any tool on the side can.

Rustup has 4 different ways to select the toolchain; at least what I described only has 2. :-)
https://rust-lang.github.io/rustup/overrides.html

[josharian]

One minor downside to this is that it will increase Go cache pressure. If I'm working across a handful of projects, each with a slightly different toolchain, I'm going to end up with a lot more in my Go cache--not just N toolchains, but N different compiled versions of shared dependencies.

[rsc]

Definitely true.

Worth noting that if you have C -> B -> A in one project and D -> B -> A2 in a different project, you're already getting two different compiled copies of B, because of the different As sending different export data up to B's imports. And if you are working hard to share exactly the same set of dependency versions across projects, it seems like the same sync mechanism could handle toolchain too.

[mvdan]

Perhaps this proposal should go hand in hand with a slightly better heuristic around the sizes of Go caches, like #29561.

[thepudds]

It might be worthwhile to be more explicit about the minor Go revision in the overall discussion (that is, the Y in go1.X.Y).

For example:

By default, if the go line in go.mod in the current module or workspace lists a newer version of Go than the current one being run, the current one goes and gets the new one and reinvokes it.

My understanding from the discussion is that would mean the latest minor Go revision?

Or if we look at:

Continuing the examples above, if you go get a new or updated dependency that has a go.mod that says go 1.100, then the current module’s go.mod toolchain line would be updated to toolchain go1.100, with a message to standard error about the update.

With regular modules, you can write require foo v1.100 (without the third digit) in your go.mod, but then a go mod tidy or similar will update it to read v1.100.9 (if that is the latest foo v1.100.*), which among other things helps with consistency for consumers of your module. It sounds like similar behavior is not being suggested for the go directive in go.mod, but I'm not sure what the suggested behavior is for the minor go version on a toolchain line that is being set based on a dependency's go directive as in the example quoted just above. (If the toolchain line is updated with the latest minor revision, which might have been published by the Go team five minutes ago, then that seems like it might be a small-ish step away from the high-fidelity build behavior of MVS for regular modules?)

[thomasf]

When it comes to Go minor versions I would maybe want some way to at blacklist some of them.

I have had programs that breaks/panics on a couple of 1.x.0 Go releases that were fixed in the .1 fix release. I don't know if a minimum minor Go version is the right solution for that situation because both v1.100.1 and v1.99.0 could be compatible while v1.100.0 isn't because of compiler/runtime bugs. A subtle aspect of this issue is that it can be a bug that only happens on a particular OS or architecture but for simplicity sake I would be satisfied with flat out reject a whole Go minor version anyway, the most important thing is that go install/build makes a fuss when you try to build something that is known to be broken.

It could also be argued that exempting Go version from the same semver rules that applies to Go modules would be a bad idea because it is less straight forward and not with it. The scenario described above isn't really possible to express within MVS though.

[golightlyb]

How does / how should this interact with (rare) security issues in the Go toolchain (e.g. https://go.dev/blog/path-security). Could a malicious package author specify an older toolchain that hasn't recieved the security update in order to achieve RCE in the build environment?

[rsc]

For the go.mod go line, the toolchain will only auto-upgrade, not auto-downgrade.
For the toolchain line, yes, it would make it possible to downgrade to an older toolchain.
Remote build environments should almost certainly be sandboxed anyway,
but they could also force GOTOOLCHAIN=auto to override the toolchain line but still allow upgrades to newer versions.

[randall77]

On the other hand, if you change the line to go 1.88, Go 1.90 will keep running and will do its best to emulate Go 1.88. (See also this discussion about backward compatibility.)

Why can't we download and run go1.88?
Maybe "can't" isn't the right question here. Maybe "Why shouldn't we download and run go1.88?" is the right question.
I guess we want everyone to upgrade to go1.90 eventually, so letting people pin forever is maybe unsatisfying. But that's unsatisfying for us, maybe for users that's exactly what they want. (Modulo dependencies that upgrade their toolchain dependencies, in which case you have to upgrade the toolchain used, and emulate on the modules that ask for older semantics.)

[rsc]

If you've gone to the trouble of installing Go 1.90 on your machine, I think that's a strong sign that you want to use it, even to compile older Go code. You can always override that if you want, of course, by setting GOTOOLCHAIN or adding a toolchain line to go.mod.

[mdempsky]

How does toolchain selection affect non-gc toolchains, like gccgo and TinyGo?

[rsc]

I assume that people using those toolchains would be unaffected. But part of the reason for the 'go' prefix in 'toolchain go1.90' is so that we could distinguish 'toolchain gccgo1.90' or 'toolchain tinygo1.90' at some point in the future.

[vpereira01]

Interesting idea 👍 , a concern and an opinion

1. Automatic binary download/execution seems a bit dirty (something that doesn't feel right to me)
   • The security risk can be mitigated but a good implementation and security review in go implementation but seems an increased risk even if small.
   • On first release it might trigger several security tools, as it will show some behavior similar to an infected compiler (binary download from the internet -> execution -> binary produced from compilation is now slightly different than without that previous download/execution)
   • I'm not sure security personal and system administrators will be OK to have go 1.20 installed on a build agent and have builds running with go 1.21, go 1.22, go 1.33.... With each execution downloading binaries and running them.
   • I might be overthinking the risk but one think that makes me uncomfortable in other tools/languages is doing "ACME build" and see in the log messages "downloading x..... executing x".
2. For most of the stated problems to be fixed, temporary tool chain changes, I think the environment variable is the best option.
   • On go.mod I think it will spread for the whole Go ecosystem. Since the stated use cases seem uncommon to me, I think this brings unnecessary complexity to new comers. They will need to understand what's the go.mod go directive and also the toolchain directive. Currently, they only need to know to have a go runtime installed greater than what's in go.mod

[minux]

1. Automatic binary download/execution seems a bit dirty (something that doesn't feel right to me)

I second this concern.

Is it possible to download the source code of the requested toolchain and build it on the fly? It might alleviate some of the concern on downloading & executing new binaries. This might be a little trickier to do but as for now, Go module builds generally build everything from source (except .syso files) and do not download or run any binary blobs (again, with the exception of syso files; but at least those are not executed during the build.)

At the very least, I'd want a documented way to instruct the Go command to not download unavailable toolchains and instead error out and in the error message, point out the exact reason (which module requires which Go toolchain version, etc.)

[rsc]

To opt out you would set GOTOOLCHAIN=local or go env -w GOTOOLCHAIN=local.

Obviously we have to get the implementation right, but at the moment you already trust commands like 'go run' or 'go test' to download bits from the internet, read those bits into memory, interpret them in some way, write out different bits to a new file, set the execute bit on that file, and run it. I honestly don't see how any security scanner will distinguish what the go command already does from what's contemplated in this discussion. Perhaps an overzealous security scanner could intercept the HTTPS connection and see that the zip file being downloaded contains an executable and block the download?

[vpereira01]

If I understand the problem to solve I think this should be opt-in, as in, trying non-released versions seems a non-standard use case to me. Even if it's no the case I would still prefer an explicit opt-in for this for at least some point release given how big is the change in behavior.

Obviously we have to get the implementation right, but at the moment you already trust commands like 'go run' or 'go test'

From a security perspective I think there's a big difference because of go.sum. In the case of "go run" and "go test" the download is of known files because there's an hash to validate the downloads. In the case of downloading a new toolchain it would be unknown because the local "go" binary has no information even if the "toolchain 1.73" exists.

Perhaps an overzealous security scanner could intercept the HTTPS connection and see that the zip file being downloaded contains an executable and block the download?

I have seen that behaviour in corporate machines. Usually developers ask an opt-out of those security scanners/firewalls rules or need to ask IT to add it to the allow-list.

[jhenstridge]

Is it possible to download the source code of the requested toolchain and build it on the fly?

Note that there's no guarantee that the existing toolchain will be able to compile the requested toolchain. You might need to go through some intermediate toolchain versions if it relies on language features not present in the existing toolchain.

[zephyrtronium]

It seems like this would make for a very poor DX on Alpine or other musl systems, where GOOS/GOARCH look like a supported pair but you can't use the official binary distributions.

[rsc]

That may not be true in Go 1.20. I made some changes at the start of the current dev cycle to try to make the distributed toolchains work equally well on musl and non-musl systems. If the toolchain for Go 1.20 beta 1 doesn't work out of the box on Alpine, please file an issue. Thanks!

[mvdan]

For reference, I think that would be https://go-review.googlesource.com/c/go/+/420774.

[zephyrtronium]

If I'm still behind the times and using Go 1.113 while Go 1.119 has been released, would I be able to use go mod edit -go=1.119 to select the new version? What if Go 1.119 is still in beta, or the tree just opened for the 1.119 cycle?

[rsc]

Yes, that's the idea. If Go 1.119 is still in beta, then you will get an error when you try the next command. You could fix that with 'toolchain go1.119beta1'.

[mbyio]

I think this would be helpful, mainly because we could make the go.mod file the "source of truth" for which version of Go to use.

My company currently has to keep the Go version we use for each application updated in several places:

• Dockerfiles used for building the app
• Dockerfiles used for running tests (if they're different, it might be the same Dockerfile, in which case we have to update in multiple places in the same file)
• CI pipeline configuration
• go.mod
• ... I'm sure there are others I'm forgetting

We currently use a script to keep them all synchronized. It is just annoying enough that we don't always stay up-to-date.

I wish we could just have all tools read the version in go.mod, and in Dockerfiles we could use FROM golang:alpine3.16 and it would just do the right thing.

To prevent this from slowing down builds, it would be great if the go mod download command could also fetch the go toolchain. That would make sense to me since it is in go.mod as well.

[rsc]

Thanks for sharing your experiences with the current system. These kinds of examples are very helpful.

go mod download would in fact fetch the new Go toolchain, but because of the go, not the mod download.
Other commands like go version or go help would also download the new Go toolchain.

[ChrisHines]

Automatically downloading binary distributions may run into issues similar to #43996 when people (for whatever reason) are stuck with an old OS. I encountered this recently when we upgraded Go to 1.16. We had to change the affected Docker image to build Go from source instead of downloading the prepackaged binaries. I understand that would still be possible and using GOTOOLCHAIN=local might help avoid confusing error messages. I mention this simply as another way the automation can break down.

[rsc]

The specific problem in #43996 is caused by shipping .o files built on one machine in the distribution and expecting them to work on a different machine. We would like to not do that at all (#4719) and are moving slowly toward that.

[ChrisHines]

Thanks for the clarification. #4719 is a blast from the past, I'd forgotten that was still ongoing. Your comment here #4719 (comment) was a good reminder to me that:

$GOROOT/pkg is difficult to kill because we want to keep allowing the building of programs with cgo-enabled package net from systems without a C compiler. We will probably have $GOROOT/pkg or an equivalent for quite some time.

At the risk of being off topic, is it still a goal to support building cgo-enabled package net without a C compiler? If so, how would that work without shipping those .o files with the downloaded toolchain?

[rsc]

Either (1) we drop that goal or (2) we find a way to make the default builds not use any C code. The latter may be possible on any system with a dynamically linked C library, which at least covers Linux, Mac, and Windows. We'd have to maintain by hand the linker directives that cgo derives automatically today, and we'd have to make sure that rebuilds still happen properly when important settings change (like trying to do a static build). It's still up in the air but I think doable.

[mvdan]

I like this proposal. I have thought that Go needed some form of "target" Go version in go.mod files, given that the only version that it has today is generally treated as only a minimum. I also like that the toolchain version is a precise version rather than only a major version. Below are some itemized thoughts.

1. One minor disadvantage is that Go master would get less real-world testing. I daily drive Go tip for personal and work projects, and I know at least a handful of others who do as well. Over the years this has resulted in finding bugs after big features have landed or early regressions which weren't covered by tests yet. If my go build starts instead downloading and using previous Go releases, that benefit would get lost unless I went out of my way with GOTOOLCHAIN=local.

   Arguably this is in general a net benefit, because entire teams or companies could start testing beta and RC versions across all machines and environments, which today doesn't happen nearly often enough. But the quality of those pre-releases might drop a bit given that Go master would be used in fewer downstream builds and tests over time.

2. I also wonder to what degree this could encourage Go developers to not pay attention to what Go version they have installed locally. If we allow that version to lag considerably over time, that could lead to some fun and confusing problems. Imagine if the ancient and recent Go versions use the shared module cache differently, for example.

3. Similar to the last point, I wonder if this change could make it easier for projects or companies to stay on older versions of Go for a longer time. As of today, if a project is stuck on Go 1.17.x when 1.19.x is the latest, that is an increasingly significant problem given that the old release is no longer getting security fixes nor backports. And, given the inconvenience that is to install and use an older Go version, it's likely that someone will look into a fix. With the proposal, I imagine that the only external reminder that the project would have is when dependencies are upgraded and require a newer Go toolchain. That might be enough in many cases, but I'm not sure.

4. I imagine that the early consumption of go.mod to check the go and toolchain lines should be a pretty forgiving or lazy parse of the file, to allow us to add more features to the file in the future without breaking this mechanism in older Go versions.

[mvdan]

(Not content-related, but how did you fix my list so that the two-paragraph first bullet rendered properly? I noticed that but did not know how to avoid it.)

[rsc]

Thanks for the careful thoughts. To reply to a few of them:

(1) Builds using Go's dev branches would certainly handle any go version listed in go.mod, so the only possible time they'd delegate to an alternate toolchain would be an explicit toolchain line. Since this is a discussion and not a proposal, I don't have every detail worked out: maybe such builds would default to GOTOOLCHAIN=local instead of GOTOOLCHAIN=auto, under the theory that you're working on a bleeding edge Go and want to use it in preference to others. I'm not sure.

(2) There's always the possibility of added confusion, but most problems I can think of would also be caused simply by having two different Go toolchains on a machine. If things like 'go install golang.org/dl/go1.19@latest; go1.19 download; go1.19 build' are OK today, then this should be OK too. We were pretty careful in the design of the module cache and build cache to allow different versions of Go to coexist.

(3) I believe the point is that 'toolchain go1.17' makes it easy for a project to keep using an older version of Go, and contributors working on it will "auto-downgrade" and not necessarily provide motivation for updating. I suppose that is true, although I'm not sure how common that would be. Perhaps we can do something to flag this more aggressively, although I am not quite sure what.

(4) Yes, definitely. The go.mod file has no multiline comments, so it suffices to do something like a search for (?m)^\s*go\s+(.*)$ in the file.

Thanks again.

[rsc]

(In Markdown you make extra paragraphs continue the list item by indenting them at least 3 spaces.)

[mvdan]

Thanks for your reply!

1. Making tip default to the local toolchain is an interesting idea. I think I would like that, personally. If I'm using Go tip, it is to explicitly test out that newer Go version. I wonder where that would leave beta and RC pre-releases; I guess those should work like stable releases.

2. It's true that this is already a problem - it would just make using multiple Go versions at once much more common. I guess this depends on how much you anticipate to change how Go's caches work, or e.g. the default location of GOMODCACHE. I am worried about changes like cmd/go: add GOMODCACHE #34527, for example, which could lead to different Go versions using different cache locations.

3. I am indeed thinking that we'll want some way to flag this to users. Perfectly fine to leave that out of the initial implementation, but I think it should be mentioned in the proposal somehow.

[msiner]

The features described here make some assumptions about the environment go is used in that don't match with some of my main use cases. I'll attempt to describe those here as data points for consideration. If implemented as currently described, I believe I would be using GOTOOLCHAIN=local in most cases.

I build and deploy custom Linux images, using Yocto, for single board computers. Those images contain both go (currently version 1.15.8) and custom software written in Go (with some requiring Cgo). Yocto builds and/or version controls both the software used to build the image and the software built for the image. If multiple or different versions of go are required, it would need to be explicitly specified at a higher level than go.mod. While my experience is limited to Yocto, I assume this is a model shared by many Linux distro build systems.

Once deployed, software written in Go can be built on those single board computers. This is mostly done for faster development iteration and for troubleshooting. Typically, those hosts are not connected to the Internet or even a network that might host something like Gitlab or a Go modules proxy. Vendoring is used to enable building on these hosts. Also, installing multiple versions of go would have a negative effect on image size.

[rsc]

Thanks for sharing this use case. I really appreciate these kinds of details.

In this case, it sounds like you control both the toolchain and the source code being loaded into the SBC, including the go.mod files. As long as the go.mod files do not specify a newer Go version in their 'go' lines, nor a toolchain line listing a different version of Go than what's installed, everything would continue to behave as before.

Setting GOTOOLCHAIN=local would override any go.mod toolchain line. If the go.mod says 'go 1.101' when the toolchain is Go 1.100, then GOTOOLCHAIN=local will cause the build to fail, but so would not having the internet, so there's not too much difference. Either way, since you control the go.mod files on the computer, it sounds like you can easily ensure that they don't ask for something that the local toolchain can't already supply.

[zikaeroh]

One thing I haven't seen mentioned is how distributions / package maintainers are expected to handle this sort of thing; my understanding is that this proposal is effectively saying that Go will download prebuilt binaries, rather than using what was actually installed on the system. I think that's going to end up really surprising (or further, undesirable) to package maintainers, where the expectation is that users are getting the version of Go that they installed. It would be really surprising for someone like Ubuntu to backport a fix to their Go distribution, only to have Go completely ignore that and invoke some other, unpackaged version.

Also, distros like Nix/NixOS cannot be used out-of-the-box with prebuilt binaries intended for a "regular" Linux system; trying to execute releases straight from golang.org is very likely to fail, because things like libc and such live in completely different locations.

In the same vein, I'm aware of corporate environments which have their own custom "approved" Go distribution; those similarly would be harmed by logic that explicitly tries to invoke a toolchain that's not the installed one. Those environments already require some element of patching, so maybe it's not such a big deal to also set a different default, but it's another knob that upstream isn't expecting to be changed.

[jech]

I am using Go for undergraduate teaching.

In early September, I send an e-mail to our administrators asking them to install some recent version of the Go compiler ("Please install go-1.18 or later, you may verify by typing "go version"). I then make sure that the practicals work with the version of Go that was installed. The admins might occasionally upgrade the system in the middle of the semester, sometimes without warning me.

At first sight, your proposal might cause a number of issues:

• I'd need to request a higher disk quota for the students who follow my lecture, which would inevitably lead to pressure to use a more conservative language (I'm already under some pressure to stick to Java, like most of the other lectures);
• I'd lose at least 15 minutes at the beginning of the first practical after each upgrade as 60 students download the Go toolchain simultaneously over an overloaded shared link.

My use case is probably not unique. Please make sure to take it into account.

[rsc]

Thanks for the note. It should suffice to use something like 'toolchain local' in the go.mod you give your students or else export GOTOOLCHAIN=local.

[rsc]

This discussion has been very helpful. Thanks everyone!

[candlerb]

Personally, I'm uncomfortable with downloading the whole toolchain automatically on demand, just because some dependency happens to be using a newer version of go than me (which may or may not need new features; it just happens to declare the latest version in go.mod because that's what the author has on their system).

Also I might want my software built with 1.18, say, because I'm not ready for some upcoming potentially-breaking feature in 1.19. I do realise those cases are rare (async preemption is one that springs to mind).

However, a mechanism like pyenv/rustup to switch between explicitly-installed versions, and/or to install them on request, would be useful.

Right now, the go binary release tarballs (for Linux at least) have go/ as the top-level directory name, on the assumption that you'll most likely unpack this under /usr/local or your home directory, and only have one version installed.

What I find myself doing is to untar, then rename the outer directory to (say) go-1.19.2, then move this under /usr/local/, and then symlink /usr/local/go to go-1.19.2 (or I can change my $PATH). It does feel like I'm fighting against the packaging though. What if the release tarballs were to have go-1.19.2/ or go/1.19.2/ as the top-level directory?

The difference between "go" doing this, and a separate "goup" or whatever, is about whether it takes place automatically. If "go" itself does auto-upgrades, then in the limit, you could turn "go" into a wrapper which knows nothing except how to download a published toolchain into your local package cache and invoke it.

Minor point: on multi-user systems with N users and M different newer versions of go referenced from dependencies, you'll get NxM copies of the toolchain installed. But who uses timesharing these days? :-)

[philpennock]

In support of the need for something like this proposal: we already see CI systems trying to overload the meaning of the go version in the go.mod file. In https://github.com/actions/setup-go, the baseline Go environment setup action for GitHub Actions, the go-version-file directive is used to take the version of Go from the go.mod file.

It seems to me that this fits in neatly with the proposal: the action would:

1. Switch to preferring the toolchain directive in the go.mod file if present
2. Export GOTOOLCHAIN=local for the runtime environment of Go, particularly to handle matrix versions where the maintainer has explicitly configured testing with specific versions of Go
3. Continue to use their existing cache and cache-aging mechanisms which they already have to have anyway
4. Encourage people to stop scattering version numbers in multiple different places inside a repo and instead default to the go.mod toolchain directive where not explicitly overruled otherwise

So I am very much in favor of providing a directive guiding tooling to "which version of Go should be used", vs "which is the baseline set of semantics which should be used when compiling this code, which might be several releases behind the minimum supported because the code just isn't using anything new".

None of the above requires the automatic download side of things, and I can see that running into compliance and regulatory issues for developer environments (even if automatic download of modules is somehow handwaved away already). So I'm neutral on the auto-download, but very firmly in favor of the directive itself (even if the default Go tools end up doing absolutely nothing with it).

[rsc]

This is now a proposal at #57001.

[prologic]

@rsc I can see what you're trying to do here, but this is really "over engineering". I've never seen build tools attempt to reach out to the internet as much as much as what you are proposing here and in the other controversial issue. Why do you feel the need to make go do "magical"™ things that have undesired, unwanted or even not well understood side-effects?

If I want to use a new Go compiler version, I will just go upgrade to that new version. I really do not want go to go download a temporary Go compiler of a newer version for me. That's just "eww". Name one other compiler that does this?

Please keep Go simple. It's a great language.

[dmitris]

Assume that we publish all binary Go distributions as module zip files (in addition to the usual zip files and other installers). Then if the go command notices it needs a different distribution, it can download it from the module proxy and authenticate it using the checksum database. (We don't need to invent a separate authenticated download channel, nor a separate caching mechanism.)

I think this could go against (or at least have some "friction" with) a possible corporate practice and requirement to only install the Go toolchain from the internal mirrors (ex. corporate Artifactory). Please consider this scenario if/when moving to the implementation! 😄

As an aside - it would be fantastic to have a goup tool similar to rustup and ghcup that would provide a standard way to download, install and update the Go toolchain as well as switch between different versions (maybe even including the beta/nightly). Also to have some equivalents to RUSTUP_UPDATE_ROOT and RUSTUP_DIST_SERVER that would allow to download the toolchain from the given mirror server with something similar to https://github.com/panamax-rs/panamax! 🙏 😆

[prologic]

The problem I have is that I do not want my compiler to be reaching out to the Internet and installing a new version of the compiler somewhere without my control.

[willfaught]

@prologic You can do GOTOOLCHAIN=local for that.

[gophun]

The go tool is not the compiler, it's an all-round tool to make a Go programmer's life easy.

[prologic]

@prologic You can do GOTOOLCHAIN=local for that.

I shouldn't have to do this 🤦‍♂️

[DeedleFake]

I somewhat agree, but if you run go env -w GOTOOLCHAIN=local it'll set it for all future calls by that user.