Updated COMPOSE_ENV_FILES in env files

[ls-aron-kyle]

Updated the docker compose command to recursively search through any specified .env files for the COMPOSE_ENV_FILES parameter.

When found, the paths are followed recursively to extend the list of .env files used in the command.
A cache is kept that prevents circular dependencies.
Recursion is depth first.

Related issue

Example fix for #11122

(not mandatory) A picture of a cute animal, if possible in relation to what you did

[ndeloof]

👎 IMHO having an env file to declare COMPOSE_ENV_FILES demonstrates a mistake organizing config
It seems what you're looking for is some kind f support for include in dot env file. We have no plan yet for such a feature

[ls-aron-kyle]

👎 IMHO having an env file to declare COMPOSE_ENV_FILES demonstrates a mistake organizing config It seems what you're looking for is some kind f support for include in dot env file. We have no plan yet for such a feature

@ndeloof

The situation we're trying to solve is to standardize our env files while still providing a place to put local values. i.e. we'd like to commit the .env file but optionally reference a second file that devs can use for their personal config values, something like ~/.env.local. By standardizing the locations it's easier to add automation to provision the ~/.env.local file.

Judging by the comments in thread in #11122 there is interest in this feature.

I'm open to discussing an include feature, in particular how it might differ from this solution. At the end of the day I think we all just want the best user experience possible.

[ShadowLNC]

There seems to be a common issue that some config is tracked/dynamically pulled in one file, and some is managed by the user on a personal basis in a separate file (which is not tracked). Right now, Compose has no way to support this structure without changing the CLI invocation.

We can't set COMPOSE_ENV_FILES globally in the shell configuration, because there might be other projects which don't manage .env files the same way. We also can't expect users to remember to specify the "magic" --env-file ... arguments (in many cases, the users don't care about these files at all, and may not even know they exist if they're working on code in a container and not the Compose configuration itself).

For now, I can't use COMPOSE_ENV_FILES at all, so I've got a wrapper script around docker compose to add in the --env-files argument. This has attracted additional scrutiny on my code, where I'm being asked "why aren't we just letting users directly invoke docker compose?".

I wonder if we can settle on a "halfway" point with the following restrictions (which may help alleviate @ndeloof's concerns both here and on the linked issue):

• If Compose is not given the flag --env-file or the shell env var COMPOSE_ENV_FILES, it will first parse .env and see if COMPOSE_ENV_FILES exists
• If .env does have COMPOSE_ENV_FILES:
  • If it doesn't list .env, the remainder of .env is ignored
  • For any file that is listed (and parsed), including .env, no further attempt is made to read COMPOSE_ENV_FILES
• If .env does not have COMPOSE_ENV_FILES, then we follow the current main behaviour

[ndeloof]

I'm being asked "why aren't we just letting users directly invoke docker compose?".

Compose comes with many flags to support customization and various usages, it is expected docker compose up can't cover all needs out of the box. It's 100% legitimate to have a launcher script, typically a Makefile to select adequate flags depending on your use-case: run in development, run with debugger enabled, run for tests, etc.