Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
INFENG-382: Release redesign (#10002)
INFENG-848: Fix gen-versions.py (#9908) INFENG-809: Update CircleCI tag logic and add release dryrun workflow (#9877) INFENG-812: fix model_hub Docker tag (#9809) * Update package-and-push-system-dev and package-and-push-system-dev-ee CircleCI jobs to build dev Docker images correctly. Specifically, fix a bash if statement to check for if we're running the job from the "main" branch or a branch prefixed with "release-". The previous behavior would match on any git branch regardless. INFENG-780: Remove CircleCI bumpversion (#9785) * Remove bump2version ("bumpversion", colloquially) dependency, both literally from requirements.txt and also the last remaining reference in its config file, .bumpversion.cfg, along with the config file itself. The remaining reference was in .circleci/real_config.yml. Removing it, though, was somewhat tricky, and had a few knock-on effects that had to be solved. The version string itself was a default for the pipeline parameter det-version, which is threaded through the entirety of the build process as a source of truth for the Determined version information. Instead of updating the CircleCI config itself, we can dynamically generate it in .circleci/config.yml and set the parameter there. There's still some work left to do to get tag builds working correctly, but for now, I call version.sh to set det-version, which should work correctly for all branch builds, by returning the local version string, e.g. 0.34.0+2e616a3d7 * Update version.sh to fix a small bug that caused git tags with subsequent 'v's in them, e.g. v1.0.3-dev0, to be munged incorrectly into 1.0.3-de0. Now, we use ${parameter#word} shell expansion to remove the first singular 'v' from tags. * Update two direct invocations of setup.py in CircleCI to instead use the build module. Invoking setup.py directly is deprecated. * Update the agent creation code to URL-escape the query parameter that's used to set the agent version. Under the new versioning scheme, local build versions use a short SHA after a plus + for uniqueness, and the plus sign was being interpreted as being part of a URL, where it was then turned into a space. This broke tests that attempt to fetch the agent version (it's also just plain incorrect). Now, the version string is properly escaped, and the proper agent version is returned. * Update how setup.py and the determined (harness/) Python package manage versions. I kept running into test errors that indicated that the correct version wasn't being set somewhere in setuptools. At first, I mistakenly assumed that pytest was, somehow, using the bare source for testing. This is incorrect, though (and makes no sense anyway). pytest still needs to import packages, and it does so after we pip install an editable install of determined. Editable installs still end up calling setuptools, which means the built package has package metadata, and so we can still use importlib.metadata locally to fetch version information. The only bit that really had to be changed was harness/setup.py and model_hub/setup.py. I did tweak harness/determined/__init__.py, but that's just to catch the possible importlib.metadata.PackageNotFoundError exception. I believe this will never happen, but I've been wrong before. And there really is no excuse for __init__.py to ever do anything that isn't successful. So now, and it feels mildly gross, but is not too different from how other large projects like numpy manage versions, setup.py simply shells out to version.sh to get a version number during package build, if VERSION isn't already set. I realize that I also check VERSION in version.sh, but a belt and suspenders won't hurt. If VERSION is set, we use that; otherwise, we get the local (+SHA) version. This enables us to do editable installations, which a bunch of things assume anyway, without having to ensure VERSION is set. Before, setup.py assumed VERSION was set, or else it would error out. But when you pip install -e harness directly, like in CI or even locally (which can happen with pip install -r requirements.txt, as -e harness is the first line), that variable won't necessarily be set. So instead, setup.py checks VERSION, and then, if it's empty, attempts to fetch the version itself. If setup.py and __init__.py can't retrieve versions correctly, they'll return default values. setup.py will return 1!0.0.0+unknown, whereas __init__.py will return 2!0.0.0+unknown. My hope is that using two different versions can help indicate where the error is, and using epochs should make package requirement resolution still pass, as both 1 and 2 are greater than the implicit 0 epoch that our version numbers use now. * Update/remove comments that reference bumpversion. * Add additional VERSION environment variable to CircleCI config. * Update Python file to comport with black. * Add return types to functions in setup.py to satisfy mypy. INFENG-743: Remove Helm chart bumpversion * Update Chart.yaml version strings to dummy values: 0.0.0, and instead pass VERSION in through --version and --app-version flags during helm package in helm/Makefile. INFENG-744: Remove VERSION file * Remove dependency on static VERSION file and replace it with a bash script, version.sh, that either fetches the version from git tags, usually used for local builds, or returns an existing environment variable, mainly used in CI. * The script uses a few different git tools to find the most recent previous tag for a given commit, a combination of git-describe, git-tag, and git-merge-base. Additional implementation details are available in the comments in that file. The resulting version, though, is <prev-tag>+<HEAD-short-SHA>. This ensures that certain Python dependency constraints are able to be satisfied when building Determined locally. Otherwise, if VERSION is already set, as will be the case in CI, the script simply returns that instead. * Replace existing `cat`s of the VERSION file with invocations of this shell script; specifically, in the following files: - .devcontainer/server.sh - .github/workflows/lint-python.yml - agent/Makefile - docs/Makefile - docs/deploy/Makefile - helm/Makefile - master/Makefile - model_hub/Makefile * Invoke version.sh in the Python Determined module Makefile: harness/Makefile. This was necessary to replace my previous attempt at fetching version string information from git using a Pythom setuptools plugin called setuptools_scm. Although that library works and is quite handy, and I sincerely thank Ronny Pfannschmidt and all other contributors for their work, configuring it was non-trivial, at times, and it was becoming difficult to cleanly get it to do what I wanted. I also decided that it makes more sense to have a single, canonical version string (not unlike the VERSION file itself) that we use for as many things as possible, notwithstanding certain potential incongruences between how different systems parse version strings. Now, instead of using setuptools_scm, we read the environment variable in harness/setup.py, and pass that to setuptools as a dynamic version. This also helps ensure that there aren't multiple ways to fetch a canonical version, which could become out of sync if we aren't careful. It does, however, mean that you need to use the Makefile to build the determined module; if you want to invoke `python -m build` directly, you'll have to set VERSION yourself. This is unlikely. If VERSION is unset, setup.py will raise an exception. * Remove the --version-file flag from docs/deploy/upload.py. I don't think it's even used, but it's certainly unnecessary now. * Replace reading the VERSION file in docs/deploy/upload.py with fetching the VERSION environment variable, which should be set by the docs makefile. I've done the same thing in docs/conf.py. * Although I've described what version.sh does, the diff will show some minor tweaks, like editing comments and moving some tag-munging commands around. I've also removed `set -x`, because I forgot that executing the script locally, even in a subshell, will set shell options globally, and apply after the script exits. This is undesirable. There are ways to mitigate this, but it's not a pressing issue. * Remove VERSION file from .bumpversion.cfg. INFENG-739: remove version string from Vite config * Replace the process.env.VERSION variable explicitly defined in vite.config.mts with either the actual VERSION environment variable, if it's set, or the most recent git tag, followed by an optional commit counter and partial unique SHA hash. This shells out to git, and accounts for the rare but possible case that the config is being from outside of a git repository. In that situation, the version string is useless anyway, so we return "unknown" from the shell. * During the release, we will set VERSION ourselves based on the tag, so we don't have to go over the top trying to parse it out here. The local version doesn't matter too much, either, as it's only referenced during development, and, as far as I can tell, is really only used to display the current version to the user. During the release, we can set VERSION from CircleCI directly, as it already provides the tag if the run was triggered by one. INFENG-726: Generate versions.json from git tags This changeset introduces a new Python script, docs/gen-versions.py, to generate the existing versions.json file from git tags, while maintaining the previous structure of the file and incorporating new tags as they get created. My initial thought was to use the existing versions.json file as a template, and then append subsequent entries to it from new git tags as they get created and pushed. This felt brittle, though, and very special case-y. Instead, I use the GitPython version of git-rev-list to walk the DAG from all tags back to right before 0.21.0. I then filter out all tags that aren't a major, minor, or patch release, and exclude versions that have tags, but without corresponding entries in the original versions.json file. This gives us all relevant doc tags between now and 0.21.0 (inclusive), which we can use to generate versions.json. It also works for any new tags we use going forward. The resultant file still winds up where it normally would, and should get packaged and released like it would if it were stored in git on disk. The summary of changes is as follows: * Add gen-versions.py, which generates versions.json from git tags. * Add a new docs/Makefile target, gen-versions, to run gen-versions.py as part of the docs build process. * Remove versions.json from .bumpversion.cfg. * Remove versions.json. INFENG-709: Remove CloudFormation versions * Remove hardcoded Determined versions from CloudFormation templates. Because we're already passing through an optional command line parameter anyway, we can instead set that default value to the existing determined.__version__ variable pulled from package metadata, instead of setting a default value in the `Version` parameter of CloudFormation templates. INFENG-704: Remove bumpversion * Update determined Python package to fetch its own version dynamically from package metadata using importlib.metadata, which seems to be the canonical way to fetch the version string at runtime. In lieu of something more complicated, I assigned a module-level __version__ variable in __init__.py to avoid touching the rest of the package code that relies on this behavior. * Migrate all of the setup.py options into the existing pyproject.toml, except options that are no longer necessary, leaving the long_description in because pyproject.toml seems to be unable to reference README files above the pyproject.toml root. * Remove harness/determined/__version__.py and model_hub/model_hub/__version__.py, as they are no longer necessary. The __version__ dunder is now provided by __init__.py. See PEP-396 <https://peps.python.org/pep-0396/> for more information about why this is no longer desirable. * Add explicit dependencies on setuptools>=64 and setuptools_scm>=8 in harness/pyproject.toml and model_hub/pyproject.toml to allow tagging versions dynamically from git. * Add setuptools==70.0.0 and setuptools-scm==8.1.0 as dependencies to support setuptools dynamically using git tag information. * Set build-backend to setuptools.build_meta to use setuptools_scm. * Move all setuptools.setup() args into pyproject.toml, because newer version of setuptools, if I understand correctly, have support for them. * Remove bumpversion config for setup.py and __version__.py.
- Loading branch information