INFENG-382: Release redesign (#10002)

INFENG-848: Fix gen-versions.py (#9908)

INFENG-809: Update CircleCI tag logic and add release dryrun workflow (#9877)

INFENG-812: fix model_hub Docker tag (#9809)

* Update package-and-push-system-dev and package-and-push-system-dev-ee
   CircleCI jobs to build dev Docker images correctly. Specifically, fix
   a bash if statement to check for if we're running the job from the
   "main" branch or a branch prefixed with "release-". The previous
   behavior would match on any git branch regardless.

INFENG-780: Remove CircleCI bumpversion (#9785)

* Remove bump2version ("bumpversion", colloquially) dependency, both
  literally from requirements.txt and also the last remaining reference
  in its config file, .bumpversion.cfg, along with the config file
  itself. The remaining reference was in .circleci/real_config.yml.
  Removing it, though, was somewhat tricky, and had a few knock-on
  effects that had to be solved.

  The version string itself was a default for the pipeline parameter
  det-version, which is threaded through the entirety of the build
  process as a source of truth for the Determined version
  information. Instead of updating the CircleCI config itself, we can
  dynamically generate it in .circleci/config.yml and set the parameter
  there. There's still some work left to do to get tag builds working
  correctly, but for now, I call version.sh to set det-version, which
  should work correctly for all branch builds, by returning the local
  version string, e.g. 0.34.0+2e616a3d7

* Update version.sh to fix a small bug that caused git tags with
  subsequent 'v's in them, e.g. v1.0.3-dev0, to be munged incorrectly
  into 1.0.3-de0. Now, we use ${parameter#word} shell expansion to
  remove the first singular 'v' from tags.

* Update two direct invocations of setup.py in CircleCI to instead use
  the build module. Invoking setup.py directly is deprecated.

* Update the agent creation code to URL-escape the query parameter
  that's used to set the agent version. Under the new versioning scheme,
  local build versions use a short SHA after a plus + for uniqueness,
  and the plus sign was being interpreted as being part of a URL, where
  it was then turned into a space. This broke tests that attempt to
  fetch the agent version (it's also just plain incorrect). Now, the
  version string is properly escaped, and the proper agent version is
  returned.

* Update how setup.py and the determined (harness/) Python package
  manage versions. I kept running into test errors that indicated that
  the correct version wasn't being set somewhere in setuptools. At
  first, I mistakenly assumed that pytest was, somehow, using the bare
  source for testing. This is incorrect, though (and makes no sense
  anyway). pytest still needs to import packages, and it does so after
  we pip install an editable install of determined. Editable installs
  still end up calling setuptools, which means the built package has
  package metadata, and so we can still use importlib.metadata locally
  to fetch version information.

  The only bit that really had to be changed was harness/setup.py and
  model_hub/setup.py. I did tweak harness/determined/__init__.py, but
  that's just to catch the possible
  importlib.metadata.PackageNotFoundError exception. I believe this will
  never happen, but I've been wrong before. And there really is no
  excuse for __init__.py to ever do anything that isn't successful. So
  now, and it feels mildly gross, but is not too different from how
  other large projects like numpy manage versions, setup.py simply
  shells out to version.sh to get a version number during package build,
  if VERSION isn't already set. I realize that I also check VERSION in
  version.sh, but a belt and suspenders won't hurt. If VERSION is set,
  we use that; otherwise, we get the local (+SHA) version.

  This enables us to do editable installations, which a bunch of things
  assume anyway, without having to ensure VERSION is set. Before,
  setup.py assumed VERSION was set, or else it would error out. But when
  you pip install -e harness directly, like in CI or even locally (which
  can happen with pip install -r requirements.txt, as -e harness is the
  first line), that variable won't necessarily be set. So instead,
  setup.py checks VERSION, and then, if it's empty, attempts to fetch
  the version itself.

  If setup.py and __init__.py can't retrieve versions correctly, they'll
  return default values. setup.py will return 1!0.0.0+unknown, whereas
  __init__.py will return 2!0.0.0+unknown. My hope is that using two
  different versions can help indicate where the error is, and using
  epochs should make package requirement resolution still pass, as both
  1 and 2 are greater than the implicit 0 epoch that our version numbers
  use now.

* Update/remove comments that reference bumpversion.

* Add additional VERSION environment variable to CircleCI config.

* Update Python file to comport with black.

* Add return types to functions in setup.py to satisfy mypy.

INFENG-743: Remove Helm chart bumpversion

* Update Chart.yaml version strings to dummy values: 0.0.0, and instead
  pass VERSION in through --version and --app-version flags during helm
  package in helm/Makefile.

INFENG-744: Remove VERSION file

* Remove dependency on static VERSION file and replace it with a bash
  script, version.sh, that either fetches the version from git tags,
  usually used for local builds, or returns an existing environment
  variable, mainly used in CI.

* The script uses a few different git tools to find the most recent
  previous tag for a given commit, a combination of git-describe,
  git-tag, and git-merge-base. Additional implementation details are
  available in the comments in that file. The resulting version, though,
  is <prev-tag>+<HEAD-short-SHA>. This ensures that certain Python
  dependency constraints are able to be satisfied when building
  Determined locally. Otherwise, if VERSION is already set, as will be
  the case in CI, the script simply returns that instead.

* Replace existing `cat`s of the VERSION file with invocations of this
  shell script; specifically, in the following files:

- .devcontainer/server.sh
- .github/workflows/lint-python.yml
- agent/Makefile
- docs/Makefile
- docs/deploy/Makefile
- helm/Makefile
- master/Makefile
- model_hub/Makefile

* Invoke version.sh in the Python Determined module Makefile:
  harness/Makefile. This was necessary to replace my previous attempt at
  fetching version string information from git using a Pythom setuptools
  plugin called setuptools_scm. Although that library works and is quite
  handy, and I sincerely thank Ronny Pfannschmidt and all other
  contributors for their work, configuring it was non-trivial, at times,
  and it was becoming difficult to cleanly get it to do what I wanted. I
  also decided that it makes more sense to have a single, canonical
  version string (not unlike the VERSION file itself) that we use for as
  many things as possible, notwithstanding certain potential
  incongruences between how different systems parse version strings.

  Now, instead of using setuptools_scm, we read the environment
  variable in harness/setup.py, and pass that to setuptools as a
  dynamic version. This also helps ensure that there aren't multiple
  ways to fetch a canonical version, which could become out of sync
  if we aren't careful. It does, however, mean that you need to use
  the Makefile to build the determined module; if you want to invoke
  `python -m build` directly, you'll have to set VERSION
  yourself. This is unlikely. If VERSION is unset, setup.py will
  raise an exception.

* Remove the --version-file flag from docs/deploy/upload.py. I don't
  think it's even used, but it's certainly unnecessary now.

* Replace reading the VERSION file in docs/deploy/upload.py with
  fetching the VERSION environment variable, which should be set by the
  docs makefile. I've done the same thing in docs/conf.py.

* Although I've described what version.sh does, the diff will show some
  minor tweaks, like editing comments and moving some tag-munging
  commands around. I've also removed `set -x`, because I forgot that
  executing the script locally, even in a subshell, will set shell
  options globally, and apply after the script exits. This is
  undesirable. There are ways to mitigate this, but it's not a pressing
  issue.

* Remove VERSION file from .bumpversion.cfg.

INFENG-739: remove version string from Vite config

* Replace the process.env.VERSION variable explicitly defined in
  vite.config.mts with either the actual VERSION environment variable,
  if it's set, or the most recent git tag, followed by an optional
  commit counter and partial unique SHA hash. This shells out to git,
  and accounts for the rare but possible case that the config is being
  from outside of a git repository. In that situation, the version
  string is useless anyway, so we return "unknown" from the shell.

* During the release, we will set VERSION ourselves based on the tag, so
  we don't have to go over the top trying to parse it out here. The
  local version doesn't matter too much, either, as it's only referenced
  during development, and, as far as I can tell, is really only used to
  display the current version to the user. During the release, we can
  set VERSION from CircleCI directly, as it already provides the tag if
  the run was triggered by one.

INFENG-726: Generate versions.json from git tags

This changeset introduces a new Python script, docs/gen-versions.py, to
generate the existing versions.json file from git tags, while
maintaining the previous structure of the file and incorporating new
tags as they get created.

My initial thought was to use the existing versions.json file as a
template, and then append subsequent entries to it from new git tags as
they get created and pushed. This felt brittle, though, and very special
case-y.

Instead, I use the GitPython version of git-rev-list to walk the DAG
from all tags back to right before 0.21.0. I then filter out all tags
that aren't a major, minor, or patch release, and exclude versions that
have tags, but without corresponding entries in the original
versions.json file. This gives us all relevant doc tags between now and
0.21.0 (inclusive), which we can use to generate versions.json. It also
works for any new tags we use going forward.

The resultant file still winds up where it normally would, and should
get packaged and released like it would if it were stored in git on disk.

The summary of changes is as follows:

* Add gen-versions.py, which generates versions.json from git tags.

* Add a new docs/Makefile target, gen-versions, to run gen-versions.py
  as part of the docs build process.

* Remove versions.json from .bumpversion.cfg.

* Remove versions.json.

INFENG-709: Remove CloudFormation versions

* Remove hardcoded Determined versions from CloudFormation
  templates. Because we're already passing through an optional command
  line parameter anyway, we can instead set that default value to the
  existing determined.__version__ variable pulled from package metadata,
  instead of setting a default value in the `Version` parameter of
  CloudFormation templates.

INFENG-704: Remove bumpversion

* Update determined Python package to fetch its own version dynamically
  from package metadata using importlib.metadata, which seems to be the
  canonical way to fetch the version string at runtime. In lieu of
  something more complicated, I assigned a module-level __version__
  variable in __init__.py to avoid touching the rest of the package code
  that relies on this behavior.

* Migrate all of the setup.py options into the existing pyproject.toml,
  except options that are no longer necessary, leaving the
  long_description in because pyproject.toml seems to be unable to
  reference README files above the pyproject.toml root.

* Remove harness/determined/__version__.py and
  model_hub/model_hub/__version__.py, as they are no longer
  necessary. The __version__ dunder is now provided by __init__.py. See
  PEP-396 <https://peps.python.org/pep-0396/> for more information about
  why this is no longer desirable.

* Add explicit dependencies on setuptools>=64 and setuptools_scm>=8 in
  harness/pyproject.toml and model_hub/pyproject.toml to allow tagging
  versions dynamically from git.

* Add setuptools==70.0.0 and setuptools-scm==8.1.0 as dependencies to
  support setuptools dynamically using git tag information.

* Set build-backend to setuptools.build_meta to use setuptools_scm.

* Move all setuptools.setup() args into pyproject.toml, because newer
  version of setuptools, if I understand correctly, have support for
  them.

* Remove bumpversion config for setup.py and __version__.py.

.circleci/config.yml

@@ -87,6 +87,16 @@ jobs:
               mv -v tmpfile "<<pipeline.parameters.params_basename>>"
             fi
 
+      - run:
+          name: Set det-version parameter.
+          command: |
+            VERSION=$(./version.sh)
+            echo "Version is: [${VERSION}]"
+            echo "CIRCLE_TAG is: [${CIRCLE_TAG}]"
+            jq --arg version "${VERSION}" '. += {"det-version": $version}' < "<<pipeline.parameters.params_basename>>" > tmpfile
+            mv -v tmpfile "<<pipeline.parameters.params_basename>>"
+            cat "<<pipeline.parameters.params_basename>>"
+
       # this must be last; persist the file to the workspace
       - persist_to_workspace:
           root: .