up-to-date/base-image: cache dockerhub image listing #47447
Merged
+51
−24
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
up-to-date
: to find the latest base image we have to list all the base images available and fetch their digest. If this operation is not cached it can easily lead to DockerHub rate limiting issue.cache_ttl_seconds
parameters to cache the interaction with thecrane
client. Inup-to-date
we set this cache TTL to 1 day.This pull request introduces a caching mechanism to avoid DockerHub rate limiting and updates various files to support this new feature. The most important changes include adding a cache TTL for base image listings, updating the
CraneClient
and related methods to accept a cache TTL, and adding logging for digest fetching.Caching Mechanism:
airbyte-ci/connectors/base_images/README.md
: Added a changelog entry for version 1.1.0, mentioning the cache TTL addition.airbyte-ci/connectors/base_images/base_images/utils/docker.py
: Modified theCraneClient
class to accept acache_ttl_seconds
parameter and updated thebare_container
initialization to use this TTL.airbyte-ci/connectors/base_images/base_images/version_registry.py
: Updated methods to pass thecache_ttl_seconds
parameter to theCraneClient
. [1] [2] [3] [4]airbyte-ci/connectors/pipelines/README.md
: Documented the use ofcache_ttl
for base image registry listing in version 4.41.8.airbyte-ci/connectors/pipelines/pyproject.toml
: Bumped the version to 4.41.8.Logging Enhancements:
airbyte-ci/connectors/base_images/base_images/utils/docker.py
: Added logging statements for digest fetching to help identify bottlenecks and improve debugging.