drone-cache

A Drone plugin for caching current workspace files between builds to reduce your build times. drone-cache is a small CLI program, written in Go without any external OS dependencies (such as tar, etc).

With drone-cache, you can provide your own cache key templates, specify archive format (tar, tar.gz, etc) and you can use popular object storage as storage for your cached files, even better you can implement your custom storage backend to cover your use case.

For detailed usage information and a list of available options please take a look at usage and examples. If you want to learn more about custom cache keys, see cache key templates.

If you want to learn more about the story behind drone-cache, you can read our blogpost Making Drone Builds 10 Times Faster!!

Supported Storage Backends

• AWS S3
  • Configuration
  • Example
  • Other AWS API compatible stores:
    • Minio
    • Red Hat Ceph
    • IBM Object Store
    • and many many others
• Azure Storage
  • Configuration
  • Example
• Google Cloud Storage
  • Configuration
  • Example
• Alibaba OSS Storage
  • Configuration
  • Example
• or any mounted local volume
  • Configuration
  • Example

How does it work

drone-cache stores mounted directories and files under a key at the specified backend (by default S3).

Use this plugin to cache data that makes your builds faster. In the case of a cache miss or an empty cache restore it will fail silently in won't break your running pipeline.

The best example would be to use this with your package managers such as Mix, Bundler or Maven. After your initial download, you can build a cache and then you can restore that cache in your next build.

With restored dependencies from a cache, commands like mix deps.get will only need to download new dependencies, rather than re-download every package on each build.

Example Usage of drone-cache

The following example configuration file (.drone.yml) shows the most common use of drone-cache.

Simple (with AWS S3 backend)

kind: pipeline
name: default

steps:
  - name: restore-cache
    image: meltwater/drone-cache
    environment:
      AWS_ACCESS_KEY_ID:
        from_secret: aws_access_key_id
      AWS_SECRET_ACCESS_KEY:
        from_secret: aws_secret_access_key
    pull: true
    settings:
      restore: true
      cache_key: '{{ .Commit.Branch }}-{{ checksum "go.mod" }}' # default if ommitted is {{ .Commit.Branch }}
      bucket: drone-cache-bucket
      region: eu-west-1
      mount:
        - 'vendor'

  - name: build
    image: golang:1.18.4
    pull: true
    commands:
      - make drone-cache

  - name: rebuild-cache
    image: meltwater/drone-cache
    pull: true
    environment:
      AWS_ACCESS_KEY_ID:
        from_secret: aws_access_key_id
      AWS_SECRET_ACCESS_KEY:
        from_secret: aws_secret_access_key
    settings:
      rebuild: true
      cache_key: '{{ .Commit.Branch }}-{{ checksum "go.mod" }}' # default if ommitted is {{ .Commit.Branch }}
      bucket: drone-cache-bucket
      region: eu-west-1
      mount:
        - 'vendor'

More Examples

• examples for Drone, see docs/examples/drone-1.0.md

Usage

Using executable (with CLI args)

NAME:
   Drone cache plugin - Drone cache plugin

USAGE:
   drone-cache [global options] command [command options] [arguments...]

VERSION:
   v1.4.0

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --access-key value                    AWS access key [$PLUGIN_ACCESS_KEY, $AWS_ACCESS_KEY_ID, $CACHE_AWS_ACCESS_KEY_ID]
   --acl value                           upload files with acl (private, public-read, ...) (default: "private") [$PLUGIN_ACL, $AWS_ACL]
   --alibaba.access-key value            AlibabaOSS access key [$PLUGIN_ALIBABA_ACCESS_KEY, $ALIBABA_ACCESS_KEY_ID, $CACHE_ALIBABA_ACCESS_KEY_ID]
   --alibaba.secret-key value            AlibabaOSS access secret [$PLUGIN_ALIBABA_ACCESS_SECRET, $ALIBABA_ACCESS_SECRET, $CACHE_ALIBABA_ACCESS_SECRET]
   --archive-format value                archive format to use to store the cache directories (tar, gzip, zstd) (default: "tar") [$PLUGIN_ARCHIVE_FORMAT]
   --azure.account-key value             Azure Blob Storage Account Key [$PLUGIN_ACCOUNT_KEY, $AZURE_ACCOUNT_KEY]
   --azure.account-name value            Azure Blob Storage Account Name [$PLUGIN_ACCOUNT_NAME, $AZURE_ACCOUNT_NAME]
   --azure.blob-container-name value     Azure Blob Storage container name [$PLUGIN_CONTAINER, $AZURE_CONTAINER_NAME]
   --azure.blob-max-retry-requets value  Azure Blob Storage Max Retry Requests (default: 4) [$AZURE_BLOB_MAX_RETRY_REQUESTS]
   --azure.blob-storage-url value        Azure Blob Storage URL (default: "blob.core.windows.net") [$AZURE_BLOB_STORAGE_URL]
   --backend value                       cache backend to use in plugin (s3, filesystem, sftp, azure, gcs) (default: "s3") [$PLUGIN_BACKEND]
   --backend.operation-timeout value     timeout value to use for each storage operations (default: 3m0s) [$PLUGIN_BACKEND_OPERATION_TIMEOUT, $BACKEND_OPERATION_TIMEOUT]
   --bucket value                        AWS bucket name [$PLUGIN_BUCKET, $S3_BUCKET, $GCS_BUCKET]
   --build.created value                 build created (default: 0) [$DRONE_BUILD_CREATED]
   --build.deploy value                  build deployment target [$DRONE_DEPLOY_TO]
   --build.event value                   build event (default: "push") [$DRONE_BUILD_EVENT]
   --build.finished value                build finished (default: 0) [$DRONE_BUILD_FINISHED]
   --build.link value                    build link [$DRONE_BUILD_LINK]
   --build.number value                  build number (default: 0) [$DRONE_BUILD_NUMBER]
   --build.started value                 build started (default: 0) [$DRONE_BUILD_STARTED]
   --build.status value                  build status (default: "success") [$DRONE_BUILD_STATUS]
   --cache-key value                     cache key to use for the cache directories [$PLUGIN_CACHE_KEY]
   --commit.author.avatar value          git author avatar [$DRONE_COMMIT_AUTHOR_AVATAR]
   --commit.author.email value           git author email [$DRONE_COMMIT_AUTHOR_EMAIL]
   --commit.author.name value            git author name [$DRONE_COMMIT_AUTHOR]
   --commit.branch value                 git commit branch (default: "master") [$DRONE_COMMIT_BRANCH]
   --commit.link value                   git commit link [$DRONE_COMMIT_LINK]
   --commit.message value                git commit message [$DRONE_COMMIT_MESSAGE]
   --commit.ref value                    git commit ref (default: "refs/heads/master") [$DRONE_COMMIT_REF]
   --commit.sha value                    git commit sha [$DRONE_COMMIT_SHA]
   --compression-level value             compression level to use for gzip/zstd compression when archive-format specified as gzip/zstd
                                             (check https://godoc.org/compress/flate#pkg-constants for available options for gzip
                                             and https://pkg.go.dev/github.com/klauspost/compress/zstd#EncoderLevelFromZstd for zstd) (default: -1) [$PLUGIN_COMPRESSION_LEVEL]
   --debug                               debug (default: false) [$PLUGIN_DEBUG, $DEBUG]
   --encryption value                    server-side encryption algorithm, defaults to none. (AES256, aws:kms) [$PLUGIN_ENCRYPTION, $AWS_ENCRYPTION]
   --endpoint value                      endpoint for the s3/cloud storage connection [$PLUGIN_ENDPOINT, $S3_ENDPOINT, $GCS_ENDPOINT]
   --filesystem.cache-root value         local filesystem root directory for the filesystem cache (default: "/tmp/cache") [$PLUGIN_FILESYSTEM_CACHE_ROOT, $FILESYSTEM_CACHE_ROOT]
   --gcs.acl value                       upload files with acl (private, public-read, ...) (default: "private") [$PLUGIN_GCS_ACL, $GCS_ACL]
   --gcs.api-key value                   Google service account API key [$PLUGIN_API_KEY, $GCP_API_KEY]
   --gcs.encryption-key value            server-side encryption key, must be a 32-byte AES-256 key, defaults to none
                                             (See https://cloud.google.com/storage/docs/encryption for details.) [$PLUGIN_GCS_ENCRYPTION_KEY, $GCS_ENCRYPTION_KEY]
   --gcs.json-key value                  Google service account JSON key [$PLUGIN_JSON_KEY, $GCS_CACHE_JSON_KEY]
   --help, -h                            show help (default: false)
   --local-root value                    local root directory to base given mount paths (default pwd [present working directory]) [$PLUGIN_LOCAL_ROOT]
   --log.format value                    log format to use. ('logfmt', 'json') (default: "logfmt") [$PLUGIN_LOG_FORMAT, $LOG_FORMAT]
   --log.level value                     log filtering level. ('error', 'warn', 'info', 'debug') (default: "info") [$PLUGIN_LOG_LEVEL, $LOG_LEVEL]
   --mount value                         cache directories, an array of folders to cache  (accepts multiple inputs) [$PLUGIN_MOUNT]
   --override                            override even if cache key already exists in backend (default: true) [$PLUGIN_OVERRIDE]
   --path-style                          AWS path style to use for bucket paths. (true for minio, false for aws) (default: false) [$PLUGIN_PATH_STYLE, $AWS_PLUGIN_PATH_STYLE]
   --prev.build.number value             previous build number (default: 0) [$DRONE_PREV_BUILD_NUMBER]
   --prev.build.status value             previous build status [$DRONE_PREV_BUILD_STATUS]
   --prev.commit.sha value               previous build sha [$DRONE_PREV_COMMIT_SHA]
   --rebuild                             rebuild the cache directories (default: false) [$PLUGIN_REBUILD]
   --region value                        AWS bucket region. (us-east-1, eu-west-1, ...) [$PLUGIN_REGION, $S3_REGION]
   --remote-root value                   remote root directory to contain all the cache files created (default repo.name) [$PLUGIN_REMOTE_ROOT]
   --remote.url value                    git remote url [$DRONE_REMOTE_URL]
   --repo.avatar value                   repository avatar [$DRONE_REPO_AVATAR]
   --repo.branch value                   repository default branch [$DRONE_REPO_BRANCH]
   --repo.fullname value                 repository full name [$DRONE_REPO]
   --repo.link value                     repository link [$DRONE_REPO_LINK]
   --repo.name value                     repository name [$DRONE_REPO_NAME]
   --repo.namespace value                repository namespace [$DRONE_REPO_NAMESPACE]
   --repo.owner value                    repository owner (for Drone version < 1.0) [$DRONE_REPO_OWNER]
   --repo.private                        repository is private (default: false) [$DRONE_REPO_PRIVATE]
   --repo.trusted                        repository is trusted (default: false) [$DRONE_REPO_TRUSTED]
   --restore                             restore the cache directories (default: false) [$PLUGIN_RESTORE]
   --role-arn value                      AWS IAM role ARN to assume [$PLUGIN_ASSUME_ROLE_ARN, $AWS_ASSUME_ROLE_ARN]
   --s3-bucket-public value              Set to use anonymous credentials with public S3 bucket [$PLUGIN_S3_BUCKET_PUBLIC, $S3_BUCKET_PUBLIC]
   --secret-key value                    AWS secret key [$PLUGIN_SECRET_KEY, $AWS_SECRET_ACCESS_KEY, $CACHE_AWS_SECRET_ACCESS_KEY]
   --sftp.auth-method value              sftp auth method, defaults to none. (PASSWORD, PUBLIC_KEY_FILE) [$SFTP_AUTH_METHOD]
   --sftp.cache-root value               sftp root directory [$SFTP_CACHE_ROOT]
   --sftp.host value                     sftp host [$SFTP_HOST]
   --sftp.password value                 sftp password [$PLUGIN_PASSWORD, $SFTP_PASSWORD]
   --sftp.port value                     sftp port [$SFTP_PORT]
   --sftp.public-key-file value          sftp public key file path [$PLUGIN_PUBLIC_KEY_FILE, $SFTP_PUBLIC_KEY_FILE]
   --sftp.username value                 sftp username [$PLUGIN_USERNAME, $SFTP_USERNAME]
   --skip-symlinks                       skip symbolic links in archive (default: false) [$PLUGIN_SKIP_SYMLINKS, $SKIP_SYMLINKS]
   --sts-endpoint value                  Custom STS endpoint for IAM role assumption [$PLUGIN_STS_ENDPOINT, $AWS_STS_ENDPOINT]
   --version, -v                         print the version (default: false)
   --yaml.signed                         build yaml is signed (default: false) [$DRONE_YAML_SIGNED]
   --yaml.verified                       build yaml is verified (default: false) [$DRONE_YAML_VERIFIED]

Using Docker (with Environment variables)

$ docker run --rm \
      -v "$(pwd)":/app \
      -e DRONE_REPO=octocat/hello-world \
      -e DRONE_REPO_BRANCH=master \
      -e DRONE_COMMIT_BRANCH=master \
      -e PLUGIN_MOUNT=/app/node_modules \
      -e PLUGIN_RESTORE=false \
      -e PLUGIN_REBUILD=true \
      -e PLUGIN_BUCKET=<bucket> \
      -e AWS_ACCESS_KEY_ID=<token> \
      -e AWS_SECRET_ACCESS_KEY=<secret> \
      meltwater/drone-cache

Development

Usage:
  make <target>

Targets:
  setup          	  Setups dev environment
  drone-cache    	  Runs drone-cache target
  clean          	  Cleans build resourcess
  docs           	  Generates docs
  generate       	  Generate documentation, website and yaml files,
  vendor         	  Updates vendored copy of dependencies
  compress       	  Creates compressed binary
  container      	  Builds drone-cache docker image with latest tag
  container-push 	  Pushes latest $(CONTAINER_REPO) image to repository
  test           	  Runs tests
  test-integration	  Runs integration tests
  test-unit      	  Runs unit tests
  lint           	  Runs golangci-lint analysis
  fix            	  Runs golangci-lint fix
  format         	  Runs gofmt
  help           	  Shows this help message

Releases

Release management handled by the CI pipeline. When you create a tag on master branch, CI handles the rest.

You can find released artifacts (binaries, code, archives) under releases.

You can find released images at DockerHub.

PLEASE DO NOT INTRODUCE BREAKING CHANGES

Keep in mind that users usually use the image tagged with latest in their pipeline, please make sure you do not interfere with their working workflow. Latest stable releases will be tagged with the latest.

Versioning

drone-cache uses SemVer for versioning. For the versions available, see the tags on this repository.

As the versioning scheme dictates, drone-cache respects backward compatibility within the major versions. However, the project only offers guarantees regarding the command-line interface (flags and environment variables). Any exported public package can change its API.

Authors and Acknowledgement

See the list of all contributors.

• @kakkoyun - Thank you Kemal for bringing drone-cache to life, and building most of the initial version.
• @AdamGlazerMW - Special thanks to Adam for the amazing artwork!
• @dim - Thanks for the original work that inspired drone-cache!

Inspiration

• github.com/bsm/drone-s3-cache (original work)
• github.com/Drillster/drone-volume-cache
• github.com/drone/drone-cache-lib

Contributing

Please read CONTRIBUTING.md to understand how to submit pull requests to us, and also see our code of conduct.

Future work

All ideas for new features and bug reports will be kept in github.com/meltwater/drone-cache/issues.

One bigger area of future investment is to add a couple of new storage backends for caching the workspace files.

License and Copyright

This project is licensed under the Apache License 2.0.