diff --git a/README.md b/README.md index 2ad7e039..d9511267 100644 --- a/README.md +++ b/README.md @@ -288,11 +288,11 @@ $ python tests/.py ``` ## Project Status -Release 2.2.0 is out! See the [release notes](docs/releases/v2_2_0.md) for more information. +Release 2.3.0 is out! See the [release notes](docs/releases/v2_3_0.md) for more information. We try to keep the [project roadmap](./docs/project-roadmap.md) as up to date as possible. We are currently working on Release 3.0.0. -## Releases +## Previous Releases * [v2.2.0](docs/releases/v2_2_0.md) * [v2.1.0](docs/releases/v2_1_0.md) * [v2.0.0](docs/releases/v2_0_0.md) diff --git a/docs/releases/release_checklist.md b/docs/releases/release_checklist.md index a6371494..b674d083 100644 --- a/docs/releases/release_checklist.md +++ b/docs/releases/release_checklist.md @@ -5,20 +5,21 @@ This is a checklist for cutting a release - [ ] Prepare Release PR. * Freeze development on master. * Prepare your local development environment by committing or stashing your changes. Work at the tip of master. + * Create a branch for the release: `git checkout -b `. * In a separate folder, create a fresh environment and activate it. * Clone the `tern/master` repository by running `git clone --single-branch git@github.com:tern-tools/tern.git` and `cd` into it. - * Create a branch for the release: `git checkout -b `. - [ ] Update direct dependencies and run tests. - * Run `pip install wheel pip-tools twine`. + * In the fresh environment, run `pip install wheel pip-tools twine`. * Run `pip-compile --upgrade --output-file upgrade.txt`. - * Compare the dependency versions from the output of the pip-compile command to the current dependency versions listed in the `requirements.txt` file. Upgrade `requirements.txt` if necessary. - * Run `pip install .` to install tern. + * Compare the module versions in upgrade.txt with requirements.txt in the development environment. Bump up versions if needed. + * In the fresh environment, run `pip install .` to install tern. * Run appropriate tests. Roll back requirements if necessary. * When satisfied, run `pip-compile --generate-hashes --output-file v-requirements.txt` where is of the form `major_minor_patch`. + * Copy this file to the `docs/releases/` folder in the development environment. - [ ] Write release notes. - * Create a new file for the release notes: `docs/releases/v.md` + * In the development environment, create a new file for the release notes: `docs/releases/v.md` * If you are writing release notes for a patched release, only include: - A link to the primary release notes. - A brief summary of what the patched release changes do. @@ -39,15 +40,8 @@ This is a checklist for cutting a release * Update the Project Status part of the README.md to reflect this release and add it to the list of releases. -- [ ] Commit release notes and create patch for your changes - * `git add` and `git commit` any changes. This will likely include`v-requirements.txt`, any changes to `requirements.txt` and `v.md`. **Do not push these changes to master!** - * Run `git format-patch -n1`. This will create a patch file of the release changes you just committed called `0001-.patch`. - * Open a new terminal and `cd` into a development virtual environment that contains your forked version of the Tern repo. `cd` into the forked Tern repo directory. - * Create a new branch. You will use this branch to submit a PR for the release changes. - * Copy the patch file you just created into your new forked repo environment. - * Run `git am 0001-.patch`. - * Run `git push origin ` to push the changes to your forked repo. - * The changes are now available in your forked repo. You can verify this by running `git log` and looking at the top commit from the output. +- [ ] Commit release notes and submit a PR + * `git add` and `git commit` any changes. This will likely include`v-requirements.txt`, any changes to `requirements.txt` and `v.md`. * Open a pull request in the Tern project repository for your release changes. * Request a review from another maintainer. Update PR as needed based on feedback. Merge the PR. This commit is where the release will be tagged. diff --git a/docs/releases/v2_3_0-requirements.txt b/docs/releases/v2_3_0-requirements.txt new file mode 100644 index 00000000..7b1e7ea8 --- /dev/null +++ b/docs/releases/v2_3_0-requirements.txt @@ -0,0 +1,116 @@ +# +# This file is autogenerated by pip-compile +# To update, run: +# +# pip-compile --generate-hashes --output-file=v2_3_0-requirements.txt +# +attrs==20.3.0 \ + --hash=sha256:31b2eced602aa8423c2aea9c76a724617ed67cf9513173fd3a4f03e3a929c7e6 \ + --hash=sha256:832aa3cde19744e49938b91fea06d69ecb9e649c93ba974535d08ad92164f700 \ + # via debut +certifi==2020.11.8 \ + --hash=sha256:1f422849db327d534e3d0c5f02a263458c3955ec0aae4ff09b95f195c59f4edd \ + --hash=sha256:f05def092c44fbf25834a51509ef6e631dc19765ab8a57b4e7ab85531f0a9cf4 \ + # via requests +chardet==3.0.4 \ + --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \ + --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 \ + # via debut, requests +debut==0.9.8 \ + --hash=sha256:b353e1d826d0be80a7268762efd99ba05f9d1df1aef0553fb7ea17c670bee85c \ + --hash=sha256:edd4ff3d265ca5bf645c73d6863a886d34743152d215a5de094c4d31fa6943e3 \ + # via -r requirements.in +docker==4.3.1 \ + --hash=sha256:13966471e8bc23b36bfb3a6fb4ab75043a5ef1dac86516274777576bed3b9828 \ + --hash=sha256:bad94b8dd001a8a4af19ce4becc17f41b09f228173ffe6a4e0355389eef142f2 \ + # via -r requirements.in +dockerfile-parse==1.1.0 \ + --hash=sha256:80ea4b88694ab014001e39e62335aa2f4feb695b80de751377e994a344fa5952 \ + --hash=sha256:f37bfa327fada7fad6833aebfaac4a3aaf705e4cf813b737175feded306109e8 \ + # via -r requirements.in +idna==2.10 \ + --hash=sha256:b307872f855b18632ce0c21c5e45be78c0ea7ae4c15c828c20788b26921eb3f6 \ + --hash=sha256:b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 \ + # via requests +pbr==5.5.1 \ + --hash=sha256:5fad80b613c402d5b7df7bd84812548b2a61e9977387a80a5fc5c396492b13c9 \ + --hash=sha256:b236cde0ac9a6aedd5e3c34517b423cd4fd97ef723849da6b0d2231142d89c00 \ + # via -r requirements.in, stevedore +pyyaml==5.3.1 \ + --hash=sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97 \ + --hash=sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76 \ + --hash=sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2 \ + --hash=sha256:6034f55dab5fea9e53f436aa68fa3ace2634918e8b5994d82f3621c04ff5ed2e \ + --hash=sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648 \ + --hash=sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf \ + --hash=sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f \ + --hash=sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2 \ + --hash=sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee \ + --hash=sha256:ad9c67312c84def58f3c04504727ca879cb0013b2517c85a9a253f0cb6380c0a \ + --hash=sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d \ + --hash=sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c \ + --hash=sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a \ + # via -r requirements.in +regex==2020.11.13 \ + --hash=sha256:02951b7dacb123d8ea6da44fe45ddd084aa6777d4b2454fa0da61d569c6fa538 \ + --hash=sha256:0d08e71e70c0237883d0bef12cad5145b84c3705e9c6a588b2a9c7080e5af2a4 \ + --hash=sha256:1862a9d9194fae76a7aaf0150d5f2a8ec1da89e8b55890b1786b8f88a0f619dc \ + --hash=sha256:1ab79fcb02b930de09c76d024d279686ec5d532eb814fd0ed1e0051eb8bd2daa \ + --hash=sha256:1fa7ee9c2a0e30405e21031d07d7ba8617bc590d391adfc2b7f1e8b99f46f444 \ + --hash=sha256:262c6825b309e6485ec2493ffc7e62a13cf13fb2a8b6d212f72bd53ad34118f1 \ + --hash=sha256:2a11a3e90bd9901d70a5b31d7dd85114755a581a5da3fc996abfefa48aee78af \ + --hash=sha256:2c99e97d388cd0a8d30f7c514d67887d8021541b875baf09791a3baad48bb4f8 \ + --hash=sha256:3128e30d83f2e70b0bed9b2a34e92707d0877e460b402faca908c6667092ada9 \ + --hash=sha256:38c8fd190db64f513fe4e1baa59fed086ae71fa45083b6936b52d34df8f86a88 \ + --hash=sha256:3bddc701bdd1efa0d5264d2649588cbfda549b2899dc8d50417e47a82e1387ba \ + --hash=sha256:4902e6aa086cbb224241adbc2f06235927d5cdacffb2425c73e6570e8d862364 \ + --hash=sha256:49cae022fa13f09be91b2c880e58e14b6da5d10639ed45ca69b85faf039f7a4e \ + --hash=sha256:56e01daca75eae420bce184edd8bb341c8eebb19dd3bce7266332258f9fb9dd7 \ + --hash=sha256:5862975b45d451b6db51c2e654990c1820523a5b07100fc6903e9c86575202a0 \ + --hash=sha256:6a8ce43923c518c24a2579fda49f093f1397dad5d18346211e46f134fc624e31 \ + --hash=sha256:6c54ce4b5d61a7129bad5c5dc279e222afd00e721bf92f9ef09e4fae28755683 \ + --hash=sha256:6e4b08c6f8daca7d8f07c8d24e4331ae7953333dbd09c648ed6ebd24db5a10ee \ + --hash=sha256:717881211f46de3ab130b58ec0908267961fadc06e44f974466d1887f865bd5b \ + --hash=sha256:749078d1eb89484db5f34b4012092ad14b327944ee7f1c4f74d6279a6e4d1884 \ + --hash=sha256:7913bd25f4ab274ba37bc97ad0e21c31004224ccb02765ad984eef43e04acc6c \ + --hash=sha256:7a25fcbeae08f96a754b45bdc050e1fb94b95cab046bf56b016c25e9ab127b3e \ + --hash=sha256:83d6b356e116ca119db8e7c6fc2983289d87b27b3fac238cfe5dca529d884562 \ + --hash=sha256:8b882a78c320478b12ff024e81dc7d43c1462aa4a3341c754ee65d857a521f85 \ + --hash=sha256:8f6a2229e8ad946e36815f2a03386bb8353d4bde368fdf8ca5f0cb97264d3b5c \ + --hash=sha256:9801c4c1d9ae6a70aeb2128e5b4b68c45d4f0af0d1535500884d644fa9b768c6 \ + --hash=sha256:a15f64ae3a027b64496a71ab1f722355e570c3fac5ba2801cafce846bf5af01d \ + --hash=sha256:a3d748383762e56337c39ab35c6ed4deb88df5326f97a38946ddd19028ecce6b \ + --hash=sha256:a63f1a07932c9686d2d416fb295ec2c01ab246e89b4d58e5fa468089cab44b70 \ + --hash=sha256:b2b1a5ddae3677d89b686e5c625fc5547c6e492bd755b520de5332773a8af06b \ + --hash=sha256:b2f4007bff007c96a173e24dcda236e5e83bde4358a557f9ccf5e014439eae4b \ + --hash=sha256:baf378ba6151f6e272824b86a774326f692bc2ef4cc5ce8d5bc76e38c813a55f \ + --hash=sha256:bafb01b4688833e099d79e7efd23f99172f501a15c44f21ea2118681473fdba0 \ + --hash=sha256:bba349276b126947b014e50ab3316c027cac1495992f10e5682dc677b3dfa0c5 \ + --hash=sha256:c084582d4215593f2f1d28b65d2a2f3aceff8342aa85afd7be23a9cad74a0de5 \ + --hash=sha256:d1ebb090a426db66dd80df8ca85adc4abfcbad8a7c2e9a5ec7513ede522e0a8f \ + --hash=sha256:d2d8ce12b7c12c87e41123997ebaf1a5767a5be3ec545f64675388970f415e2e \ + --hash=sha256:e32f5f3d1b1c663af7f9c4c1e72e6ffe9a78c03a31e149259f531e0fed826512 \ + --hash=sha256:e3faaf10a0d1e8e23a9b51d1900b72e1635c2d5b0e1bea1c18022486a8e2e52d \ + --hash=sha256:f7d29a6fc4760300f86ae329e3b6ca28ea9c20823df123a2ea8693e967b29917 \ + --hash=sha256:f8f295db00ef5f8bae530fc39af0b40486ca6068733fb860b42115052206466f \ + # via -r requirements.in +requests==2.25.0 \ + --hash=sha256:7f1a0b932f4a60a1a65caa4263921bb7d9ee911957e0ae4a23a6dd08185ad5f8 \ + --hash=sha256:e786fa28d8c9154e6a4de5d46a1d921b8749f8b74e28bde23768e5e16eece998 \ + # via -r requirements.in, docker +six==1.15.0 \ + --hash=sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259 \ + --hash=sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced \ + # via docker, dockerfile-parse +stevedore==3.2.2 \ + --hash=sha256:5e1ab03eaae06ef6ce23859402de785f08d97780ed774948ef16c4652c41bc62 \ + --hash=sha256:f845868b3a3a77a2489d226568abe7328b5c2d4f6a011cc759dfa99144a521f0 \ + # via -r requirements.in +urllib3==1.26.2 \ + --hash=sha256:19188f96923873c92ccb987120ec4acaa12f0461fa9ce5d3d0772bc965a39e08 \ + --hash=sha256:d8ff90d979214d7b4f8ce956e80f4028fc6860e4431f731ea4a8c08f23f99473 \ + # via requests +websocket-client==0.57.0 \ + --hash=sha256:0fc45c961324d79c781bab301359d5a1b00b13ad1b10415a4780229ef71a5549 \ + --hash=sha256:d735b91d6d1692a6a181f2a8c9e0238e5f6373356f561bb9dc4c7af36f452010 \ + # via docker diff --git a/docs/releases/v2_3_0.md b/docs/releases/v2_3_0.md new file mode 100644 index 00000000..2d4061fe --- /dev/null +++ b/docs/releases/v2_3_0.md @@ -0,0 +1,102 @@ +# Release 2.3.0 + +## Summary +This release contains a big code refactor which fixed a good number of technical debt issues. It also delivers support for [multistage Dockerfiles](https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds), which is valuable as Docker removes the intermediate stages leaving only the final deploy container image to analyze. Teams building applications using this method can now get a Sofware Bill of Materials for each stage. A special thanks to Junlai Wang (@ForgetMe17 on GitHub) for laying the groundwork to finally implement this feature. We also have a Dockerfile for building Tern with Scancode-Toolkit. To build this image, simply run `docker built -t ternscancode -f docker/Dockerfile.scancode .` and then `docker_run.sh ternscancode "report -x scancode -i "`. Thanks to Jeroen Knoops (@JeroenKnoops on GitHub) for contributing this Dockerfile. + +A note about this release: Although this is a minor version bump, the short `-d` for `--driver` is now `-dr` to prevent confusion between `-d` for passing a Dockerfile. + +As always, we would like to thank our community for contributing to this release. + +## New Features +* [Preliminary support for multistage Dockerfiles](https://github.com/tern-tools/tern/issues/612): Tern can now generate reports in HTML, JSON, YAML and human-readable formats for multistage Dockerfiles. Note that this is the case only for Dockerfiles, not container images that may have been built using Dockerfiles. We think this is pretty cool! + +## Bug Fixes +* [Fix crash when an image is not found by the Docker API](https://github.com/tern-tools/tern/issues/828) +* [Fix crash when a script invocation fails](https://github.com/tern-tools/tern/issues/822) +* [Fix parsing tabs in a Docker image's created_by value](https://github.com/tern-tools/tern/issues/812) +* Many bugs were fixed as a result of the code refactor. + + +## Resolved Technical Debt +* Parts of a larger code refactor: + * [Move container pull and dump operations to a new module](https://github.com/tern-tools/tern/issues/802) + * [Move setup and teardown checks into a new module](https://github.com/tern-tools/tern/issues/808) + * [Re-organize tern/analyze folder](https://github.com/tern-tools/tern/issues/803) +* [Resolving all code complexity debt](https://github.com/tern-tools/tern/issues/789) + +## Future Work +* A "step" subcommand to step through container image layers and analyze them individually. +* Analysis for OCI style images. +* Continuing code cleanup + +The next release will be a Beta release 3.0.0. Since it will be the first in 2021, and the US holidays are upon us, expect the next release by March or April. Watch the [Beta Release Milestone](https://github.com/tern-tools/tern/milestone/13) for progress. We're really excited about this release! + +## Changelog + +Note: This changelog will not include these release notes + +Changelog generated by command: `git log --pretty=format:"%h %s" v2.1.0..master` + +``` +bb38e14 merge: Enable analysis for multistage Dockerfiles +906edac Fix ci build for locking a Dockerfile +24b4e51 Fixes for reading and writing Dockerfiles +daab1d4 Fix Dockerfile build with context +142c74e Enable multistage Dockerfile analysis +453fad6 Replace the short driver option with -dr +4ca9b88 Add subroutine to analyze multistage Dockerfiles +3e2325e Update code navigation document +a8ec222 Add Dockerfile for scancode +ad2b97c Add 'apt' Snippet In Command Library +e420355 Fix crash when a chroot command fails +e33357d Fix Dockerfile analysis if no base image is found +1621437 Gracefully exit if there is no image to analyze +222a138 Fix unbound local error when repo digest is given +cfb8d10 Recognize assignments before command in script +14c2dca merge: Organize code under tern/analyze +85bbd09 Fix tests after refactor +e7b3b6a Shorten fill_package_metadata function +0c0d587 Re-enable Dockerfile lock +f0ff818 Fix operation errors after refactor +fe1de25 Refactor functions with too many branches +716b1e0 Complete Dockerfile analysis +a991b0f Fix multi-layer container analysis +c2e8dfa Fix single layer analysis +5f24e3e More moving of code into logical places +43f64af Organized code in the analyze folder +2f5f4c6 Move multi-layer analysis to default +e8a8228 Move command_lib into default and organize +4b67c87 Create new folder for default operation +9b181d3 merge: Move external interactions to load directory +21156d0 Remove container.py and some deprecated functions +5681dac Fix checksum parsing and Dockerfile building +5f4b0f5 Fixed tests and linting for common.py and Package +90cd6cb Fix loading package files from cache +5706b2b Hook up docker_api to setup and teardown +70fdc09 classes: Use load functions in DockerImage +c5cc233 load: New code section for external interactions +338fde3 merge: Map layer files to packages +056c309 Fix error caused by tabs in ENV +bebbb18 Add file info for packages +2d29c8d Extract file info for packages +d561fce docs: Add GitHub Action link in README +1139109 ci: Update python version for GHA +7f6ab45 Refactor Dockerfiles +``` + +## Contributors + +``` +asifjoardar mrsparrow04@gmail.com +HeroicHitesh hiteshkumar_1mv17cs042@sirmvit.edu +Isac Sund isac@isacsund.com +Jeroen Knoops jeroen.knoops@philips.com +PrajwalM2212 prajwalmmath@gmail.com +WangJL hazard15020@gmail.com +Yann Jorelle yann.jorelle@nokia.com +``` + +## Contact the Maintainers + +Nisha Kumar: nishak@vmware.com +Rose Judge: rjudge@vmware.com diff --git a/requirements.in b/requirements.in index 78230b7d..730bc1b1 100644 --- a/requirements.in +++ b/requirements.in @@ -1,4 +1,4 @@ -# Copyright (c) 2019 VMware, Inc. All Rights Reserved. +# Copyright (c) 2019-2020 VMware, Inc. All Rights Reserved. # SPDX-License-Identifier: BSD-2-Clause # # This file is used by pip-tools for release management diff --git a/requirements.txt b/requirements.txt index 82266ac7..9f5502c1 100644 --- a/requirements.txt +++ b/requirements.txt @@ -8,9 +8,9 @@ PyYAML>=5.3 docker~=4.3 -dockerfile-parse~=1.0 -requests~=2.24 +dockerfile-parse~=1.1 +requests~=2.25 stevedore>=3.2 -pbr>=5.4 +pbr>=5.5 debut>=0.9 -regex>=2020.7 +regex>=2020.11