Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add zlib support to extractGzip #713

Merged
merged 8 commits into from
Dec 18, 2024
Merged

Conversation

egibs
Copy link
Member

@egibs egibs commented Dec 17, 2024

We've historically had issues extracting zlib-compressed files such as these compressed Pickle files:

time=2024-12-17T07:22:02.609-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.10.0_compressed_pickle_py27_np17.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.10.0_compressed_pickle_py27_np17.gz: failed to create gzip reader: gzip: invalid header"
time=2024-12-17T07:22:02.610-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.10.0_compressed_pickle_py34_np19.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.10.0_compressed_pickle_py34_np19.gz: failed to create gzip reader: gzip: invalid header"
time=2024-12-17T07:22:02.610-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.10.0_compressed_pickle_py33_np18.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.10.0_compressed_pickle_py33_np18.gz: failed to create gzip reader: gzip: invalid header"
time=2024-12-17T07:22:02.610-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.10.0_compressed_pickle_py35_np19.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.10.0_compressed_pickle_py35_np19.gz: failed to create gzip reader: gzip: invalid header"
time=2024-12-17T07:22:02.622-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.8.4_compressed_pickle_py27_np17.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.8.4_compressed_pickle_py27_np17.gz: failed to create gzip reader: gzip: invalid header"
time=2024-12-17T07:22:02.623-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.2_compressed_pickle_py27_np16.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.2_compressed_pickle_py27_np16.gz: failed to create gzip reader: gzip: invalid header"
time=2024-12-17T07:22:02.623-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.2_compressed_pickle_py27_np17.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.2_compressed_pickle_py27_np17.gz: failed to create gzip reader: gzip: invalid header"
time=2024-12-17T07:22:02.623-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.2_compressed_pickle_py34_np19.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.2_compressed_pickle_py34_np19.gz: failed to create gzip reader: gzip: invalid header"
time=2024-12-17T07:22:02.624-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.2_compressed_pickle_py35_np19.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.2_compressed_pickle_py35_np19.gz: failed to create gzip reader: gzip: invalid header"
time=2024-12-17T07:22:02.626-06:00 level=ERROR source=~/repos/chainguard-dev/malcontent/pkg/action/scan.go:330 msg="unable to process ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz: extract to temp: failed to extract ~/Downloads/nemo/usr/share/nemo/.venv/lib/python3.11/site-packages/joblib/test/data/joblib_0.9.4.dev0_compressed_cache_size_pickle_py35_np19.gz: failed to create gzip reader: gzip: invalid header"

Since zlib files don't have a specific extension, I added the functionality to the gzip function since the files we've encountered have had the .gz extension. Ideally, we'd handle this independently of extension but that's for a future PR (probably when I refactor archive.go.)

With these changes:

$ go run cmd/mal/mal.go --all analyze ~/Downloads/nemo/joblib_0.10.0_compressed_pickle_py27_np17.gz
🔎 Scanning "~/Downloads/nemo/joblib_0.10.0_compressed_pickle_py27_np17.gz"

Signed-off-by: egibs <20933572+egibs@users.noreply.github.com>
@egibs egibs requested a review from tstromberg December 17, 2024 15:07
Signed-off-by: egibs <20933572+egibs@users.noreply.github.com>
Signed-off-by: egibs <20933572+egibs@users.noreply.github.com>
@egibs egibs requested a review from tstromberg December 17, 2024 22:21
@egibs egibs enabled auto-merge (squash) December 18, 2024 00:32
@egibs egibs merged commit f297f5d into chainguard-dev:main Dec 18, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants