Hello from the ClamAV team; Some concerns about freshclam data usage w/ new containers

[micahsnyder]

Hi!

I just saw your message on the #clamav IRC space. This is a neat project. 👍 By the way, I noticed that you didn't stick around. If you want, try using our Discord server or our user mailing list - which offer a better experience for drive by messages because you can check back later to see if you got any responses ;-) https://docs.clamav.net/#mailing-lists-and-irc-channel

We (the ClamAV team) recently added our own docker images on Docker Hub: https://hub.docker.com/r/clamav/clamav
Currently we don't have a stable release out yet -- our first will be 0.104. We haven't backported our image to older versions because we're using CMake to build ClamAV and the CMake build system was not complete in the current stable release.

You'll note that we offer 2 images. The first (eg "unstable") is very large and includes a signature database set which we'll update once per month. This will save the ClamAV project a lot of bandwidth when a new container is started from the image because freshclam won't have to download the entire database set from scratch. The second image has a _base suffix and does not include any databases. This one should only be used when mounting a volume that stores existing signature databases for the database directory. See https://github.com/Cisco-Talos/clamav-devel/blob/dev/0.104/README.Docker.md#persisting-the-virus-database-volume

Using either an image which includes databases (that are newer than 90 days old) or mounting a database directory volume not only saves the ClamAV project 💰 on our CDN bill, but it also reduces the chance that new containers will get rate limited by our CDN. Rate limiting is particularly common in AWS and other public clouds because the public IP for different users is the same. It occurs when freshclam tries to download a whole database file instead of a database patch file (.cdiff file).

If your project could ensure that users do not download the entire database set when starting a new container, that would really help.

I haven't put a lot of time into reviewing the code here yet, just a cursory glance. I may try to look a little bit more but don't know that I'll provide much input. If you have any specific questions I can probably help better.

Also, I recently was working on documentation improvements. In reviewing the community projects list, I found there are 2 other projects for scanning S3 buckets. I haven't spent any time with them but you may find it helpful to review these projects, :

• https://docs.clamav.net/manual/Installing/Community-projects.html#clara
• https://docs.clamav.net/manual/Installing/Community-projects.html#bucket-antivirus-function

[dontirun]

Hello!

I understand the concern about CDN costs and rate limitation. Hopefully the system's approach addresses that concern.

There are two Lambda functions that utilize ClamAV in the docker image (which is built locally at launch time and on system updates) . One for downloading definitions and one for scanning

1. One time only (on system initialization) run freshclam to download database files to a definitions S3 Bucket using the download definitions code
2. On scan use the S3 bucket as the remote database for freshclam using the scanning code
3. Hourly, call the download definitions code to run freshclam against the definitions S3 Bucket files and update them if necessary

Thanks for thoughtful response!

[micahsnyder]

This sounds like a reasonable approach, but I do notice here:

cdk-serverless-clamscan/assets/lambda/code/download_defs/lambda.py

Line 45 in 02e3f4f

files = ["bytecode.cvd", "daily.cvd", "main.cvd", "freshclam.conf"]

that you aren't taking the .cld files into account. When a .cvd (or .cld) database is updated by FreshClam, FreshClam downloads one or more .cdiff patch files. It unpacks the .cvd (or .cld) and applies the patches in order. When complete, it creates a new database archive with a .cld extension. So TL;DR is that the .cvd update process replaces the file with a .cld file.

Perhaps as a disclaimer, in a future version we may add more database archives than just daily, main, and bytecode. It may be best to try to get all *.cvd and *.cld + the freshclam config just in case we do add more.

[dontirun]

Perhaps as a disclaimer, in a future version we may add more database archives than just daily, main, and bytecode. It may be best to try to get all *.cvd and *.cld + the freshclam config just in case we do add more.

That makes sense. I've opened an issue for that.

In regards to the .cld files. The download_defs lambda function downloads all the .cvd files from the definitions S3 bucket, then uploads all local files so that it can be used as a Private Mirror for the scanning lambda functions. The scanning function run FreshClam pointing to the S3 mirror against a shared local database on an Elastic File System (NFS). That file system contains the .cld files.

With that being said. Should the download_defs lambda function be downloading the .cdiff files from the S3 Private Mirror as well?

[micahsnyder]

No the download_defs lambda function should be downloading the .cvd and .cld files from the bucket so that freshclam can update them. The .cdiff files are short-lived; freshclam deletes the .cdiff files immediately after the update is complete, so you probably will never see them.