Processing large sets of data and questions about modularity

[srjsml]

Hello, I'm currently working on an antivirus solution for an archiving service. Users can upload large collections (anywhere from 100s of GB to TB) of data. All the files will be uploaded to a landing bucket and an S3 event will trigger an AV scan on all the files uploaded. Depending on the results of the scan, the files will then move onto a staging or quarantine bucket for further processing.

Specifically, I'm wondering if this construct can be integrated into a pre-existing AWS project?

I'm also curious why the file needs to be transferred from the S3 bucket to an EFS. Is it possible to scan a S3 bucket instead? Or would this be inefficient? Sorry if my questions are rudimentary, I'm fairly new to AWS.

[dontirun]

Specifically, I'm wondering if this construct can be integrated into a pre-existing AWS project?

It can be added into existing S3 buckets and scan any new objects placed in them

I'm also curious why the file needs to be transferred from the S3 bucket to an EFS. Is it possible to scan a S3 bucket instead?

The primary reason is that Lambda functions don't support mounting S3 buckets as a filesystem, and ClamAV can't scan objects directly from a S3 bucket that isn't mounted as a filesystem

[srjsml]

Thanks for the reply. That makes sense - I tried implementing it and quickly realized that Lambda cannot mount S3 buckets. I ended up creating a solution that auto-scales EC2 instances to run clamscan on uploaded files.