EFS cleanup

[fvdaak]

Hello together!

First of all: thank you for this nice project!
We are running the cdk-serverless-clamscan for about 4 month with about 1000-2000 files, scanned per day.

We have now discovered that the process causes very high costs. Currently around $5000/month. The costs increase by approximately $1000 per month!

The main cost driver is the EFS Share. We have discovered that there are now thousands of directories and files, and the number is constantly growing.
By analyzing the underlying code, we saw that the process deletes the scanned files, but leaves the directories. Also a lot of files were not deleted. This results - we don't know exactly why - in a huge and increasing number of expensive READ's.
Another consequence is that the lambdas take longer and therefore incur ever higher costs.

How can we avoid this? We currently think about some kind of housekeeping process, that mount's the EFS share, to delete all files, older than one day.
But is this a good solution for our problem?

Any help would be appreciated!

[dontirun]

Have you encountered any situations with scans that time out/fail to complete?

Regardless I think it would be beneficial to include a secondary cleanup process in this construct to prevent this case

[fvdaak]

Yes, sometimes, the scan times out. I think, that in this cases, the whole files will stay forever.

And yes: a cleanup process is essential

[fvdaak]

Hi!

Any news about the cleanup?

[dontirun]

Can you can open a feature request for this? I can't guarantee that it will be worked on, but at least that will make it visible for other users to potentially work on or see as a current limitation