Update ScanCore & Virus Definitions

[zelon88]

I really want to find a way to update these definitions in a clean manner, and automate the collection of IOC's somehow.

I think that we need to;

• Write a command for ScanCore to check for definition updates from the official Github repository, or a defined URL in config.php & update them automatically.
• Create a shell or Python script to scrape a list of sources to gather potential IOC hashes as well as interesting data, articles, & specimins for inclusion into the ScanCore definitions. Perhaps leverage Github Actions or Pull Requests?
• Create a quick and easy process for reviewing & approving new IOC's to the list of definitions. Perhaps leverage Github Actions or Pull Requests?
• Create a shell or Python script for easily turning the list of approved IOC's into an update list of definitions.
• Create a way to organize definitions into "packs" that distinguish/track definition age, infection type, and severity. That way we can eventually provide these as options to the user, so they can pick the definitions they want to use. We can also use this to selectively scan for high or low risk items, or specific IOCs.
• Create logic in HRConvert2 to take advantage of the new ScanCore features to trigger and manage A/V updates.

[zelon88]

Working towards;
Write a command for ScanCore to check for definition updates from the official Github repository, or a defined URL in config.php & update them automatically.

The command is done but I'm not ready to incorporate the new ScanCore into HRConvert2 yet. I'll get into specific ScanCore changes in a moment.

I started by creating a repository for ScanCore Definitions at https://github.com/zelon88/ScanCore_Definitions and breaking our existing definitions into three separate categories.

The current categories are;

• Virus
• Malware
• PUP

These definitions are known as "Definition Subscriptions" and defined by...

1. ...their filename beginning with "ScanCore_" and...
2. ...the file definition of ".def".

Other than the filename preface and the file extension, subscriptions are arbitrary. For example, we could create a subscription for...
North_American_Manufacturing_Trojans
...specifically for detecting highly targetted samples. The filename for such a subscription would be...
ScanCore_North_American_Manufacturing_Trojans.def
Any ScanCore client that was subscribed to the "ScanCore_North_American_Manufacturing_Trojans" definition will automatically acquire this definition during update operations.

To accomodate this functionality I recently bumped ScanCore to v1.1. This update (combined with v1.0) is a total refactor and adds the --updatedefinitions & -ud arguments.

These new arguments are functionally identical and will clone the Git repository defined in "ScanCore_Config.php" for definition updates. If the repository is reachable it will be cloned and any subscriptions will be gathered from it. The subsription data will then be written to a custom "Combined" definitions file that is specifiic to the agent performing the update.

Because each agent can be configured to gather separate definition subscriptions, it's possible to have separate agents installed on one machine that only scan for specific types of infections. Maybe you want to schedule scans for malware twice as often as your scans for PUPs. Maybe it's the other way around. You can control your destiny.

In regards to;
Create a shell or Python script to scrape a list of sources to gather potential IOC hashes as well as interesting data, articles, & specimins for inclusion into the ScanCore definitions. Perhaps leverage Github Actions or Pull Requests?

I'm interested in ThreatFox database and it's API because they offer bounties for quality sample submissions. I've scanned through IOCDB and AlienVault and I found many false positives, including very blatant ones. I feel like the bounty reward model that ThreatFox uses can be a key that we search off of in our automated search for IOC's. Evidence of a high bounty being paid for a malware sample would indicate that sample to be authentic. More attention needs to be paid to this to determine what already exists and what we can create for automating the collection of IOCs from ThreatFox, and other similar sources (but ThreatFox seems like a good place to start).