AIL Project v6.0: Improved Dark Web Analysis with a New Dashboard

The AIL project team is thrilled to announce the release of AIL v6.0, a significant milestone in our continuous effort to deliver cutting-edge tools for analyzing information leaks and malicious activities on the dark web. This release introduces a host of new features, improvements, and fixes designed to streamline workflows and provide deeper insights for analysts. Here's what you can expect in this version:

New Features and Enhancements

1. Updated Dashboard for Enhanced Usability

• Tag Monitoring: Detection are now displayed by tags, enabling quicker categorization and prioritization.
• Tracker Descriptions: A new feature that displays tracker descriptions directly on the dashboard.
• Crawler Stats & Object Tooltips: Gain a comprehensive view of crawler statistics and in-depth information about objects through intuitive tooltips.
• Enhanced Object Metrics: Real-time updates and insights into the number of objects per day, supported by a new WebSocket-enabled interface.
• EChart Feeder Graph: A sleek, interactive graph for visualizing data, along with a cleanup of outdated graph libraries.

2. Backend Improvements

• Module Management Overhaul: Removal of legacy modules and module loaders, paving the way for a leaner and more efficient system.
• PubSubLogger Upgrade: The old redis_logger has been retired for a more modern and reliable implementation.
• Queue Enhancements: The queue system now records module start times and process IDs for better process tracking.

3. New Object Type: Barcode

• AIL v6.0 introduces support for barcodes, allowing analysts to extract and analyze barcode objects efficiently.

4. Visual and UI Upgrades

• Migration to FontAwesome v6.6.0, ensuring compatibility with the latest icon set and a more polished visual interface.
• A dedicated date view for daily analysis, improving workflow for time-sensitive investigations.

🛠️ Fixes and Optimizations

• Dashboard Fixes:
  • Resolved formatting issues with day display.
  • Corrected feeder names for better clarity.
• Retro Hunts: Enhanced functionality by removing outdated objects and fixing tag addition errors.
• Object Item Display: Fixed URL visibility for item objects.
• Barcode Message Card: Addressed issues with the display of barcode-related messages.
• Sidebar Fixes: Improved layout and display for organizational information.

Why Upgrade to AIL v6.0?

This release is a leap forward for dark web analysts and cybersecurity professionals. With a more intuitive dashboard, enhanced tracking capabilities, and support for new object types, AIL v6.0 empowers users to handle complex investigations with greater speed and accuracy.

Get Started with AIL v6.0 Today!

To upgrade, after the update, the launch script can be called doing the update automatically. Detailed instructions and documentation are available in our official repository.

Let’s shape the future of open-source dark web analysis together. Stay tuned for more updates, and as always, your feedback is invaluable to us!

AIL Framework v5.9 Released – New Features such as dom-hash correlation, improvements and many bug fixes

AIL Project Release v5.9 - 2024-10-18

We are glad to announce the release of AIL v5.9, packed with numerous updates and fixes that enhance the performance and features of the framework. This release focuses on improvements in qrcode handling, dom-hash support, title processing, and crawler operations, among other critical updates.

dom-hash is a structural fingerprint of the HTML’s Document Object Model (DOM) originally developed by CERT.PL.

The fingerprint is calculated by extracting all the tag names (ignoring the content itself as well as attributes of the HTML Page). The tag names are concatenated with a pipe value |, hashed using the SHA-256 algorithm, and truncated to the first 32 characters.

Software such as LookyLoo or MISP have implemented the algorithm, the AIL framework now support the dom-hash algorithm to cluster and group similar page structure.

What's New in v5.9

Changes

• [qrcodes] Daterange search: Sort qrcode by content. [terrtia]
• [dom-hash] Add dom-hash object to compute dom-hash for domains and crawled items. [terrtia]
• [CEDetector] Tag domains. [terrtia]
• [CEDetector] Add detect message functionality. [terrtia]
• [titles] Setup Titles queues and integrate CEDetector module. [terrtia]
• [tools] Add reprocess option for Titles + CEDetector. [terrtia]
• [crawler] Disable unsafe title auto-tagging. [terrtia]
• [module] General debugging and improvements. [terrtia]
• [module] Add CEDetector module. [terrtia]
• [crawler] Tag domain by vanity. [terrtia]
• [crawler] Crawl list URLs: Filter duplicates. [terrtia]
• [crawler] Submit free text of URLs to crawl. [terrtia]
• [onion module] Filter out Onion v2 domains. [terrtia]
• [show domain] Display title content and fix item tags. [terrtia]
• [crawler] Update TOR user agent for more efficient crawling. [terrtia]
• [message image] Show extracted QR codes in messages. [terrtia]
• [domain lookup] Extract domains from URL input. [terrtia]
• [api] Rename domain lookup function. [terrtia]

Fixes

• [module] Fixed a typo in the module. [terrtia]
• [tag] Tag delete: Fixed update for first/last seen times. [terrtia]
• [show domain] Fixed the correlation button when correlation count is zero. [terrtia]
• [crawler] Filter lookup tags for better accuracy. [terrtia]
• [crawler] Filter lookup for parent + domain daterange. [terrtia]

We hope these updates enhance your experience with the AIL framework. Stay tuned for future updates as we continue to improve and expand AIL’s capabilities.

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on the misp-lea.org website.

AIL Framework v5.8 Released – New Features such as QR code extraction, improvements and fixes

We announce the release of AIL Framework v5.8, packed with new features, improvements, and bug fixes to facilitate the usage. This release focuses on expanding functionality and improving efficiency in key areas such as QR code extraction, domain lookups, image handling, and more.

We would like to thank LEAs and police officers for their feedback during the ENISA / Europol EC3 Workshop.

What’s New in v5.8?

Enhanced Features:

• QR Code Search by Tags: You can now search for QR codes using tags, making it easier to identify and track items based on tag data.
• Domain Lookup API: A domain lookup function has been added to the API, streamlining investigations with a direct lookup feature.
• Image Lazy Loading in Chats Explorer: To optimize performance, image lazy loading has been implemented, ensuring quicker and smoother browsing through chat logs.
• Favicon Search by Date Range: Favicons can now be searched by date range, coupled with lazy loading, for an efficient exploration process.
• Improved QR Code Extraction: The QR code extractor has been upgraded to provide better extraction from images and screenshots, with added correlation features.
• New Text Wrapping Option in Item Display: You can now replace the canvas with image blur for improved visuals and use a new button to wrap text for clearer readability.
• Username Search: We’ve introduced the ability to search by username, making it easier to locate users in your dataset.
• Tags Search Improvement: If no result is found when searching by tags, the last associated date is now shown for better context.

User Interface Enhancements:

• Hunter Sidebar Icon Alignment: Icons in the hunter sidebar are now aligned for a cleaner and more intuitive interface.
• Improved Domain Screenshot Display: The canvas display for domain screenshots has been replaced by an image blur effect for a smoother visual experience.
• Organization Selector Search in User Creation/Edit: Searching for organizations is now faster and more user-friendly with the new org selector search feature.

Key Fixes:

• QR Code Extractor Improvements: Multiple issues related to the QR code extractor have been resolved, including fixing exceptions, handling empty content, filtering invalid images, and debugging image formats like JPEGs.
• Bitcoin Bech32 Address Validator: A bug affecting the Bech32 address validator has been addressed, improving accuracy when validating Bitcoin addresses.
• Improved Mail Module: Minor typos in the mail module have been fixed for better clarity.
• Crawler Dashboard: A fix has been applied to the domains up/down links, ensuring accurate daily domain tracking.
• Investigation Object Table: Long object IDs in the investigation table have been fixed to prevent display issues.
• UI Fixes: Several visual improvements were made, including resolving overlapping table/image displays and updating crawler bar chart colors for up/down states.

Conclusion:

With the release of AIL Framework v5.8, we continue to build on our commitment to delivering a powerful and intuitive tool for your intelligence investigations. From improved search functionality to enhanced image handling and bug fixes, this release is designed to optimize your day-to-day investigations.

As always, feel free to provide feedback or report any issues to help us make AIL even better.

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on the misp-lea.org website.

AIL Project version 5.7 released with many improvements with 2FA support, multi-organisation support, improved chat monitoring and more.

AIL Framework v5.7 Released!

We are thrilled to announce the release of AIL Framework version 5.7. This update brings a host of new features, improvements, and bug fixes designed to enhance performance, security, and user experience.

Highlights

• Two-Factor Authentication (2FA): Enhance your account security with new TOTP and HOTP authentication methods.
• Organizational Support: Introduced organization-level access controls and management for users, trackers, retro hunts, investigations, and cookie jars.
• Improved Chat Monitoring: Added support for new chat types, placeholders, and enhanced message viewing with usernames and relationships.
• Dashboard Enhancements: Real-time updates with event streams replacing interval requests for a smoother experience.
• User Management Overhaul: Refactored user creation and editing processes, including organization assignments and session management.

---

What's New

Security Enhancements

• Two-Factor Authentication (2FA):

  • Implemented TOTP and HOTP methods for additional account security.
  • Users and admins can manage 2FA settings directly from their profiles.

• User Session Management:

  • Administrators can now manually log out users or terminate user sessions.
  • Added metadata for users, including creation date, last edit, last login, last seen, and login status.

Organizational Features

• Organization-Level Access Control:

  • Introduced organizations to structure data and access.
  • Trackers, retro hunts, investigations, and cookie jars now support organization-specific ACLs.
  • Users can view their organization information in their profiles.

• User Roles Update:

  • Renamed the "coordinator" role to "org_admin" for clarity.
  • Refactored user roles to better align with organizational structures.

Chat and Messaging Improvements

• Chat Monitoring:

  • Added support for additional chat types and placeholders.
  • Chats now display usernames for better context.
  • Implemented chat monitoring requests in the chats explorer.

• Message Relationships:

  • Enhanced relationships by adding message mentions linking chats and user accounts.
  • Introduced chord diagrams to visualize message flow between chats and users.

User Interface and Experience

• Dashboard Updates:

  • Replaced interval-based requests with event streams for real-time updates.
  • Improved performance and reduced server load.

• Error Handling:

  • Enhanced UI to gracefully handle 403 and 404 errors.
  • Refined logs to filter out unnecessary SSL errors when clients disconnect.

• Visualization Tools:

  • Updated D3.js to the latest version.
  • Migrated heatmap to version 7 with improved tooltips.
  • Circos graphs now display the number of inbound and outbound messages in tooltips.

Other Notable Additions

• Crawler Management:

  • Added functions to delete schedules and manually clear queues.
  • Improved crawler statistics with monthly domain-type stats.

• Export and Import:

  • Filtered out non-MISP objects during MISP exports.
  • Updated MISP taxonomies and galaxies to the latest versions.

---

Bug Fixes

• Crawler Queue Statistics:

  • Resolved multiple issues causing inaccurate crawler queue stats.

• User Management:

  • Fixed role editing and user validation processes.
  • Corrected issues with users changing their own passwords.

• Trackers and Retro Hunts:

  • Addressed ACL issues for global trackers.
  • Fixed webhook exports and post-filter selectors.

• Cookie Jar:

  • Resolved problems when adding cookies with UUIDs.

• Object Handling:

  • Fixed errors when retrieving objects with None values.
  • Corrected display issues in the object subtype dashboard.

• User Accounts:

  • Fixed tooltips in chord graphs.
  • Corrected last username timestamp displays.

---

Upgrading to v5.7

To upgrade to the latest version:

1. Pull the Latest Changes: Update your local repository to include the latest commits.
2. Restart Services: Restart the AIL Framework services to apply the new changes. The update script is started automatically.

---

For a detailed list of changes, visit our GitHub repository.

---

Funding

MISP-LEA, a collaborative endeavor between Shadowserver and CIRCL, is a 24-month initiative funded by the European Union. The project’s central aim is to establish operational and enduring MISP and AIL instances dedicated specifically to law enforcement agencies. This setup will facilitate a smoother exchange of evidence between law enforcement agencies and improve the onset of collaborative investigations. For this purpose, the system will ingest data from Shadowserver’s ransomware and C2 infrastructure tracking.

Law enforcement agencies willing to discover and leverage the MISP-LEA platform can apply on the misp-lea.org website.

Stay Connected:

• Website: ail-project.org
• GitHub: github.com/ail-project/ail-framework
• Mastodon @ail_project
• Twitter/X/N*zi platform: @AIL_Project

Thank you for your continued support. We look forward to your feedback!

AIL Project version 5.6 released with many improvements in the OCR and correlation functions and many updates.

Release Notes for AIL Project - Version 5.6 (2024-05-10)

What's New in v5.6

Changes

• TRON Cryptocurrency Detection & Correlation: Added detection and correlation features for TRON cryptocurrency. Thanks to @pventuzelo (from Fuzzinglabs for this contribution.
• Tag Search Enhancements: Improved the search functionality for OCRs and images by tags and fixed issues with OCR and filtering invalid images.
• Correlation Graph Updates: Added a separator for date first seen and last seen in the correlation graph. Also, fixed the display of OCR object tags.

Fixes

• Language Detector Stability: Fixed an issue where an exception occurred if the LibreTranslate URL was not specified.
• OCR Error Handling: Implemented catching of cv2 errors in OCR processes.
• PGPDump Installer Update: Updated the installer to handle new pgpdump version requirements, including launching autoreconf.
• OCR File Handling: Enhanced error handling with a catch for OSError on MP4 files.
• OCR Language Support: Fixed issues with supported languages in OCR and updated the filter for OCR supported languages.
• Language Detector Improvement: Updated the language detector to return an empty list when no language is detected, enhancing the robustness of language detection.
• OCR Tracker Fixes: Fixed the type of object accepted by the OCR tracker for better accuracy and performance.

Documentation

• New Introduction Presentation: A new introductory presentation is now available. It includes overview diagrams and can be accessed here.

AIL Framework 5.5 Released: New OCR Module for Images, Report Generator for Tracker Module, and Numerous Improvements.

AIL Framework 5.5 Released: New OCR Module for Images, Report Generator for Tracker Module, and Numerous Improvements.

Changes

• Documentation: Improved including overview diagrams.
• OCR Features:
  • Introduced the OCR object and added functionality to retrieve OCRs by daterange with fixes.
  • Added OCR Extractor module and implemented a filter for GIF images in the reprocess tool.
  • Enhanced OCR capabilities with cache addition, correlation between OCR and chat messages, and enabled the OCR extractor by default.
  • Added language detection and translation, a new view for OCR results, and a blueprint for additional languages.
  • Developed a model to get languages, group extracted content by line, process OCR objects, and retrieve all associated images.
• Performance Enhancements: Reduced memory usage.
• Domain-Specific Updates:
  • Fixed the last check in card template domain.
• Tracking and Reporting:
  • Introduced an experimental report generator in the tracker module.
  • Implemented heatmap visualization for the number of user messages.
  • Enhanced message module to show tracker and module matches.
  • Updated trackers UI to allow removal of objects.
• User Account Management:
  • Added display features for chats and subchannels.

Fixes

• OCR and UI Improvements:
  • Fixed an issue where 'None' was copied incorrectly in date fields.
  • Resolved UI correlation issues in the language block.
• Correlation Fixes:
  • Multiple fixes applied to the objects selector to enhance functionality.
• Export and Logging:
  • Fixed export functionality in TheHive component.
• UI Enhancements:
  • Resolved issues with empty matches and added handling for overlapping matches in the UI matches extractor.
• Language Processing:
  • Fixed the minimum probability setting for item languages to improve accuracy.

Overview of AIL features

The new AIL OCR module in action

A sample correlation between chat users

JTAN

Development of the AIL framework is co-funded by the European Union CEF program and CIRCL.

The Action will establish a Joint Threat Analysis Network, an open collaboration group of European computer security incident response teams (CSIRTs) with the focus on collecting, sharing and analysis of technical, operational and strategic threat intelligence. The purpose of this collaboration is to combine unique advantages of different teams to obtain comprehensive situational awareness and actionable information to effectively defend constituencies in each Member State, from critical infrastructure operators targeted by state-sponsored actors to individual citizens affected by cybercrime. The main part of the Action addresses gaps in the Cyber Threat Intelligence (CTI) tooling that is currently used by the national level CSIRTs in Europe. By strengthening individual tools and interconnecting them, the beneficiaries will achieve a new level of common situational awareness and they will benefit from shared knowledge and tooling.

AIL framework 5.4 released with many semantic improvements in chat channels, new Tor vanity domain explorer, Favicon Correlations and various improvements.

The AIL Project version 5.4, released on March 25, 2024, includes a series of changes, fixes, and other updates.

Changes:

• Language Enhancements:

  • Add thread languages stats.
  • Add nb languages stats by chat/subchannel objects.
  • Improve language detection + UI for manual translation.

• New Features and Improvements:

  • Add lexilang and demoji to requirements.
  • Add default and basic cards for user accounts, chat-subchannels, image objects, etc.
  • Performance improvements in the global module.
  • Reprocessing tools for objects by type and improved error outputs in modules.
  • RetroHunt on messages, Direct Correlations UI changes, and heatmap for messages per hour in chats.
  • API enhancements for message retrieval and object investigations.
  • New crawler stats, pie charts, and stacked bar charts.
  • Basic API tests and updates in gitignore.
  • Refactor API blueprint.
  • Favicon Correlations.
  • Update vanity domains, including a vanity domain explorer.
  • Sort onion vanity names and new APIs for titles.
  • Enhanced tracker listings and retro hunt features.

Fixes:

• Chat and Subchannel Issues:

  • Fix subchannel-message correlation and empty chat card issues.
  • Address language detection in empty messages.

• Dashboard and Tracker Corrections:

  • Fix object links in the dashboard and Yara content errors in the tracker.
  • Correct typos.

• Module and Domain Adjustments:

  • Fix SQL Injection Detection object ID and domain-related issues.
  • Fix favicon crawler and export issues.

• Testing and Documentation:

  • Adjust tests and update documentation for chat JSON fields and chat translation.
  • Various fixes in monthly crawled domain stats, favicon export, vanity titles, and chat image importer.

For detailed overview of the all the changes, the changelog provides an extensive overview of all the changes.

JTAN

Development of the AIL framework is co-funded by the European Union CEF program and CIRCL.

The Action will establish a Joint Threat Analysis Network, an open collaboration group of European computer security incident response teams (CSIRTs) with the focus on collecting, sharing and analysis of technical, operational and strategic threat intelligence. The purpose of this collaboration is to combine unique advantages of different teams to obtain comprehensive situational awareness and actionable information to effectively defend constituencies in each Member State, from critical infrastructure operators targeted by state-sponsored actors to individual citizens affected by cybercrime. The main part of the Action addresses gaps in the Cyber Threat Intelligence (CTI) tooling that is currently used by the national level CSIRTs in Europe. By strengthening individual tools and interconnecting them, the beneficiaries will achieve a new level of common situational awareness and they will benefit from shared knowledge and tooling.

AIL framework 5.3 released with chat explorer, Discord and Telegram monitoring, automatic translation, new features and various bugs fixed

Chat Explorer

The Chat Explorer in AIL v5.3 streamlines chat data analysis. It allows users to easily navigate through messages, threads, and subchannels, offering a straightforward approach to monitoring chat activities and extracting insightful information.

Discord and Telegram chats can now be imported using new importers/feeders:

• Discord Feeder: Monitors and imports Discord chat data. Learn more
• Telegram Feeder: Monitors and imports Telegram chat data. Learn more

Future releases will aim to further simplify the import process for other chat platforms. Basic documentation on the JSON format for importing messages is available here.

New Chat Features

AIL v5.3 introduces various new chat features, including:

• Chat Metadata: Name, username, icon, description, participants, etc.
• Chat Subchannels
• Threads in Chats/Subchannels/Messages
• Emoticons
• Images in Messages
• Message Replies
• User Metadata: Account ID, name, icon, additional info, username, etc.

To address language barriers in chat analysis, AIL v5.3 integrates LibreTranslate, an open-source, self-hosted machine translation tool. This feature allows for the translation of chat messages, facilitating easier analysis and comprehension of conversations in different languages without using external services. The information collected via the different chat are processed in the AIL framework and benefit from all the different analysis modules.

AIL v5.3 enhances the correlation of User Accounts, Chats, Images, and Messages, providing a more integrated view of the data and enabling deeper analytical insights.

Improvements

• Significant improvements in the crawling capabilities and integration with Lacus.
• Support for HHHash correlation added to correlate server per HTTP header profile.
• DomainClassifier improved especially on the performance aspect and caching.
• Kvrocks back-end updated to the latest version.

For detailed overview of the all the changes, the changelog provides an extensive overview of all the changes.

AIL images in LXD are available as download and can be used directly in production environment.

JTAN

Development of the AIL framework is co-funded by the European Union CEF program and CIRCL.

The Action will establish a Joint Threat Analysis Network, an open collaboration group of European computer security incident response teams (CSIRTs) with the focus on collecting, sharing and analysis of technical, operational and strategic threat intelligence. The purpose of this collaboration is to combine unique advantages of different teams to obtain comprehensive situational awareness and actionable information to effectively defend constituencies in each Member State, from critical infrastructure operators targeted by state-sponsored actors to individual citizens affected by cybercrime. The main part of the Action addresses gaps in the Cyber Threat Intelligence (CTI) tooling that is currently used by the national level CSIRTs in Europe. By strengthening individual tools and interconnecting them, the beneficiaries will achieve a new level of common situational awareness and they will benefit from shared knowledge and tooling.

AIL framework 5.2 released with new features and various bugs fixed

The latest release, version 5.2 of the AIL project framework, introduces several changes, fixes, and improvements. Some notable changes include the removal of old updates prior to version 5.0, a refactoring of the background updater along with the addition of the v5.2 update, and the introduction of a new etag object. The correlation graph now offers an option to hide objects/nodes and reset functionality using the "H" key. Additionally, an object comment feature has been added to investigations.

Several fixes have been implemented in this release. The environment issues in the updater and background update modules have been resolved. The crawler now includes timeouts for Unknown captures and exception handling for ping_lacus. It also performs an existence check for screenshot sets. The decoding process for downloaded files has been fixed, and the tag functionality for correlation objects has been improved. Fixes have also been made to the updater module, including the removal of old ARDB environment references. Lastly, issues with MISP event JSON export and retro hunt date search and description in the hunter module have been addressed.

In terms of contributors, Thirion Aurélien and fukusuket have made significant contributions to this release, addressing specific issues and providing fixes.

Detailed Change Log

v5.2 (2023-07-12)

Changes

• [update] remove old updates < 5.0. [Terrtia]

• [updater] refactor background updater + add v5.2 update. [Terrtia]

• [crawler har] compress HAR. [Terrtia]

• [correlation] correlation graph, add an option to hide an object/node by pressing H + reset correlation graph. [Terrtia]

• [etag] add new etag object. [Terrtia]

• [investigation] add object comment. [Terrtia]

Fix

• [updater] fix env. [Terrtia]

• [background update] fix logger. [Terrtia]

• [crawler] add timeout to Unknown captures. [Terrtia]

• [crawler] add exception handing for ping_lacus. [fukusuket]

• [crawler] added existence check for screenshot set. [fukusuket]

• [decoded] fix download file. [Terrtia]

• [updater] fix db checker. [Terrtia]

• [correlation tags] fix tag all objects. [Terrtia]

• [correlation card decoded meta] mimetype + size. [Terrtia]

• [correlation card decoded meta] mimetype + size. [Terrtia]

• [updater] remove old ARDB env. [Terrtia]

• [hunter + misp export] fix misp event json export + retro hunt date search and description. [Terrtia]

Other

• Merge pull request #174 from fukusuket/fix-500-erro-when-invalid-lacus-url. [Thirion Aurélien]

  fix: [crawler] add exception handing for ping_lacus

• Merge pull request #176 from fukusuket/fix-500-error-when-crawler-screenshot-setting-off. [Thirion Aurélien]

  fix: [crawler] added existence check for screenshot set

AIL framework 5.1 released with new features and many bugs fixed

Version 5.1 (2023-06-26) includes several changes, fixes, and updates. The changes include fixing gzipped pastes in the pystemon importer, showing a message when the maximum number of nodes is reached in the correlation graph, and adding the ability to auto tag crawled domains. Additionally, new features were added such as pagination for title searches, the ability to search title IDs and contents, and the inclusion of a favicon object.

Several fixes were implemented, including resolving issues with base64 encoding in the pystemon importer, maintaining the same capture UUID for already crawled domains in the crawler, and handling empty queues in the IPAdress module. Other fixes addressed issues with title searches returning empty results, incomplete responses in the crawler, and errors related to user tokens and deletion.

Various improvements were made to different modules and objects, such as the addition of a new cookie-name object along with its correlation, enhancements to importers, improvements to the HOWTO guide, and updates to correlation graphs and statistics. The Phone module was also updated to filter invalid phone numbers and display extracted information in the user interface.

In addition to the changes and fixes, there were updates to the MISP export, domains explorer, daterange object, tracker module, and various other components. The README.md file and CI badge were corrected, the installer was fixed for YARA and pycld3 installations, and tests were updated and replaced.

Lastly, there were some general updates, including merging changes from the old CIRCL/AIL-framework repository (the official repository is ail-project/ail-framework and incorporating a pull request related to email categorization.

Overall, version 5.1 introduced new features, addressed several issues, and included various updates and improvements to different parts of the system.

Detailed Change Log

v5.1 (2023-06-26)

Changes

• [pystemon importer] fix gzipped pastes. [Terrtia]

• [correlation graph] show message if max_nodes reached + fix cookie-name sparkline. [Terrtia]

• [crawler] auto tag crawled domains. [Terrtia]

• [correlation] add an option to remove max number of nodes if max_node == 0. [Terrtia]

• [object cookie-name] add new cookie-name object + correlation. [Terrtia]

• [title search] add pagination. [Terrtia]

• [titles] add title IDs and contents search. [Terrtia]

• [favicon object] add favicon object. [Terrtia]

• [sow item] show item investigations. [Terrtia]

• [kvrocks migration] mv update/v.50. [Terrtia]

• [redis] update minimal version. [Terrtia]

• [doc] add AIL v5.0 + objects + Importers + sync. [Terrtia]

• [correlation] filter blank screenshots. [Terrtia]

• [importers] improve abstract class and logs. [Terrtia]

• [domains explorer] unsafe tag default image. [Terrtia]

• [README.md] update. [Terrtia]

• [HOWTO] improve HOWTO. [Terrtia]

• [correlation graph] update node legend. [Terrtia]

• [correlation graph] select correlation depth. [Terrtia]

• [correlation] correlation graph: filter title objects. [Terrtia]

• [correlation] add direct correlation stats. [Terrtia]

• [new title object] add new title object + correlation on page title. [Terrtia]

• [Phone module] Filter Invalid Phone numbers + UI Show extracted. [Terrtia]

• [importers] add Dir/File Importer. [Terrtia]

Fix

• [pystemon importer] fix base64 encoding. [Terrtia]

• [crawler] same capture uuid if a domain is already crawled. [Terrtia]

• [IPAdress module] empty queue if no IP ranges provided. [Terrtia]

• [retro hunt] fix object tag queue + decoded content. [Terrtia]

• [daterange object] fix objects by date. [Terrtia]

• [title] fix title search empty result. [Terrtia]

• [crawler] fix incomplete response. [Terrtia]

• [user] fix get user token #163. [Terrtia]

• [user] fix user delete #163. [Terrtia]

• [MISP export] fix ail object first/last seen + obj logger. [Terrtia]

• [MISP export] fix empty event. [Terrtia]

• [d4] change enable d4. [Terrtia]

• [kvrocks migration] [Terrtia]

• [objects] fix investigation + ail2ail + screenshot MISP export. [Terrtia]

• [domains explorer] None screeenshot. [Terrtia]

• [show domains] fix down domains. [Terrtia]

• [domains explorer] domain screeenshot. [Terrtia]

• [domains explorer] fix empty screenshots. [Terrtia]

• [correlation] fix tagging nb nodes. [Terrtia]

• [README.md] fix CI badge. [Terrtia]

• [README.md] fix logo. [Terrtia]

• [module.cfg] fix templateModule example. [Terrtia]

• [module extractor] fix tracker extractor. [Terrtia]

• [tracker] fix tracker delete. [Terrtia]

• [tracker] fix webhook. [Terrtia]

• [crawler] fix undefined capture status. [Terrtia]

• [correlation btc info] catch btc txs error. [Terrtia]

• [Phone module] Filter Invalid Phone numbers. [Terrtia]

• [phone] fix phone module. [Terrtia]

• [domain search] fix template domain types filter. [Terrtia]

• [domain search] fix template domain types filter. [Terrtia]

• [MISP auto export] fix module input message. [Terrtia]

• [tests] replace unmaintened nose by nose2. [Terrtia]

• [tests] fix tests. [Terrtia]

• [instaler] fix yara and pycld3 install. [Terrtia]

• [tests] github workflow. [Terrtia]

• [tests] github workflow. [Terrtia]

• [flask] remove old import. [Terrtia]

Other

• Merge github.com:CIRCL/AIL-framework. [Terrtia]

• Merge pull request #592 from shadow2033/patch-2. [Thirion Aurélien]

  Update Categ Mail

• Update Mail. [shadow2033]

  ///English
  added (inbox; zoho)

  ///Russian
  добавлен (inbox; zoho)