Merge pull request #466 from idaholab/v24.04.0_merge_idaholab

Malcolm v24.04.0

* Features and enhancements
    - Zeek-extracted files scanned and preserved on a [Hedgehog Linux](https://idaholab.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#HedgehogZeekFileExtraction) sensor can now be accessed via [the extracted files download user interface](https://idaholab.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtractionUI) (#331).
    - Improvements to creation of index templates, dashboards, and other saved objects on startup (#208) to ensure that saved objects get created correctly upon upgrade (see [this comment](#208 (comment)) for more details on this feature).
    - [Populating the NetBox inventory via passively-gathered network traffic metadata](https://idaholab.github.io/Malcolm/docs/asset-interaction-analysis.html#NetBoxPopPassive) now uses network traffic logs for DNS, NTLM, and DHCP to identify assets' host names when possible for use when populating device and VM names (#415). Autopopulated devices now have their *status* field set to `Active` rather than `Stage`, and uses *tags* instead to indicated that they were created through autopopulation.
    - Users can now specify pruning thresholds for [carved files](https://idaholab.github.io/Malcolm/docs/file-scanning.html#ZeekFileExtraction) so that old files are deleted in order to avoid filling available storage (#453). See a new section of documentation on [Managing disk usage](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about this and similar settings.
    - Users can now specify a prefix that will be prepended to dashboards as they are imported into OpenSearch Dashboards or Kibana, allowing users who have dashboards from other sources to differentiate between those and Malcolm's (#455).
    - The default anomaly detectors created for the OpenSearch Anomaly Detection plugin are now created with [category fields for high cardinality](https://opensearch.org/docs/latest/observing-your-data/ad/index/#optional-set-category-fields-for-high-cardinality) to allow for better breakdown of contributing values to anomalies discovered (#464).
    - Include [JA4+ plugin in Arkime](https://arkime.com/settings#ja4plus). See #419 for status on upcoming full JA4+ support in Malcolm.
    - Hedgehog Linux sensors can now [periodically refresh](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/hedgehog-iso/interface/sensor_ctl/control_vars.conf#L75) their [Zeek inteligence files](https://idaholab.github.io/Malcolm/docs/hedgehog-config-zeek-intel.html#HedgehogZeekIntel).
        + **NOTE**: Due to an oversight, a value is missing from the default Hedgehog Linux configuration in this release, preventing the intel refresh cron job from executing. As a workaround, appending the line `export INTEL_DIR=/opt/sensor/sensor_ctl/zeek/intel` to `/opt/sensor/sensor_ctl/control_vars.conf` and restarting the sensor services will remedy the situation. This will be corrected in the next Malcolm release.
    - Assorted documentation improvements.
* Component version updates
    - Arkime to [v5.1.2](https://github.com/arkime/arkime/blob/bcd9d7e68be8e4a52a17c35211c5d5a7fdcc1a1c/CHANGELOG#L36-L41)
    - OpenSearch and OpenSearch Dashboards to [v2.13.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.13.0.md)
    - Beats to [v8.13.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.13.2.html)
    - Logstash to [v8.13.2](https://www.elastic.co/guide/en/logstash/current/logstash-8-13-2.html)
    - gunicorn to v22.0.0 to address [CVE-2024-1135](GHSA-w3h3-4rj7-4ph4).
    - elasticsearch-dsl to [v8.13.0](https://github.com/elastic/elasticsearch-dsl-py/releases/tag/v8.13.0)
    - elasticsearch-py to [v8.13.0](https://github.com/elastic/elasticsearch-py/releases/tag/v8.13.0)
    - idna to v3.7 to address [CVE-2024-3651](GHSA-jjg7-2v4v-x38h)
    - Fluent Bit to [v3.0.3](https://fluentbit.io/announcements/v3.0.3/)
* Bug fixes
    - The documentation for [Windows host system configuration](https://idaholab.github.io/Malcolm/docs/host-config-windows.html#HostSystemConfigWindows) was out of date and has been updated for the latest version of Microsoft Windows Subsystem for Linux (#421).
    - An issue was fixed in which Malcolm's list of users and their password hashes could become corrupted if the file did not initially end with a newline character (#426).
    - The manner in which Zeek intel files are generated has been changed to avoid problems found in Kubernetes deployments when scaling out the number of `zeek-live` containers (#456). See [this comment](#456 (comment)) for more details.
    - Removed the version top-level element from `docker-compose.yml` files as it is [now obsolete](https://docs.docker.com/compose/compose-file/04-version-and-name/) and caused a warning message that sometimes was not handled correctly.
    - Fix Malcolm ISO not correctly detecting if it's in a live boot ISO environment or installed mode.
    - Restart live Zeek instances with `zeekctl deploy` instead of `zeekctl restart`.
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/blob/v24.04.0/config))
    - `ARKIME_QUERY_ALL_INDICES` in [`arkime.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/arkime.env.example#L9-L11) can be set to control the [`queryAllIndices` setting](https://arkime.com/settings#queryAllIndices) in Arkime's `config.ini`.
    - `DASHBOARDS_PREFIX` in [`dashboards-helper.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/dashboards-helper.env.example#L3C1-L4C19) has been added for #455 (see above in **Features and Enhancements**).
    - `LOGSTASH_NETBOX_ENRICHMENT_DATASETS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L13) has been changed to include `zeek.dhcp`, `zeek.dns`, and `zeek.ntlm` to support #415 (see above in **Features and Enhancements**).
    - `LOGSTASH_ZEEK_IGNORED_LOGS` in [`logstash.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/logstash.env.example#L15) has been changed to remove `capture_loss` and `stats` so that those diagnostic Zeek logs can be parsed without the user having to manually change this variable.
    - `ZEEK_CRON` has been removed from [`zeek-live.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-live.env.example) and `ZEEK_INTEL_REFRESH_CRON_EXPRESSION` was removed from [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example) and moved to the "offline" version of the container in [`zeek-offline.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek-offline.env.example#L17-L19) for #456.
    - `EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE`, `EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT`, and `EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS` were added to [`zeek.env`](https://github.com/idaholab/Malcolm/blob/bceee4616dd5676a010a3dd7b0410856257948e8/config/zeek.env.example#L32-L37) for #453. See a new section of documentation on [Managing disk usage](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#DiskUsage) for more information about these and similar settings.

.github/workflows/api-build-and-push-ghcr.yml

@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

.github/workflows/arkime-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/dashboards-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/dashboards-helper-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/file-upload-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/filebeat-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/freq-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/hedgehog-iso-build-docker-wrap-push-ghcr.yml

@@ -99,6 +99,9 @@ jobs:
           cp -r ./arkime/patch ./hedgehog-iso/shared/arkime_patch
           mkdir -p ./hedgehog-iso/suricata
           cp -r ./suricata/rules-default ./hedgehog-iso/suricata/
+          mkdir -p ./hedgehog-iso/nginx
+          cp -r ./nginx/landingpage/css ./hedgehog-iso/nginx/
+          cp -r ./nginx/landingpage/js ./hedgehog-iso/nginx/
           pushd ./hedgehog-iso
           echo "${{ steps.extract_malcolm_version.outputs.mversion }}" > ./shared/version.txt
           echo "${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}" > ./shared/maxmind_license.txt

.github/workflows/htadmin-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/logstash-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml

@@ -9,6 +9,8 @@ on:
       - 'malcolm-iso/**'
       - 'shared/bin/*'
       - '!shared/bin/configure-capture.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/zeek*'
       - '!shared/bin/suricata*'
       - '.trigger_iso_workflow_build'

.github/workflows/netbox-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/nginx-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/agg-init.sh'
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/os-disk-config.py'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'

.github/workflows/opensearch-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

.github/workflows/pcap-capture-build-and-push-ghcr.yml

@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

.github/workflows/pcap-monitor-build-and-push-ghcr.yml

@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

.github/workflows/postgresql-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

.github/workflows/redis-build-and-push-ghcr.yml

@@ -12,6 +12,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

.github/workflows/suricata-build-and-push-ghcr.yml

@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

.github/workflows/zeek-build-and-push-ghcr.yml

@@ -13,6 +13,8 @@ on:
       - '!shared/bin/common-init.sh'
       - '!shared/bin/sensor-init.sh'
       - '!shared/bin/os-disk-config.py'
+      - '!shared/bin/extracted_files_http_server.py'
+      - '!shared/bin/web-ui-asset-download.sh'
       - '!shared/bin/preseed_late_user_config.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'

Dockerfiles/arkime.Dockerfile

@@ -7,7 +7,7 @@ ENV TERM xterm
 ENV PYTHONDONTWRITEBYTECODE 1
 ENV PYTHONUNBUFFERED 1
 
-ENV ARKIME_VERSION "v5.0.1"
+ENV ARKIME_VERSION "v5.1.2"
 ENV ARKIME_DIR "/opt/arkime"
 ENV ARKIME_URL "https://github.com/arkime/arkime.git"
 ENV ARKIME_LOCALELASTICSEARCH no
@@ -16,7 +16,8 @@ ENV ARKIME_INET yes
 ADD arkime/scripts/bs4_remove_div.py /opt/
 ADD arkime/patch/* /opt/patches/
 
-RUN apt-get -q update && \
+RUN export DEBARCH=$(dpkg --print-architecture) && \
+    apt-get -q update && \
     apt-get -y -q --no-install-recommends upgrade && \
     apt-get install -q -y --no-install-recommends \
         binutils \
@@ -73,7 +74,10 @@ RUN apt-get -q update && \
     make install && \
     npm cache clean --force && \
     rm -f ${ARKIME_DIR}/wiseService/source.* ${ARKIME_DIR}/etc/*.systemd.service && \
-    bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded"
+    bash -c "file ${ARKIME_DIR}/bin/* ${ARKIME_DIR}/node-v*/bin/* | grep 'ELF 64-bit' | sed 's/:.*//' | xargs -l -r strip -v --strip-unneeded" && \
+    mkdir -p "${ARKIME_DIR}"/plugins && \
+    curl -fsSL -o "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so" "https://github.com/arkime/arkime/releases/download/${ARKIME_VERSION}/ja4plus.${DEBARCH}.so" && \
+    chmod 755 "${ARKIME_DIR}/plugins/ja4plus.${DEBARCH}.so"
 
 FROM debian:12-slim
 

Dockerfiles/dashboards-helper.Dockerfile

@@ -61,7 +61,7 @@ ADD scripts/malcolm_utils.py /data/
 
 RUN apk update --no-cache && \
     apk upgrade --no-cache && \
-    apk --no-cache add bash python3 py3-pip curl openssl procps psmisc npm rsync shadow jq tini && \
+    apk --no-cache add bash python3 py3-pip curl openssl procps psmisc moreutils npm rsync shadow jq tini && \
     npm install -g http-server && \
     pip3 install --break-system-packages supervisor humanfriendly requests && \
     curl -fsSLO "$SUPERCRONIC_URL" && \
@@ -95,7 +95,7 @@ RUN apk update --no-cache && \
                                 /opt/templates && \
     chmod 755 /data/*.sh /data/*.py /data/init && \
     chmod 400 /opt/maps/* && \
-    (echo -e "*/2 * * * * /data/create-arkime-sessions-index.sh\n0 10 * * * /data/index-refresh.py --index MALCOLM_NETWORK_INDEX_PATTERN --template malcolm_template --unassigned\n30 */2 * * * /data/index-refresh.py --index MALCOLM_OTHER_INDEX_PATTERN --template malcolm_beats_template --unassigned\n*/20 * * * * /data/opensearch_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
+    (echo -e "*/2 * * * * /data/shared-object-creation.sh\n0 10 * * * /data/index-refresh.py --index MALCOLM_NETWORK_INDEX_PATTERN --template malcolm_template --unassigned\n30 */2 * * * /data/index-refresh.py --index MALCOLM_OTHER_INDEX_PATTERN --template malcolm_beats_template --unassigned\n*/20 * * * * /data/opensearch_index_size_prune.py" > ${SUPERCRONIC_CRONTAB})
 
 EXPOSE $OFFLINE_REGION_MAPS_PORT
 

Dockerfiles/dashboards.Dockerfile

@@ -1,4 +1,4 @@
-FROM opensearchproject/opensearch-dashboards:2.12.0
+FROM opensearchproject/opensearch-dashboards:2.13.0
 
 LABEL maintainer="malcolm@inl.gov"
 LABEL org.opencontainers.image.authors='malcolm@inl.gov'
@@ -20,7 +20,7 @@ ENV PUSER_PRIV_DROP true
 ENV TERM xterm
 
 ENV TINI_VERSION v0.19.0
-ENV OSD_TRANSFORM_VIS_VERSION 2.12.0
+ENV OSD_TRANSFORM_VIS_VERSION 2.13.0
 
 ARG NODE_OPTIONS="--max_old_space_size=4096"
 ENV NODE_OPTIONS $NODE_OPTIONS
@@ -40,8 +40,8 @@ RUN yum upgrade -y && \
     /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin remove securityDashboards --allow-root && \
     cd /tmp && \
         # unzip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
-        # sed -i "s/2\.12\.0/2\.12\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
-        # sed -i "s/2\.12\.0/2\.12\.0/g" opensearch-dashboards/transformVis/package.json && \
+        # sed -i "s/2\.12\.0/2\.13\.0/g" opensearch-dashboards/transformVis/opensearch_dashboards.json && \
+        # sed -i "s/2\.12\.0/2\.13\.0/g" opensearch-dashboards/transformVis/package.json && \
         # zip transformVis.zip opensearch-dashboards/transformVis/opensearch_dashboards.json opensearch-dashboards/transformVis/package.json && \
         cd /usr/share/opensearch-dashboards/plugins && \
         /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin install file:///tmp/transformVis.zip --allow-root && \

Dockerfiles/file-monitor.Dockerfile

@@ -34,6 +34,9 @@ ARG EXTRACTED_FILE_SCANNER_START_SLEEP=10
 ARG EXTRACTED_FILE_LOGGER_START_SLEEP=5
 ARG EXTRACTED_FILE_MIN_BYTES=64
 ARG EXTRACTED_FILE_MAX_BYTES=134217728
+ARG EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE=1TB
+ARG EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT=0
+ARG EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS=300
 ARG VTOT_API2_KEY=0
 ARG VTOT_REQUESTS_PER_MINUTE=4
 ARG EXTRACTED_FILE_ENABLE_CLAMAV=false
@@ -65,6 +68,9 @@ ENV EXTRACTED_FILE_SCANNER_START_SLEEP $EXTRACTED_FILE_SCANNER_START_SLEEP
 ENV EXTRACTED_FILE_LOGGER_START_SLEEP $EXTRACTED_FILE_LOGGER_START_SLEEP
 ENV EXTRACTED_FILE_MIN_BYTES $EXTRACTED_FILE_MIN_BYTES
 ENV EXTRACTED_FILE_MAX_BYTES $EXTRACTED_FILE_MAX_BYTES
+ENV EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE $EXTRACTED_FILE_PRUNE_THRESHOLD_MAX_SIZE
+ENV EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT $EXTRACTED_FILE_PRUNE_THRESHOLD_TOTAL_DISK_USAGE_PERCENT
+ENV EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS $EXTRACTED_FILE_PRUNE_INTERVAL_SECONDS
 ENV VTOT_API2_KEY $VTOT_API2_KEY
 ENV VTOT_REQUESTS_PER_MINUTE $VTOT_REQUESTS_PER_MINUTE
 ENV EXTRACTED_FILE_ENABLE_CLAMAV $EXTRACTED_FILE_ENABLE_CLAMAV
@@ -103,6 +109,11 @@ ENV SUPERCRONIC_SHA1SUM "cd48d45c4b10f3f0bfdd3a57d054cd05ac96812b"
 ENV SUPERCRONIC_CRONTAB "/etc/crontab"
 
 COPY --chmod=755 shared/bin/yara_rules_setup.sh /usr/local/bin/
+ADD nginx/landingpage/css "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css"
+ADD nginx/landingpage/js "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/js"
+ADD --chmod=644 docs/images/logo/Malcolm_background.png "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/assets/img/bg-masthead.png"
+COPY --chmod=644 docs/images/icon/favicon.ico "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/favicon.ico"
+COPY --chmod=755 shared/bin/web-ui-asset-download.sh /usr/local/bin/
 
 RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sources && \
     apt-get -q update && \
@@ -129,7 +140,7 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
       pkg-config \
       tini \
       unzip && \
-    apt-get  -y -q install \
+    apt-get -y -q install \
       inotify-tools \
       libzmq5 \
       psmisc \
@@ -143,6 +154,7 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
     python3 -m pip install --break-system-packages --no-compile --no-cache-dir \
       clamd \
       dominate \
+      humanfriendly \
       psutil \
       pycryptodome \
       python-magic \
@@ -170,6 +182,8 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
       rm -rf "${SRC_BASE_DIR}"/yara* && \
     cd "${YARA_RULES_SRC_DIR}" && \
       /usr/local/bin/yara_rules_setup.sh -r "${YARA_RULES_SRC_DIR}" -y "${YARA_RULES_DIR}" && \
+    cd /tmp && \
+      /usr/local/bin/web-ui-asset-download.sh -o "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css" && \
     cd /tmp && \
       curl -fsSL -o ./capa.zip "${CAPA_URL}" && \
       unzip ./capa.zip && \
@@ -190,9 +204,6 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
         libtool \
         make \
         python3-dev && \
-      apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages autoremove && \
-      apt-get clean && \
-      rm -rf /var/lib/apt/lists/* /tmp/* && \
     mkdir -p /var/log/clamav "${CLAMAV_RULES_DIR}" && \
     groupadd --gid ${DEFAULT_GID} ${PGROUP} && \
       useradd -m --uid ${DEFAULT_UID} --gid ${DEFAULT_GID} ${PUSER} && \
@@ -214,31 +225,22 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/clam_scan.py && \
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/yara_scan.py && \
       ln -r -s /usr/local/bin/zeek_carve_scanner.py /usr/local/bin/capa_scan.py && \
-      echo "0 */6 * * * /bin/bash /usr/local/bin/capa-update.sh\n0 */6 * * * /usr/local/bin/yara_rules_setup.sh -r \"${YARA_RULES_SRC_DIR}\" -y \"${YARA_RULES_DIR}\"" > ${SUPERCRONIC_CRONTAB}
+      echo "0 */6 * * * /bin/bash /usr/local/bin/capa-update.sh\n0 */6 * * * /usr/local/bin/yara_rules_setup.sh -r \"${YARA_RULES_SRC_DIR}\" -y \"${YARA_RULES_DIR}\"" > ${SUPERCRONIC_CRONTAB} && \
+  apt-get -y -q --allow-downgrades --allow-remove-essential --allow-change-held-packages autoremove && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists/* /tmp/*
 
 USER ${PUSER}
 
 RUN /usr/bin/freshclam freshclam --config-file=/etc/clamav/freshclam.conf
 
 USER root
 
-ADD nginx/landingpage/css "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css"
-ADD nginx/landingpage/js "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/js"
-ADD --chmod=644 docs/images/logo/Malcolm_background.png "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/assets/img/bg-masthead.png"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u_w4BMUTPHjxsI9w2_Gwfo.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u8w4BMUTPHjxsAXC-v.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u_w4BMUTPHjxsI5wq_Gwfo.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u9w4BMUTPHh7USSwiPHA.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6uyw4BMUTPHjx4wWw.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 https://fonts.gstatic.com/s/lato/v24/S6u9w4BMUTPHh6UVSwiPHA.ttf "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/"
-ADD --chmod=644 'https://cdn.jsdelivr.net/npm/bootstrap-icons@1.5.0/font/fonts/bootstrap-icons.woff2?856008caa5eb66df68595e734e59580d' "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/bootstrap-icons.woff2"
-ADD --chmod=644 'https://cdn.jsdelivr.net/npm/bootstrap-icons@1.5.0/font/fonts/bootstrap-icons.woff?856008caa5eb66df68595e734e59580d' "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/css/bootstrap-icons.woff"
-
-COPY --chmod=644 docs/images/icon/favicon.ico "${EXTRACTED_FILE_HTTP_SERVER_ASSETS_DIR}/favicon.ico"
 COPY --chmod=755 shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
+COPY --chmod=755 shared/bin/prune_files.sh /usr/local/bin/
 COPY --chmod=755 shared/bin/service_check_passthrough.sh /usr/local/bin/
 COPY --chmod=755 shared/bin/zeek_carve*.py /usr/local/bin/
-COPY --chmod=755 file-monitor/scripts/*.py /usr/local/bin/
+COPY --chmod=755 shared/bin/extracted_files_http_server.py /usr/local/bin/
 COPY --chmod=644 shared/bin/watch_common.py /usr/local/bin/
 COPY --chmod=644 scripts/malcolm_utils.py /usr/local/bin/
 COPY --chmod=644 file-monitor/supervisord.conf /etc/supervisord.conf

Dockerfiles/filebeat.Dockerfile

@@ -1,4 +1,4 @@
-FROM docker.elastic.co/beats/filebeat-oss:8.12.1
+FROM docker.elastic.co/beats/filebeat-oss:8.13.2
 
 # Copyright (c) 2024 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="malcolm@inl.gov"

Dockerfiles/logstash.Dockerfile

@@ -1,4 +1,4 @@
-FROM docker.elastic.co/logstash/logstash-oss:8.12.1
+FROM docker.elastic.co/logstash/logstash-oss:8.13.2
 
 LABEL maintainer="malcolm@inl.gov"
 LABEL org.opencontainers.image.authors='malcolm@inl.gov'

Dockerfiles/opensearch.Dockerfile

@@ -1,4 +1,4 @@
-FROM opensearchproject/opensearch:2.12.0
+FROM opensearchproject/opensearch:2.13.0
 
 # Copyright (c) 2024 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="malcolm@inl.gov"