Publish AST v0.6.0

.env-example

@@ -2,3 +2,7 @@
 # These should be updated to more secure values outside of testing environments.
 GF_SECURITY_ADMIN_USER=admin
 GF_SECURITY_ADMIN_PASSWORD=admin
+
+# Optional Parameters Required for metrics export to F5 DataFabric
+SENSOR_SECRET_TOKEN="YOUR_TOKEN"
+SENSOR_ID="YOUR_ID"

.github/workflows/deploy-pages.yml

@@ -0,0 +1,26 @@
+name: GitHub Pages
+run-name: ${{ github.actor }} is running this workflow
+on:
+  push:
+    tags:
+      - 'v[0-9]+.[0-9]+.[0-9]+'
+jobs:
+  # Single deploy job no building
+  deploy:
+    environment:
+      name: github-pages
+      url: ${{steps.deployment.outputs.page_url}}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Setup Pages
+        uses: actions/configure-pages@v3
+      - name: Upload Artifact
+        uses: actions/upload-pages-artifact@v2
+        with:
+          # upload pages directory
+          path: './pages'
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v2

README.md

@@ -1,11 +1,16 @@
 # Application Study Tool
 
+> 🚨🚨**Notice**🚨🚨
+> 
+> Configuration for the Application Study Tool has changed significantly in the v0.6.0 release. To
+update a legacy configuration, see [docs/config_migration.md](docs/config_migration.md).
+
 ## Overview
 
 The Application Study Tool is intended to provide enhanced insights into (classic) BIG-IP products, leveraging best in class
 open source telemetry tools. The full installation includes:
 
-* Custom Instance of OpenTelemetry Collector with enhanced BIG-IP data receivers (data fetched via iControlRest).
+* Custom Instance of OpenTelemetry Collector with enhanced BIG-IP data receivers (data fetched via iControlRest) [Full List of Metrics Collected](docs/receiver_metrics.md).
 * Prometheus timeseries database for storing and querying collected data.
 * Grafana Instance with pre-configured dashboards for quick insights at the device and "fleet" levels.
 
@@ -34,42 +39,102 @@ cd application-study-tool
 cp .env-example .env
 # Edit the following file with device secrets as required (see "Configure Device Secrets" below)
 cp .env.device-secrets-example .env.device-secrets
-# Edit the config file with device / connection info (see "Configure Devices To Scrape" below)
-vi ./config/big-ips.json
+# Edit the default settings for your environment as required
+# (see "Configure Default Device Settings" below)
+vi ./config/ast_defaults.yaml
+# Edit the config file with device / connection info
+# (see "Configure Devices To Scrape" below)
+vi ./config/bigip_receivers.yaml
+# Run the configuration generator
+docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config
 # Start the tool
 docker-compose up
 ```
 
-### Configure Devices To Scrape
-Application Study Tool includes an init container which builds an OpenTelemetry
-Collector Configuration file based on a provided list of BIG-IPs in JSON format.
-
-Edit config/big-ips.json to reflect your list of BIG-IPs and their access credentials:
-```json
-[
-  {
-    // Set this to the management IP for the device. This must be
-    // reachable from the Application Study Tool host.
-    "endpoint": "https://10.0.0.1",
-    // Set this to the desired account's user name
-    "username": "admin",
-    // This field tells the collector the name of an environment variable
-    // which contains the password for the device.
-    // This field does not contain the password itself.
-    "password_env_ref": "BIGIP_PASSWORD_1",
-    // Secure TLS communication requires mounting the certificate bundle
-    // used to sign the BigIP certificates. Though not recommended, in the
-    // case of self-signed certificates or for testing purposes, you can skip
-    // this check by setting this field to true.
-    "tls_insecure_skip_verify": false,
-    // The path to a CA File used to validate BIG-IP certificates. This is required
-    // if tls_insecure_skip_verify is set to false. See below for details.
-    "ca_file": "",
-  }
-]
-```
-
-### Configure Device Secrets
+## Configuration
+
+For additional configuration management background, see
+[docs/config-management.md](docs/config-management.md).
+The below assumes you're using the config_helper script for assisted management.
+
+
+Application Study Tool config management relies on default configs in
+[/configs/ast_defaults.yaml](/configs/ast_defaults.yaml) and device specific information in
+[/configs/bigip_receivers.yaml](/configs/bigip_receivers.yaml).
+
+Settings in the bigip_receivers.yaml override those in ast_defaults.yaml.
+
+To update a legacy (pre v0.6.0) configuration, to the new scheme see
+[docs/config_migration.md](docs/config_migration.md)
+
+## Configure Default Device Settings
+
+Edit config/ast_defaults.yaml to reflect common values for your BIG-IPs:
+```yaml
+# These configs are applied to each entry in the bigip_receivers file
+# where they don't contain an equivalent / overriding entry.
+bigip_receiver_defaults:
+  # The time to wait between metric collection runs
+  collection_interval: 60s
+  # The username to login to the device with
+  username: admin
+  # The password (not recommended) or a reference to an env variable (recommended)
+  # Below tells the collector to look for an environment variable named
+  # BIGIP_PASSWORD_1
+  password: "${env:BIGIP_PASSWORD_1}"
+  # The data_types that should be enabled or disabled.
+  # DNS and GTM are disabled by default and users can enable those modules
+  # on all devices by setting the below to true.
+  # A full list of data_types is in /docs/receiver_readme.md.
+  data_types:
+    f5.dns:
+      enabled: false
+    f5.gtm:
+      enabled: false
+  # The TLS settings to use. Either a CA file must be specified or
+  # insecure_skip_verify set to true (not recommended).
+  tls:
+    # Secure TLS communication requires mounting the certificate bundle
+    # used to sign the BigIP certificates. Though not recommended, in the
+    # case of self-signed certificates or for testing purposes, you can skip
+    # this check by setting this field to true.
+    insecure_skip_verify: false
+    # The path to a CA File used to validate BIG-IP certificates. This is required
+    # if tls_insecure_skip_verify is set to false. See below for details.
+    ca_file: ""
+```
+
+## Configure Devices To Scrape
+Edit the device list in config/bigip_receivers.yaml:
+```yaml
+#### Values not explicitly configured here inherit values in ast_defaults.yaml.
+#### Each entry must have a unique name, starting with bigip/ 
+#### (e.g. bigip/1, bigip/2)
+bigip/1:
+  #### Endpoint must be specified for each device
+  #### because there's no rational default.
+  #### Set this to the management IP for the device. This must be
+  #### reachable from the Application Study Tool host.
+  endpoint: https://10.0.0.1
+  #### Override some default settings with device specific values
+  username: SOME_OVERRIDE_ACCOUNT_NAME
+  password: "${SOME_OTHER_ENV_VAR_WITH_ANOTHER_PASSWORD}"
+  #### Everything commented out here gets the value from default
+  # collection_interval: 30s
+  # data_types:
+  #   f5.dns:
+  #     enabled: false
+  #   f5.gtm:
+  #     enabled: false
+  # tls:
+  #   insecure_skip_verify: true
+  #   ca_file:
+bigip/2:
+  endpoint: https://10.0.0.2
+```
+
+
+## Configure Device Secrets
 The application study tool default configuration relies on environment variables
 which contain device access credentials. There are a number of ways to manage and
 inject secrets into a container environment (modifications to the docker-compose file
@@ -84,12 +149,78 @@ BIGIP_PASSWORD_2=bar-foo123!
 ```
 
 The variable name (the part on the left of the equal sign) must match the configured
-value for the devices that use this password in config/big-ips.json.
+value for the devices that use this password in config/ast_defaults.yaml or device specifc
+cofig in config/bigip_receivers.yaml. In the following example, bigip/1 uses BIGIP_PASSWORD_1
+from  the defaults and bigip/2 uses BIGIP_PASSWORD_2 from the device settings:
+
+```
+##############################
+## config/ast_defaults.yaml 
+##############################
+
+bigip_receiver_defaults:
+  ...
+  password: "${env:BIGIP_PASSWORD_1}"
+  ...
+
+##############################
+## config/bigip_receivers.yaml
+##############################
+
+# This gets the default "${env:BIGIP_PASSWORD_1}"
+bigip/1:
+  endpoint: https://10.0.0.1
 
-#### Account Permissions
+# This overrides it with "${env:BIGIP_PASSWORD_2}"
+bigip/2:
+  endpoint: https://10.0.0.1
+  password: ${env:BIGIP_PASSWORD_2}
+
+```
+
+
+## Configure Periodic Metric Data Export To F5
+The application study tool can be configured to periodically (every 5 minutes) export a snapshot of your
+BigIP metrics to F5. Contact your F5 Sales Representative for a "Sensor ID" (a unique string used to associate
+your metrics with your Organization) and a "Sensor Secret Token" (used to authenticate to the F5 Datafabric as
+an authorized data sender for your Org).
+
+This functionality is enabled as follows:
+
+1. Enable the flag in [config/ast_defaults.yaml](config/ast_defaults.yaml) file as follows:
+
+```yaml
+# Set this true to enable periodic metric export to F5 DataFabric.
+# Requires adding your sensor ID and secret token to the container environment (see .env-example).
+# Contact your F5 sales rep to obtain the ID / secret token.
+f5_data_export: true
+```
+
+2. Add the Sensor ID and Secret Token to the .env file, or otherwise attach it to the Opentelemetry Collector container
+as SENSOR_ID and SENSOR_SECRET_TOKEN (see [.env-example](./.env-example) for example).
+
+3. Run the configuration helper script (see below).
+
+## Run The Configuration Helper
+The config helper script can be run natively or via docker to merge the default and device
+level configs into the final OTEL Collector config as follows:
+```shell
+# Run the configuration generator
+docker run --rm -it -w /app -v ${PWD}:/app --entrypoint /app/src/bin/init_entrypoint.sh python:3.12.6-slim-bookworm --generate-config
+```
+
+This will write 2 new files in the services/otel_collector directory:
+
+* `receivers.yaml` - The final list of scraper configs and their settings.
+* `pipelines.yaml` - The final pipeline configs that map receievers to output destinations
+(prometheus, and optionally F5).
+
+## Account Permissions
 The vast majority of telemetry data can be collected with read-only access to the BigIP. Some
-granular stats are only available as output to a iControl Rest 'bash' shell command, and these require
-read-write access. If a read-only account is used, the following metrics are unavailable:
+granular stats are only available as output to a iControl Rest 'bash' shell command, and
+these require read-write access.
+
+If a read-only account is used, the following metrics are unavailable:
 
 ```
 f5_virtual_server_profile_client_ssl_connection_count{}
@@ -117,39 +248,35 @@ f5_plane_cpu_utilization_5s{}
 This will impact data output in several dashboards/panels (denoted with description fields indicating as such).
 
 ### Configure CA File
-AST expects a valid TLS cert bundle unless `tls_insecure_skip_verify` is
+AST expects a valid TLS cert bundle unless `tls.insecure_skip_verify` is
 set to true for each device. In order to mount and use your CA file, you must
-configure the docker-compose.yaml file in this directory, and set the `ca_file` parameter to the resulting path. Example:
+configure the docker-compose.yaml file in this directory, and set the `ca_file` parameter to
+the resulting path. Example:
 
 docker-compose.yaml:
 ```yaml
   ...
   otel-collector:
     ...
     volumes:
-      - otel_collector:/etc/otel-collector-config
+      - ./services/otel_collector:/etc/otel-collector-config
       - ./config/ca_bundle.pem:/etc/ssl/ca_bundle.pem
 ```
 
-big-ips.json:
-```json
-[
-  { // device 1
-    ...
-    "ca_file": "/etc/ssl/ca_bundle.pem",
-  },
-  { // device 2
-    ...
-    "ca_file": "/etc/ssl/ca_bundle.pem",
-  },
-]
+config/ast_defaults.yaml (or the tls section of each device in config/bigip_receivers.yaml):
+```yaml
+bigip_receiver_defaults:
+  ...
+  tls:
+    insecure_skip_verify: false
+    ca_file: "/etc/ssl/ca_bundle.pem"
 ```
 
-The configuration parameter `tls_insecure_skip_verify` defaults to false. Installers
- that would like to opt-in to run in an insecure TLS mode must set
- `tls_insecure_skip_verify: true` for each BIG-IP in the config array and understand
- that the connection between the OTEL collector and the BIG-IP does not have secure
- TLS termination.
+The configuration parameter `tls.insecure_skip_verify` defaults to false. Installers
+who would like to opt-in to run in an insecure TLS mode must set
+`tls.insecure_skip_verify: true` and understand
+that the connection between the OTEL collector and the BIG-IP does not have secure
+TLS termination.
 
 ### Configure Grafana
 The Grafana instance can be configured via environment variables using their standard

VERSION

@@ -1 +1 @@
-v0.5.0
+v0.6.0

_config.yml

@@ -0,0 +1 @@
+markdown: kramdown

config/README.md

@@ -0,0 +1,8 @@
+# Configuration
+
+Files in this directory can be used to configure aspects of the Application Study Tool.
+
+For additional detail, see:
+
+* [Configuration Management](/docs/config_management.md) (new users start here)
+* [Pre v0.6.0 Config Migration](/docs/config_migration.md)