Opni Workload DRAIN Service
This service serves as a cache for log messages that have been inferred on by creating a log template for each messages. It is based on the DRAIN3 log templatization algorithm. If a log message comes to this service and it is not currently a part of the cache, it will be sent to the Deep Learning service to be inferred on. Once the anomaly level of the log has been determined, the inferencing service will send back this log to the DRAIN workload service and the log message with the corresponding anomaly level will be added to the cache. The result will then be sent to the Opensearch Updating Service
- Python
- Create log templates for workload log messages.
- Serve as a caching mechanism with the log template and anomaly level which is determined via the Deep Learning model.
- Persist cache into S3 every hour if cache has been modified in any way.
- Send log message insights including log template and anomaly level to the Opensearch Updating Service.
| Component | Type | Description |
|---|---|---|
| preprocessed_logs_workload | Nats subject | The logs received from the raw_logs Nats subject come from the ingest plugin. |
| model_workload_parameters | Nats subject | The payload received from the model_workload_parameters Nats subject is a nested JSON dictionary with the workloads of interest selected by the user. This payload is sent by the training controller service when the user updates the watchlist of workloads/ The preprocessing service will then take the payload |
| model_inferenced_workload_logs | Nats subject | Once the logs have been inferred by the Deep Learning model trained on the designated workload logs, the workload DRAIN service will receive the inferred results and add the results to its cache. |
| batch_processed_workload | Nats subject | Anytime the DRAIN3 model cache has been updated in any form, it will send over payload to the batch_processed_workload Nats subject. If it has been at least an hour since the DRAIN cache model was last persisted into S3, then the updated cache will be saved into S3. |
| Component | Type | Description |
|---|---|---|
| inferenced_logs | Nats subject | For any log messages which are stored within the DRAIN cache, obtain the anomaly level and then publish the updated payload to the inferenced_logs Nats subject which will be received by the Opensearch Updating Service. |
| templates_index | Nats subject | When a new log template has been created or a log template has been modified, publish to the templates_index Nats subject so that it can be updated by the Opensearch Updating service. |
| opnilog_workload_logs | Nats subject | For any log messages that are not currently stored in the DRAIN cache, send over those logs to the Deep Learning model for inferencing by publishing to the opnilog_workload_logs Nats subject. |
| batch_processed_workload | Nats subject | Anytime the DRAIN3 model cache has been updated in any form, it will send over payload to the batch_processed_workload Nats subject. |
- Does not support an HA setup properly.
- Multiple workload DRAIN service pods means multiple caches have to be maintained.
- Unit tests
- Integration tests
- e2e tests
- Manual testing