Real Time Logging

When running in (near) real-time logging mode, the pilot (via the API:s below) sends the tail of the payload log as soon as there is a detected update. The tail is visible on the web interface (the PanDA monitor job page in the case of ATLAS) typically within a few seconds.

The pilot supports three types of (near) real-time logging.

1. Google Cloud Logging. Requires google.cloud.logging and google.cloud.logging_v2.handlers python modules. Google Cloud Platform provides a web interface for viewing the logs. Used by Rubin in production
2. Fluentd. Requires fluent python module. Tested by Rubin
3. Logstash. Requires logstash python module. Logs are sent to a logstash server at CERN. Used by ATLAS in production (both production jobs and user jobs can enable real-time logging)

The logging server (url:port) must be defined in the pilot config file (default.cfg).