Log Stacking

[akshat-zopsmart]

Background

In case of production systems, log level is often set to ERROR and hence any messages logged at lower levels are not printed. However, in case of an error encountered, such logs are crucial. Application teams most end up changing the log level to lower values in the hope of encountering the error again, with the detailed logs available this time.

Challenges with current process (lowering the log level)

• High turn-around time for the application team to isolate the issue
• Increased log volumes causing spike in the cost of processing & storing logs. May also impact the performance of the application as well due to excessive logging.

Requirement

In case of an error level log in a request, all messages logged in the request till that point (within the context) should also get printed so that the application team does not need to change the log level to get the detailed logs.

Implementation Proposed

Framework to maintain the messaged, logged at lower levels, in memory and print them out whenever a message is logged at ERROR or higher level. Stack trace being printed in the ERROR log may also be removed given that it does not have much utility for scenarios where messages are logged at ERROR level.

[vikash]

This sounds like a good idea. However, it will significantly increase the memory uses. Because, we will not know which request will result into 500, which means ALL request related logs will need to be kept in memory. At very high loads, this can be problematic. For a corner case like this, not sure if increased memory is a good tradeoff.