Disable logging response body

[razorcd]

Hello,

The response body of the AWS services is being logged in DEBUG mode by boto library. This is an issue when requesting AWS secrets, as it will expose private information.

Most developers temporarily enable DEBUG mode over the entire application to debug issues. This automatically enabled DEBUG mode in boto modules too.

To avoid this risk, it would be better to remove logging the body completely. Or at least to put it behind a external configuration that is disabled by default.

Source: https://github.com/boto/botocore/blob/develop/botocore/parsers.py#L241C25-L241C25

Thank you,
Cristian

[tim-finnigan]

Hi @razorcd thanks for reaching out. The current behavior is called out here in the documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/boto3.html#boto3.set_stream_logger

Be aware that when logging anything from 'botocore' the full wire trace will appear in your logs. If your payloads contain sensitive data this should not be used in production.

This also overlaps with a related issue: boto/boto3#2292. Here are a couple comments that are relevant in particular:

boto/boto3#2292 (comment)

The logging message in question is a wire-level representation of the request and as such may include sensitive information. At the wire-level it's not feasible to apply a log filter to catch secrets as the secrets are arbitrary and cannot be comprehensively caught by any pattern. In general, enabling debug logs on production workloads is a double-edged sword that requires extreme caution, and botocore/boto3 is no different. The pattern based filtering techniques mentioned are certainly applicable when particular formats are known, and there is nothing stopping you from applying filtering formats relevant to secrets in your application. While the logger setup by boto3 directly could include some filter (if a reasonable one could be defined), manual logger configuration such as the example shown originally would bypass it anyways.

In regards to a pattern-based filter to anonymize secret strings:

boto/boto3#2292 (comment)

Such a filter could be applied, but it's not comprehensive. It certainly won't stop all secrets in all APIs, and likely won't even catch all secrets within the secretsmanager API. At which point we're playing whac-a-mole with a mystical logging pattern hoping everything is caught which is potentially more dangerous than not having it as it could lead to a false sense of security. And again, any pattern we put in place within boto3 can easily be circumvented by custom/manual logging configuration.

To be clear, I'm not saying there's no improvements that can be made here. I'm just not convinced pattern based filtering is a robust enough solution.

Considering those comments from a maintainer, it does not seem like there is a clear path forward to address this.