Spike: solution for automatically disabling non-wrapper instrumentation with too many errors

[nr-ahemsath]

Agent instrumentation that is implemented as a wrapper (which is most agent instrumentation) has a couple of features that help protect the stability of customer's applications. See

newrelic-dotnet-agent/src/Agent/NewRelic/Agent/Core/Wrapper/WrapperService.cs

Line 149 in 9e355f3

try

1. Any exceptions thrown by a wrapper are caught by the WrapperService automatically so they don't affect the customer's app
2. If too many exceptions are thrown by a wrapper, it will be prevented from running again for the lifetime of that agent instance.

We have other instrumentation, however, that isn't implemented as a wrapper. Rather, a wrapper intercepts some configuration method in a library, and the agent "installs" some other component - a piece of middleware, a Serilog sink, a MassTransit filter, etc. Unhandled exceptions which occur in this kind of instrumentation can and will affect the customer's app because there is no generic try/catch protecting it.

A generic solution for both protecting customers apps from exceptions in this kind of instrumentation, and the ability to have the agent notice such exceptions and disable the offending instrumentation is desired. The outcome of the spike should be a document describing a proposed implementation, or else what was tried and rejected and why. A stretch goal would be stories for implementing the solution with estimates.

[workato-integration]

https://new-relic.atlassian.net/browse/NR-181670