Permanent error: unsupported metric http.server.duration (Histogram)

[asitkumar13]

I am using otel-collector chart 0.8.1 and app version is 0.37.1.
I have update batch processors like below and used in metrics & traces service pipeline.
batch/2:
timeout: 100ms
send_batch_size: 1000
send_batch_max_size: 1100

reason to reduce timeout to 100 millisecond as I was observing few traces from different app services were getting batched together, which were generated in gap of milliseconds.

When I updated otel javagant v 1.10.1 in my apps services, I started getting below error message in otel-collector pod.

2022-02-03T14:11:40.636Z warn batchprocessor/batch_processor.go:185 Sender failed {"kind": "processor", "name": "batch/2", "error": "Permanent error: unsupported metric http.server.duration (Histogram); Permanent error: unsupported metric http.client.duration (Histogram); Permanent error: unsupported metric http.client.duration (Histogram)", "errorCauses": [{"error": "Permanent error: unsupported metric http.server.duration (Histogram)"}, {"error": "Permanent error: unsupported metric http.client.duration (Histogram)"}, {"error": "Permanent error: unsupported metric http.client.duration (Histogram)"}]}

Please help me here.

[mx-psi]

What exporters are you using? This looks like an error from the NewRelic exporter cc @alanwest @jack-berg @nrcventura

[asitkumar13]

yes, I am using newrelic exporter.

  service:
    pipelines:
      metrics:
        exporters:
          - newrelic
        processors:
          - batch/2
          - memory_limiter
        receivers:
          - otlp
      traces:
        exporters:
          - newrelic
        processors:
          - batch/2
          - memory_limiter
        receivers:
          - otlp

[jack-berg]

This error occurs when you send metrics to New Relic with cumulative aggregation temporality. New Relic currently only supports delta aggregation temporality. Best option is to configure the application emitting the data to export with delta temporality instead of the default cumulative.

If thats not an option, there is a cumulativetodeltaprocessor you can add to your metrics pipeline to do the conversion in the collector. However, the cumulativetodeltaprocessor is experimental so proceed with caution.

Also note that because New Relic not supports OTLP ingest, the newrelicexporter is deprecated. Please use the OTLP gRPC exporter instead as described in the README.

Apologies for the inconvenience!