Unable to send metrics using googlecloudexporter to custom endpoint with incorrect errors

[Mona3087]

Component(s)

exporter/googlecloud

What happened?

Description

We are using otel collector to collect custom metrics from our Kubernetes cluster and export it to Google Cloud Monitoring for alerts. When have lots of pods in our cluster, our clusters could be generating more than 200k time series per minute. Otel at that point starts logging error and is not able to push metrics in cloud. The error is talking about hitting 200 quota limit and it is misleading. Ref: https://cloud.google.com/monitoring/quotas

Steps to Reproduce

Create sample pod/app that pushes more than 200k time series on single monitored resource type.

Expected Result

Error should specify that which quota limit otel exporter is hitting.

Actual Result

2024-11-19T17:18:39.278Z error internal/queue_sender.go:92 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = ResourceExhausted desc = One or more TimeSeries could not be written: timeSeries[0-53]: write for resource=k8s_cluster{location:us-east1,cluster_name:ctl-gke-4501} failed with: Monitored resource has too many time series (custom metrics).\nerror details: name = Unknown desc = total_point_count:54 errors:{status:{code:8} point_count:54}", "dropped_items": 54}

Collector version

v0.114.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
GKE 1.31.1-gke.2105000

OpenTelemetry Collector configuration

processors: 
  batch:       
    send_batch_size: 200       
    send_batch_max_size: 200
    timeout: 200ms
  memory_limiter:      
    check_interval: 1s       
    limit_mib: 1600       
    spike_limit_mib: 320
exporters:    
  googlecloud:      
    project: ""      
    metric:       
      prefix: "custom.googleapis.com"
service:  
  pipelines:    
    metrics:     
      recievers: []    
      processors: [batch, memory_limiter]     
      exporters: [googlecloud]

Log output

2024-11-19T17:18:39.278Z	error	internal/queue_sender.go:92	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = ResourceExhausted desc = One or more TimeSeries could not be written: timeSeries[0-53]: write for resource=k8s_cluster{location:us-east1,cluster_name:ctl-gke-4501} failed with: Monitored resource has too many time series (custom metrics).\nerror details: name = Unknown  desc = total_point_count:54  errors:{status:{code:8}  point_count:54}", "dropped_items": 54}

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.