[exporter/datadog] API key validation neither succeeds nor fails when API key is passed in as env var with quotes v0.111.0+

[jackgopack4]

Component(s)

exporter/datadog

What happened?

Description

If Datadog API key is passed in with single and double quotations, e.g. '"my-api-key"', Datadog exporter fails to validate but does not produce an error message. This causes 403 errors to be thrown when telemetry is sent to Datadog API and doesn't leave it clear what is happening.

This also happens if it is passed in via env var and that env var contains double quotations
testfile.txt:

"my-api-key"

var=$(cat testfile.txt)
export DATADOG_API=$var

config.yaml:

...
exporters:
  datadog:
    api:
      key: ${env:DATADOG_API}
...

Steps to Reproduce

follow export steps above, or manually set api: '"my-api-key"' in the config.yaml

Expected Result

API key validates and exporter sends telemetry

Actual Result

api key validation starts but never completes. never fails either.

Collector version

v0.111.0 and onward

Environment information

Environment

OS: Ubuntu:noble
Compiler used linux/amd64 ubuntu package, collector contrib

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  batch:
connectors:
  datadog/connector:
exporters:
#  debug:
  datadog/exporter:
    #hostname: ${env:OTEL_COLLECTOR}
    api:
      key: ${env:DATADOG_API}
      site: datadoghq.com
#      fail_on_invalid_key: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [datadog/connector, datadog/exporter]
    metrics:
      receivers: [datadog/connector, otlp] 
      processors: [batch]
      exporters: [datadog/exporter]

Log output

2024-11-22T20:36:07.168Z	info	service@v0.111.0/service.go:136	Setting up own telemetry...
2024-11-22T20:36:07.172Z	info	telemetry/metrics.go:70	Serving metrics	{"address": "localhost:8888", "metrics level": "Normal"}
2024-11-22T20:36:37.345Z	info	provider/provider.go:71	Resolved source	{"kind": "exporter", "data_type": "metrics", "name": "datadog/exporter", "provider": "system", "source": {"Kind":"host","Identifier":"e4eac38e404a"}}
2024-11-22T20:36:37.353Z	info	clientutil/api.go:40	Validating API key.	{"kind": "exporter", "data_type": "metrics", "name": "datadog/exporter"}
2024-11-22T20:36:37.357Z	info	datadogexporter@v0.111.0/factory.go:359	Trace metrics are now disabled in the Datadog Exporter by default. To continue receiving Trace Metrics, configure the Datadog Connector or disable the feature gate.	{"kind": "exporter", "data_type": "traces", "name": "datadog/exporter", "documentation": "https://docs.datadoghq.com/opentelemetry/guide/migration/", "feature gate ID": "exporter.datadogexporter.DisableAPMStats"}
2024-11-22T20:36:37.358Z	info	datadogexporter@v0.111.0/zaplogger.go:38	Starting Agent with processor trace buffer of size 0	{"kind": "exporter", "data_type": "traces", "name": "datadog/exporter"}
2024-11-22T20:36:37.362Z	info	datadogexporter@v0.111.0/zaplogger.go:38	Receiver configured with 8 decoders and a timeout of 0ms	{"kind": "exporter", "data_type": "traces", "name": "datadog/exporter"}
2024-11-22T20:36:37.366Z	info	datadogexporter@v0.111.0/zaplogger.go:38	Trace writer initialized (climit=5 qsize=1 compression=gzip)	{"kind": "exporter", "data_type": "traces", "name": "datadog/exporter"}
2024-11-22T20:36:37.368Z	info	clientutil/api.go:40	Validating API key.	{"kind": "exporter", "data_type": "traces", "name": "datadog/exporter"}
2024-11-22T20:36:37.369Z	info	datadogexporter@v0.111.0/zaplogger.go:38	Processing Pipeline configured with 16 workers	{"kind": "exporter", "data_type": "traces", "name": "datadog/exporter"}
2024-11-22T20:36:37.373Z	info	datadogconnector@v0.111.0/factory.go:62	Datadog connector using the native OTel API to ingest OTel spans and produce APM stats. To revert to the legacy processing pipeline, disable the feature gate	{"kind": "connector", "name": "datadog/connector", "exporter_in_pipeline": "traces", "receiver_in_pipeline": "metrics", "feature gate": "connector.datadogconnector.NativeIngest"}
2024-11-22T20:36:37.373Z	info	datadogconnector@v0.111.0/connector_native.go:62	Building datadog connector for traces to metrics	{"kind": "connector", "name": "datadog/connector", "exporter_in_pipeline": "traces", "receiver_in_pipeline": "metrics"}
2024-11-22T20:36:37.381Z	info	service@v0.111.0/service.go:208	Starting otelcol-contrib...	{"Version": "0.111.0", "NumCPU": 16}
2024-11-22T20:36:37.381Z	info	extensions/extensions.go:39	Starting extensions...
2024-11-22T20:36:37.384Z	info	datadogconnector@v0.111.0/connector_native.go:91	Starting datadogconnector	{"kind": "connector", "name": "datadog/connector", "exporter_in_pipeline": "traces", "receiver_in_pipeline": "metrics"}
2024-11-22T20:36:37.385Z	warn	internal@v0.111.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-11-22T20:36:37.387Z	info	otlpreceiver@v0.111.0/otlp.go:112	Starting GRPC server	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4317"}
2024-11-22T20:36:37.387Z	warn	internal@v0.111.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks.	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-11-22T20:36:37.388Z	info	otlpreceiver@v0.111.0/otlp.go:169	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "metrics", "endpoint": "0.0.0.0:4318"}
2024-11-22T20:36:37.389Z	info	service@v0.111.0/service.go:234	Everything is ready. Begin running and processing data.

Additional context

something must have changed between v0.110.0 and v0.111.0+ on how environment variables are read into configuration maps.

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @dineshg13 @liustanley @songy23 @mackjmr @ankitpatel96 @jade-guiton-dd

See Adding Labels via Comments if you do not have permissions to add labels yourself.