[BUG][opensearch] cant connect to metrics endpoint

[gabriel-suela]

Describe the bug
Im trying to see the metrics for opensearch but it seems that it isn't working.

To Reproduce

serviceMonitor:
  enabled: true

Expected behavior
See all metrics

Chart Name
Specify the Chart which is affected?

Screenshots

Host/Environment (please complete the following information):

• Helm Version: 3.15.4
• Kubernetes Version: 1.29.6

Additional context
Also, prometheus is alerting that the metrics are unreachable

[BernhardGruen]

Same problem here. Thank you for your report.

[eyenx]

TBH this looks more like a configuration issue. I have the same problem. Looking into it, probably opensearch is either missing a configuration option or even the prometheus-exporter-plugin

[eyenx]

Looking at what port is by default configured (9600) I'd think the idea is to use the performance analyzer which is already included and installed as plugin on the opensearch container, but not enabled for what I can tell
https://opensearch.org/docs/latest/monitoring-your-cluster/pa/index/

[eyenx]

Okay, when trying to start it inside the container I just see that the address is already bound but not reachable.

I'm gonna continue troubleshoot this.

sh-5.2$ OPENSEARCH_HOME=/usr/share/opensearch/ OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk/ OPENSEARCH_PATH_CONF=/usr/share/opensearch/config/ bin/opensearch-performance-analyzer/performance-analyzer-agent-cli &
[1] 1181
sh-5.2$ 09:05:50.833 [main] ERROR org.opensearch.performanceanalyzer.PerformanceAnalyzerWebServer - Could not create HttpServer on port 9600
java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method) ~[?:?]
        at java.base/sun.nio.ch.Net.bind(Net.java:565) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:344) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:301) ~[?:?]
        at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocke
        ```

[eyenx]

According to the initial PR, the SM is meant to be used in conjunction with https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch

I'll try to integrate it and make sure it's installed by default when enabling serviceMonitor

Reference: #537 (comment)

[gabriel-suela]

According to the initial PR, the SM is meant to be used in conjunction with https://github.com/Aiven-Open/prometheus-exporter-plugin-for-opensearch

I'll try to integrate it and make sure it's installed by default when enabling serviceMonitor

Reference: #537 (comment)

with the plugin i managed to access the metrics on port 9200, but the ServiceMonitor manifest still have a port pointing to metrics port name:

 spec:                                                                                                                                                                                                                                                                    │
│   endpoints:                                                                                                                                                                                                                                                             │
│   - interval: 10s                                                                                                                                                                                                                                                        │
│     path: /_prometheus/metrics                                                                                                                                                                                                                                           │
│     port: metrics    

i believe that is why prometheus keeps warning me that the opensearch metrics are unreachable even so i can access it through port forward.

[eyenx]

You can configure the port with metricsPort