The backend storage ES index could not be found[Bug]

[AiTian1211]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

OAP server (apache/skywalking)

What happened

The backend storage ES index could not be found
OAP version：9.7.0
configs:
SW_CORE_RECORD_DATA_TTL=15
SW_CORE_METRICS_DATA_TTL=15

What you expected to happen

The index data configuration retains data for 15 days. After 15 days, Elasticsearch creates a new index and clears the old index data. The indices sw-sys00008-cgpt-prod_metrics-all-20240623 and sw-sys00008-cgpt-prod_metrics-all-20240613 are created and the old index data is purged, but OAP (OpenAgent Platform or similar) is still requesting data from the old index. Shouldn't OAP normally use index aliases to query data instead of directly referencing the index names?

How to reproduce

Use OAP version 9.7.0, es version 7.16.2 and use the following configuration

Anything else

Specify the retention period for index data

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

Shouldn't OAP normally use index aliases to query data instead of directly referencing the index names?

AFAIK, no, we don't use that. Because in the large scale, it is slow.

I noticed you set TTL=15, and Day_step is large 10. I don't know whether this works. TTL and DAY STEP should not be so close. TTL should at least 3 times larger of DAY_STEP.

[AiTian1211]

Based on this configuration, I suggest you perform further testing.

[AiTian1211]

Another phenomenon that arises when multiple OAP instances share the same backend Elasticsearch (ES) cluster account and rely solely on "different index name prefixes" to differentiate between OAP services is the issue of "one OAP service querying another OAP service's indexes." For instance, if:

OAP-A is configured with SW_NAMESPACE=test,
OAP-B is configured with SW_NAMESPACE=prob,
occasionally, OAP-A may encounter errors when it inadvertently queries indexes prefixed with prob, which are intended for OAP-B.