Do not rely on _source for cloud_security_posture plugin queries in Kibana

[maxcold]

Motivation

While working on the AWS Security Hub integration Service Integration team pointed out a problem with the cloud_security_posture plugin relying on querying _source in Kibana, eg. for data grid queries. It has at least two consequences:

• in integrations, it's common to remove constant_keyword fields from the _source to optimize storage. As a result we don't have this data in our data gird. It happened with observer.vendor for example.
• in serverless and in ESS 9.0 querying _source has a performance penalty due to the need to recreate the _source from the fields.

We need to stop relying on the _source field for queries and use the fields directly

Definition of done

• _source field is not queried in Kibana plugin cloud_security_posture

Out of scope

Related tasks/epics

• Syntethic _source docs https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-source-field.html#synthetic-source
• https://github.com/elastic/security-team/issues/10427
• https://github.com/elastic/kibana-team/issues/1174
• https://github.com/elastic/security-team/issues/11020

Team tag

@elastic/kibana-cloud-security-posture

[maxcold]

More context on why this is still important even if not required for 9.0 in here https://github.com/elastic/security-team/issues/10427#issuecomment-2468680312

[andrewkroh]

@maxcold FYI, the workaround that we did in the AWS integration (elastic/integrations#11608 (comment)) to always incorporate the constant_keyword value into _source will not solve the problem under LogsDB. This is because in synthesized _source, constant_keyword fields are never incorporated into the value. The values can only be accessed through fields. Relates to elastic/elasticsearch#117182.