aws.securityhub_findings: Add fields to _source as needed by CDR workflows.

[kcreddy]

Proposed commit message

Add cloud.provider, event.kind, and observer.vendor fields to
_source as needed by CDR workflows.

The commit here removed the fields from _source. But the fields are required to be
present in _source for Cloud Detection and Response (CDR) workflows. This PR reverts
the changes made in that commit and re-adds the fields into the ingest pipeline.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

cd packages/aws && elastic-package stack down && elastic-package build && elastic-package stack up --version=8.16.0-SNAPSHOT -d -v && eval "$(elastic-package stack shellinit)" && elastic-package test pipeline --generate -v --data-streams=securityhub_findings

--- Test results for package: aws - START ---
╭─────────┬──────────────────────┬───────────┬──────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM          │ TEST TYPE │ TEST NAME                                                │ RESULT │ TIME ELAPSED │
├─────────┼──────────────────────┼───────────┼──────────────────────────────────────────────────────────┼────────┼──────────────┤
│ aws     │ securityhub_findings │ pipeline  │ (ingest pipeline warnings test-securityhub-findings.log) │ PASS   │ 322.486291ms │
│ aws     │ securityhub_findings │ pipeline  │ test-securityhub-findings.log                            │ PASS   │ 274.104459ms │
╰─────────┴──────────────────────┴───────────┴──────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: aws - END   ---
Done

Related issues

• Relates https://github.com/elastic/security-team/issues/11020

Sample documents after the change:

1. Source index: _source-updated-document-destination-index.json
2. Destination index: _source-updated-document-source-index.json

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

@maxcold What is the reason that the CDR workflows depend on _source instead of fields? We should not need to sacrifice on storage efficiency.

[maxcold]

@andrewkroh in general there is quite a lot of code in Cloud Security that relies on the _source , eg. https://github.com/elastic/kibana/blob/main/x-pack/plugins/cloud_security_posture/public/pages/configurations/latest_findings/latest_findings_table.tsx#L55
I don't know if there is any specific reason for that or was it just smth that worked for us as we only relied on the data coming from cloudbeat and our native integration. For us to switch from this in our codebase won't be a small thing. I can bring it up with the team and have a github issue to track the effort of not relying directly on the _source

[andrewkroh]

I can bring it up with the team and have a github issue to track the effort of not relying directly on the _source

I think that would be the prudent course of action. We can add these fields into _source as a temporary workaround.

Another important reason to avoid _source is synthetic source, which is now enabled by default in Serverless as part of LogsDB and will become the default in Elasticsearch in the next major. Every query for _source incurs the penalty of synthesizing the _source value from the doc values.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 6661262

cc @kcreddy

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[efd6]

To provide context, the relevant PR and comment are here.

[efd6]

Approving as a work-around while the world is brought into order.

[elastic-vault-github-plugin-prod]

Package aws - 2.31.1 containing this change is available at https://epr.elastic.co/search?package=aws

[maxcold]

@andrewkroh here is the issue to track getting rid of _source in our queries:

• Do not rely on _source for cloud_security_posture plugin queries in Kibana kibana#198642