[Cloud Security][Telemetry][Discussion] Cloudbeat indexing Policy ID in order to track account feature adoption

[Omolola-Akinleye]

Currently, we don’t have the telemetry ability to track feature adoption by account. As of today, the onboarding features such as cloud formation, cloud shell, and single vs organization account are only configured in a package policy.

From the findings index, we can track cloud environment ids( cloud.account.id or orchestrator.cluster.id) and map rule.benchmark.id to CIS_GCP, CIS_AWS, CIS_EKS, CIS_K8S.

• We define how many customers have CSPM installed on GCP cloud accounts
• We define how many customers have CSPM installed on aws cloud accounts
• We define how many customers have KSPM installed on EKS clusters
• We define how many customers have KSPM installed on vanilla clusters

However, since accounts are not connected to the package, we can't capture the following metrics:

• how many GCP or AWS accounts used an automatic installation?
• how many AWS accounts are used in an organization account vs a single account?

Problem:
The findings and vulnerabilities index lacks an efficient connection point to retrieve package policy internal configuration. In the findings/vulnerabilities index. The agent.id is indexed; however, for one agent.id will entail creating multiple API requests inducing a bottleneck of network requests.

Solution:

• Cloudbeat could index agent policy id per finding or vulnerability. This will effectively connect accounts to package policy.
• Once we have the policy_id, we can send A request to capture package policy feature configurations such as
  • Manual vs automatic
  • Single vs Organization account
  • Any other onboarding feature implementations

Below, I have an 8.10 deployment, where we can run tests to investigate the indexes in the current telemetry and also API services such as package policy and agent policy.

1. Sign in with credentials
   Here is an Kibana Env to test runtime mappings per solution and data sources.
   Username: elastic
   Password
2. Go to Dev Tools
3. Run the following

POST kbn:/api/telemetry/v2/clusters/_stats
{

  "unencrypted": true
}

GET kbn:/api/fleet/package_policies

GET kbn:/api/fleet/agent_policies?full=true

// kspm accounts
POST /logs-cloud_security_posture.findings_latest-default/_search
{
  "size": 1,
  "_source": false,
  "query": {
    "match_all": {}
  },
  "runtime_mappings": {
    "asset_identifier": {
      "type": "keyword",
      "script": {
        "source": """
         def orchestratorIdAvailable = doc.containsKey("orchestrator.cluster.id") &&
          !doc["orchestrator.cluster.id"].empty;
        def clusterIdAvailable = doc.containsKey("cluster_id") &&
          !doc["cluster_id"].empty;

        if (orchestratorIdAvailable) {
          emit(doc["orchestrator.cluster.id"].value);
        } else if (clusterIdAvailable) {
          emit(doc["cluster_id"].value);
        }
        """
      }
    }
  },
  "aggs": {
    "accounts": {
      "terms": {
        "field": "asset_identifier",
        "order": {
          "_count": "desc"
        },
        "size": 100
      },
      "aggs": {
        "latest_timestamp": {
          "top_metrics": {
            "metrics": {
              "field": "@timestamp"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        },
        "nodes_count": {
          "cardinality": {
            "field": "host.name"
          }
        },
        "agents_count": {
          "cardinality": {
            "field": "agent.id"
          }
        },
      
        "benchmark_version": {
          "top_metrics": {
            "metrics": {
              "field": "rule.benchmark.version"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        },
        "benchmark_name": {
          "top_metrics": {
            "metrics": {
              "field": "rule.benchmark.name"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        },
        "kubernetes_version": {
          "top_metrics": {
            "metrics": {
              "field": "cloudbeat.kubernetes.version"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        },
        "passed_findings_count": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "term": {
                          "result.evaluation": "passed"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                }
              ]
            }
          }
        },
        "failed_findings_count": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "term": {
                          "result.evaluation": "failed"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                }
              ]
            }
          }
        },
        "resources": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "term": {
                          "resource.sub_type": "Pod"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                }
              ]
            }
          },
          "aggs": {
            "pods_count": {
              "cardinality": {
                "field": "resource.id"
              }
            }
          }
        },
        "cloud_provider": {
          "top_metrics": {
            "metrics": {
              "field": "cloud.provider"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        }
      }
    }
  }
}

// cspm accounts
POST /logs-cloud_security_posture.findings_latest-default/_search
{
  "size": 1,
  "_source": false,
  "query": {
    "match_all": {}
  },
  "runtime_mappings": {
    "asset_identifier": {
      "type": "keyword",
      "script": {
        "source": """def cloudAccountIdAvailable = doc.containsKey("cloud.account.id") &&
          !doc["cloud.account.id"].empty;
        if (cloudAccountIdAvailable) {
          emit(doc["cloud.account.id"].value);
        }"""
      }
    }
  },
  "aggs": {
    "accounts": {
      "terms": {
        "field": "asset_identifier",
        "order": {
          "_count": "desc"
        },
        "size": 100
      },
      "aggs": {
        "latest_timestamp": {
          "top_metrics": {
            "metrics": {
              "field": "@timestamp"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        },
        "benchmark_version": {
          "top_metrics": {
            "metrics": {
              "field": "rule.benchmark.version"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        },
        "benchmark_name": {
          "top_metrics": {
            "metrics": {
              "field": "rule.benchmark.name"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        },
        "passed_findings_count": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "term": {
                          "result.evaluation": "passed"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                }
              ]
            }
          }
        },
        "failed_findings_count": {
          "filter": {
            "bool": {
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "term": {
                          "result.evaluation": "failed"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                }
              ]
            }
          }
        },
        "cloud_provider": {
          "top_metrics": {
            "metrics": {
              "field": "cloud.provider"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        }
      }
    }
  }
}

// cnvm accounts 
POST /logs-cloud_security_posture.vulnerabilities_latest-default/_search
{
  "size": 1,
  "_source": false,
  "query": {
    "match_all": {}
  },
  "runtime_mappings": {
    "asset_identifier": {
      "type": "keyword",
      "script": {
        "source": """def cloudAccountIdAvailable = doc.containsKey("cloud.account.id") &&
          !doc["cloud.account.id"].empty;
        if (cloudAccountIdAvailable) {
          emit(doc["cloud.account.id"].value);
        }"""
      }
    }
  },
  "aggs": {
    "accounts": {
      "terms": {
        "field": "asset_identifier",
        "order": {
          "_count": "desc"
        },
        "size": 100
      },
      "aggs": {
         "latest_timestamp": {
          "top_metrics": {
            "metrics": {
              "field": "@timestamp"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        },
         "cloud_provider": {
          "top_metrics": {
            "metrics": {
              "field": "cloud.provider"
            },
            "size": 1,
            "sort": {
              "@timestamp": "desc"
            }
          }
        }
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[CohenIdo]

Hey @Omolola-Akinleye, thanks for this great summary.

Another option to achieve the same can achieved if cloudbeat will send the relevant data in dedicated fields. ie: installation_type: manual / automatic, account_type: single / manual etc.
On the one hand, it will be repetitive on all findings, on the other hand, the alternative will require us to do these steps for every account:

• get the package policy by the package policy id
• parse the package policy
• fetch the relevant data.

[Omolola-Akinleye]

Hey @CohenIdo, I like the idea of storing the relevant feature fields. This approach would be certainly easier in extracting the telemetry data. One challenge might be overhead as features are being developed per sprint we would have to coordinate with the Cloudbeat team to store the relevant fields for feature adoption in releases.

I like the first approach. How should we define and coordinate that process with Cloudbeat?

Also, if an overhead of tracking feature adoption fields, another solution we could get the package policy info via bulk package policy API - POST /package_policies/_bulk_get. So for all accounts:

• Get a list of package policy IDs
• bulk fetch the package policies
• Parse package policies
• Apply relevant data to accounts

There is still some logic we would have to apply here with this solution.

[CohenIdo]

@Omolola-Akinleye and @kfirpeled, I am summarizing the decisions we made:

1. We would like cloudbeat to attach the package policy id to the findings (and not the agent id).
2. For coupling between installation params (i.e.: cloud provider, single/multi account etc. ) we will merge the installation stats data with the accounts stats data using the common package policy id.

@Omolola-Akinleye, can you confirm if CloudBeat has a task to attach the package policy ID to the findings? If so, we can close this issue.

[Omolola-Akinleye]

@CohenIdo @kfirpeled

2. For coupling between installation params (i.e.: cloud provider, single/multi account etc. ) we will merge the installation stats data with the accounts stats data using the common package policy id.

We want to merge account stats and installation stats in a single table in Big Query. Correct?

Confirmed. I created the cloudbeat package policy id ticket

[kfirpeled]

Sounds ok,
But I would like you to confirm first we don't have other use-cases that lean on these tables as standalone tables.
Before making that merge.