Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented Generation (RAG) for Alerts_ Feature Flag #173809

Conversation

andrew-goldstein
Copy link
Contributor

@andrew-goldstein andrew-goldstein commented Dec 21, 2023

[Security Solution] [Elastic AI Assistant] Delete the Retrieval Augmented Generation (RAG) for Alerts Feature Flag

This PR deletes the assistantRagOnAlerts feature flag introduced in [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts #172542.

Deleting the assistantRagOnAlerts feature flag makes the Alerts toggle available in the assistant settings, per the screenshot below:

alerts_setting

This PR should not be merged until the docs describing the feature in elastic/security-docs#4456 have been merged.

This PR also includes @benironside improvements to the Alerts setting in the video below:

alerts_setting.mov

Desk testing

To desk test this change:

  1. Delete the following assistantRagOnAlerts feature flag from your local config/kibana.dev.yml:
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
  1. Start Kibana

  2. Generate alerts with a variety of severity (e.g. low, medium, high, and critical)

  3. Navigate to Security > Alerts

  4. Click the AI Assistant button to open the assistant

  5. Click the X button to clear the conversation

  6. Click the assistant's Settings gear

  7. Click the Knowledge Base category

Expected result

  • The Alerts toggle shown in the screenshot below is available

alerts_setting

  1. Click the Alerts toggle to enable the feature

  2. Click the Save button to close settings

  3. Enter the following prompt:

How many open alerts do I have?

Expected result

  • A response with alert counts grouped by workflow status will be returned, similar to the example below:
You currently have 48 open alerts in your system. These are categorized by severity as following: 19 of them are low severity, 16 are high severity, 12 are of medium severity and 1 is of critical severity. There is also 1 critical severity alert which is acknowledged.
  1. Enter the following prompt:
Which alerts should I look at first?

Expected result

A response with alert details, similar to the following is returned:

Based on the latest information, the alerts to prioritize first are those related to a mimikatz process starting on the hosts, which have a critical severity and the highest risk score of 99. There are also a series of alerts related to an EQL process sequence with a high severity and risk scores of 73. There is one alert about an Elastic Endpoint Security alert with a medium severity and risk score of 47.
  1. Once again, click the assistant's Settings gear

  2. Click the Knowledge Base category

  3. Click the Alerts toggle to disable the feature

  4. Click the Save button to close settings

  5. Once again, enter the following prompt:

How many open alerts do I have?

Expected result

  • The assistant does NOT respond with a breakdown of alerts by severity. Instead it replies with something like the following example response:
I'm sorry for any confusion, but as an AI, I don't have real-time access to your data or system to provide the number of your current open alerts. You can check your Elastic Security dashboard or use the appropriate querying commands to get the updated count of your open alerts.
  1. One more time, enter the following prompt:
Which alerts should I look at first?

Expected result

  • The assistant does NOT respond with alert details. Instead it replies with something like the following example response:
As an AI model, I don't have the capability to access real-time data. However, when it comes to managing alerts in Elastic Security, it's generally recommended to first look at the ones with the highest severity and risk score. Alerts related to malware, unauthorized access attempts, and abnormal data transfers or process activities, for example, may need immediate attention due to their potential high impact.

@andrew-goldstein andrew-goldstein added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting:Investigations Security Solution Investigations Team Feature:Security Assistant Security Assistant v8.12.0 v8.13.0 labels Dec 21, 2023
@andrew-goldstein andrew-goldstein requested a review from a team as a code owner December 21, 2023 08:40
@andrew-goldstein andrew-goldstein self-assigned this Dec 21, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

Copy link
Contributor

@stephmilovic stephmilovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, glad to see it go live 🎆 LGTM

@andrew-goldstein
Copy link
Contributor Author

Thanks to @benironside for the improvements to the Alerts setting in the video below:

alerts_setting.mov

…ugmented Generation (RAG) for Alerts_ Feature Flag

This PR deletes the `assistantRagOnAlerts` feature flag introduced in [[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts elastic#172542](elastic#172542).

Deleting the `assistantRagOnAlerts` feature flag makes the `Alerts` toggle available in the assistant settings, per the screenshot below:

![alerts_setting](https://github.com/elastic/kibana/assets/4459398/f8d20751-48e0-4d75-ac0b-66f63c10b7cb)

This PR should not be merged until the docs describing the feature in <elastic/security-docs#4456> have been merged.

### Desk testing

To desk test this change:

1) Delete the following `assistantRagOnAlerts` feature flag from your local `config/kibana.dev.yml`:

```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```

2) Start Kibana

3) Generate alerts with a variety of severity (e.g. `low`, `medium`, `high`, and `critical`)

4) Navigate to Security > Alerts

5) Click the `AI Assistant` button to open the assistant

6) Click the `X` button to clear the conversation

7) Click the assistant's `Settings` gear

8) Click the `Knowledge Base` category

**Expected result**

- The `Alerts` toggle shown in the screenshot below is available

![alerts_setting](https://github.com/elastic/kibana/assets/4459398/f8d20751-48e0-4d75-ac0b-66f63c10b7cb)

9) Click the `Alerts` toggle to enable the feature

10) Click the `Save` button to close settings

11) Enter the following prompt:

```
How many open alerts do I have?
```

**Expected result**

- A response with alert counts grouped by workflow status will be returned, similar to the example below:

```
You currently have 48 open alerts in your system. These are categorized by severity as following: 19 of them are low severity, 16 are high severity, 12 are of medium severity and 1 is of critical severity. There is also 1 critical severity alert which is acknowledged.
```

12) Enter the following prompt:

```
Which alerts should I look at first?
```

**Expected result**

A response with alert details, similar to the following is returned:

```
Based on the latest information, the alerts to prioritize first are those related to a mimikatz process starting on the hosts, which have a critical severity and the highest risk score of 99. There are also a series of alerts related to an EQL process sequence with a high severity and risk scores of 73. There is one alert about an Elastic Endpoint Security alert with a medium severity and risk score of 47.
```

13) Once again, click the assistant's `Settings` gear

14) Click the `Knowledge Base` category

15) Click the `Alerts` toggle to disable the feature

16) Click the `Save` button to close settings

17) Once again, enter the following prompt:

```
How many open alerts do I have?
```

**Expected result**

- The assistant does NOT respond with a breakdown of alerts by severity. Instead it replies with something like the following example response:

```
I'm sorry for any confusion, but as an AI, I don't have real-time access to your data or system to provide the number of your current open alerts. You can check your Elastic Security dashboard or use the appropriate querying commands to get the updated count of your open alerts.
```

18) One more time, enter the following prompt:

```
Which alerts should I look at first?
```

**Expected result**

- The assistant does NOT respond with alert details. Instead it replies with something like  the following example response:

```
As an AI model, I don't have the capability to access real-time data. However, when it comes to managing alerts in Elastic Security, it's generally recommended to first look at the ones with the highest severity and risk score. Alerts related to malware, unauthorized access attempts, and abnormal data transfers or process activities, for example, may need immediate attention due to their potential high impact.
```
@andrew-goldstein andrew-goldstein force-pushed the remove_rag_on_alerts_feature_flag branch from ada3a90 to 05ba636 Compare December 21, 2023 19:31
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 11.3MB 11.3MB -977.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
securitySolution 63.8KB 63.8KB -24.0B

History

  • 💚 Build #184715 succeeded ada3a90329dd9dae09b866f799ae45f6ccfda522

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @andrew-goldstein

@andrew-goldstein
Copy link
Contributor Author

✅ merging this PR because both security-docs PRs for this feature are available, and LGTM:

@andrew-goldstein andrew-goldstein merged commit ec05dd7 into elastic:main Dec 21, 2023
37 checks passed
@andrew-goldstein andrew-goldstein deleted the remove_rag_on_alerts_feature_flag branch December 21, 2023 23:01
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Dec 21, 2023
…ented Generation (RAG) for Alerts_ Feature Flag (elastic#173809)

## [Security Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented Generation (RAG) for Alerts_ Feature Flag

This PR deletes the `assistantRagOnAlerts` feature flag introduced in [[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts elastic#172542](elastic#172542).

Deleting the `assistantRagOnAlerts` feature flag makes the `Alerts` toggle available in the assistant settings, per the screenshot below:

![alerts_setting](https://github.com/elastic/kibana/assets/4459398/1647a92c-653b-49de-926a-d0a3b65d270a)

This PR should not be merged until the docs describing the feature in <elastic/security-docs#4456> have been merged.

This PR also includes @benironside improvements to the Alerts setting in the video below:

https://github.com/elastic/kibana/assets/4459398/73ea2717-ad2a-4998-afe2-cc154d8d19a9

### Desk testing

To desk test this change:

1) Delete the following `assistantRagOnAlerts` feature flag from your local `config/kibana.dev.yml`:

```
xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']
```

2) Start Kibana

3) Generate alerts with a variety of severity (e.g. `low`, `medium`, `high`, and `critical`)

4) Navigate to Security > Alerts

5) Click the `AI Assistant` button to open the assistant

6) Click the `X` button to clear the conversation

7) Click the assistant's `Settings` gear

8) Click the `Knowledge Base` category

**Expected result**

- The `Alerts` toggle shown in the screenshot below is available

![alerts_setting](https://github.com/elastic/kibana/assets/4459398/1647a92c-653b-49de-926a-d0a3b65d270a)

9) Click the `Alerts` toggle to enable the feature

10) Click the `Save` button to close settings

11) Enter the following prompt:

```
How many open alerts do I have?
```

**Expected result**

- A response with alert counts grouped by workflow status will be returned, similar to the example below:

```
You currently have 48 open alerts in your system. These are categorized by severity as following: 19 of them are low severity, 16 are high severity, 12 are of medium severity and 1 is of critical severity. There is also 1 critical severity alert which is acknowledged.
```

12) Enter the following prompt:

```
Which alerts should I look at first?
```

**Expected result**

A response with alert details, similar to the following is returned:

```
Based on the latest information, the alerts to prioritize first are those related to a mimikatz process starting on the hosts, which have a critical severity and the highest risk score of 99. There are also a series of alerts related to an EQL process sequence with a high severity and risk scores of 73. There is one alert about an Elastic Endpoint Security alert with a medium severity and risk score of 47.
```

13) Once again, click the assistant's `Settings` gear

14) Click the `Knowledge Base` category

15) Click the `Alerts` toggle to disable the feature

16) Click the `Save` button to close settings

17) Once again, enter the following prompt:

```
How many open alerts do I have?
```

**Expected result**

- The assistant does NOT respond with a breakdown of alerts by severity. Instead it replies with something like the following example response:

```
I'm sorry for any confusion, but as an AI, I don't have real-time access to your data or system to provide the number of your current open alerts. You can check your Elastic Security dashboard or use the appropriate querying commands to get the updated count of your open alerts.
```

18) One more time, enter the following prompt:

```
Which alerts should I look at first?
```

**Expected result**

- The assistant does NOT respond with alert details. Instead it replies with something like  the following example response:

```
As an AI model, I don't have the capability to access real-time data. However, when it comes to managing alerts in Elastic Security, it's generally recommended to first look at the ones with the highest severity and risk score. Alerts related to malware, unauthorized access attempts, and abnormal data transfers or process activities, for example, may need immediate attention due to their potential high impact.
```

(cherry picked from commit ec05dd7)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.12

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Dec 22, 2023
…al Augmented Generation (RAG) for Alerts_ Feature Flag (#173809) (#173883)

# Backport

This will backport the following commits from `main` to `8.12`:
- [[Security Solution] [Elastic AI Assistant] Delete the _Retrieval
Augmented Generation (RAG) for Alerts_ Feature Flag
(#173809)](#173809)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Andrew
Macri","email":"andrew.macri@elastic.co"},"sourceCommit":{"committedDate":"2023-12-21T23:01:15Z","message":"[Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag (#173809)\n\n## [Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag\r\n\r\nThis PR deletes the
`assistantRagOnAlerts` feature flag introduced in [[Security Solution]
[Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
#172542](https://github.com/elastic/kibana/pull/172542).\r\n\r\nDeleting
the `assistantRagOnAlerts` feature flag makes the `Alerts` toggle
available in the assistant settings, per the screenshot
below:\r\n\r\n![alerts_setting](https://github.com/elastic/kibana/assets/4459398/1647a92c-653b-49de-926a-d0a3b65d270a)\r\n\r\nThis
PR should not be merged until the docs describing the feature in
<elastic/security-docs#4456> have been
merged.\r\n\r\nThis PR also includes @benironside improvements to the
Alerts setting in the video
below:\r\n\r\nhttps://github.com/elastic/kibana/assets/4459398/73ea2717-ad2a-4998-afe2-cc154d8d19a9\r\n\r\n###
Desk testing\r\n\r\nTo desk test this change:\r\n\r\n1) Delete the
following `assistantRagOnAlerts` feature flag from your local
`config/kibana.dev.yml`:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental:
['assistantRagOnAlerts']\r\n```\r\n\r\n2) Start Kibana\r\n\r\n3)
Generate alerts with a variety of severity (e.g. `low`, `medium`,
`high`, and `critical`)\r\n\r\n4) Navigate to Security >
Alerts\r\n\r\n5) Click the `AI Assistant` button to open the
assistant\r\n\r\n6) Click the `X` button to clear the
conversation\r\n\r\n7) Click the assistant's `Settings` gear\r\n\r\n8)
Click the `Knowledge Base` category\r\n\r\n**Expected result**\r\n\r\n-
The `Alerts` toggle shown in the screenshot below is
available\r\n\r\n![alerts_setting](https://github.com/elastic/kibana/assets/4459398/1647a92c-653b-49de-926a-d0a3b65d270a)\r\n\r\n9)
Click the `Alerts` toggle to enable the feature\r\n\r\n10) Click the
`Save` button to close settings\r\n\r\n11) Enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- A response with alert
counts grouped by workflow status will be returned, similar to the
example below:\r\n\r\n```\r\nYou currently have 48 open alerts in your
system. These are categorized by severity as following: 19 of them are
low severity, 16 are high severity, 12 are of medium severity and 1 is
of critical severity. There is also 1 critical severity alert which is
acknowledged.\r\n```\r\n\r\n12) Enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\nA response with alert
details, similar to the following is returned:\r\n\r\n```\r\nBased on
the latest information, the alerts to prioritize first are those related
to a mimikatz process starting on the hosts, which have a critical
severity and the highest risk score of 99. There are also a series of
alerts related to an EQL process sequence with a high severity and risk
scores of 73. There is one alert about an Elastic Endpoint Security
alert with a medium severity and risk score of 47.\r\n```\r\n\r\n13)
Once again, click the assistant's `Settings` gear\r\n\r\n14) Click the
`Knowledge Base` category\r\n\r\n15) Click the `Alerts` toggle to
disable the feature\r\n\r\n16) Click the `Save` button to close
settings\r\n\r\n17) Once again, enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with a breakdown of alerts by severity. Instead it replies with
something like the following example response:\r\n\r\n```\r\nI'm sorry
for any confusion, but as an AI, I don't have real-time access to your
data or system to provide the number of your current open alerts. You
can check your Elastic Security dashboard or use the appropriate
querying commands to get the updated count of your open
alerts.\r\n```\r\n\r\n18) One more time, enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with alert details. Instead it replies with something like the
following example response:\r\n\r\n```\r\nAs an AI model, I don't have
the capability to access real-time data. However, when it comes to
managing alerts in Elastic Security, it's generally recommended to first
look at the ones with the highest severity and risk score. Alerts
related to malware, unauthorized access attempts, and abnormal data
transfers or process activities, for example, may need immediate
attention due to their potential high
impact.\r\n```","sha":"ec05dd7afddaef353d27f0bcbc7046ff09c0a5d6","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","Team:Threat Hunting:Investigations","Feature:Elastic
AI
Assistant","v8.12.0","v8.13.0"],"number":173809,"url":"https://github.com/elastic/kibana/pull/173809","mergeCommit":{"message":"[Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag (#173809)\n\n## [Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag\r\n\r\nThis PR deletes the
`assistantRagOnAlerts` feature flag introduced in [[Security Solution]
[Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
#172542](https://github.com/elastic/kibana/pull/172542).\r\n\r\nDeleting
the `assistantRagOnAlerts` feature flag makes the `Alerts` toggle
available in the assistant settings, per the screenshot
below:\r\n\r\n![alerts_setting](https://github.com/elastic/kibana/assets/4459398/1647a92c-653b-49de-926a-d0a3b65d270a)\r\n\r\nThis
PR should not be merged until the docs describing the feature in
<elastic/security-docs#4456> have been
merged.\r\n\r\nThis PR also includes @benironside improvements to the
Alerts setting in the video
below:\r\n\r\nhttps://github.com/elastic/kibana/assets/4459398/73ea2717-ad2a-4998-afe2-cc154d8d19a9\r\n\r\n###
Desk testing\r\n\r\nTo desk test this change:\r\n\r\n1) Delete the
following `assistantRagOnAlerts` feature flag from your local
`config/kibana.dev.yml`:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental:
['assistantRagOnAlerts']\r\n```\r\n\r\n2) Start Kibana\r\n\r\n3)
Generate alerts with a variety of severity (e.g. `low`, `medium`,
`high`, and `critical`)\r\n\r\n4) Navigate to Security >
Alerts\r\n\r\n5) Click the `AI Assistant` button to open the
assistant\r\n\r\n6) Click the `X` button to clear the
conversation\r\n\r\n7) Click the assistant's `Settings` gear\r\n\r\n8)
Click the `Knowledge Base` category\r\n\r\n**Expected result**\r\n\r\n-
The `Alerts` toggle shown in the screenshot below is
available\r\n\r\n![alerts_setting](https://github.com/elastic/kibana/assets/4459398/1647a92c-653b-49de-926a-d0a3b65d270a)\r\n\r\n9)
Click the `Alerts` toggle to enable the feature\r\n\r\n10) Click the
`Save` button to close settings\r\n\r\n11) Enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- A response with alert
counts grouped by workflow status will be returned, similar to the
example below:\r\n\r\n```\r\nYou currently have 48 open alerts in your
system. These are categorized by severity as following: 19 of them are
low severity, 16 are high severity, 12 are of medium severity and 1 is
of critical severity. There is also 1 critical severity alert which is
acknowledged.\r\n```\r\n\r\n12) Enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\nA response with alert
details, similar to the following is returned:\r\n\r\n```\r\nBased on
the latest information, the alerts to prioritize first are those related
to a mimikatz process starting on the hosts, which have a critical
severity and the highest risk score of 99. There are also a series of
alerts related to an EQL process sequence with a high severity and risk
scores of 73. There is one alert about an Elastic Endpoint Security
alert with a medium severity and risk score of 47.\r\n```\r\n\r\n13)
Once again, click the assistant's `Settings` gear\r\n\r\n14) Click the
`Knowledge Base` category\r\n\r\n15) Click the `Alerts` toggle to
disable the feature\r\n\r\n16) Click the `Save` button to close
settings\r\n\r\n17) Once again, enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with a breakdown of alerts by severity. Instead it replies with
something like the following example response:\r\n\r\n```\r\nI'm sorry
for any confusion, but as an AI, I don't have real-time access to your
data or system to provide the number of your current open alerts. You
can check your Elastic Security dashboard or use the appropriate
querying commands to get the updated count of your open
alerts.\r\n```\r\n\r\n18) One more time, enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with alert details. Instead it replies with something like the
following example response:\r\n\r\n```\r\nAs an AI model, I don't have
the capability to access real-time data. However, when it comes to
managing alerts in Elastic Security, it's generally recommended to first
look at the ones with the highest severity and risk score. Alerts
related to malware, unauthorized access attempts, and abnormal data
transfers or process activities, for example, may need immediate
attention due to their potential high
impact.\r\n```","sha":"ec05dd7afddaef353d27f0bcbc7046ff09c0a5d6"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/173809","number":173809,"mergeCommit":{"message":"[Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag (#173809)\n\n## [Security
Solution] [Elastic AI Assistant] Delete the _Retrieval Augmented
Generation (RAG) for Alerts_ Feature Flag\r\n\r\nThis PR deletes the
`assistantRagOnAlerts` feature flag introduced in [[Security Solution]
[Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts
#172542](https://github.com/elastic/kibana/pull/172542).\r\n\r\nDeleting
the `assistantRagOnAlerts` feature flag makes the `Alerts` toggle
available in the assistant settings, per the screenshot
below:\r\n\r\n![alerts_setting](https://github.com/elastic/kibana/assets/4459398/1647a92c-653b-49de-926a-d0a3b65d270a)\r\n\r\nThis
PR should not be merged until the docs describing the feature in
<elastic/security-docs#4456> have been
merged.\r\n\r\nThis PR also includes @benironside improvements to the
Alerts setting in the video
below:\r\n\r\nhttps://github.com/elastic/kibana/assets/4459398/73ea2717-ad2a-4998-afe2-cc154d8d19a9\r\n\r\n###
Desk testing\r\n\r\nTo desk test this change:\r\n\r\n1) Delete the
following `assistantRagOnAlerts` feature flag from your local
`config/kibana.dev.yml`:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental:
['assistantRagOnAlerts']\r\n```\r\n\r\n2) Start Kibana\r\n\r\n3)
Generate alerts with a variety of severity (e.g. `low`, `medium`,
`high`, and `critical`)\r\n\r\n4) Navigate to Security >
Alerts\r\n\r\n5) Click the `AI Assistant` button to open the
assistant\r\n\r\n6) Click the `X` button to clear the
conversation\r\n\r\n7) Click the assistant's `Settings` gear\r\n\r\n8)
Click the `Knowledge Base` category\r\n\r\n**Expected result**\r\n\r\n-
The `Alerts` toggle shown in the screenshot below is
available\r\n\r\n![alerts_setting](https://github.com/elastic/kibana/assets/4459398/1647a92c-653b-49de-926a-d0a3b65d270a)\r\n\r\n9)
Click the `Alerts` toggle to enable the feature\r\n\r\n10) Click the
`Save` button to close settings\r\n\r\n11) Enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- A response with alert
counts grouped by workflow status will be returned, similar to the
example below:\r\n\r\n```\r\nYou currently have 48 open alerts in your
system. These are categorized by severity as following: 19 of them are
low severity, 16 are high severity, 12 are of medium severity and 1 is
of critical severity. There is also 1 critical severity alert which is
acknowledged.\r\n```\r\n\r\n12) Enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\nA response with alert
details, similar to the following is returned:\r\n\r\n```\r\nBased on
the latest information, the alerts to prioritize first are those related
to a mimikatz process starting on the hosts, which have a critical
severity and the highest risk score of 99. There are also a series of
alerts related to an EQL process sequence with a high severity and risk
scores of 73. There is one alert about an Elastic Endpoint Security
alert with a medium severity and risk score of 47.\r\n```\r\n\r\n13)
Once again, click the assistant's `Settings` gear\r\n\r\n14) Click the
`Knowledge Base` category\r\n\r\n15) Click the `Alerts` toggle to
disable the feature\r\n\r\n16) Click the `Save` button to close
settings\r\n\r\n17) Once again, enter the following
prompt:\r\n\r\n```\r\nHow many open alerts do I
have?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with a breakdown of alerts by severity. Instead it replies with
something like the following example response:\r\n\r\n```\r\nI'm sorry
for any confusion, but as an AI, I don't have real-time access to your
data or system to provide the number of your current open alerts. You
can check your Elastic Security dashboard or use the appropriate
querying commands to get the updated count of your open
alerts.\r\n```\r\n\r\n18) One more time, enter the following
prompt:\r\n\r\n```\r\nWhich alerts should I look at
first?\r\n```\r\n\r\n**Expected result**\r\n\r\n- The assistant does NOT
respond with alert details. Instead it replies with something like the
following example response:\r\n\r\n```\r\nAs an AI model, I don't have
the capability to access real-time data. However, when it comes to
managing alerts in Elastic Security, it's generally recommended to first
look at the ones with the highest severity and risk score. Alerts
related to malware, unauthorized access attempts, and abnormal data
transfers or process activities, for example, may need immediate
attention due to their potential high
impact.\r\n```","sha":"ec05dd7afddaef353d27f0bcbc7046ff09c0a5d6"}}]}]
BACKPORT-->

Co-authored-by: Andrew Macri <andrew.macri@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Security Assistant Security Assistant release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting:Investigations Security Solution Investigations Team v8.12.0 v8.13.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants