Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 3 #174168

Open
64 of 80 tasks
banderror opened this issue Jan 3, 2024 · 7 comments
Open
64 of 80 tasks
Labels
8.18 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Meta Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.14.0 v8.15.0 v8.16.0 v8.17.0 v8.18.0

Comments

@banderror
Copy link
Contributor

banderror commented Jan 3, 2024

Epic: https://github.com/elastic/security-team/issues/1974 (internal)
Milestones: <<>>

Status: In development.

Summary

Milestone 3: Add support for customizing prebuilt rules. Extend the rule upgrade workflow with full support for 3-way diffs and conflict resolution. Allow users to:

  • Edit and customize prebuilt rules
  • Export and import prebuilt rules, including customized ones
  • Upgrade prebuilt rules while keeping the user customizations whenever possible

User stories

Prebuilt rule customization workflow:

  • User can edit a single prebuilt rule
    • User can click "edit" button for prebuilt rules and customize (almost) any field on the Rule Editing page, just like it's possible to do with custom rules
    • User can't edit the Author and License fields
  • User can bulk edit multiple prebuilt rules via bulk actions
  • User can see if the rule is customized on the Rule Details page
  • User can see which rules are customized on the Rule Management page

Prebuilt rule upgrade workflow:

  • User can upgrade a single prebuilt rule to its latest version with previewing the incoming updates
    • User can preview updates from Elastic, for each rule field that has an update from Elastic
    • User can preview their customizations, for each rule field that was customized
    • User can compare their customizations with updates from Elastic and see if there are any conflicts between them, per each rule field
    • User can manually resolve conflicts between their customizations and updates from Elastic, per each rule field
    • User can edit the final field values before submitting the update
    • User can upgrade a rule if its type has been changed by Elastic in the latest version, but can only accept the incoming changes
  • User can upgrade a single prebuilt rule to its latest version without previewing the incoming updates
  • User can bulk upgrade multiple prebuilt rules to their latest versions

Prebuilt rule export/import workflow:

  • User can export a single prebuilt rule
    • Pages: Rule Details, Rule Management
    • It can be a prebuilt non-customized or prebuilt customized rule
  • User can bulk export multiple prebuilt rules via bulk actions
    • Pages: Rule Management
    • We support exporting prebuilt non-customized, prebuilt customized, and custom rules in any combination
  • User can bulk import multiple prebuilt rules
    • Pages: Rule Management
    • We support importing prebuilt non-customized, prebuilt customized, and custom rules - in any combination

Useful info

Design

Technical design

Preview Give feedback
  1. 8.14 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp
    jpdjere
  2. 8.14 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp discuss
    approksiu banderror
    jpdjere
  3. Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp discuss release_note:skip skip-ci
    banderror

UI/UX design

Preview Give feedback
  1. 8.14 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp design

Preparatory changes

Preparatory changes is something we can work on before starting to hide functionality behind a feature flag. This will reduce the overall complexity introduced by feature toggling.

Missing UI for editing certain rule fields

Preview Give feedback
  1. 8.14 candidate Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.14.0
    dplumlee
  2. 8.15 candidate Feature:Rule Creation Feature:Rule Details Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0
    ARWNightingale dplumlee
  3. 8.15 candidate Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0
    ARWNightingale maximpn
  4. 8.15 candidate Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0
    ARWNightingale nikitaindik

Missing UI for editing certain rule fields (docs)

Preview Give feedback
  1. 3 of 3
    Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.14.0
    joepeeples
  2. 3 of 3
    Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.15.0
    joepeeples
  3. 3 of 3
    Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.15.0
    joepeeples
  4. 3 of 3
    Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.15.0
    joepeeples

Schema-related changes

Preview Give feedback
  1. 8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0
    xcrzx
  2. 8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0
    xcrzx
  3. Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp backport:skip bug impact:critical release_note:skip v8.15.0
    xcrzx
  4. 8.15 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0
    xcrzx
  5. 8.15 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring v8.15.0
    jpdjere
  6. 8.16 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring v8.15.0 v8.16.0
    nikitaindik xcrzx
  7. 8.16 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    jpdjere

Rule customization, API changes

Preview Give feedback
  1. 8.16 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    xcrzx
  2. 8.16 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    xcrzx
  3. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    dplumlee
  4. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    dplumlee

Rule upgrade, API changes

Preview Give feedback
  1. 8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0
    dplumlee
  2. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    jpdjere
  3. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    jpdjere
  4. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring
    jpdjere
  5. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0
    jpdjere
  6. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    jpdjere
  7. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp
    jpdjere
  8. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    jpdjere

Rule upgrade, diff algorithms

Preview Give feedback
  1. 8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0
    dplumlee
  2. 8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0
    dplumlee
  3. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0
    dplumlee
  4. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0
    dplumlee
  5. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0
    dplumlee
  6. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0
    dplumlee
  7. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement
    dplumlee
  8. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0
    dplumlee

Fleet package with prebuilt rules

Preview Give feedback
  1. 3 of 3
    8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet v8.16.0
    xcrzx
  2. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet technical debt
    xcrzx
  3. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet technical debt
    xcrzx
  4. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet
    xcrzx
  5. 8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0
    xcrzx
  6. Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet
    kpollich
  7. 8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet performance v8.17.0
    xcrzx
  8. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.17.0 v8.18.0
    approksiu xcrzx

Changes hidden behind the feature flag

These are changes that will need to be hidden behind the prebuiltRulesCustomizationEnabled feature flag.

Rule customization, UI changes

Preview Give feedback
  1. 8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0
    nikitaindik
  2. 8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0
    nikitaindik
  3. 8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement
    dplumlee
  4. 8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp
    dplumlee
  5. 8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp
    dplumlee

Rule upgrade, UI changes

Preview Give feedback
  1. 8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs design
    ARWNightingale jpdjere
    xcrzx
  2. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 v8.17.0 v8.18.0
    maximpn nikitaindik
  3. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0
    maximpn nikitaindik
  4. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0
    xcrzx
  5. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement
    maximpn

Rule export and import, API and UI changes

Preview Give feedback
  1. 8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement
    rylnd
  2. 8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement
    rylnd
  3. 8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp
    dplumlee
  4. 8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp
    dplumlee
  5. 8.18 candidate Feature:Prebuilt Detection Rules Feature:Rule Import/Export Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0
    rylnd

Licensing

Preview Give feedback

Telemetry

Preview Give feedback
  1. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs product telemetry
    xcrzx

Before release

Bugs

Preview Give feedback
  1. 35 of 60
    8.18 candidate Feature:Prebuilt Detection Rules Meta Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 v8.17.0 v8.18.0
    banderror dplumlee
    jkelas jpdjere maximpn nikitaindik xcrzx

Testing

Preview Give feedback
  1. 8.18 candidate Feature:Prebuilt Detection Rules Feature:Rule Details Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test-coverage v8.18.0
    dplumlee nikitaindik
  2. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage test-plan v8.18.0
    dplumlee
  3. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage test-plan v8.18.0
    jpdjere maximpn
  4. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage test-plan v8.18.0
    dplumlee
  5. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0
    ARWNightingale approksiu
  6. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0
    MadameSheema pborgonovi

Documentation

Preview Give feedback
  1. v8.18.0
    nastasha-solomon
  2. v8.18.0
    joepeeples

Release

Preview Give feedback
  1. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0

After release

Last changes after releasing the feature

Preview Give feedback
  1. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0
    pborgonovi
  2. 8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp
@banderror banderror added Meta Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area labels Jan 3, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@banderror
Copy link
Contributor Author

banderror commented Feb 23, 2024

Draft plan for Milestone 3

UPD: the plan has been moved to the ticket description.

maximpn added a commit that referenced this issue Sep 27, 2024
…Update Workflow (#193531)

**Epic:** #174168
**Addresses:** #171520

## Summary

This PR introduces a new `Update` tab allowing users to resolve rule upgrade conflicts. It's a result of combination of read-only components implemented in #193261 and rule upgrade state implemented in #191721.

## Details

The goal of this PR is to provide intermediate integration between rule upgrade state ([PR](#191721)) and components displaying the diff and read-only state ([PR](#193261)). It will facilitate further development of rule field editable components and streamline rule upgrade functionality developing.

## How to test?

The functionality is hidden under `prebuiltRulesCustomizationEnabled` feature flag. Add the following to your Kibana config

```yaml
xpack.securitySolution.enableExperimental:
  - prebuiltRulesCustomizationEnabled
```

When the above feature flag enabled the new `Update` tab is displayed instead of the old one.

## Screenshots

Suggested components design 
![image](https://github.com/user-attachments/assets/b5aaf571-286a-4595-9bd4-fdaf9a423b03)

New `Update` tab
<img width="1718" alt="image" src="https://github.com/user-attachments/assets/28aa6bb3-f805-4109-a808-d67e58c7c5b8">
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Sep 27, 2024
…Update Workflow (elastic#193531)

**Epic:** elastic#174168
**Addresses:** elastic#171520

## Summary

This PR introduces a new `Update` tab allowing users to resolve rule upgrade conflicts. It's a result of combination of read-only components implemented in elastic#193261 and rule upgrade state implemented in elastic#191721.

## Details

The goal of this PR is to provide intermediate integration between rule upgrade state ([PR](elastic#191721)) and components displaying the diff and read-only state ([PR](elastic#193261)). It will facilitate further development of rule field editable components and streamline rule upgrade functionality developing.

## How to test?

The functionality is hidden under `prebuiltRulesCustomizationEnabled` feature flag. Add the following to your Kibana config

```yaml
xpack.securitySolution.enableExperimental:
  - prebuiltRulesCustomizationEnabled
```

When the above feature flag enabled the new `Update` tab is displayed instead of the old one.

## Screenshots

Suggested components design
![image](https://github.com/user-attachments/assets/b5aaf571-286a-4595-9bd4-fdaf9a423b03)

New `Update` tab
<img width="1718" alt="image" src="https://github.com/user-attachments/assets/28aa6bb3-f805-4109-a808-d67e58c7c5b8">

(cherry picked from commit 878ba13)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.18 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Meta Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.14.0 v8.15.0 v8.16.0 v8.17.0 v8.18.0
Projects
None yet
Development

No branches or pull requests

10 participants