[Security Solution] Rule exception with a newline in comment throws a 500 on the _find api "Unable to load exception items"

[dhurley14]

Describe the bug:
After upgrading a deployment from any version below 8.16.0 to 8.16+, any rule exceptions with comments containing newline char \n will throw a 500 in the exception items _find api, like when trying to view the exceptions in the rule details page.

Importing a rule with an exception item containing a comment with a newline char \n will pass be imported succesfully, however fetching that item on the rule details page throws a 500 during response validation.

Creating an exception item with a comment containing a \n char through the UI will yield a 400 as it fails validation. That validation is happening here:

kibana/packages/kbn-openapi-common/schemas/primitives.gen.ts

Line 28 in 8487bc8

.regex(/^(?! *$).+$/);

The above is the same validation that causes the response validation to fail. Introduced here: #185865 and validation began in 8.16.0 when validating response was added in a separate PR here: https://github.com/elastic/kibana/pull/185951/files#diff-a17301a97f3e637c351f1d698c12976dc05185b2a4b15e45eb5784c00faf4468R85

Kibana/Elasticsearch Stack version:
8.16.0+

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):
Detection and Response

Steps to reproduce:

1. Have a deployment version < 8.16.0 and a rule with an exception item containing a comment with a newline char \n
2. upgrade deployment to 8.16.x or higher and try to load the exception items in the rules' details page.
3. 500 error

Current behavior:

Expected behavior:
Exception items should be visible

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[dhurley14]

Only known workaround is to export rules with exceptions that are failing, delete any newline chars and re-import them AND select the option on import to Overwrite existing exception lists with conflicting "list_id"

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[nicpenning]

I am experiencing this same issue as well. Upgraded from 8.15.2 - 8.16.1 and created an exception that had new line characters in it, then could no longer access any exceptions. Major issue!

https://elasticstack.slack.com/archives/CNRTGB9A4/p1732843176608059

[nicpenning]

From what I am seeing, this could have been broken before I made the exception.

[nicpenning]

@dhurley14 Could you please break this down on how to do this? I can't even list the exceptions without getting an error.

[paul-tavares]

@dhurley14 - FYI: two issues have now been reported from customers after upgrade to 8.16.1 - I was initially assigned to this one: https://github.com/elastic/sdh-security-team/issues/1122

[rylnd]

Known workarounds

For custom rules, as mentioned above, one can:

1. Export the rule (with its exceptions) via the UI
2. Modify the NDJSON file so that comments no longer contain newline characters
3. Re-import (while allowing overwriting of existing exception lists) the modified NDJSON file via the "Import Rules" UI

For prebuilt rules, which do not support exporting/importing via the UI, we can instead use the Exception list API to modify the exceptions themselves:

1. Fetch the exception list id(s) associated with the rule. We need id, list_id, and namespace_type in order to retrieve the exception list via the API. Luckily, all three are contained in the response of the Read Rule API.
   1. Find the id of the rule in question: it can be found in the URL of the rule's details page in the Kibana UI, e.g. app/security/rules/id/167a5f6f-2148-4792-8226-b5e7a58ef46e
   2. Use the id to fetch the rule's details via the Read Rule API:

      curl -H 'kbn-xsrf: true' -H 'elastic-api-version: 2023-10-31' http://localhost:5601/api/detection_engine/rules?id=167a5f6f-2148-4792-8226-b5e7a58ef46e -u elastic:changeme

   3. The JSON response will contain the appropriate identifiers within the exceptions_list key, e.g.:

      {
        "id": "167a5f6f-2148-4792-8226-b5e7a58ef46e",
        "exceptions_list": [
          {
            "id": "490525a2-eb66-4320-95b5-88bdd1302dc4",
            "list_id": "f75aae6f-0229-413f-881d-81cb3abfbe2d",
            "namespace_type": "single"
          }
        ]
      }

2. Retrieve the exception list via the export exceptions API
   1. Using the values obtained in 1.3 above, plug them into the following API call:

      curl -XPOST -u elastic:changeme -H 'kbn-xsrf: true' -H 'elastic-api-version: 2023-10-31' 'http://localhost:5601/api/exception_lists/_export?list_id=f75aae6f-0229-413f-881d-81cb3abfbe2d&id=490525a2-eb66-4320-95b5-88bdd1302dc4&namespace_type=single' -o list.ndjson

3. Modify the exception list NDJSON to ensure that no comments[].comment values contain newline (\n) characters.
4. Re-import the modified exception list via the Exceptions List UI (/app/security/exceptions)
   1. The initial import will fail due to the list already existing; after the initial failure, a "Overwrite the existing list" checkbox will appear. Check that box, and submit again. The list should now be updated, and the issue resolved.

[nicpenning]

Any timeline on a fix? Just wondering if it's worth waiting to upgrade instead of applying this work around.

[nicpenning]

Also, this should be listed as known issue in the docs.

[gils3nan]

We've been seeing the same error when it comes to exceptions. On Kibana 8.16.1.
Example of error:

Error
    at fetch_Fetch.fetchResponse (https://customer-y.kb.eu-west-2.aws.cloud.es.io/c8b46e87c4d6/bundles/core/core.entry.js:16:231834)
    at async https://customer-y.kb.eu-west-2.aws.cloud.es.io/c8b46e87c4d6/bundles/core/core.entry.js:16:229826
    at async https://customer-y.kb.eu-west-2.aws.cloud.es.io/c8b46e87c4d6/bundles/core/core.entry.js:16:229783

Any updates on fixes?

[craiglawson]

Any timeline on a fix? Just wondering if it's worth waiting to upgrade instead of applying this work around.

It has been indicated to us that the fix will be released at the end of December...

[nicpenning]

Yikes okay.

[nicpenning]

FYI

You can export the Endpoint Security rules from the exception lists. We hadn't tried this before but rather tried from the Endpoint Security Rule. The only difference it seems is that this is at the end of the UI exported list:

Then we used this PowerShell script to adjust the script:

$rules = Get-Content .\endpoint_list.ndjson | ConvertFrom-Json

#Zayn's Brilliance
foreach ($rule in $rules){foreach($comment in $rules.comments){$comment.comment = $comment.comment.replace("`n","|####WASANEWLINE####|")}}

$rules | ForEach-Object {
    $_ | ConvertTo-Json -Compress -Depth 25
} | Out-File list_modified.ndjson

Now we will see how it imports. We had over 200+ exceptions.

[nicpenning]

Confirmed this worked. Also, ours didn't have a special list ID it was endpoint_list

[MakoWish]

+1 Also seeing this and waiting for the fix.

[yomduf]

Should the fix be implemented in 8.16.2 or 8.17.0 as the tag suggest ?

[yctercero]

Thank you everyone for your patience with this one and apologize for the frustration it caused.

@yomduf it has been backported to 8.16 to be included in 8.16.2 and 8.17. Though with the timing, it will only make 8.17.1.