[Data Views] Disable scripted field creation in the Data Views manage…

…ment page (elastic#202250)

## Summary

Scripted fields in data views have been deprecated since 7.13, and
superseded by runtime fields.

The ability to create new scripted fields has been removed from the Data
Views management page in 9.0. Existing scripted fields can still be
edited or deleted, and the creation UI can be accessed by navigating
directly to
`/app/management/kibana/dataViews/dataView/{dataViewId}/create-field`,
but it is recommended to migrate to runtime fields or ES|QL instead to
prepare for removal.

Additionally, the scripted fields entry in the Upgrade Assistant has
been updated to reflect these changes, improve migration instructions,
and surface the full list of data views across all spaces that contain
scripted fields as well as their associated spaces. New documentation
has been added to the "Manage data views" page with examples and
instructions to help users migrate off scripted fields to runtime fields
or ES|QL, which is also linked to from the Upgrade Assistant entry.

Data Views management page:

![management](https://github.com/user-attachments/assets/ed42310c-d7fa-48ba-8430-533e9230b48d)

Upgrade Assistant:
<img
src="https://github.com/user-attachments/assets/4ff27866-4fe8-4357-82e8-cd4c503ab50b"
width="500" />

Resolves elastic#182067.

### Checklist

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [x] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

---------

Co-authored-by: Matt Kime <matt@mattki.me>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

docs/management/manage-data-views.asciidoc

@@ -170,7 +170,7 @@ Edit the settings for runtime fields, or remove runtime fields from data views.
 [[scripted-fields]]
 === Add scripted fields to data views
 
-deprecated::[7.13,Use {ref}/runtime.html[runtime fields] instead of scripted fields. Runtime fields support Painless scripts and provide greater flexibility.]
+deprecated::[7.13,Use {ref}/runtime.html[runtime fields] instead of scripted fields. Runtime fields support Painless scripting and provide greater flexibility. You can also use the {ref}/esql.html[Elasticsearch Query Language (ES|QL)] to compute values directly at query time.]
 
 Scripted fields compute data on the fly from the data in your {es} indices. The data is shown on
 the Discover tab as part of the document data, and you can use scripted fields in your visualizations. You query scripted fields with the <<kuery-query, {kib} query language>>, and can filter them using the filter bar. The scripted field values are computed at query time, so they aren't indexed and cannot be searched using the {kib} default
@@ -193,33 +193,123 @@ For more information on scripted fields and additional examples, refer to
 https://www.elastic.co/blog/using-painless-kibana-scripted-fields[Using Painless in {kib} scripted fields]
 
 [float]
-[[create-scripted-field]]
-==== Create scripted fields
+[[migrate-off-scripted-fields]]
+==== Migrate to runtime fields or ES|QL queries
 
-Create and add scripted fields to your data views.
+The following code snippets demonstrate how an example scripted field called `computed_values` on the Kibana Sample Data Logs data view could be migrated to either a runtime field or an ES|QL query, highlighting the differences between each approach.
 
-. Go to the *Data Views* management page using the navigation menu or the <<kibana-navigation-search,global search field>>.
+[float]
+[[scripted-field-example]]
+===== Scripted field
 
-. Select the data view you want to add a scripted field to.
+In the scripted field example, variables are created to track all values the script will need to access or return. Since scripted fields can only return a single value, the created variables must be returned together as an array at the end of the script.
 
-. Select the *Scripted fields* tab, then click *Add scripted field*.
+[source,text]
+----
+def hour_of_day = $('@timestamp', ZonedDateTime.parse('1970-01-01T00:00:00Z')).getHour();
+def time_of_day = '';
+
+if (hour_of_day >= 22 || hour_of_day < 5)
+  time_of_day = 'Night';
+else if (hour_of_day < 12)
+  time_of_day = 'Morning';
+else if (hour_of_day < 18)
+  time_of_day = 'Afternoon';
+else
+  time_of_day = 'Evening';
+
+def response_int = Integer.parseInt($('response.keyword', '200'));
+def response_category = '';
+
+if (response_int < 200)
+  response_category = 'Informational';
+else if (response_int < 300)
+  response_category = 'Successful';
+else if (response_int < 400)
+  response_category = 'Redirection';
+else if (response_int < 500)
+  response_category = 'Client Error';
+else
+  response_category = 'Server Error';
+
+return [time_of_day, response_category];
+----
 
-. Enter a *Name* for the scripted field, then enter the *Script* you want to use to compute a value on the fly from your index data.
+[float]
+[[runtime-field-example]]
+===== Runtime field
 
-. Click *Create field*.
+Unlike scripted fields, runtime fields do not need to return a single value and can emit values at any point in the script, which will be combined and returned as a multi-value field. This allows for more flexibility in the script logic and removes the need to manually manage an array of values.
 
-For more information about scripted fields in {es}, refer to {ref}/modules-scripting.html[Scripting].
+[source,text]
+----
+def hour_of_day = $('@timestamp', ZonedDateTime.parse('1970-01-01T00:00:00Z')).getHour();
+
+if (hour_of_day >= 22 || hour_of_day < 5)
+  emit('Night');
+else if (hour_of_day < 12)
+  emit('Morning');
+else if (hour_of_day < 18)
+  emit('Afternoon');
+else
+  emit('Evening');
+
+def response_int = Integer.parseInt($('response.keyword', '200'));
+
+if (response_int < 200)
+  emit('Informational');
+else if (response_int < 300)
+  emit('Successful');
+else if (response_int < 400)
+  emit('Redirection');
+else if (response_int < 500)
+  emit('Client Error');
+else
+  emit('Server Error');
+----
+
+[float]
+[[esql-example]]
+===== ES|QL query
+
+Alternatively, ES|QL can be used to skip the need for data view management entirely and simply compute the values you need at query time. ES|QL supports computing multiple field values in a single query, using computed values with its rich set of commands and functions, and even aggregations against computed values. This makes it an excellent solution for one-off queries and realtime data analysis.
+
+[source,esql]
+----
+FROM kibana_sample_data_logs
+  | EVAL hour_of_day = DATE_EXTRACT("HOUR_OF_DAY", @timestamp)
+  | EVAL time_of_day = CASE(
+      hour_of_day >= 22 OR hour_of_day < 5, "Night",
+      hour_of_day < 12, "Morning",
+      hour_of_day < 18, "Afternoon",
+      "Evening"
+    )
+  | EVAL response_int = TO_INTEGER(response)
+  | EVAL response_category = CASE(
+      response_int < 200, "Informational",
+      response_int < 300, "Successful",
+      response_int < 400, "Redirection",
+      response_int < 500, "Client Error",
+      "Server Error"
+    )
+  | EVAL computed_values = MV_APPEND(time_of_day, response_category)
+  | DROP hour_of_day, time_of_day, response_int, response_category
+----
 
 [float]
 [[update-scripted-field]]
 ==== Manage scripted fields
 
+WARNING: The ability to create new scripted fields has been removed from the *Data Views* management page in 9.0. Existing scripted fields can still be edited or deleted, and the creation UI can be accessed by navigating directly to `/app/management/kibana/dataViews/dataView/{dataViewId}/create-field`, but we recommend migrating to runtime fields or ES|QL queries instead to prepare for removal.
+
 . Go to the *Data Views* management page using the navigation menu or the <<kibana-navigation-search,global search field>>.
 
 . Select the data view that contains the scripted field you want to manage.
 
 . Select the *Scripted fields* tab, then open the scripted field edit options or delete the scripted field.
 
+For more information about scripted fields in {es}, refer to {ref}/modules-scripting.html[Scripting].
+
 WARNING: Built-in validation is unsupported for scripted fields. When your scripts contain errors, you receive
 exceptions when you view the dynamically generated data.
 

docs/upgrade-notes.asciidoc

@@ -164,5 +164,19 @@ resolving issues if any deprecated features are enabled.
 To access the assistant, go to **Stack Management** > **Upgrade Assistant**.
 
 
+[discrete]
+[[deprecation-202250]]
+.Scripted field creation has been disabled in the Data Views management page (9.0.0)
+[%collapsible]
+====
+*Details* +
+The ability to create new scripted fields has been removed from the *Data Views* management page in 9.0. Existing scripted fields can still be edited or deleted, and the creation UI can be accessed by navigating directly to `/app/management/kibana/dataViews/dataView/{dataViewId}/create-field`, but we recommend migrating to runtime fields or ES|QL queries instead to prepare for removal.
 
+For more information, refer to {kibana-pull}202250[#202250].
 
+*Impact* +
+It will no longer be possible to create new scripted fields directly from the *Data Views* management page.
+
+*Action* +
+Migrate to runtime fields or ES|QL instead of creating new scripted fields. Existing scripted fields can still be edited or deleted.
+====

packages/core/deprecations/core-deprecations-common/index.ts

@@ -8,6 +8,7 @@
  */
 
 export type {
+  DeprecationDetailsMessage,
   BaseDeprecationDetails,
   ConfigDeprecationDetails,
   FeatureDeprecationDetails,

packages/core/deprecations/core-deprecations-common/src/types.ts

@@ -7,6 +7,11 @@
  * License v3.0 only", or the "Server Side Public License, v 1".
  */
 
+export interface DeprecationDetailsMessage {
+  type: 'markdown' | 'text';
+  content: string;
+}
+
 /**
  * Base properties shared by all types of deprecations
  *
@@ -22,7 +27,7 @@ export interface BaseDeprecationDetails {
    * The description message to be displayed for the deprecation.
    * Check the README for writing deprecations in `src/core/server/deprecations/README.mdx`
    */
-  message: string | string[];
+  message: string | DeprecationDetailsMessage | Array<string | DeprecationDetailsMessage>;
   /**
    * levels:
    * - warning: will not break deployment upon upgrade

src/platform/packages/shared/kbn-doc-links/src/get_doc_links.ts

@@ -345,6 +345,7 @@ export const getDocLinks = ({ kibanaBranch, buildFlavor }: GetDocLinkOptions): D
       fieldFormattersNumber: `${KIBANA_DOCS}numeral.html`,
       fieldFormattersString: `${KIBANA_DOCS}managing-data-views.html#string-field-formatters`,
       runtimeFields: `${KIBANA_DOCS}managing-data-views.html#runtime-fields`,
+      migrateOffScriptedFields: `${KIBANA_DOCS}managing-data-views.html#migrate-off-scripted-fields`,
     },
     addData: `${KIBANA_DOCS}connect-to-elasticsearch.html`,
     kibana: {

src/platform/packages/shared/kbn-doc-links/src/types.ts

@@ -304,6 +304,7 @@ export interface DocLinks {
     readonly fieldFormattersNumber: string;
     readonly fieldFormattersString: string;
     readonly runtimeFields: string;
+    readonly migrateOffScriptedFields: string;
   };
   readonly addData: string;
   readonly kibana: {

src/plugins/data_view_management/public/components/edit_index_pattern/scripted_fields_table/components/call_outs/call_outs.tsx

@@ -24,11 +24,12 @@ export const CallOuts = ({ deprecatedLangsInUse, painlessDocLink }: CallOutsProp
 
   return (
     <>
+      <EuiSpacer size="m" />
       <EuiCallOut
         title={
           <FormattedMessage
-            id="indexPatternManagement.editIndexPattern.scripted.deprecationLangHeader"
-            defaultMessage="Deprecation languages in use"
+            id="indexPatternManagement.editIndexPattern.scripted.deprecatedLangHeader"
+            defaultMessage="Deprecated languages in use"
           />
         }
         color="danger"
@@ -38,7 +39,7 @@ export const CallOuts = ({ deprecatedLangsInUse, painlessDocLink }: CallOutsProp
           <FormattedMessage
             id="indexPatternManagement.editIndexPattern.scripted.deprecationLangLabel.deprecationLangDetail"
             defaultMessage="The following deprecated languages are in use: {deprecatedLangsInUse}. Support for these languages will be
-            removed in the next major version of Kibana and Elasticsearch. Convert you scripted fields to {link} to avoid any problems."
+            removed in the next major version of Kibana and Elasticsearch. Convert your scripted fields to {link} to avoid any problems."
             values={{
               deprecatedLangsInUse: deprecatedLangsInUse.join(', '),
               link: (
@@ -53,7 +54,6 @@ export const CallOuts = ({ deprecatedLangsInUse, painlessDocLink }: CallOutsProp
           />
         </p>
       </EuiCallOut>
-      <EuiSpacer size="m" />
     </>
   );
 };