[Index pattern] Add support for object runtime field "subFields"

[sebelga]

Problem

Object runtime fields give the user the possibility to declare multiple sub fields inside a parent object through a script. For that the user will emit multiple value from the painless script as a set of key/value pairs. The problem is: this single runtime field, once saved in the Kibana index pattern, will appear as one field and none of the sub fields will be exposed. When an app (Lens, Discover) will ask for the list of fields on the index pattern it will then receive this top object runtime field and none of the associated sub fields.

Proposal

I suggest the following:

Returning a subFields map of leaf runtime fields

interface SubFields {
  [name: string]: RuntimeField;
}

The field editor would return a new optional subFields property on the runtime field (following the naming convention in place for concrete fields)

Adding a parent prop to the RuntimeField interface

interface RuntimeField {
  ...
  parent?: string; // new
}

This is the new interface returned by the field editor (it is internal to the plugin)

{
  "name": "myObject",
  "type": "object",
  "script": "// script to emit multiple sub fields",
  "subFields": { // the editor returns this map of runtime fields
    "foo": {
      "name": "foo",
      "type": "keyword",
      "format": { ... },
      "customLabel": "myFooField",
      "priority: 1,
      "parent": "myObject",
    },
    "bar": {
      "name": "bar",
      "type": "long",
      "format": { ... },
      "customLabel": "myBarField",
      "priority": 2,
      "parent": "myObject",
    }
  }
}

Add a new top level property runtimeObjects to the index pattern

interface RuntimeObject  {
  name: string;
  script: {
    source: string;
  };
  subFields: string[]; // Array of subFields **names**
}

interface RuntimeObjects {
  [name: string]: RuntimeObject;
}

So the IndexPattern interface becomes

interface IndexPattern {
  ...
  fields: IIndexPatternFieldList; // no change, same as today
  runtimeObjects: RuntimeObjects; // the above interface
}

Option 1: Update the addRuntimeField() handler to now receive a map of fields and optional runtime object

Here is the current handler.

addRuntimeField(fields: { [name: string]: RuntimeField }, runtimeObject?: RuntimeObject) {
  if (runtimeObject) {
    // 1. We check if the runtimeObject exists already
    // 2. If it does we **remove** all the subFields associated with it (reading its "subFields" array property)
    // 3. We add the runtimeObject to the indexPattern
  }

  Object.entries(fields).forEach(([name, runtimeField]) => {
    // Same as before inside the loop
  });
}

Option 2: Add a new addRuntimeObject() handler to save runtime objects

We keep the current addRuntimeField() handler as it is and we add a new handler to save runtime objects along with its sub fields

addRuntimeObject(runtimeObject: RuntimeObject, subFields: { [name: string]: Omit<RuntimeField, "script"> }) {
  // Here we have the validation to make sure that:
  // 1. All subFields have their parent set to the runtimeObject
  // 2. The array of subFields name declared on the runtimeObject correspond to the map of sub fields

  // We can then add the runtime object to the index pattern and loop the sub fields
  // calling the current "addRuntimeField()" for each of them

  // **Important**: We will also need first to check if the runtime object already exists. If it does we will first remove all
  // of its current sub fields before adding the ones passed to this handler.
}

This will allow us the normal CRUD operation like other runtime fields:

• List: No changes should be needed here as we have basically added multiple runtime fields in one operation, nothing more.

• Create: As mentioned above we will update the addRuntimeField() to support saving multiple runtime fields + optionally a runtime object.

• Edit: When editing an leaf object (e.g. myRuntimeObject.foo) we will:

  • check that it is a runtime field
  • if so: check if it has a parent which is also a runtime field (with a runtimeField definition + a script)
  • if so: we would open the editor without the script section, just like any concrete field (with custom label, format, priority). We would add an callout indicating to the user that this field comes from an object runtime field with a link to edit the parent object field to change its value.

• Delete: To delete a leaf object field we would open a modal saying that this field has been generated by an object runtime field script and that the user needs to modify the parent's script to delete the field. There will be a button for the user to open the field editor and edit the script in order to remove the child field.

Support format for sub fields

We also need to update the logic to add the format definition for each subField to the index pattern attributes (here: index pattern field editor)

Update the logic to add runtime fields to the search query

We will also need to update the code where we add the runtime fields to the search query. This should.be straightforward as we will simply need to filter out the runtime fields that have a parent and only add the parent (the runtimeObject associated).

cc @mattkime @stacey-gammon

[stacey-gammon]

If the user has a timestamp field and wants to extract month and year from it, what happens? Do you end up with an index pattern field list of:

[{
  "name": "timestamp",
  "type": "date",
}, {
 "name": "timestampParentField1",
 "type": "Object",
 "subFields": [{
   name: "month"
 }, 
 {
 name: "year"
 }]
}]

or do the subFields get added to the timestamp field, thereby making it impossible for the user to use that field anymore as a date field?

[sebelga]

Yes the list would be

[
  {
    "name": "timestamp",
    "type": "date"
  },
  {
    "name": "timestampExtract",
    "type": "Object",
    "subFields": [
      {
        "name": "month"
      },
      {
        "name": "year"
      }
    ]
  }
]

With 3 fields:

• timestamp
• timestampExtract.month
• timestampExtract.year

[stacey-gammon]

Can you add the proposed Type/Public API Item changes? What is the SubField type? is it of IFieldType? Or IndexPatternField?

[ppisljar]

what about scripts with if statements in there or some other complex code, how will we detect all the possible things it can emit ? are we relying in executing the script to see what fields it generates or will we parse the script ?

[sebelga]

This is all done on the ES side. We run the painless _execute script and whatever comes back are the fields the user will be able to save in the index pattern.

[mattkime]

I think there are four (!) distinct interfaces we need to define

• The return value(s) of the field editor
  • do we have agreement here? We're at least close. I think I'm comfortable with this.
• The index pattern class changes
  • Main thing I think needs to be addressed here is that the runtimeObjects need to store the subfield names and types and as best I can tell thats it. Everything else pertaining to fields is stored elsewhere (formatting and other attributes such as description) and applied when the field is created in the index pattern initialization process
• The index pattern spec changes
  • Can turn our attention to this after the class changes are in place
• The index pattern saved object
  • Just serialize the runtimeObjects

[mattkime]

Is the runtime object expected to be the same as the ES runtime object interface?

interface RuntimeObject  {
  name: string;
  script: {
    source: string;
  };
  subFields: string[]; // Array of subFields **names**
}``

I'm tempted to change `subFields` to `map<string, { type: string }>` so we store the type info as well. 

Any thoughts on that @sebelga ?

[sebelga]

Is the runtime object expected to be the same as the ES runtime object interface?

In ES there is no concept of "runtime object". There are simply runtime fields. The type: "object" has yet to be defined, it might be named differently.

I'm tempted to change subFields to map<string, { type: string }> so we store the type info as well.

I am not seeing the need right now of having the type stored inside the RuntimeObject. We can always change it later if we see the need but for now the array of field name is enough.

[mattkime]

@sebelga I think the index pattern field list should only be leaf fields and the 'parent' definition should only exist in the runtime field definitions. Introducing an 'object' field type would potentially mean making changes to all code that takes the field list as an argument. Instead, we could add a 'runtimeParent' attribute to the IndexPatternField when its the product of an object runtime field. This would also sidestep the need to modify the field formatting code to work with subfields.

[sebelga]

I think we need to Zoom for me to understand this. Introducing an object field type is what is coming from ES and that the field editor will need to handle.
I am not sure that we need to narrow "parent" name to "runtime fields" as it would be a property on any index pattern field (be them runtime or indexed).
Let's find a time to discuss online as maybe I am mis interpreting something. 👍

[sebelga]

After discussing with @mattkime we will do something slightly different to completely align with what is currently in place and not have suddenly object parent field in the list of fields of the index pattern.

object runtime field are runtime field without custom label, format and priority. So fields just with a name, type and script.

With that in mind we are going to add a new top level properties runtimeObjects (name to be defined as the ES API might still change) at the same level as fields today. This way we won't break apps which access directly the fields on the index pattern without using a handler.

When creating an object runtime field we will:

1. Add the runtime object to the runtimeObjects map
2. Add all the subFields as we currently do with single value runtime fields.

cc @stacey-gammon

[sebelga]

@mattkime @stacey-gammon I've updated the discussion with the updated proposal if you want to have another look 👍

[mattkime]

@ppisljar - We should get your thoughts

[ppisljar]

its gonna take some time to process this, give me a day or two

[sebelga]

I've updated the description with a second option to have a dedicated addRuntimeObject() handler on the index pattern.

[ppisljar]

if i am reading this correct we have an agreement on how to proceed ?

• addRuntimeField signature does not change
• when adding runtime field of type object its not stored in the fields list but in runtimeObjects and all the leaf fields are added to fields list

questions:

• i guess the leaf nodes (in the fields array) still need to link to the parent ?
• what are the benefits of keeping the object fields in a separate list ? if we just add them to the main list (fields) won't it still work ok, just as it would if you have any field type that's not supported by our aggs/filters/... ? so don't we just need to make sure all the leaf nodes are also added to fields list as top level entry so that we can configure the type and format and have it appear in all the places around kibana without refactoring ?

[stacey-gammon]

I am waiting for that agreement from @mattkime or @stacey-gammon 😊

I felt like I was getting involved in the details too early on, before the area lead had a chance to weigh in, so I was going to let you folks iterate and I'll review when there is consensus. Does that sound good?

[ppisljar]

i was under assumption that we already have fields in the field list that apps can't handle (nested fields are of this kind but i think there are more es types we don't fully support). so applications already need to perform some kind of logic to filter this down to a list of fields they know how to handle.

[sebelga]

I can't answer that @ppisljar. It was a concern from @mattkime. I would prefer to find a solution that wouldn't require the apps to make modification to their code if possible.

[mattkime]

What @ppisljar says is true but I don't see a reason to rely on that when we know the components of the runtime field objects are supported types. Additionally, fields stored on the index pattern are added to the field list during the constructor so we have control over how this happens.

i guess the leaf nodes (in the fields array) still need to link to the parent ?

Yes, for editing and when consumed by searchsource.

what are the benefits of keeping the object fields in a separate list ?

There are a couple of different concerns that are conflated. There's how we store the index pattern and there's the instance we pass around within kibana. Where we store the runtime field object is a matter of code style but I think @sebelga and I are in agreement that storing like with like is nice. This is in contrast to the current state where scripted fields are saved on the 'fields' array but we have to loop over the whole field list to find them. (As I say this out loud I'm realizing this could be optimized but time is better spent elsewhere)

As for the index pattern instance, I don't see a reason to have the runtime field object represented on the field list.

so don't we just need to make sure all the leaf nodes are also added to fields list as top level entry so that we can configure the type and format and have it appear in all the places around kibana without refactoring ?

Absolutely.

[ppisljar]

yes, lets def try to find solution that can be kept an internal implementation detail and doesn't need updates from all the consumers