Kibana console needs to support passing version header

[mattkime]

The kibana dev console allows calling Kibana apis via the kbn: prefix. Unfortunately there's no way to pass a version header - something that's actively being rolled out to apis for backward compatibility support. Passing a version header is enforced in dev environments.

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[jughosta]

DevTools currently responds with:

{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Please specify a version via elastic-api-version header. Available versions: [2023-10-31]"
}

[jloleysens]

TL;DR we need to think a little bit about the intended utility of kbn: in Console. It could be used to work against a lot of the efforts to lock down Kibana HTTP APIs. Personally I'd like to see it usable against only public Kibana HTTP APIs.

For a start, this could be as "dumb" as a map/set of public method+path (remember public HTTP APIs don't change often). If a request is not in the map, it's a 4xx. One alternate idea: Kibana plugins could register their endpoints against Console (more effort up front, but less maintenance).

Open to ideas regarding how we "fix" this in dev environments 😅 . Perhaps paths with their corresponding versions, although this increases maintenance.

Here is a 🧠 ⬇️

---

The behaviour requiring a version is different for public HTTP APIs in dev: we require all calls to versioned HTTP APIs in dev to specify a version as a way to ensure we are hitting specific handlers (strong guarantees about behaviour).

This is driven by defaulting server.versioned.versionResolution to none in dev.

kibana/packages/core/http/core-http-server-internal/src/http_config.ts

Line 186 in 08db8ee

versionResolution: schema.conditional(

More practically:

Starting Kibana with yarn start will get you this behaviour:

GET kbn:/api/data_views
=> 400

Starting Kibana with yarn start --server.versioned.versionResolution=oldest will get you:

GET kbn:/api/data_views
=> 200

---

Currently it will result in 400 for all versioned internal endpoints as they require a version in all contexts. When serverless=... is specified 90+% of Kibana endpoints will require setting our special internal header or get 400.

[alisonelizabeth]

Thanks for your input @jloleysens!

TL;DR we need to think a little bit about the intended utility of kbn: in Console. Currently it can be used to work against a lot of the efforts to lock down Kibana HTTP APIs. Personally I'd like to see it usable against only public Kibana HTTP APIs.

++ The Management team started this discussion earlier in the week. While the kbn: utility is a great feature, I don't think there was much thought put into how it would be used at the time it was developed. Now that Kibana HTTP APIs are evolving, I think it's important to rethink its intended usage. I agree that it makes sense to restrict it to public APIs.

[lukeelmers]

Just my two cents, I could imagine doing something like this:

• in dev mode kbn: applies the x-elastic-internal-origin header to allow hitting internal APIs for testing purposes
• in production kbn: doesn't apply the header, thus restricting requests to public APIs only

The versioning problem is more tricky. IMO we made things harder on ourselves by applying a blanket check for this in dev mode... ideally there'd be a way to make console exempt from this check because it is such a different use case. I am nervous about adding more complexity to console or the http service to "fix" what's essentially a self-imposed problem that only exists in dev-mode and only affects console.

[TinaHeiligers]

IMO we made things harder on ourselves by applying a blanket check for this in dev mode...

The restriction was implemented when the default for API's was public. Changing the default to internal had more repercussions than we expected.

[jloleysens]

The versioning problem is more tricky. IMO we made things harder on ourselves...

Yeah, I agree this adds friction that is questionable in these early stages. However I think it serves a valuable purpose: the time devs are running Kibana we know that our requests are explicit about what handlers they need. This gives us more confidence (quality?) moving forward, but at the cost of running into situations like this today. I'm definitely open to talking through this again though!

Depending on how we resolve this #163678, as @lukeelmers noted, we may have a way to allow access to internal routes (preferably just in dev). However I'd argue that Kibana devs (and users actually) have alternative ways of calling internal Kibana routes besides console and wouldn't optimise for this.

That leaves public routes. The simplest way forward today would be to add 2023-10-31 as the version by default in Console and then work towards something like allowing a way for plugins to register against Console. WDYAT?

[mattkime]

I'm concerned about locking things down too tightly. Then again, I might not be up to speed on the 'why' of what we're doing.

My understanding is that we don't want our users building workflows around internal apis. It seems to me that the additional headers make abundantly clear what is and isn't internal. I think its reasonable for our customers to respect this.

I think the main problem we've had in the past is a lack of consistency and follow through. APIs stay in Technical Preview forever, users come to expect they're there, and then we determine we need to support them.

Hitting internal apis via the console can be extremely helpful for debugging. I've often asked for the result of a manual fields_for_wildcard request to establish which layer is causing a problem. Removing this would result in more guessing at where the problem lies.

[mattkime]

@jloleysens

The simplest way forward today would be to add 2023-10-31 as the version by default in Console and then work towards something like allowing a way for plugins to register against Console. WDYAT?

This would work for now.

What if the version param could also be passed via query string?

[jloleysens]

What if the version param could also be passed via query string?

This is possible today on an opt-in basis via:

https://github.com/elastic/kibana/blob/f38a37c0c175a7f25c7dd8474de1cb732c1da09e/packages/core/http/core-http-server/src/versioning/types.ts#L55C6-L55C6

It is intended as an escape hatch when there is no other option as described in the doc comment.

Hitting internal apis via the console can be extremely helpful for debugging

That's a good point but you could also use curl to do this (does not have to be Console).

main problem we've had in the past is a lack of consistency and follow through

IMO we'll take a big step toward consistency by removing ambiguity and not allowing our users to craft requests against internal APIs. If it really is something users need access to we should make it public.

[lukeelmers]

From the platform/core side I think we should consider this work complete by a combination of the existing query param for versions and the PR for passing the internal header. Together, these will allow developers to continue to interact with Kibana APIs in console. External users who aren't aware of these hacks will still be able to hit public APIs in console and get back the default version for an API. I believe this would meet the near term needs here.

Longer term, the Deployment Management team can determine whether it's worth eventually prioritizing a UI of some kind that allows users to select/specify versions in their requests.

[jloleysens]

Deployment Management team can determine

++ it'd be nice to support this in future

the existing query param for versions

just to be clear: this will only make the (e.g.) apiVersion=1 query param option available for routes that opt-in. By far most internal versioned routes won't be accessible from Console, even in prod.

[yuliacech]

Related to #110881

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)

[ghudgins]

hi all, the dashboard API that's coming is increasing the urgency for a fix here so customers can enjoy the new API in the dev console more seamlessly

[lukeelmers]

We currently only have one API version in both serverless & non-serverless, and we expect new versions to be very infrequent.

The version header is also optional. For serverless we set versionResolution: latest, and for non-serverless we set versionResolution: oldest

Which is to say - I'm not sure this is blocking users' ability to experiment with the new dashboard APIs. They should be able to access them in dev console if they're public without a version header being required.

[nickpeihl]

and for non-serverless we set versionResolution: oldest

IIUC, we default to versionResolution: oldest in distributions, but not when running in development. I think that's where I was confused. So users in serverless and traditional distributions can use Console to send kbn: prefixed API requests. But when running in development, the dev context causes versionResolution to default to none.

So developers would need to explicitly add server.versionedversionResolution: oldest to their kibana.dev.yml file to make API requests via the Console.

[jloleysens]

So developers would need to explicitly add server.versionedversionResolution: oldest to their kibana.dev.yml file to make API requests via the Console

++ this is true, and an unintended side effect is that the original intention to encourage us to send headers at dev time is disabled. Core is discussing changing the dev default to also be server.versionedversionResolution: oldest