Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x] [inference] Add cancelation support for chatComplete and output (#203108) #204588

Merged
merged 2 commits into from
Dec 17, 2024

Conversation

kibanamachine
Copy link
Contributor

Backport

This will backport the following commits from main to 8.x:

Questions ?

Please refer to the Backport tool documentation

…tic#203108)

## Summary

Fix elastic#200757

Add cancelation support for `chatComplete` and `output`, based on an
abort signal.

### Examples

#### response mode

```ts
import { isInferenceRequestAbortedError } from '@kbn/inference-common';

try {
  const abortController = new AbortController();
  const chatResponse = await inferenceClient.chatComplete({
    connectorId: 'some-gen-ai-connector',
    abortSignal: abortController.signal,
    messages: [{ role: MessageRole.User, content: 'Do something' }],
  });
} catch(e) {
  if(isInferenceRequestAbortedError(e)) {
    // request was aborted, do something
  } else {
    // was another error, do something else
  }
}

// elsewhere
abortController.abort()
```

#### stream mode

```ts
import { isInferenceRequestAbortedError } from '@kbn/inference-common';

const abortController = new AbortController();
const events$ = inferenceClient.chatComplete({
  stream: true,
  connectorId: 'some-gen-ai-connector',
  abortSignal: abortController.signal,
  messages: [{ role: MessageRole.User, content: 'Do something' }],
});

events$.subscribe({
  next: (event) => {
    // do something
  },
  error: (err) => {
    if(isInferenceRequestAbortedError(e)) {
      // request was aborted, do something
    } else {
      // was another error, do something else
    }
  }
});

abortController.abort();
```

(cherry picked from commit 0b74f62)
@kapral18
Copy link
Contributor

Please do not merge this pull request. We disabled auto-merge because we are trying to merge a this big PR as part of sustainable architecture migration which is impossible with ever increasing stream of backports. We will resume the automerge after our PR is merged. Reach out to #sustainable-kibana-architecture for more info.

@kapral18 kapral18 disabled auto-merge December 17, 2024 16:38
@kapral18
Copy link
Contributor

Auto-merge has been re-enabled. Thank you for your patience. :heart

@stephmilovic stephmilovic enabled auto-merge (squash) December 17, 2024 19:34
@pgayvallet
Copy link
Contributor

@elasticmachine merge upstream

@stephmilovic stephmilovic merged commit abde299 into elastic:8.x Dec 17, 2024
8 checks passed
@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/inference-common 43 40 -3

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
inference 7.4KB 7.5KB +74.0B
Unknown metric groups

API count

id before after diff
@kbn/inference-common 136 141 +5

History

cc @pgayvallet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants