Supporting AI tools

[antonpirker]

Hey everybody!

Not long ago we added an OpenAI integration to the Sentry SDK. Right now we have PRs for two more AI related integrations open, Anthropic and Langchain.

If you work in the AI sector or are an AI enthusiast, I would like to ask you a couple of questions:

• What data from your AI project would you like to see in Sentry?
• What data would help to improve your AI pipelines, the training of your models, help you improve your AI project?
• What other AI related features would you love to see in Sentry?
• Are there any other tools you would like to see automatic instrumentation for?

Let's get a discussion going, so together we can build kick ass AI support into Sentry!

[smeubank]

After talking with some folks internally and from the community (downloads and stars etc), I believe the most useful platforms for folks would be as follows, grouped by product/service, and then popularity within. ✅ Means we already have a PR in the works for them

LLM Providers

• openai ✅
• huggingface-hub https://github.com/huggingface/huggingface_hub
• anthropic ✅

Framework

• langchain ✅
• llama-index https://github.com/run-llama/llama_index
• haystack https://github.com/deepset-ai/haystack

Vector DBs

• chromadb https://github.com/chroma-core/chroma
• cohere https://github.com/cohere-ai
• qdrant-client https://github.com/qdrant/qdrant-client
• weaviate-client https://github.com/weaviate/weaviate-python-client

[czyber]

Hey hey! Sure I'll give a vector db a spin. If you have any preferences on which to integrate first, I'm open to do any of them. Otherwise I'll have time on the weekend to have a deeper look 👍

[antonpirker]

@colin-sentry you probably know best what vector db @czyber could look into. You do cohere. Chroma is the one that is included in Langchain, so maybe this is a good candidate? Or should we wait until you do a cohere integration and just copy it and modify it to fit another db?

[colin-sentry]

I was just going to do the cohere LLM, not their vector DB yet.

We do have the embeddings measured in the OpenAI integration, which is most of the interesting bits for most vector DBs

[smeubank]

in the future we may have better/more native OTel support in the Python SDK

but until then :) @czyber if you want it might be interesting to work on chromadb, and refer the implementation here for inspiration

https://github.com/traceloop/openllmetry/tree/main/packages/opentelemetry-instrumentation-chromadb

[czyber]

Sure thing!

[smeubank]

Interesting use cases:

the following I am regurgitating from @czyber 😆 for full transparency 🚀

An interesting metric to observe for LLMs in general could be the output tokens from the LLM categorized by prompts.

Lets take the following example:
I have the use case that I want the LLM to create studying material for students (e.g. task descriptions for math problems),

The prompt is a template, and has some dynamic inputs (such as 'output_language', 'grade', 'topic'). It would be interesting to have
some metrics (maybe even alerts) if the LLM responses deviate from the expectation value of the output_token count.

[smeubank]

Hugging Face does provide some support in the way that you can indicate an OTLP endpoint (Text Generation Interface: Distributed Tracing). Which appear to be what openLLMetry takes advantage of here

[antonpirker]

Just a quick update:

• Our LLM monitoring is now out of Beta: https://docs.sentry.io/product/insights/llm-monitoring/
• The Python libraries we support right now can be found here: https://docs.sentry.io/platforms/python/integrations/#ai