Chains Streaming, fixes BT-10339

[marius-baseten]

🚀 What

Support raw string/byte streaming and streaming of structured (pydantic) data with separate item and optional header/footer types.

• New chains example (also used for testing).
• Not supported (not recommended) for sync functions.
• Remove CI check on doc sync (no useful).

💻 How

• Adapt code gen and framework/validations to allow AsyncIterator[bytes|str] endpoints. Enforce that generator functions have this annotation and vice versa.
• Rename internal terminology from generator to streaming.
• Bugfix in truss schema to be invariant against Iterator/Generator annotations.
• Fix docker internal URLs in DockerTrussService.
• Add method to consume streaming responses in stub (with retries/error handling).
• Helper classes StreamWriter and StreamReader, linked by StreamTypes to consistently deserialize and serialize structured data. Some fun typing to make this work variadically...

🔬 Testing

• E2E testing for "happy" case both local and docker mode.
• Unittesting for streaming.py.

[linear]

BT-10339 Support (async) Generators

[tyranitar]

Looks good! Just a few minor comments.

[squidarth]

This looks great! One implementation question -- what's the story for how this could be consumed by the end user?

[squidarth]

if the final chain streams output, what would be an easy way of consuming that?

[marius-baseten]

You have 3 choices:

• raw strings: works directly.
• raw bytes: works directly.
• structured/typed pydantic models: you create stream_reader with the same model definitions client side.

For the last one, we can discuss ways to distribute that implementation. It depends only on pydantic and builtins, so you wouldn't need to install the whole truss package, you just need that source file. Or we could even generate a "client".

[squidarth]

For the last one, we can discuss ways to distribute that implementation. It depends only on pydantic and builtins, so you wouldn't need to install the whole truss package, you just need that source file. Or we could even generate a "client".

Sounds good

[squidarth]

just to be clear, if someone has binary data (say audio) that they are sending back, they don't have to use this right? They would just stream the bytes directly?

I'm not sure how a pydantic model would serialize a field that contained binary data

[marius-baseten]

If you want to return raw binary data, you don't need any of these streamers, you can directly stream the bytes.