[]byte vs io.Reader for message contents

[markus-wa]

Hey @Jeffail

First of all, this looks like an awesome project, thanks for building this!

I was wondering if Benthos should support io.Reader in the future for large message contents.
This could be useful for sources with large files (S3. files, etc.), where reading the whole content in at once is not desirable.

I realise this might be a rather large change and if it's not in the scope of the project to support this that is perfectly understandable.

Edit: This would probably only be useful for plugins(?)

[Jeffail]

Hey @markus-wa, this is definitely possible and probably not too much effort since things like structured parsing to/from JSON is already lazily evaluated, so we could expand that concept to include byte consumption being lazily read and offer the io.Reader interface as an advanced API.

However, this will need to wait for a major version bump as it'd definitely involve chopping and restructuring some of the fundamental component APIs. I'm currently in the loose planning phase for a new plugin API (just created a ticket for it) that would start the ball rolling on Benthos V4. Once that's done we can take a look at this and weigh in the pros/cons and think about how the APIs should look.

[markus-wa]

Cool, I'd be happy to help out (if I have the time for it). Feel free to give me a ping here whenever you think this is ready to be designed / worked on 🙂

[dilipkk-foyernet]

Yes, it makes sense to have "io.Reader" exposed else, @Jeffail is it possible to provide configuration options like reading line by line or number of bytes to be read at once, and based on that we can emit one or more messages. I feel this way we don't need to change API drastically, I am still understanding this library so my understanding might be incomplete. Please forgive me in case this is misunderstood.

[nicktelford]

We've been having a problem processing large, compressed batches of messages from S3. Benthos tries to read in and decompress the entire file before processing any of the messages.

I believe switching to io.Reader should enable Benthos to process these files incrementally, including both reading and decompressing.

This is a high priority for us.

[Jeffail]

Just realised that I forgot to update this issue, but as a short term measure I've added a new codec field to a few file-based inputs, which currently includes file as well as the new aws_s3 which will eventually replace s3. These codecs are currently quite limited (line-by-line, tar archives, and csv) but the interface is fairly straight forward to implement so I'm hoping I'll be able to keep on top of any requests for new codecs coming in.

The long term plan is to gradually expand codecs and eventually, once we're happy that their common interface is stable, I will allow you to add codec plugins the same way processor and other plugins are possible now, which would allow you to build custom definitions for how an io.Reader can be streamed into benthos messages.

[markus-wa]

nice one @Jeffail - thanks for the update