What is the reason of having offset changed only after commit?

[trialoid]

Hi @castorm! Thank you very much for the work you've done on this connector, very well done and extension points really help moving forward with custom solutions!

I'm currently making some changes for AWS Neptune streams CDC and what confused me a bit is the way how offsets are modified: not in the poll() but inside commit(). This leads to connector being "stuck" on same offsets and producing duplicated records until commit happens. What was the reason for such implementation?

Because in my case it's desirable to keep records polling as fast as possible (have very low http.timer.interval.millis) and not be dependent on offset.flush.interval.ms server value (which is usually high), only in case of failure have duplicated records as a side effect.

[castorm]

Hi @Serious7Sam,

Thanks for your kind words.

Your use case is definitely different from the one this connector was originally designed for. Having a relatively long interval was acceptable.

Having said that, I can't really remember why specifically commitRecord()/commit() were implemented, sorry!

I'd like to think there was a reason, but in absence of an explanation you might as well get rid of them until proven necessary.

[trialoid]

Got it, thank you!