Split and Join

[ruszkowsk1]

Hello,

One of our clients is pushing to use Split and Join in AS4 communication in order to reduce errors and bandwith usage while sending large files.
I saw that another solution on the market has such technology.

I would like to ask what stops you from implementing it to phase4, and if there are any plans regarding it.

[phax]

I simply didn't had the necessity yet for implementing it, as I think the network transport layer should take care of that (see e.g. HTTP chunked encoding). However, I need to dig deeper into the specification to see what that means and what the impact would be.

[phax]

So based on my research I found the following:

• Split & Join is not part of the AS4 profile. Split & Join is an ebMS 3.0 extended functionality
• I looked at the implementation and I think it may be implemented on top of the existing functionality - kind of like a "wrapper" around it, but I would need more reasearch to give a final answer on that
• Honestly I think, that the added value is limited compared to the effort needed (splitting, joining, caching of partial messages, error handling, ...) over the use of "Chunked Encoding" on the HTTP level

Do you want to use it in the context of Peppol or something else? If you think of it from a Peppol Perspective, it is not supported anyway.... hth

[ruszkowsk1]

The context we use is not Peppol, it is RDE.

I guess Chunked Encoding is helpful with packet loss, but won't help much with broken connections and situations when there is a need to restart download I guess, and Split & Join would help then from what I understand.

[phax]

The HTTP protocol is quite versatile and also supports restart at a certain position etc.
See e.g. https://www.rfc-editor.org/rfc/rfc9110#name-range-requests for some specification text around this.

I am pretty confident that the tools of the HTTP protocol are way more suitable then an extra layer on top of the AS4 protocol. Also interoperability seems simpler...

[ruszkowsk1]

Hmm, you might be right - if it's same functionality on lower level, then it must be more efficient. But then - why other libraries introduce it on library level? What's the reasoning behind this?

[phax]

Please don't ask me. It is standardised in ebMS 3. Everything that is standardised becomes implemented one way or the other.
I assume it is done, because ebMS 3 is defined in a transport-protocol agnostic level, so it can be used with HTTP, SMTP etc.
If you are however bound to a specific transport protocol that is imho not needed

[ri4a]

Sounds like a solution looking for a problem.

[ruszkowsk1]

Could you please elaborate? We noticed that other popular AS4 libraries (i.e. Domibus) introduced Split&Join option for AS4.

[ri4a]

What I mean is: which problem are you solving, what is the use-case. For instance in OpenPeppol the largest document size allowed is 2GB. Is this really a problem in a machine-to-machine environment?

[ruszkowsk1]

The problem is following - our client is planning to start sending big attachments through AS4. They have government hardware structure, that is sometimes unreliable. They told us that about 5-10% connections may get broken when sending big files.

With traditional way - without split & join - this would cause lots of data being "lost". Using split & join would allow to redownload only missing parts of file, instead of redownloading whole file after broken connection.