Tagged @liam-lloyd since he has been looking at multipart upload too.

I was about to write in the issue but realized the PR is probably the best place for it at this point -- In testing this out locally I realized that the "local buffer" implementation doesn't quite work out with the way sftp sends data (and, in fact, wouldn't have worked even without the local buffer, and maybe wouldn't even have worked necessarily in the original implementation!).

The SFTP protocol includes an offset:

`offset' is the offset (in bytes) from the
   beginning of the file where to start writing, and `data' is the data
   to be written.

   The write will extend the file if writing beyond the end of the file.
   It is legal to write way beyond the end of the file; the semantics
   are to write zeroes from the end of the file to the specified offset
   and then the data.  On most operating systems, such writes do not
   allocate disk space but instead leave "holes" in the file.

Which is to say, writes are done in parallel AND are not guaranteed to be sequential. I'm currently working to figure out a good way to handle this.

It looks like the SFTP protocol (v3) does not provide a way to signal to clients that writes must be sequential.

I think this means that, well, we need to support non-sequential writes and know how to honor the offset.

The trouble here is we now have some conflicting constratings:

1. We want a finite cap on the amount of local data that can be stored per connection.
2. S3 requires us to write 5MB chunks.

If data is being written sequentially we have no issue: just append to a single local temporary file and when it gets large enough, upload it.

Since data is NOT being written sequentially, we have no way of ensuring that we will get a 5MB chunk.

For example, one could imagine imagine an SFTP client that breaks a 100MB file into twenty separate 5MB segments, and sends parts of each segment in a round robin style. This would functionally result in storing all 100MB in local temporary storage before writing anything to S3!

It does appear that rclone is writing sequentially, by the way, it's just that we need to appreciate that this is not guaranteed.

I'm going to reflect on alternate approaches given these revelations. One approach could be to use s3 for even the "temporary" storage as a series of individual, small files that are indexed by their offsets and ultimately get coalesced. I am concerned that this will lead to massive overhead in creating so many tiny S3 objects (and I am also unsure if S3 prices per object, which I could see making this tactic very expensive).

Another approach could be to simply implement our current plan, knowing that generally clients are generally writing sequentially. We could also add limits to the allowed local storage (but those limits might be higher than 5MB). This could represent a "sweet spot" of preventing DDOS risk but also could lead to confusing file upload failures for users if their SFTP client has an obnoxious write fragmentation algorithim

Marking this as draft for now since there have been enough odd curve balls that it warrants stepping back and making sure this is still the approach we want to take.

I also just spoke with @kfogel and I'm going to at least update the current (full tmp storage) implementation to properly account for offset.