Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multithreaded uploading to s3 #7

Open
inconduit opened this issue Feb 15, 2013 · 2 comments
Open

multithreaded uploading to s3 #7

inconduit opened this issue Feb 15, 2013 · 2 comments

Comments

@inconduit
Copy link

not an issue but a question - would it be possible to modify the upload mechanism to implement multithreaded uploading to an s3 bucket? i mean multithreaded for a single file, where the chunks are uploaded concurrently to maximize bandwidth.

this looks like a great project, thanks a lot for sharing it. i've just started digging into the source and learning about s3 chunked uploads, so i'm unsure if multipart-concurrent uploading is possible.

@stakach
Copy link
Member

stakach commented Feb 15, 2013

Multipart concurrency for a single file is very possible.

I originally avoided it as browsers

  • Have a limited number of connections and at the time was more interested in parallel uploads
  • Not all cloud providers support parallel parts (thinking google here, although I could be wrong)
  • Adds quite a bit of complexity, especially when resuming

As my main use case will ultimately be larger files I may re-visit implementing parallel parts (as an option) although feel free to send a pull request!

@inconduit
Copy link
Author

thanks for the fast reply. can you email me at adam@inconduit.com ? i'd like to discuss the possibility of contracting you to implement this feature, and github issues aren't really the place for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants