Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uploading large dir of images hangs #109

Open
jvasile opened this issue Mar 17, 2023 · 4 comments
Open

Uploading large dir of images hangs #109

jvasile opened this issue Mar 17, 2023 · 4 comments

Comments

@jvasile
Copy link

jvasile commented Mar 17, 2023

Uploading a largish (500+ items) directory of largish files (images) and some html files, rclone just hung after 146 files.

The files are all images and web pages from Astronomy Photo Of The Day. I obtained that dir by cloning https://github.com/OpenTechStrategies/permanent-rclone-qa/ and running ./apod-downloader.py. Then I tried to upload them with ./upload-test.py test-tree/apod --remote-dir=apod. Things went fine, and it zipped along with 4 parallel uploads. But after 146 files, it looks like each upload hung, one by one, as it hit a file that refused to either complete or fail.

Here's the log from this run:
apod.txt

If you want to reproduce this, you're welcome to use my apod-downloader and upload-test script, but I also suspect you can try to upload just about any largish set of largish files. Maybe I'll try it with a largish set of small files next.

This might be related to #99 but I got way further than just 6 jpgs before things went sideways.

@slifty
Copy link
Contributor

slifty commented Mar 22, 2023

I've made a whole slew of changes that I'm hoping will resolve this and other issues.

Two things may have been happening:

  1. file handles were being closed twice, which was (sometimes) resulting in absolute chaos! This is resolved by Prevent double closing #112
  2. If anything failed on the permanent backend it would result in a failure, and rclone would then try to delete the file. Our service doesn't support deletion but that lack of support meant no response was being provided at all, and so rclone would sit there waiting for a response. This is resolved by Return failure status on remove operations #110

These changes will be deployed this afternoon, at which point I think it would be worth trying this again.

@jvasile
Copy link
Author

jvasile commented Mar 28, 2023

@fenn-cs, can you keep testing the above and then add your results here?

@nfebe
Copy link
Contributor

nfebe commented Mar 28, 2023

@jvasile Sure! Would follow up on it.

@slifty
Copy link
Contributor

slifty commented Apr 19, 2023

@fenn-cs Any word on this? Would love to close it if it isn't repeating.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants