big file upload complete_multipart network request error

[livegoplayer]

Describe the bug

在解决这个https://github.com/huggingface/huggingface_hub/issues/2152 问题的时候，我感觉的自己写的脚本很好用，而且那个时候，没有这样的错误，从今年开始，我会陆续收到上传失败的错误，网络没有问题，上传没有问题，就是在上传的最后一步老是报错，在几个月之前，它是偶发性的，我只需要重新上传即可，到了这几个月，我发现我的脚本几乎不能上传了（几乎都是大文件，小文件我没怎么测试过），错误都是下面描述的错误，在上传完毕之后会请求一个接口
requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='hf-mirror.com', port=443): Max retries exceeded with url: /api/complete_multipart?uploadId=HuyPi8JwJPR6YHq675qpeVhtqm29L5iBGn3NfC5fzneUqIzTSWi8H70VemnplKdIRgoBnYW_lHCNPFnjynxuLs68.YTrjZh.sLAOW68hN.M3eLMcxON4Y8PP5uLxAo39&bucket=hf-hub-lfs-us-east-1&prefix=repos%2Fc5%2F61%2Fc5613e7083eb34df0e57d45010b95bf9024f3efc99a3e975d7a37d95d82e7bcc&expiration=Wed%2C+24+Jul+2024+06%3A44%3A59+GMT&signature=6e70faf8c80aff0f0ea882d0ed946f4778592ed3f31992157e4a73738abcb9e5 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000015AE6EA3FD0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))"), '(Request ID: 7bfd1f94-471f-47d7-abae-3b416ba3b76f)')

这个接口总是这样报错，详情请看下面的报错信息

Reproduction

https://github.com/livegoplayer/scripts/tree/main/huggingface_io

Logs

uploading file F:\Desktop\imgs\开天\开天.tgz to aigc11/tengxunhunyuan//开天.tgz...
开天.tgz:   0%|                                                                            | 0.00/5.00G [00:00<?, ?B/s]'(MaxRetryError("HTTPSConnectionPool(host='hf-hub-lfs-us-east-1.s3-accelerate.amazonaws.com', port=443): Max retries exceeded with url: /repos/c5/61/c5613e7083eb34df0e57d45010b95bf9024f3efc99a3e975d7a37d95d82e7bcc/a70135c79b6bc2d5d92cabd00a6e11ad2674c42bcc03145abfd8712e2ecf7766?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA2JU7TKAQLC2QXPN7%2F20240723%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240723T064459Z&X-Amz-Expires=86400&X-Amz-Signature=25b9789214afa66d863262d1777fd2c6fcccf83f97ac779158c48f98610b7031&X-Amz-SignedHeaders=host&partNumber=1&uploadId=HuyPi8JwJPR6YHq675qpeVhtqm29L5iBGn3NfC5fzneUqIzTSWi8H70VemnplKdIRgoBnYW_lHCNPFnjynxuLs68.YTrjZh.sLAOW68hN.M3eLMcxON4Y8PP5uLxAo39&x-id=UploadPart (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000015AE6EA1BA0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))"), '(Request ID: 5d9ba429-f31e-4b06-9ac4-308c503aced9)')' thrown while requesting PUT https://hf-hub-lfs-us-east-1.s3-accelerate.amazonaws.com/repos/c5/61/c5613e7083eb34df0e57d45010b95bf9024f3efc99a3e975d7a37d95d82e7bcc/a70135c79b6bc2d5d92cabd00a6e11ad2674c42bcc03145abfd8712e2ecf7766?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA2JU7TKAQLC2QXPN7%2F20240723%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240723T064459Z&X-Amz-Expires=86400&X-Amz-Signature=25b9789214afa66d863262d1777fd2c6fcccf83f97ac779158c48f98610b7031&X-Amz-SignedHeaders=host&partNumber=1&uploadId=HuyPi8JwJPR6YHq675qpeVhtqm29L5iBGn3NfC5fzneUqIzTSWi8H70VemnplKdIRgoBnYW_lHCNPFnjynxuLs68.YTrjZh.sLAOW68hN.M3eLMcxON4Y8PP5uLxAo39&x-id=UploadPart
Retrying in 1s [Retry 1/5].
开天.tgz: 100%|███████████████████████████████████████████████████████████████████| 5.00G/5.00G [26:37<00:00, 3.13MB/s]
Traceback (most recent call last):
  File "D:\Software\python\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "D:\Software\python\lib\site-packages\urllib3\util\connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "D:\Software\python\lib\socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Software\python\lib\site-packages\urllib3\connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "D:\Software\python\lib\site-packages\urllib3\connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "D:\Software\python\lib\site-packages\urllib3\connectionpool.py", line 1058, in _validate_conn
    conn.connect()
  File "D:\Software\python\lib\site-packages\urllib3\connection.py", line 363, in connect
    self.sock = conn = self._new_conn()
  File "D:\Software\python\lib\site-packages\urllib3\connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x0000015AE6EA3FD0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Software\python\lib\site-packages\requests\adapters.py", line 489, in send
    resp = conn.urlopen(
  File "D:\Software\python\lib\site-packages\urllib3\connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "D:\Software\python\lib\site-packages\urllib3\util\retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='hf-mirror.com', port=443): Max retries exceeded with url: /api/complete_multipart?uploadId=HuyPi8JwJPR6YHq675qpeVhtqm29L5iBGn3NfC5fzneUqIzTSWi8H70VemnplKdIRgoBnYW_lHCNPFnjynxuLs68.YTrjZh.sLAOW68hN.M3eLMcxON4Y8PP5uLxAo39&bucket=hf-hub-lfs-us-east-1&prefix=repos%2Fc5%2F61%2Fc5613e7083eb34df0e57d45010b95bf9024f3efc99a3e975d7a37d95d82e7bcc&expiration=Wed%2C+24+Jul+2024+06%3A44%3A59+GMT&signature=6e70faf8c80aff0f0ea882d0ed946f4778592ed3f31992157e4a73738abcb9e5 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000015AE6EA3FD0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Software\python\lib\site-packages\huggingface_hub\_commit_api.py", line 402, in _wrapped_lfs_upload
    lfs_upload(operation=operation, lfs_batch_action=batch_action, headers=headers, endpoint=endpoint)
  File "D:\Software\python\lib\site-packages\huggingface_hub\lfs.py", line 243, in lfs_upload
    _upload_multi_part(operation=operation, header=header, chunk_size=chunk_size, upload_url=upload_url)
  File "D:\Software\python\lib\site-packages\huggingface_hub\lfs.py", line 345, in _upload_multi_part
    completion_res = get_session().post(
  File "D:\Software\python\lib\site-packages\requests\sessions.py", line 635, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "D:\Software\python\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\Software\python\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "D:\Software\python\lib\site-packages\huggingface_hub\utils\_http.py", line 68, in send
    return super().send(request, *args, **kwargs)
  File "D:\Software\python\lib\site-packages\requests\adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='hf-mirror.com', port=443): Max retries exceeded with url: /api/complete_multipart?uploadId=HuyPi8JwJPR6YHq675qpeVhtqm29L5iBGn3NfC5fzneUqIzTSWi8H70VemnplKdIRgoBnYW_lHCNPFnjynxuLs68.YTrjZh.sLAOW68hN.M3eLMcxON4Y8PP5uLxAo39&bucket=hf-hub-lfs-us-east-1&prefix=repos%2Fc5%2F61%2Fc5613e7083eb34df0e57d45010b95bf9024f3efc99a3e975d7a37d95d82e7bcc&expiration=Wed%2C+24+Jul+2024+06%3A44%3A59+GMT&signature=6e70faf8c80aff0f0ea882d0ed946f4778592ed3f31992157e4a73738abcb9e5 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000015AE6EA3FD0>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))"), '(Request ID: 7bfd1f94-471f-47d7-abae-3b416ba3b76f)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "F:\projects\scripts\huggingface_io\huggingface.py", line 605, in <module>
    main()
  File "F:\projects\scripts\huggingface_io\huggingface.py", line 598, in main
    list_projects()
  File "F:\projects\scripts\huggingface_io\huggingface.py", line 202, in list_projects
    select_current_project(start, end)
  File "F:\projects\scripts\huggingface_io\huggingface.py", line 227, in select_current_project
    show_project_details()
  File "F:\projects\scripts\huggingface_io\huggingface.py", line 280, in show_project_details
    upload_files(current_project)
  File "F:\projects\scripts\huggingface_io\huggingface.py", line 468, in upload_files
    api.upload_file(repo_id=repo_id, path_or_fileobj=absolute_path, path_in_repo=relative_path + "/"  + file_name, commit_message=commit_message, token=load_access_token(), repo_type=get_repo_type(current_project))
  File "D:\Software\python\lib\site-packages\huggingface_hub\utils\_validators.py", line 119, in _inner_fn
    return fn(*args, **kwargs)
  File "D:\Software\python\lib\site-packages\huggingface_hub\hf_api.py", line 1230, in _inner
    return fn(self, *args, **kwargs)
  File "D:\Software\python\lib\site-packages\huggingface_hub\hf_api.py", line 4275, in upload_file
    commit_info = self.create_commit(
  File "D:\Software\python\lib\site-packages\huggingface_hub\utils\_validators.py", line 119, in _inner_fn
    return fn(*args, **kwargs)
  File "D:\Software\python\lib\site-packages\huggingface_hub\hf_api.py", line 1230, in _inner
    return fn(self, *args, **kwargs)
  File "D:\Software\python\lib\site-packages\huggingface_hub\hf_api.py", line 3586, in create_commit
    self.preupload_lfs_files(
  File "D:\Software\python\lib\site-packages\huggingface_hub\hf_api.py", line 4087, in preupload_lfs_files
    _upload_lfs_files(
  File "D:\Software\python\lib\site-packages\huggingface_hub\utils\_validators.py", line 119, in _inner_fn
    return fn(*args, **kwargs)
  File "D:\Software\python\lib\site-packages\huggingface_hub\_commit_api.py", line 412, in _upload_lfs_files
    _wrapped_lfs_upload(filtered_actions[0])
  File "D:\Software\python\lib\site-packages\huggingface_hub\_commit_api.py", line 404, in _wrapped_lfs_upload
    raise RuntimeError(f"Error while uploading '{operation.path_in_repo}' to the Hub.") from exc
RuntimeError: Error while uploading '开天.tgz' to the Hub.

System info

- huggingface_hub version: 0.23.0.dev0
- Platform: Windows-10-10.0.22635-SP0
- Python version: 3.10.11
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: F:\torch_models_cacahe\token
- Has saved token ?: True
- Configured git credential helpers:
- FastAI: N/A
- Tensorflow: 2.12.0
- Torch: 2.1.2+cu121
- Jinja2: 3.1.2
- Graphviz: N/A
- keras: 2.12.0
- Pydot: N/A
- Pillow: 10.1.0
- hf_transfer: N/A
- gradio: 3.41.2
- tensorboard: 2.6.2.2
- numpy: 1.26.2
- pydantic: 2.5.3
- aiohttp: 3.8.6
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: F:\torch_models_cacahe\hub
- HF_ASSETS_CACHE: F:\torch_models_cacahe\assets
- HF_TOKEN_PATH: F:\torch_models_cacahe\token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10