Skip to content

Commit

Permalink
fix(requests): Set HTTP pool connections/maxsize to max workers
Browse files Browse the repository at this point in the history
This allows requests to open and save/cache up to *max_workers* amount of TCP connections. In most situations it will still only save and re-use one TCP Connection since it always tries to re-use the connection if one is available.

However, in situations where downloads are from more than 10 Host/Port combinations (the default pool connections/maxsize) then this will improve download speeds.
  • Loading branch information
rlaphoenix committed Mar 12, 2024
1 parent 5376e4c commit 1bff87b
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions devine/core/downloaders/requests.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from typing import Any, Generator, MutableMapping, Optional, Union

from requests import Session
from requests.adapters import HTTPAdapter
from rich import filesize

from devine.core.constants import DOWNLOAD_CANCELLED
Expand Down Expand Up @@ -211,6 +212,12 @@ def requests(
]

session = Session()
session.mount("https://", HTTPAdapter(
pool_connections=max_workers,
pool_maxsize=max_workers
))
session.mount("http://", session.adapters["https://"])

if headers:
headers = {
k: v
Expand Down

0 comments on commit 1bff87b

Please sign in to comment.