Connections in the pool never get to the idle state when a proxy is being used

[rafikahmed]

So we're using httpx in production. We have a service that receives a list of URLs and parses them. For each URL we make two concurrent requests proxy and non-proxy. The first one that is fulfilled first we return it.

Now, over time and while checking the traces on DD we noticed that the proxy requests takes some time to start (laddering effect).
To understand what was going on, we started logging as a metric the connection pool info, and we noticed that for proxy requests idle connections are always 0 although we've explicitly configured the client to use x number of max_keepalive_connections.

In contrast non-proxy requests are working fine and the resources are well managed:

Here's the code I'm running locally just to test:

import asyncio
import traceback
import httpx

from pydantic.dataclasses import dataclass

GLOBAL_HTTP_TIMEOUT = 10


@dataclass(frozen=True)
class RequestResponse:
    url: str
    status_code: int
    content: str
    elapsed_time: float


@dataclass(frozen=True)
class RequestError:
    url: str
    traceback: str


def _log_as_metric_http_pool_info(client: httpx.AsyncClient) -> None:
    httpx_connection_pool_info = str(client._transport._pool)  # type: ignore
    print(httpx_connection_pool_info)
    print("-" * 10)


transport = httpx.AsyncHTTPTransport(
    verify="zyte-proxy-ca.crt",
    proxy="proxy_key"
    limits=httpx.Limits(
        max_connections=20,
        keepalive_expiry=30,
        max_keepalive_connections=20,
    ),
)
http_proxy_client = httpx.AsyncClient(
    transport=transport,
    follow_redirects=True,
    timeout=httpx.Timeout(timeout=3),
)


async def request(
    client: httpx.AsyncClient,
    url: str,
) -> RequestResponse | RequestError:
    request_response: RequestResponse | RequestError
    try:
        response = await client.get(url)
        request_response = RequestResponse(
            url=str(response.url),
            status_code=response.status_code,
            content=response.text,
            elapsed_time=response.elapsed.total_seconds(),
        )
        response.raise_for_status()
    except (
        httpx.ConnectTimeout,
        httpx.ReadTimeout,
        httpx.ReadError,
        httpx.WriteTimeout,
        httpx.PoolTimeout,
        httpx.HTTPStatusError,
        asyncio.CancelledError,
    ):
        request_response = RequestError(
            url=url, traceback=traceback.format_exc()
        )

    _log_as_metric_http_pool_info(client)
    return request_response


async def proxy_request(
    proxy_client: httpx.AsyncClient, url: str
) -> RequestResponse | RequestError:
    return await request(proxy_client, url)


urls = [
    "https://www.pcgamer.com/wooting-60he-review/",
    "https://www.rtings.com/keyboard/reviews/wooting/60he",
    "https://www.zdnet.com/article/wooting-60he-gaming-keyboard-review/",
    "https://www.goodhousekeeping.com/appliances/g20719709/best-deep-fryers/",
    "https://www.verywellhealth.com/best-blood-pressure-monitors-4158050",
    "https://www.goodhousekeeping.com/health-products/g44620450/best-blood-pressure-monitors/",
    "https://www.seriouseats.com/best-cast-iron-skillet",
    "https://www.thespruceeats.com/best-cast-iron-pans-to-buy-4140095",
    "https://www.nytimes.com/wirecutter/reviews/best-mechanical-pencils/",
    "https://www.nytimes.com/wirecutter/reviews/best-electric-toothbrush/",
]


async def main():
    for i in range(5):
        await asyncio.gather(
            *[proxy_request(http_proxy_client, url) for url in urls]
        )


asyncio.run(main())

Output:

<AsyncHTTPProxy [Requests: 9 active, 0 queued | Connections: 9 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 8 active, 0 queued | Connections: 8 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 7 active, 0 queued | Connections: 7 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 6 active, 0 queued | Connections: 6 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 5 active, 0 queued | Connections: 5 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 4 active, 0 queued | Connections: 4 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 3 active, 0 queued | Connections: 3 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 2 active, 0 queued | Connections: 2 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 1 active, 0 queued | Connections: 1 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 0 active, 0 queued | Connections: 0 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 9 active, 0 queued | Connections: 9 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 8 active, 0 queued | Connections: 8 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 7 active, 0 queued | Connections: 7 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 6 active, 0 queued | Connections: 6 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 5 active, 0 queued | Connections: 5 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 4 active, 0 queued | Connections: 4 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 3 active, 0 queued | Connections: 3 active, 0 idle]>
----------
<AsyncHTTPProxy [Requests: 2 active, 0 queued | Connections: 2 active, 0 idle]>

The output shows that new connections are always acquired and the connection never gets to the idle state (Again this is only when a proxy is being used).

[rafikahmed]

Just to emphasize more here's the output when no proxy is being used:

<AsyncConnectionPool [Requests: 9 active, 0 queued | Connections: 9 active, 1 idle]>
----------
<AsyncConnectionPool [Requests: 8 active, 0 queued | Connections: 8 active, 2 idle]>
----------
<AsyncConnectionPool [Requests: 7 active, 0 queued | Connections: 7 active, 3 idle]>
----------
<AsyncConnectionPool [Requests: 6 active, 0 queued | Connections: 6 active, 4 idle]>
----------
<AsyncConnectionPool [Requests: 5 active, 0 queued | Connections: 5 active, 4 idle]>
----------
<AsyncConnectionPool [Requests: 4 active, 0 queued | Connections: 4 active, 4 idle]>
----------
<AsyncConnectionPool [Requests: 3 active, 0 queued | Connections: 3 active, 5 idle]>
----------
<AsyncConnectionPool [Requests: 2 active, 0 queued | Connections: 2 active, 5 idle]>

Connections change state from active to idle as expected.