-6

If a scraper starts throwing ReadTimeout and the first instinct is to double every timeout, read the message again. It can point at the wrong place.

We made each phase of a proxied HTTPS request stall on purpose in a local lab (curl 8.5.0, requests 2.34.2, httpx 0.28.1). For a proxy that accepts the connection and never answers CONNECT, curl said "Proxy CONNECT aborted due to timeout". requests raised ReadTimeout with the target's hostname in it, though the target was never contacted. httpx said ReadTimeout too, but only after its read timer, so 8 seconds instead of 5.

The tell in requests: if the number in "(read timeout=N)" equals your connect value, the stall was in the tunnel or TLS, not on the site.

Two more from the same run. An unreachable proxy raises ProxyError, which isn't a subclass of requests.exceptions.Timeout, so code that only catches Timeout lets it crash the worker. And with no timeout set at all, curl and requests both sat there for 134 seconds, which is the Linux kernel's SYN retries and not the client.

curl -w with time_connect, http_connect, time_appconnect and time_starttransfer splits the four phases, and honestly that's the first thing I'd run.

Full table: https://proxymint.com/proxy-timeout/

Disclosure: our blog, we run a proxy service.

top 2 comments
sorted by: hot top new old
[-] onlinepersona@programming.dev 2 points 4 days ago

I don't mind the disclaimer, but I want to know why you checked this. What's the point?

[-] proxymint@programming.dev 2 points 4 days ago

Fair. We run proxies, so "proxy or site?" is the question we care about, and I didn't trust the error messages to answer it.

The point is what you do after. If the proxy hung you rotate it out. If the site hung you slow down on that site. requests blames the site by name either way, so you can end up throttling a site that's fine and keeping a dead proxy. That's why.

this post was submitted on 12 Nov 2026
-6 points (31.2% liked)

Programming

29306 readers
493 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 3 years ago
MODERATORS