Consuming HTTP APIs with requests: Sessions, Auth, Retries & Errors

Almost every Python program you write from here on will talk to something over HTTP. A payment gateway, a weather feed, your own microservice, an LLM endpoint, the GitHub API. And requests makes the first 90% of that so easy that most people never learn the other 10% — until it’s 3 a.m. and a worker has been wedged for six hours on a socket that will never answer.

Here is the line everybody writes:

data = requests.get(url).json()

It works. It also contains three separate production bugs, and this lesson is largely about them: there’s no timeout, so it can hang forever; there’s no status check, so a 500 gets parsed as data; and there’s no Session, so it pays for a fresh TLS handshake every call. None of those show up in your test. All three show up in production.

Everything below targets Python 3.12+ and requests 2.32+. Every output block is real — copied from a 3.12.3 run against a live API or a local stub, not paraphrased. You’ll need one install, in a virtual environment:

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip install requests

---

Why this matters

The network is the only part of your program that is allowed to lie to you slowly. A local function either returns or raises, and it does so in microseconds. A remote call has a third option: it can accept your connection, send you nothing, and hold the socket open until the heat death of the universe. Your thread sits there. Your worker pool drains. Your health check still says “OK” because the process is alive — it’s just not doing anything. That failure mode has no exception, no log line, and no traceback, and you cannot debug it after the fact because nothing was ever written down.

The second thing that makes HTTP different is that failure arrives as data. When your database driver can’t find a row it raises. When an HTTP server can’t find your resource it sends back a perfectly well-formed response that says “404” in a header, and requests — correctly, deliberately — hands it to you without complaint. From Python’s point of view, nothing went wrong: it asked, and it got an answer. The fact that the answer was bad news is your problem to notice. Miss that, and .json() chokes on an HTML error page, and the traceback points at your parsing line instead of the request that actually failed three frames earlier.

So the mental model to carry through this lesson has three parts, and blurring them is where the bugs live:

• A response is not a success. requests raises only when it never got an HTTP answer at all — DNS died, the connection was refused, the timeout fired. If bytes came back in the shape of an HTTP response, that’s a success as far as the library is concerned, whether those bytes say 200 or 503. Status is data you must inspect.
• requests.get() is not a client — it’s a convenience. Every module-level call builds a whole Session, uses it once, and throws it away, taking the connection pool with it. A real client is a Session you keep, holding your headers, your auth, your retry policy and your live sockets.
• Time is a resource you must bound. Connect, read, retry, backoff — each has a clock, and if you don’t set them, one of them is infinite. The single most valuable line in this entire lesson is timeout=.

Get those three straight and the rest is mechanics. Miss them and you’ll ship the same three bugs I did, at roughly the same hour of the night.

---

HTTP in five minutes for the Python developer

You don’t need the RFCs. You need the shape of one exchange. A request goes out, a response comes back, and both are the same four things: a start line, some headers, a blank line, and an optional body.

GET /repos/psf/requests?per_page=3 HTTP/1.1      <- method, path + query, version
Host: api.github.com                             <- headers: metadata about the request
Accept: application/vnd.github+json
Authorization: Bearer ghp_xxx
                                                 <- blank line
                                                 <- body (empty for GET)

HTTP/1.1 200 OK                                  <- version, STATUS CODE, reason
Content-Type: application/json; charset=utf-8    <- headers: metadata about the response
Link: <...?page=2>; rel="next"
X-RateLimit-Remaining: 59

{"full_name": "psf/requests", ...}               <- body: the bit you actually want

That’s it. requests builds the top half and parses the bottom half; everything in this lesson is about controlling one of those lines.

Methods, and the word that decides your retry policy

The method is a verb — what you want done. But the property that matters most in real code isn’t what the verb means, it’s whether it’s idempotent: whether doing it twice has the same effect as doing it once. That single column decides whether you’re allowed to retry.

Safe means “read-only — changes nothing on the server.” Idempotent means “repeatable without extra effect.” PUT /users/7 {"name": "Ada"} sets the name to Ada; do it five times and the name is still Ada. POST /charges {"amount": 500} creates a charge; do it five times and your customer is out ₹2,000 extra and you are on a call with their bank.

This is not academic. It’s wired into the library: as you’ll see later, urllib3’s retry logic refuses to retry POST and PATCH by default, and that default is a feature, not an oversight.

Status codes you will actually handle

There are dozens. You will handle about a dozen. The first digit is the class, and the class alone tells you what kind of problem you have.

Two rows deserve special attention because they cause the most wasted debugging time.

401 vs 403 is the one everybody muddles. 401 is authentication — the server doesn’t know who you are. Your token is missing, malformed, or expired; retrying with the same token gives you another 401 forever, but refreshing it may genuinely fix things. 403 is authorization — the server knows exactly who you are and has decided the answer is no. Retrying a 403 is pure noise. (GitHub muddies this itself: it returns 403 for some rate-limit conditions, which is why you check for a Retry-After header rather than trusting the code alone.)

504 is the dangerous one. A gateway timeout means the proxy gave up waiting — it does not mean the work didn’t happen. Your POST /charges may have gone through and the confirmation got lost on the way back. Retrying that is how you double-charge someone. On a GET a 504 is harmless; on a POST it’s the exact scenario idempotency keys exist for.

Where do the parameters go?

Beginners guess. There’s a rule.

---

requests basics: params, bodies, and reading the response

Never build a URL with string concatenation

This is the first habit to break. It looks harmless:

# ❌ Broken the moment a value contains a space, &, =, /, or a non-ASCII character
url = "https://api.github.com/search/repositories?q=" + query + "&sort=stars"

Pass params= and let requests encode it:

import requests

r = requests.get(
    "http://127.0.0.1:8771/ok",
    params={"q": "a b&c=d", "lang": "en", "tags": ["x", "y"], "none": None, "n": 5},
    timeout=5,
)
print("r.url:", r.url)

r.url: http://127.0.0.1:8771/ok?q=a+b%26c%3Dd&lang=en&tags=x&tags=y&n=5

Read that output carefully, because four separate rules are visible in it:

That third row is a real gotcha: params={"page": page_num} where page_num is accidentally None doesn’t send page=, it sends nothing, and you silently get page 1 forever. And r.url is the single best debugging tool in the library — it’s the URL that was actually sent, after encoding.

json= vs data= vs files=

Three ways to send a body, and picking the wrong one produces a confusing 400 from a server that’s actually behaving perfectly. The difference is entirely about Content-Type and encoding:

Don’t take my word for the encodings — requests will show you exactly what it’s about to send. Request(...).prepare() builds the real thing without sending it, which is the most under-used debugging tool in the library:

import requests

req = requests.Request("POST", "http://example.com/echo", json={"name": "ada", "age": 36}).prepare()
print("method:", req.method, "| body:", req.body)
for k, v in req.headers.items():
    print(f"  {k}: {v}")

method: POST | body: b'{"name": "ada", "age": 36}'
  Content-Length: 26
  Content-Type: application/json

Note the trap in row three of that table: passing a pre-serialised JSON string to data= sends the right bytes with no Content-Type header at all. A strict server sees a body it wasn’t told how to interpret and returns 400 or 415, and you stare at a payload that looks perfect. Use json= and let the library set the header.

Reading the response

The Response object is where beginners reach for .text and then wonder why things break. Here’s the whole surface, verified against a live call:

r = requests.get("http://127.0.0.1:8771/ok", timeout=5)
print("status_code :", r.status_code, type(r.status_code).__name__)
print("json()      :", r.json(), type(r.json()).__name__)
print("headers CT  :", r.headers["content-type"], "| case-insensitive:", r.headers["CoNtEnT-TyPe"])
print("elapsed     :", type(r.elapsed).__name__, f"{r.elapsed.total_seconds():.4f}s")
print("request     :", r.request.method, r.request.url)

status_code : 200 int
json()      : {'ok': True, 'path': '/ok'} dict
headers CT  : application/json | case-insensitive: application/json
elapsed     : timedelta 0.0004s
request     : GET http://127.0.0.1:8771/ok

r.json() is json.loads(r.text) with better encoding detection — if you want the full story on how Python turns JSON into objects and where that bites, that’s Working with Data: JSON, CSV & Serialization.

---

The two lines that separate toy code from production

Everything so far is the easy 90%. This section is the 10% that decides whether your code survives contact with a real network.

⚠️ requests does NOT raise on 404 or 500

This is the single most common requests bug in the wild, and it surprises people because it’s the opposite of what the stdlib does. Watch:

import requests

r = requests.get("http://127.0.0.1:8771/404", timeout=5)
print("got here, no exception. status:", r.status_code, "| r.ok:", r.ok, "| bool(r):", bool(r))

r5 = requests.get("http://127.0.0.1:8771/500", timeout=5)
print("500 -> status:", r5.status_code, "| r.ok:", r5.ok)

got here, no exception. status: 404 | r.ok: False | bool(r): False
500 -> status: 500 | r.ok: False

No exception. No warning. Execution continues. And this is correct behaviour: requests promised to make an HTTP request and give you the response. It did. The server’s answer was “no”, but the answer arrived intact — that’s a successful request with an unsuccessful result, and only your code knows which statuses matter for your use case.

The consequence, though, is brutal. Here’s what actually happens in the wild:

# ❌ The bug, in its natural habitat
data = requests.get(url).json()      # server returned a 500 HTML error page

requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

You now have a JSON error pointing at your parsing line, and you’ll spend twenty minutes checking your JSON handling — when the real problem is that the API returned a 500 and you never looked. The traceback is telling you the truth about the wrong thing.

The fix is one line, raise_for_status():

r = requests.get("http://127.0.0.1:8771/404", timeout=5)
try:
    r.raise_for_status()
except requests.exceptions.HTTPError as e:
    print("HTTPError:", e)
    print("e.response is r:", e.response is r, "| e.request.method:", e.request.method)

# On success it returns None, so it chains cleanly:
print("200 raise_for_status returns:", repr(requests.get("http://127.0.0.1:8771/ok", timeout=5).raise_for_status()))

HTTPError: 404 Client Error: Not Found for url: http://127.0.0.1:8771/404
e.response is r: True | e.request.method: GET
200 raise_for_status returns: None

Three things worth banking. It raises HTTPError for 4xx and 5xx and nothing else. The exception carries .response and .request, so your handler can log the status, the body and the URL that caused it — no need to smuggle the response out separately. And it returns None on success, so r.raise_for_status() on its own line before r.json() reads perfectly.

The exception hierarchy: what to catch

When requests does raise, it raises from one family. This is the real tree, printed from requests.exceptions:

Three of those rows are genuinely surprising, and each one is a bug waiting to happen:

ConnectTimeout is both a ConnectionError and a Timeout. ReadTimeout is only a Timeout. So except ConnectionError: catches connect timeouts but sails straight past read timeouts:

from requests.exceptions import Timeout, ConnectionError
# ReadTimeout after a 2s read limit:
#   isinstance(e, Timeout): True | isinstance(e, ConnectionError): False
# ConnectTimeout to an unroutable IP:
#   isinstance(e, ConnectionError): True | isinstance(e, Timeout): True

SSLError is a ConnectionError. So the reflexive except ConnectionError: retry() cheerfully retries an expired certificate five times with exponential backoff. It will never succeed. It is not a transient network blip — it’s a configuration error, and it should fail loudly on the first attempt.

RequestException inherits from OSError. So a broad except OSError: around file handling will quietly swallow your HTTP errors too.

The practical policy is short:

That last row generalises a rule you’ve met before: catch the narrowest class that’s true, and give your callers one name of your own to catch. If exception design is fuzzy, Exceptions In Depth is the companion piece — everything there about raise ... from e applies directly to wrapping RequestException in an app-level error.

⚠️ ALWAYS set a timeout

If you take one line from this lesson, take this one.

requests has no default timeout. Not a long one — none. Omit the argument and there is no clock at all: a server that accepts your connection and then says nothing will hold your thread until the OS gives up, which on Linux can be a couple of hours, or forever if TCP keepalives are off. Your program isn’t crashed, so nothing restarts it. It’s just gone.

import time, requests

t = time.perf_counter()
try:
    requests.get("http://127.0.0.1:8771/slow", timeout=2)   # server sleeps 5s
except requests.exceptions.ReadTimeout as e:
    print(f"ReadTimeout after {time.perf_counter()-t:.2f}s")
    print("  msg:", e)

ReadTimeout after 2.01s
  msg: HTTPConnectionPool(host='127.0.0.1', port=8771): Read timed out. (read timeout=2)

Bounded, predictable, and loggable. Now the part almost nobody knows — timeout takes a tuple:

t = time.perf_counter()
try:
    requests.get("http://127.0.0.1:8771/slow", timeout=(3.05, 1.5))
except requests.exceptions.ReadTimeout:
    print(f"ReadTimeout after {time.perf_counter()-t:.2f}s (read leg = 1.5s)")

ReadTimeout after 1.51s (read leg = 1.5s)

Why split them? Because they fail for completely different reasons. Connect should be fast — it’s a TCP handshake to a machine that’s either there or isn’t; 3 seconds is generous. Read depends on how long the server legitimately needs to think, which might be 30 seconds for a report. One number forces you to set both to the slow one, so a dead host takes 30 seconds to notice instead of 3.

The 3.05 is a real convention, not superstition: it’s a hair over a multiple of 3, because TCP retransmits SYN packets on a 3-second schedule, so a value just above the boundary avoids cutting a legitimate retransmit off at the knees.

⚠️ The most important caveat: timeout is not a deadline for the whole request. The read timeout is the maximum gap between bytes. A server that dribbles one byte every 9 seconds will keep a timeout=(3.05, 10) call alive indefinitely, and a large download legitimately takes longer than the read timeout without ever tripping it. If you need a true wall-clock ceiling, you need something outside requests — a signal alarm, a worker-level deadline, or a supervising task.

---

Session, adapters and retries: building the production client

You now know what one call should look like. This is where it becomes a client. Before the code, here’s the whole path your request actually takes — every box below is a real object you can configure:

Follow it left to right and the design falls out. Your call is just an intent; the Session attaches identity (headers, auth, cookies) and a live socket; the HTTPAdapter owns the clock and the retry loop, which is why retries happen invisibly below your code; and on the way back, the response passes a status gate before anything tries to parse it. The two red badges are the traps: no timeout on the way out, no status check on the way in.

What a Session actually buys you

Two things, and the second one is measurable.

The first is shared state. Set a header once and every request carries it:

session = requests.Session()
session.headers.update({
    "Accept": "application/vnd.github+json",
    "User-Agent": "kloudvin-zero-to-hero/1.0",
})
session.headers["Authorization"] = f"Bearer {token}"    # set once, sent every time

The second is connection pooling, and to see why it matters, look at what a module-level requests.get() actually does. Straight from the library’s own source:

import inspect, requests.api
print(inspect.getsource(requests.api.request))

    ...
    # By using the 'with' statement we are sure the session is closed, thus we
    with sessions.Session() as session:
        return session.request(method=method, url=url, **kwargs)

Every single requests.get() builds a brand-new Session, uses it once, and closes it — destroying the connection pool along with it. So every call pays for a fresh DNS lookup, TCP handshake and full TLS negotiation. Over HTTPS that’s multiple round-trips before a single byte of your actual request goes out.

Let’s measure it. GitHub’s /rate_limit endpoint doesn’t count against your rate limit, so it’s fair game:

import requests, time

URL = "https://api.github.com/rate_limit"
N = 10

def no_session():
    t = time.perf_counter()
    for _ in range(N):
        requests.get(URL, timeout=10)          # new TCP + TLS handshake EVERY time
    return time.perf_counter() - t

def with_session():
    t = time.perf_counter()
    with requests.Session() as s:
        for _ in range(N):
            s.get(URL, timeout=10)             # one handshake, then keep-alive
    return time.perf_counter() - t

requests.get(URL, timeout=10)                  # warm DNS so we measure handshakes
a, b = no_session(), with_session()
print(f"{N} requests, fresh connection each : {a:.2f}s  ({a/N*1000:.0f} ms/req)")
print(f"{N} requests, one Session           : {b:.2f}s  ({b/N*1000:.0f} ms/req)")
print(f"Session is {a/b:.1f}x faster  ({(1-b/a)*100:.0f}% less wall time)")

10 requests, fresh connection each : 1.19s  (119 ms/req)
10 requests, one Session           : 0.41s  (41 ms/req)
Session is 2.9x faster  (65% less wall time)

2.9x, from deleting nothing and adding one object. Your exact numbers will differ with distance to the server — that’s the point: the further away and the more TLS round-trips, the bigger the win. Two thirds of the wall time in the naive version was handshakes.

⚠️ A Session holds sockets, so close it: use with requests.Session() as s: or call s.close(). In a class, keep one Session for the object’s lifetime — that’s the whole point — and expose a close().

Retries: HTTPAdapter + urllib3.Retry

Networks fail transiently. The fix isn’t a for loop in your code — it’s a policy mounted on the Session, so retries happen below your code and every call gets them for free.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=4,                                            # attempts after the first
    backoff_factor=1,                                   # ⚠️ DEFAULT IS 0 = no sleep!
    status_forcelist=(429, 500, 502, 503, 504),         # ⚠️ default is none
    allowed_methods={"GET", "HEAD", "OPTIONS"},         # safe verbs only
    respect_retry_after_header=True,                    # default True — honour 429
)
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))

Here it is against a stub that returns 503, 503, then 200:

http://127.0.0.1:8771 "GET /flaky HTTP/1.1" 503 23
Incremented Retry for (url='/flaky'): Retry(total=3, connect=None, read=None, redirect=None, status=None)
Retry: /flaky
http://127.0.0.1:8771 "GET /flaky HTTP/1.1" 503 23
Incremented Retry for (url='/flaky'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
Retry: /flaky
http://127.0.0.1:8771 "GET /flaky HTTP/1.1" 200 14
>>> RESULT 200 {'attempt': 3} after 2.01s wall

Your code called session.get() once and got a 200. The two failures never surfaced.

Two defaults there are actively dangerous. backoff_factor=0 means the default Retry retries instantly, ten times — you’ve built a tiny DDoS against a server that’s already struggling. And status_forcelist=None means a bare Retry(total=5) doesn’t retry 503s at all; it only retries connection-level errors. A Retry with neither argument set is almost never what you want.

How the backoff is actually scheduled

The formula is backoff_factor * (2 ** (retries_so_far - 1)), and it has a wrinkle that surprises everyone:

The first retry never sleeps. That’s deliberate — most transient blips are instantaneous, so an immediate retry usually just works, and only if the second attempt also fails does it conclude something’s actually wrong and start backing off. It also explains the measured 2.01s above: two retries meant 0s + 2s, not 2s + 4s.

Jitter matters the moment you have more than one machine. If fifty workers all hit a 503 at the same instant, they all back off 2s, and all fifty retry at the same instant — a thundering herd that re-kills the server just as it recovers. backoff_jitter=0.3 smears them randomly across a window. (This is urllib3 2.x; on 1.26 you don’t get it, and you also had method_whitelist instead of allowed_methods — that name was removed in 2.0 and now raises TypeError: Retry.__init__() got an unexpected keyword argument 'method_whitelist'.)

Retry only what is safe to retry

Look again at the allowed_methods default:

DEFAULT_ALLOWED_METHODS: ['DELETE', 'GET', 'HEAD', 'OPTIONS', 'PUT', 'TRACE']

That’s exactly the idempotent set from the first table. POST and PATCH are absent on purpose. Widen it to include POST and you have built a machine that occasionally charges customers twice: your POST reaches the server, the server processes it, the response is lost to a read timeout, and Retry — unable to know the difference — sends it again.

If you must retry a POST, the server has to help you: send an idempotency key (a UUID you generate per logical operation, in a header like Idempotency-Key), which lets the server recognise the duplicate and return the original result instead of doing the work twice. That’s a server-side contract — Stripe and most payment APIs offer it. Without it, don’t retry POST.

429 and Retry-After: the one you must not ignore

429 Too Many Requests is the server politely telling you to slow down. Ignore it and hammer away, and the polite 429 becomes a firewall block on your IP or a revoked key.

Usually it comes with a Retry-After header, which is not a suggestion — it’s the answer to “when may I come back?” in either seconds (Retry-After: 2) or an HTTP date. The good news: Retry honours it automatically, and it overrides your backoff schedule:

import time, requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

s = requests.Session()
s.mount("http://", HTTPAdapter(max_retries=Retry(
    total=2, backoff_factor=0.1, status_forcelist=[429, 503],
    allowed_methods={"GET"}, respect_retry_after_header=True)))

t = time.perf_counter()
try:
    r = s.get("http://127.0.0.1:8771/429", timeout=5)      # always 429, Retry-After: 2
except requests.exceptions.RetryError as e:
    print(f"RetryError after {time.perf_counter() - t:.2f}s")
    print("  msg:", str(e)[:150])

RetryError after 4.02s  (2 retries x Retry-After 2s = ~4s, NOT backoff_factor)
  msg: HTTPConnectionPool(host='127.0.0.1', port=8771): Max retries exceeded with url: /429 (Caused by ResponseError('too many 429 error responses'))

backoff_factor=0.1 would have slept ~0.2s; the server said 2, so it waited 2 — twice. That’s the contract being honoured.

The mature move is proactive: read X-RateLimit-Remaining on every response and slow down as it approaches zero, rather than sprinting into a wall and being told off. And note urllib3 only applies Retry-After to statuses {413, 429, 503} — the codes where it’s actually specified.

If you’d rather handle it yourself (necessary when the API signals rate limits with a 403, as GitHub sometimes does):

r = session.get(url, timeout=(3.05, 10))
if r.status_code == 429:
    wait = int(r.headers.get("Retry-After", "1"))
    time.sleep(wait)
    r = session.get(url, timeout=(3.05, 10))

---

Auth, secrets and TLS

The three schemes you’ll meet

Basic auth deserves a demonstration, because people assume base64 means “encoded” means “safe”:

import requests, base64
from requests.auth import HTTPBasicAuth

req = requests.Request("GET", "http://example.com/x", auth=HTTPBasicAuth("ada", "hunter2")).prepare()
h = req.headers["Authorization"]
print("  header:", h)
print("  decoded:", base64.b64decode(h.split()[1]).decode())

  header: Basic YWRhOmh1bnRlcjI=
  decoded: ada:hunter2

Base64 is an encoding, reversible by anyone in one line. Basic auth over plain http:// is your password in cleartext on the wire. Over https:// it’s fine, because TLS is doing the actual protecting. (auth=("ada", "hunter2") is shorthand for exactly the same thing — the tuple form produces a byte-identical header.)

⚠️ Never put a secret in a URL

req = requests.Request("GET", "http://api.example.com/ok", params={"api_key": "SECRET123"}).prepare()
print(req.url)

http://api.example.com/ok?api_key=SECRET123   <- this lands in every access log

That URL gets written to the server’s access log, your proxy’s log, the CDN’s log, Referer headers on any outbound link, browser history, and your own logging.debug(r.url). Headers are logged nowhere by default. Secrets go in headers. Always.

Where the token comes from

import os

token = os.environ.get("GITHUB_TOKEN")          # ✅ .get() -> None, not KeyError
if token:
    session.headers["Authorization"] = f"Bearer {token}"

Use .get() so the client degrades to unauthenticated instead of crashing — and note the asymmetry it buys you on GitHub: 60 requests/hour anonymous, 5,000/hour with a token.

⚠️ If a secret ever lands in git, deleting the line does not help. It’s in the history, on every clone, in every fork. The only fix is to rotate the credential.

TLS verification

requests.get("https://expired.badssl.com/", timeout=10)

requests.exceptions.SSLError: HTTPSConnectionPool(host='expired.badssl.com', port=443): Max retries
exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: certificate has expired ...')))

That is the system working. The certificate is bad and requests refused to talk. And then someone pastes the “fix” from Stack Overflow:

r = requests.get("https://expired.badssl.com/", verify=False, timeout=10)   # ❌

  status: 200 (cert error ignored!)
  WARNING: InsecureRequestWarning - Unverified HTTPS request is being made to host 'expired.badssl.com'.
  Adding certificate verification is strongly advised.

It “works” — you have just disabled the only thing standing between you and a man-in-the-middle. You’re still encrypted, but you have no idea who you’re encrypted to, which is the entire point of TLS. ⚠️ verify=False in production is a security incident waiting to be written up.

The right fix for a corporate proxy is to point at the CA, not to switch checking off:

session.verify = "/etc/ssl/certs/corporate-ca.pem"
# or: export REQUESTS_CA_BUNDLE=/etc/ssl/certs/corporate-ca.pem

Worth knowing: requests uses the certifi bundle it ships with, not your OS trust store. That’s why requests works on a fresh macOS python.org install while urllib.request fails with CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate — the stdlib looks for an OpenSSL cert file that doesn’t exist until you run Install Certificates.command. Different bundle, different outcome, same machine.

Proxies, briefly

proxies = {"http": "http://proxy.corp:3128", "https": "http://proxy.corp:3128"}
r = requests.get(url, proxies=proxies, timeout=(3.05, 10))

requests also reads HTTP_PROXY / HTTPS_PROXY / NO_PROXY from the environment automatically — which is a common “why is it hitting a proxy in prod?” surprise. Disable that with session.trust_env = False.

---

Pagination and streaming

Pagination: don’t fetch what you don’t need

APIs won’t hand you 40,000 issues in one response. They page. There are three common styles:

GitHub uses the Link header, and requests parses it into r.links for free:

r = s.get("https://api.github.com/repos/psf/requests/issues",
          params={"per_page": 3, "state": "closed"}, timeout=10)
print(r.headers.get("Link"))
print(r.links)

<https://api.github.com/repositories/1362490/issues?per_page=3&state=closed&after=Y3Vyc29yOnYyOpLPAAABn0sEzlDPAAAAASFFQvc%3D&page=2>; rel="next"

{'next': {'url': 'https://api.github.com/repositories/1362490/issues?per_page=3&state=closed&after=Y3Vyc29yOnYyOpLPAAABn0sEzlDPAAAAASFFQvc%3D&page=2', 'rel': 'next'}}

Look closely at that next URL: it carries an opaque after= cursor. This is exactly why you follow the server’s link instead of doing page += 1 yourself — the server is encoding state you can’t reconstruct.

The right shape for this is a generator. It yields items one at a time and only fetches the next page when the consumer actually asks, so islice(pages, 5) makes one HTTP call, not forty:

from collections.abc import Iterator
from typing import Any

def paginate(self, path: str, per_page: int = 30, **params: Any) -> Iterator[dict[str, Any]]:
    """Yield items across ALL pages, lazily - one page in memory at a time."""
    url: str | None = f"{self.base_url}{path}"
    query: dict[str, Any] | None = {**params, "per_page": per_page}
    while url:
        r = self._get(url, params=query)
        yield from r.json()                          # flatten this page's items
        url = r.links.get("next", {}).get("url")     # None on the last page -> loop ends
        query = None                                 # ⚠️ the `next` URL already has the cursor

Three details make this work. yield from flattens the page list so callers see a flat stream of items and never think about pages. r.links.get("next", {}).get("url") returns None on the last page, which ends the while — no counter, no off-by-one. And query = None after the first call is the subtle one: the next URL already contains per_page and the cursor, so passing your original params again would override the cursor and loop on page 2 forever.

The payoff is that the caller composes it with anything from itertools, and the laziness is real — this is the generator machinery from Iterators & Generators doing exactly what it’s for.

Streaming: don’t put a 2 GB file in RAM

By default requests downloads the entire body into memory before returning. Fine for a JSON doc, fatal for a big file. stream=True returns as soon as the headers land:

with requests.get(url, stream=True, timeout=(3.05, 30)) as r:
    r.raise_for_status()
    with open("big.iso", "wb") as f:
        for chunk in r.iter_content(chunk_size=64 * 1024):     # 64 KB at a time
            f.write(chunk)

streamed 5.0 MB in 64 KB chunks; peak chunk held = 64 KB

⚠️ Touching r.text or r.content on a streamed response pulls the whole body into memory anyway — a very common accidental undo, often via a stray print(r.text) in a debug line. And an un-consumed stream=True response holds its connection out of the pool until it’s closed, so always use with.

---

Choosing a client: requests vs httpx vs urllib

Three honest notes. httpx has a 5-second default timeout — the single best argument for it, since its failure mode is “too aggressive” rather than “hangs forever.” urllib raises on 404 automatically, the exact opposite of requests, which is precisely why the requests behaviour surprises people coming from the stdlib:

urllib.request.urlopen(Request("https://api.github.com/repos/psf/nope-xyz", ...))

  HTTPError: 404 Not Found  <- RAISED automatically
  requests, same URL: returns a 404 Response object, raises nothing.

And requests is in maintenance mode by design — it’s stable, not abandoned. It isn’t getting async or HTTP/2, and that’s the deal: it does one thing, hasn’t broken your code in a decade, and is installed on approximately every machine on earth. Use httpx when you need async/HTTP/2; use requests otherwise; use urllib when you cannot add a dependency.

---

Hands-on lab

You’ll build a real GitHub API client: a Session with default headers and a bolted-on default timeout, raise_for_status wrapped into one app error, a Retry adapter with backoff, a lazy pagination generator, a token from the environment, and a pytest suite that never touches the network.

⚠️ Uses only public, unauthenticated GitHub endpoints (60 requests/hour). No paid API, no account needed. A token is optional and only raises the limit.

Step 1 — Set up

mkdir gh-client && cd gh-client
python3 -m venv .venv
source .venv/bin/activate            # Windows: .venv\Scripts\activate
python -m pip install requests pytest responses

What just happened: an isolated environment with requests, pytest, and responses (the standard library for mocking requests in tests).

Step 2 — See the trap for yourself

# trap.py
import requests

r = requests.get("https://api.github.com/repos/psf/definitely-not-a-real-repo-xyz")
print("status:", r.status_code, "| no exception was raised")
print("body  :", r.text[:80])
r.raise_for_status()          # NOW it raises

status: 404 | no exception was raised
body  : {"message":"Not Found","documentation_url":"https://docs.github.com/rest/repos/re
Traceback (most recent call last):
  File "trap.py", line 7, in <module>
    r.raise_for_status()
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.github.com/repos/psf/definitely-not-a-real-repo-xyz

What just happened: the 404 sailed through silently. Only raise_for_status() turned it into an error. Note there’s no timeout= on that line either — bug number two, in a script you just wrote.

Step 3 — A server that misbehaves on demand

# flaky_server.py
"""A deliberately awful server: /flaky fails twice then works, /slow never answers in time."""
import json, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

HITS = {"n": 0}

class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    def log_message(self, *args): pass

    def do_GET(self):
        if self.path == "/slow":
            time.sleep(30)                      # you will never see this response
            body = b'{"never": true}'
        elif self.path == "/flaky":
            HITS["n"] += 1
            if HITS["n"] <= 2:
                return self._reply(503, b'{"error": "try again"}')
            body = json.dumps({"ok": True, "attempts": HITS["n"]}).encode()
        else:
            return self._reply(404, b'{"error": "no route"}')
        self._reply(200, body)

    def _reply(self, code, body):
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    print("stub server on http://127.0.0.1:8099  (Ctrl-C to stop)")
    ThreadingHTTPServer(("127.0.0.1", 8099), Handler).serve_forever()

Run it in a second terminal (same venv):

python flaky_server.py

What just happened: a local stub with reproducible failure. Real APIs are too polite to fail on cue — and public sandboxes like httpbin.org are themselves flaky (it returned 502s and read timeouts while this lesson was being written), which is exactly why a deterministic local server beats a public one for teaching failure.

Step 4 — Watch a timeout and a backoff, with real timings

# backoff_demo.py
import time, requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

BASE = "http://127.0.0.1:8099"

print("1. No timeout is a HANG. With one, you fail fast and predictably.")
t = time.perf_counter()
try:
    requests.get(f"{BASE}/slow", timeout=(3.05, 2))
except requests.exceptions.ReadTimeout:
    print(f"   ReadTimeout after {time.perf_counter() - t:.2f}s  <- bounded, as designed\n")

print("2. Retry with backoff against /flaky (503, 503, then 200):")
session = requests.Session()
session.mount("http://", HTTPAdapter(max_retries=Retry(
    total=4, backoff_factor=1, status_forcelist=(429, 500, 502, 503, 504),
    allowed_methods={"GET"},
)))
t = time.perf_counter()
r = session.get(f"{BASE}/flaky", timeout=(3.05, 5))
print(f"   -> {r.status_code} {r.json()} after {time.perf_counter() - t:.2f}s")
print("   sleeps were 0s then 2s: backoff_factor * 2**(retry-1), first retry never sleeps")

1. No timeout is a HANG. With one, you fail fast and predictably.
   ReadTimeout after 2.01s  <- bounded, as designed

2. Retry with backoff against /flaky (503, 503, then 200):
   -> 200 {'ok': True, 'attempts': 3} after 2.02s
   sleeps were 0s then 2s: backoff_factor * 2**(retry-1), first retry never sleeps

What just happened: the /slow endpoint sleeps 30s; your 2s read timeout cut it off at 2.01s. Then /flaky failed twice and your code never knew — one session.get(), one 200, attempts: 3, and a 2.02s wall time that is exactly 0s + 2s. ⚠️ Delete the timeout= from line 1 and that script hangs for 30 seconds. On a real dead host, forever.

⚠️ The /flaky counter lives in the server process, so a second run gives you attempts: 4 in 0.00s with no retries — it’s already past its two failures. Restart flaky_server.py between runs to see the backoff again. (Stateful test doubles surprising you on the second run is a preview of every integration-test suite you will ever maintain.)

Step 5 — The client

# gh_client.py
"""A small, production-shaped GitHub API client."""
from __future__ import annotations

import os
from collections.abc import Iterator
from typing import Any

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

API_ROOT = "https://api.github.com"
DEFAULT_TIMEOUT = (3.05, 10)          # (connect, read) - ALWAYS set one


class GitHubError(RuntimeError):
    """One name for callers to catch."""


class TimeoutHTTPAdapter(HTTPAdapter):
    """requests.Session has NO default timeout. This bolts one on."""

    def __init__(self, *args: Any, timeout: Any = DEFAULT_TIMEOUT, **kwargs: Any) -> None:
        self._timeout = timeout
        super().__init__(*args, **kwargs)

    def send(self, request, **kwargs):                      # type: ignore[override]
        if kwargs.get("timeout") is None:
            kwargs["timeout"] = self._timeout
        return super().send(request, **kwargs)


class GitHubClient:
    def __init__(self, token: str | None = None, base_url: str = API_ROOT,
                 timeout: Any = DEFAULT_TIMEOUT) -> None:
        self.base_url = base_url.rstrip("/")
        self.session = requests.Session()
        self.session.headers.update({
            "Accept": "application/vnd.github+json",
            "X-GitHub-Api-Version": "2022-11-28",
            "User-Agent": "kloudvin-zero-to-hero/1.0",
        })
        token = token or os.environ.get("GITHUB_TOKEN")     # NEVER hardcode
        if token:
            self.session.headers["Authorization"] = f"Bearer {token}"

        retry = Retry(
            total=4,
            backoff_factor=0.5,
            backoff_jitter=0.3,                             # urllib3 2.x
            status_forcelist=(429, 500, 502, 503, 504),
            allowed_methods=frozenset({"GET", "HEAD", "OPTIONS"}),   # safe verbs only
            respect_retry_after_header=True,
            raise_on_status=False,
        )
        adapter = TimeoutHTTPAdapter(max_retries=retry, timeout=timeout, pool_maxsize=10)
        self.session.mount("https://", adapter)
        self.session.mount("http://", adapter)

    def _get(self, url: str, params: dict[str, Any] | None = None) -> requests.Response:
        if not url.startswith("http"):
            url = f"{self.base_url}{url}"
        try:
            r = self.session.get(url, params=params)
            r.raise_for_status()
        except requests.exceptions.HTTPError as e:
            raise GitHubError(f"{e.response.status_code} for {url}") from e
        except requests.exceptions.RequestException as e:
            raise GitHubError(f"network failure for {url}: {type(e).__name__}") from e
        return r

    def repo(self, owner: str, name: str) -> dict[str, Any]:
        return self._get(f"/repos/{owner}/{name}").json()

    def paginate(self, path: str, per_page: int = 30, **params: Any) -> Iterator[dict[str, Any]]:
        """Yield items across ALL pages, lazily - one page in memory at a time."""
        url: str | None = f"{self.base_url}{path}"
        query: dict[str, Any] | None = {**params, "per_page": per_page}
        while url:
            r = self._get(url, params=query)
            yield from r.json()
            url = r.links.get("next", {}).get("url")
            query = None          # the `next` URL already carries the cursor

    def rate_limit(self) -> dict[str, Any]:
        return self._get("/rate_limit").json()["resources"]["core"]

    def close(self) -> None:
        self.session.close()

    def __enter__(self): return self
    def __exit__(self, *exc): self.close()

What just happened: every rule in this lesson, in 60 lines. Note the except ordering — HTTPError before RequestException, because it’s a subclass and Python takes the first match. And raise ... from e keeps the original traceback while giving callers one name. The __enter__/__exit__ pair makes it a context manager, so the Session always gets closed; that’s ordinary classes and objects work.

Step 6 — Run it against the real API

# demo.py
import itertools
from gh_client import GitHubClient, GitHubError

with GitHubClient() as gh:
    r = gh.repo("psf", "requests")
    print(f"{r['full_name']}: {r['stargazers_count']:,} stars, {r['open_issues_count']} open issues")

    print("\nFirst 7 closed issues (lazy, across pages of 3):")
    for i in itertools.islice(gh.paginate("/repos/psf/requests/issues", per_page=3, state="closed"), 7):
        print(f"  #{i['number']:<6} {i['title'][:48]}")

    print("\nRate limit:", gh.rate_limit())

    try:
        gh.repo("psf", "definitely-not-real-xyz")
    except GitHubError as e:
        print("\nGitHubError:", e)
        print("  __cause__:", type(e.__cause__).__name__)

psf/requests: 54,140 stars, 233 open issues

First 7 closed issues (lazy, across pages of 3):
  #7581   Fix case-insensitive JSON content type encoding
  #7578   Add snippet-level OSS license compliance audit a
  #7577   Add query method newly introduced in RFC 10008
  #7576   Fix link to AI Policy in CONTRIBUTING.md
  #7575   Support for QUERY http method
  #7573   session.prepare_request: carry a custom CookiePo
  #7572   Add query method http
Rate limit: {'limit': 60, 'remaining': 54, 'reset': 1784128296, 'used': 6}

GitHubError: 404 for https://api.github.com/repos/psf/definitely-not-real-xyz
  __cause__: HTTPError

⚠️ Scope the promise: this is a live API, so the star count, issue numbers and titles will differ when you run it — that’s what “live” means. What must match exactly is the shape: three pages fetched for 7 items at per_page=3, limit: 60 unauthenticated, and a GitHubError whose __cause__ is HTTPError.

What just happened: islice(..., 7) pulled 7 items and stopped — the generator made exactly 3 HTTP calls, not one per issue and not all 200+ pages. The 404 came back as your error class with the original HTTPError preserved as __cause__.

Step 7 — Tests that never touch the network

Tests that hit a real API are slow, flaky, rate-limited, and fail on a plane. Mock the HTTP layer:

# test_gh_client.py
"""Tests that NEVER touch the network. Fast, offline, deterministic."""
import pytest
import responses

from gh_client import GitHubClient, GitHubError

API = "https://api.github.com"


@pytest.fixture
def gh():
    with GitHubClient(token="fake-token-for-tests") as c:
        yield c


@responses.activate
def test_repo_returns_parsed_json(gh):
    responses.get(f"{API}/repos/psf/requests",
                  json={"full_name": "psf/requests", "stargazers_count": 54000})
    assert gh.repo("psf", "requests")["stargazers_count"] == 54000


@responses.activate
def test_token_is_sent_as_bearer_header(gh):
    responses.get(f"{API}/rate_limit", json={"resources": {"core": {"remaining": 59}}})
    gh.rate_limit()
    assert responses.calls[0].request.headers["Authorization"] == "Bearer fake-token-for-tests"


@responses.activate
def test_404_becomes_GitHubError_chained_from_HTTPError(gh):
    responses.get(f"{API}/repos/psf/nope", json={"message": "Not Found"}, status=404)
    with pytest.raises(GitHubError, match="404") as exc:
        gh.repo("psf", "nope")
    assert exc.value.__cause__ is not None          # original error preserved


@responses.activate
def test_paginate_follows_link_header_and_is_lazy(gh):
    page2 = f"{API}/repos/o/r/issues?page=2"
    responses.get(f"{API}/repos/o/r/issues", json=[{"number": 1}, {"number": 2}],
                  headers={"Link": f'<{page2}>; rel="next"'})
    responses.get(page2, json=[{"number": 3}])

    gen = gh.paginate("/repos/o/r/issues", per_page=2)
    assert next(gen)["number"] == 1
    assert len(responses.calls) == 1               # lazy: page 2 NOT fetched yet
    assert [i["number"] for i in gen] == [2, 3]
    assert len(responses.calls) == 2


@responses.activate
def test_retries_on_503_then_succeeds(gh):
    responses.get(f"{API}/rate_limit", json={"e": 1}, status=503)
    responses.get(f"{API}/rate_limit", json={"resources": {"core": {"remaining": 7}}})
    assert gh.rate_limit()["remaining"] == 7
    assert len(responses.calls) == 2               # it really did retry

python -m pytest test_gh_client.py -q

.....                                                                    [100%]
5 passed in 0.33s

What just happened: five real behaviours — parsing, auth headers, error mapping, lazy pagination, and retry — verified in 0.33 seconds with the network unplugged. Note test_paginate_follows_link_header_and_is_lazy: it asserts on len(responses.calls) mid-generator to prove the laziness is real, not just claimed. And test_retries_on_503_then_succeeds registers two responses for one logical call, so the second assert proves the adapter retried.

⚠️ Mocking asserts what you think the API returns. Pair it with one contract test against the real API in CI, so a breaking change upstream doesn’t pass a green suite. If you can’t add responses, unittest.mock does the same job with more boilerplate:

from unittest.mock import patch, Mock

def test_with_unittest_mock():
    fake = Mock(spec=requests.Response)
    fake.status_code, fake.links = 200, {}
    fake.json.return_value = {"full_name": "psf/requests", "stargazers_count": 54000}
    fake.raise_for_status.return_value = None
    with patch.object(requests.Session, "get", return_value=fake):
        with GitHubClient(token="t") as gh:
            assert gh.repo("psf", "requests")["stargazers_count"] == 54000

---

Common mistakes and troubleshooting

Three of these are worth more than a table row.

The timeout is the one that will actually page you. Every other bug here produces an exception, a log line, a traceback — something to grep for at 3 a.m. The missing timeout produces silence. The thread is alive, the process is healthy, the metrics look fine, and the work simply stops. It’s the only failure in this lesson with no evidence. And it’s the easiest to fix: one keyword argument, or one TimeoutHTTPAdapter subclass so you can never forget it again.

The mojibake trap is a spec quirk, not a bug. When a server sends text/* with no charset, RFC 2616 says to assume ISO-8859-1, and requests obeys — even though in 2026 the body is almost certainly UTF-8:

r = requests.get(url)              # server: Content-Type: text/html, body is UTF-8
print(r.headers['content-type'], "| r.encoding:", r.encoding)
print("r.text       :", repr(r.text))
print("apparent     :", r.apparent_encoding)
r.encoding = "utf-8"               # override, then re-read .text
print("after fix    :", repr(r.text))

text/html | r.encoding: ISO-8859-1
r.text       : '{"city": "München", "who": "café"}'
apparent     : utf-8
after fix    : '{"city": "München", "who": "café"}'

r.encoding is writable, and .text is re-decoded on access. The tell is apparent_encoding (chardet’s guess from the actual bytes) disagreeing with encoding. Note .json() sidesteps this entirely — it does its own UTF-8 detection per the JSON spec — which is another quiet argument for .json() over json.loads(r.text).

The redirect method-change is a real security and correctness trap. A 301 on a POST is permitted to become a GET — and the body just evaporates:

r = requests.post(f"{B}/r301", json={"a": 1}, timeout=5)
print("301 ->", r.json(), "| history:", [h.status_code for h in r.history], "| final url:", r.url)
r = requests.post(f"{B}/r307", json={"a": 1}, timeout=5)
print("307 ->", r.json(), "| history:", [h.status_code for h in r.history])

301 -> {'method': 'GET', 'auth': None} | history: [301] | final url: http://127.0.0.1:8772/landed
307 -> {'method': 'POST'} | history: [307]

Your POST arrived as a GET with no body, the server did nothing, and you got a cheerful 200. r.history is the evidence — a non-empty history means you were redirected, and it’s the first thing to print when a write “succeeds” but nothing changes. 307/308 exist precisely to forbid this.

Notice 'auth': None in that output too. That’s requests stripping your Authorization header because the redirect crossed to a different host — deliberate, and a good thing: without it, any server that can redirect you can harvest your token. It happens even for a header you set per-request, and it means an unexplained 401-after-redirect is usually the library protecting you from a URL you got wrong.

---

Cheat-sheet

---

Interview and exam questions

Q: Does requests.get() raise an exception on a 404 or a 500? A: No — and this is the most common requests bug. If an HTTP response came back at all, the request succeeded from the library’s point of view; the status is just data on that response. requests raises only when there’s no HTTP answer: DNS failure, refused connection, timeout (ConnectionError, Timeout). You opt into status checking with r.raise_for_status(), which raises HTTPError for 4xx/5xx and returns None otherwise. Skip it and .json() tries to parse an HTML error page, giving you JSONDecodeError: Expecting value: line 1 column 1 (char 0) pointing at your parsing line instead of the request that actually failed. (The stdlib’s urllib does the opposite — it raises HTTPError on 404 automatically — which is why this surprises people.)

Q: What happens if you don’t pass timeout? A: The request can hang forever — requests has no default timeout. A server that accepts the connection and then goes silent holds your thread until the OS gives up, which may be hours or never. The failure has no exception and no traceback: the process stays alive and healthy-looking while doing nothing, so nothing restarts it. Always pass timeout=(connect, read). ⚠️ And know its limit: it’s not a deadline for the whole call — the read timeout is the maximum gap between bytes, so a slow dribble of data can keep a call alive indefinitely. A true wall-clock ceiling needs something outside requests.

Q: What does a Session actually give you, and can you quantify it? A: Two things. Shared state — headers, cookies, auth and a mounted retry policy set once, applied to every call. And connection pooling: module-level requests.get() literally builds a new Session per call (with sessions.Session() as session: in requests/api.py) and closes it, destroying the pool, so every call repeats the DNS + TCP + TLS handshake. In the measured run in this lesson, 10 HTTPS calls took 1.19s without a Session and 0.41s with one — 2.9x faster, 65% less wall time. The gain scales with RTT to the server.

Q: Why does urllib3’s Retry exclude POST from allowed_methods by default? A: Because POST isn’t idempotent — doing it twice has a different effect than doing it once. The default is {GET, HEAD, PUT, DELETE, OPTIONS, TRACE}, precisely the idempotent verbs. The dangerous scenario: your POST reaches the server, the server processes it, and the response is lost to a read timeout. The retry layer can’t distinguish “never arrived” from “arrived, reply lost”, so it sends it again and your customer is charged twice. To retry a POST safely you need server cooperation — an Idempotency-Key header the server uses to deduplicate.

Q: With backoff_factor=1 and total=4, what are the sleeps? A: 0s, 2s, 4s, 8s — total 14s. The formula is backoff_factor * (2 ** (retries_so_far - 1)), and the first retry never sleeps, which surprises everyone. That’s intentional: most blips are instantaneous, so an immediate retry usually works, and backoff only kicks in once a second attempt has also failed. ⚠️ Two traps: backoff_factor defaults to 0, so a bare Retry(total=5) retries instantly five times — a hot loop against a struggling server. And status_forcelist defaults to None, so it won’t retry 503s at all unless you say so.

Q: A 429 comes back with Retry-After: 2. What should happen? A: Wait 2 seconds. It’s not a suggestion — ignoring it escalates a polite throttle into a revoked key or a blocked IP. Retry(respect_retry_after_header=True) is the default and honours it over your backoff schedule: with backoff_factor=0.1 and two retries against a Retry-After: 2 endpoint, the measured wall time was 4.02s, not 0.2s. Note urllib3 only applies it to {413, 429, 503}. The mature approach is proactive — read X-RateLimit-Remaining and slow down before you hit zero.

Q: Explain 401 vs 403. A: 401 Unauthorized is authentication: the server doesn’t know who you are — token missing, malformed, or expired. Retrying with the same token fails forever, but refreshing it may genuinely fix it. 403 Forbidden is authorization: the server knows exactly who you are and the answer is no. Retrying a 403 is pure noise; fix the permissions or the scope. ⚠️ Some APIs (GitHub included) return 403 for certain rate-limit conditions, so check for a Retry-After header rather than trusting the code alone.

Q: except ConnectionError: — what does it miss, and what does it wrongly catch? A: It misses ReadTimeout, which subclasses only Timeout, so a server that connects then goes silent slips past. Confusingly, ConnectTimeout subclasses both ConnectionError and Timeout, so that one is caught. And it wrongly catches SSLError, which is a ConnectionError subclass — so the reflexive “retry ConnectionError” retries an expired certificate five times with backoff, which can never succeed and is a config error that should fail loudly on attempt one. Catch RequestException (the base of everything, itself an OSError) for “the call failed”, and be explicit about what’s genuinely transient.

Q: Why is params={"q": user_input} better than f-stringing the URL? A: Encoding. params= percent-encodes values so they can’t break out of the query: "a b&c=d" becomes q=a+b%26c%3Dd, so a user-supplied &admin=true is data, not a new parameter. It also handles lists as repeated keys (["x","y"] → tags=x&tags=y) and stringifies ints. ⚠️ One gotcha: a None value is silently dropped, so params={"page": None} sends no page at all rather than page=None — a silent “stuck on page 1” bug. Check what actually went out with r.url.

Q (coding): Write a generator that yields every item from a Link-header-paginated API, lazily. A:

def paginate(session, url, **params):
    query = params or None
    while url:
        r = session.get(url, params=query, timeout=(3.05, 10))
        r.raise_for_status()
        yield from r.json()                          # flatten this page
        url = r.links.get("next", {}).get("url")     # None on last page -> stop
        query = None                                 # next URL already has the cursor

The points being tested: yield from so callers see items, not pages; r.links because requests parses Link for you; .get("next", {}).get("url") returning None to end the loop with no counter; and query = None after the first call, because the next URL already carries the cursor and re-sending the original params would overwrite it and loop forever. Laziness means islice(paginate(...), 5) makes one HTTP call, not forty.

Q (coding): How do you test an API client without touching the network? A: Mock the HTTP layer. With responses, register what the endpoint should return and assert on the client’s behaviour and on the calls made:

@responses.activate
def test_404_becomes_app_error(gh):
    responses.get("https://api.github.com/repos/psf/nope", json={"message": "Not Found"}, status=404)
    with pytest.raises(GitHubError, match="404"):
        gh.repo("psf", "nope")

Real-network tests are slow, flaky, rate-limited and fail offline. You can also assert len(responses.calls) to prove retries or laziness. unittest.mock.patch.object(requests.Session, "get", ...) does the same with more boilerplate. ⚠️ The honest caveat: mocks encode your belief about the API, so pair them with one contract test against the real thing in CI.

Q: When would you reach for httpx instead of requests? A: When you need async (AsyncClient) or HTTP/2 — requests has neither and never will; it’s deliberately in maintenance mode, which is why it hasn’t broken your code in a decade. httpx’s API is close enough that porting is mostly mechanical, and it has one meaningfully better default: a 5-second timeout where requests has none, so its failure mode is “too aggressive” instead of “hangs forever”. For sync code with no HTTP/2 need, requests remains the default — it’s ubiquitous, and Session + HTTPAdapter + Retry covers real production needs. Use stdlib urllib only when you truly cannot add a dependency; note it raises on 404 automatically and uses the OS trust store rather than certifi.

---

Key takeaways

• requests does not raise on 4xx/5xx. A 500 is a successful request with a bad answer. Call r.raise_for_status() before r.json(), every time — otherwise you parse an error page as data and the traceback blames your parser.
• ⚠️ Always pass timeout=(connect, read). There is no default; omit it and a silent server hangs your thread forever with no exception, no log, and no evidence. It’s the only failure here that leaves nothing to debug. It is not a whole-call deadline — it bounds the gap between bytes.
• Use a Session for anything that calls twice. requests.get() builds and destroys a whole Session (and its connection pool) per call. Measured: 10 HTTPS calls went from 1.19s to 0.41s — 2.9x — for one object.
• Mount retries, don’t write them. HTTPAdapter(max_retries=Retry(...)) retries below your code. ⚠️ Fix the two dangerous defaults: backoff_factor=0 (no sleep at all) and status_forcelist=None (5xx not retried). Sleeps are 0, 2, 4, 8s — the first retry never sleeps.
• Only retry what’s idempotent. allowed_methods excludes POST/PATCH on purpose. A retried POST after a lost response charges the card twice. Need it? Use a server-supported idempotency key.
• Honour 429 and Retry-After. It’s a contract, not advice — and it overrides your backoff. Better still, watch X-RateLimit-Remaining and slow down before you hit the wall.
• Secrets go in headers, from the environment. Never in the URL (logs, Referer, history), never hardcoded (git remembers — a leaked key must be rotated, not deleted). ❌ verify=False is not a fix; point verify= at the right CA.
• Page lazily with a generator, stream big bodies. yield from r.json() + r.links["next"]["url"] gives callers a flat, lazy item stream; stream=True + iter_content() keeps a 2 GB download out of RAM.
• Test with the network unplugged. responses or unittest.mock makes the suite fast, deterministic and offline — 5 real behaviours verified in 0.33s. Pair with one live contract test so upstream changes can’t pass a green suite.