NovFora Dev

How to handle rate limiting in Python's requests library?

Taylor Davis

Taylor Davis

6 months ago

Use urllib3.util.Retry with backoff factors instead of custom loops. Wrap your session in a retry adapter: requests.Session() + HTTPAdapter(max_retries=5) with backoff_factor=0.3. This handles 429s and 503s automatically while respecting the Retry-After header where available.

Liam Jackson

Liam Jackson

6 months ago

I just switched to urllib3 with retries enabled and it fixed my production issue entirely. Simple

Liam Jackson

Liam Jackson

6 months ago

use a session with max_retries set and backoff=True -- it handles exponential back

Harley Adams

Harley Adams

6 months ago

Use urllib3.util.PoolManager with maxretries and backoff_factor if you

Taylor Davis

Taylor Davis

6 months ago

requests doesn't have built-in backoff, so you typically use urllib3 retries through a custom Adapter:

from requests.adapters import HTTPAdapter
from urllib3.util import Retry

session = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
session.mount('https://', HTTPAdapter(max_retries=retries))

response = session.get('https://api.example.com/data')

The backoff_factor parameter makes wait times grow exponentially: {backoff factor} * (2 ** ({number of total retries} - 1)). With 5 retries and a factor of 1, sleep intervals are 0, 2, 4

Join the conversation to leave a reply.

Sign in to reply

Related topics