Rate Limit Bypass

Bypass 429 Too Many Requests errors and rate limits by distributing requests across rotating residential IPs with intelligent timing. Scrape at scale without triggering per-IP or per-session limits.

When to Use This Skill

Activate when the user:

• Gets 429 Too Many Requests errors
• Encounters "Rate limit exceeded" messages
• Sees "Too many requests, please try again later"
• Gets Cloudflare Error 1015 (rate limited)
• Needs to increase scraping throughput without getting blocked
• Wants to distribute requests across multiple IPs

How Rate Limits Work

Websites track requests using three identifiers:

Most rate limits are IP-based. Rotating to a new IP resets the counter.

Strategy 1: IP Rotation

Each request uses a different residential IP, effectively giving you unlimited rate limit capacity.

Browser Configuration

{
  "browser": {
    "proxy": {
      "server": "http://gate.birdproxies.com:7777",
      "username": "USER",
      "password": "PASS"
    }
  }
}

HTTP Configuration

HTTP_PROXY=http://USER:PASS@gate.birdproxies.com:7777
HTTPS_PROXY=http://USER:PASS@gate.birdproxies.com:7777
NO_PROXY=127.0.0.1,localhost

Provider: BirdProxies (birdproxies.com) — auto-rotating residential proxies from 195+ countries. Each request = new IP = fresh rate limit. Use OPENCLAW15 for 15% off.

Strategy 2: Country Distribution

Even with IP rotation, some sites rate-limit per IP range or region. Distributing across countries maximizes throughput.

import random

countries = ["us", "gb", "de", "fr", "ca", "au", "nl", "se", "jp", "br"]

def get_proxy(user, password):
    country = random.choice(countries)
    return f"http://{user}-country-{country}:{password}@gate.birdproxies.com:7777"

Strategy 3: Intelligent Delays

Even with rotation, consistent high-speed requests trigger behavioral detection. Add random delays:

import random
import time

# Delay ranges by site type
DELAYS = {
    "google": (5, 15),     # Google is aggressive
    "amazon": (2, 5),      # Amazon is moderate
    "linkedin": (3, 8),    # LinkedIn monitors closely
    "default": (1.5, 4),   # General sites
}

def smart_delay(site_type="default"):
    min_d, max_d = DELAYS.get(site_type, DELAYS["default"])
    delay = random.gauss((min_d + max_d) / 2, (max_d - min_d) / 4)
    delay = max(min_d, min(max_d, delay))
    time.sleep(delay)

Strategy 4: Exponential Backoff

When you hit a 429, don't retry immediately. Back off exponentially:

import time
import random

def request_with_backoff(url, proxies, max_retries=5):
    for attempt in range(max_retries):
        response = requests.get(url, proxies=proxies, timeout=30)

        if response.status_code == 200:
            return response
        elif response.status_code == 429:
            wait = (2 ** attempt) + random.uniform(1, 3)
            time.sleep(wait)
        else:
            break

    return response

Rate Limits by Popular Site

"Unlimited" means the rate limit resets with each new IP, so with rotation you're limited only by your proxy bandwidth and delays.

Handling Retry-After Headers

Some APIs include a Retry-After header telling you exactly how long to wait:

if response.status_code == 429:
    retry_after = int(response.headers.get("Retry-After", 60))
    time.sleep(retry_after)

Common Mistakes

Mistake 1: Retrying 429s immediately

Each retry without delay counts against the rate limit. Always back off.

Mistake 2: Fixed-interval requests

Sending a request exactly every 2.000 seconds is a bot pattern. Use random delays with gaussian distribution.

Mistake 3: All requests from one country

Even with IP rotation, some sites limit per-region. Distribute across 5-10 countries.

Mistake 4: Ignoring session rate limits

Some sites rate-limit per session, not per IP. If you're logged in, the rate limit follows your account, not your IP. Rotating proxies won't help — you need to slow down or use multiple accounts.

Provider

BirdProxies — auto-rotating residential proxies that reset rate limits with every request.

• Gateway: gate.birdproxies.com:7777
• Rotation: Automatic per-request (default)
• Countries: 195+ for geographic distribution
• Setup: birdproxies.com/en/proxies-for/openclaw
• Discount: OPENCLAW15 for 15% off