Modern web scraping has evolved far beyond sending basic curl or requests.get() commands. With platforms like Cloudflare Turnstile, DataDome, and Akamai Bot Manager analyzing browser headers, TLS handshakes, and canvas fingerprints, building a resilient scraper requires deep browser automation knowledge.

In this guide, we explore how to build a high-performance, stealth extraction engine using Python and Playwright.

1. Why Standard Headless Chrome Gets Flagged Instantly

When you launch vanilla Chromium via playwright.chromium.launch(headless=True), several JavaScript flags immediately scream "bot" to anti-bot engines:

  • navigator.webdriver is set to true.
  • WebGL vendor strings report Google Inc. (Google SwiftShader) instead of a physical GPU (NVIDIA / Intel).
  • Headless user-agents contain HeadlessChrome.
  • Window dimensions and plugin arrays are empty.

2. Setting Up Playwright Stealth

To neutralize these signals, initialize browser contexts with patched user arguments and real viewport dimensions:

from playwright.async_api import async_playwright

async def create_stealth_browser():
    p = await async_playwright().start()
    browser = await p.chromium.launch(
        headless=True,
        args=[
            "--disable-blink-features=AutomationControlled",
            "--no-sandbox",
            "--disable-infobars",
            "--window-size=1920,1080"
        ]
    )
    
    context = await browser.new_context(
        viewport={"width": 1920, "height": 1080},
        user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
        locale="en-US",
        timezone_id="America/New_York"
    )
    
    # Patch navigator.webdriver on every new frame
    await context.add_init_script("""
        Object.defineProperty(navigator, 'webdriver', {
            get: () => undefined
        });
    """)
    
    return browser, context

3. Handling Dynamic JavaScript Hydration & Infinite Scroll

Many modern web apps (built on React, Vue, or Next.js) load data via internal JSON endpoints rather than server-rendered HTML. Instead of scraping DOM elements, intercept the network requests directly:

async def intercept_pricing_api(page):
    async def handle_response(response):
        if "/api/products" in response.url and response.status == 200:
            data = await response.json()
            print(f"Extracted {len(data['items'])} products directly from internal API!")

    page.on("response", handle_response)
    await page.goto("https://example.com/catalog")
    await page.wait_for_timeout(3000)

Network interception is 10x faster and virtually immune to DOM class changes or CSS redesigns.

4. Residential Proxy Rotation Architecture

For high-volume scraping jobs (>50,000 pages per day), datacenter IPs (AWS, DigitalOcean) are permanently rate-limited. Rotate requests across residential proxy pools:

proxy_settings = {
    "server": "http://pr.oxylabs.io:7777",
    "username": "customer-user_zone-resi",
    "password": "secret_pass"
}

context = await browser.new_context(proxy=proxy_settings)

Best Practices for Responsible Scraping

  • Respect robots.txt whenever possible.
  • Apply exponential backoff with randomized jitter (e.g., sleep(uniform(1.5, 3.8))).
  • Cache repeated requests using local SQLite databases to prevent overloading origin servers.