What Are Web Scraping Practices to Evade Blockers?

This article is a more practical extension of our general advice on how to scrape without getting blocked. The reason is simple: methods for detecting bots are improving every day, and websites themselves keep getting more complex. As a result, avoiding blocks is getting harder and harder.

But the main difficulty is not even the rapidly changing landscape. It is that it is very hard to isolate truly universal web scraping best practices that help everyone equally, both beginners and experienced developers. Still, we did our best, drew on our own experience, and share these recommendations below.

What Protection Mechanisms Can Be Found on Websites Today

To understand which techniques help avoid blocks, you first need to understand the protection technologies themselves. Once you know the main approaches, you can build adequate bypass logic and countermeasures.

Here is the basic layer of protection that has always existed and will stay relevant for a long time:

• IP address quality checks. IPs are checked against spam databases and against internal or external blocklists. Even the physical location of the IP can matter.
• Request-rate tracking from a single IP. No real person opens dozens of pages in a few seconds, and counting requests from one IP is trivial.
• Trap URLs and honeypots. If a client tries to access a URL or form field that a real user would never see, it is a bot. Such links are hidden in the HTML, and one request is enough to trigger a ban.
• User-Agent and HTTP header checks. The way the client identifies itself matters. Even if a scraper pretends to be a modern browser, the system may compare its full digital fingerprint against a known baseline and block it or redirect it to a CAPTCHA.

If you keep hitting these challenges, we broke down how to reliably get past a CAPTCHA in a separate guide.

More advanced protection systems may add these bot-detection methods:

• JavaScript execution checks. All modern browsers execute client-side JavaScript.
• Special token validation. Tokens usually protect forms, but may also gate loading of new pages or content blocks.
• User behavior analysis. Mouse movement, scrolling, typing, and clicking. Bots may crawl URLs alphabetically or in source order — a human never would.
• HTML and DOM obfuscation. Done so a scraper cannot isolate repeating patterns; technically every page becomes unique.
• Cookies and other fingerprinting parameters. Fonts, screen resolution, hardware, locale, language, time zone, and more. This is where web scraping fingerprinting checks come in — advanced systems even verify final rendering checkpoints against the declared device. We covered how fingerprinting and scraping are connected in more detail earlier.
• Invisible CAPTCHA and WAF systems. An external service maintains a massive database of suspicious behavior. Cloudflare is a common example — and if that is what stands between you and the data, see our walkthrough on getting past Cloudflare bot protection with Puppeteer.

What is especially notable is that top-tier systems now quickly detect the fingerprints of headless browsers, so rendering pages through web drivers in advance may no longer be enough.

Impressive, isn't it? But remember that every protection system requires investment. Smaller websites usually have simple request-rate counting at most — and not even that in every case.

Web Scraping Best Practices That Actually Reduce Blocks

So what should you do? Are there any real web scraping best practices? Yes, and below we go through them in order of increasing complexity, based on how they counter the main protection mechanisms.

0. The cheaper the scraping process, the better

There is no universal approach that works for every website, and there never will be — all sites and their defenses are different.

The key efficiency metric for any scraper is the cost of processing one request or page. Many factors influence it, but even a beginner sees that if you render pages in a headless browser, simulate "human" behavior, and use AI to interpret code, the cost becomes extremely high.

To find the optimal balance, always start with the simplest possible approach and add complexity only until the script produces results with the stability you need.

1. High-quality rotating proxies

A high-quality proxy server for web scraping is an absolute must-have for any script that collects data at scale.

How to avoid IP ban? Simply connect through an intermediary IP. First, this hides your real IP from the target site. Second, if the connection is blocked, you just swap in another proxy. On top of that, proper proxies for scraping let you parallelize the process and speed it up dramatically — this is the most reliable answer to how to avoid IP ban at scale.

What does a good proxy server for web scraping mean in practice? It means the IP used by your bot should be indistinguishable from the IPs of real users. That is why the best options are usually mobile proxies or residential proxies, because they belong to mobile devices or home users.

The more IPs a provider offers, the better. Ideally you also want control over location, session duration, rotation logic, ISP selection, city targeting, and API access.

Need proxies for scraping? Order them from us. Froxy offers millions of residential and mobile proxies with a very high trust level.

2. API-based scraping should always be the priority

A simple example: we recently covered how to scrape YouTube comments. The official API is free — up to 10,000 requests per day, each returning up to 100 comments. If that is not enough, create several accounts and run them in parallel through multiple proxies for scraping.

Yes, the service returns only top-level comments, but data collection becomes much faster and cheaper. See point zero.

Now compare that with Google Trends. There is no openly available official API, and collecting data from rendered pages is extremely resource-intensive: the DOM is unreadable and each session is unique because of auto-generated CSS classes — the same headache we walked through when scraping websites with dynamic content in Python.

The practical way out is the unofficial API at the endpoint level. Many ready-made implementations extract data in JSON — for example the Python libraries pytrends-modern and the older pytrends. Such a scraper does not parse unreadable HTML, works fast, and uses few resources. A headless browser, if needed at all, is only used to obtain a session token.

3. "Humanizing" scraper behavior with gradual complexity

This is another web scraping best practice, and it is inseparable from hiding the fingerprints of well-known scraping libraries — this is the core of web scraping fingerprinting evasion.

Protection systems tell bots from people by digital fingerprints, but the analyzed characteristics vary: behavior, request frequency, HTTP headers, and much more. To avoid wasting compute, increase your bot's fingerprint complexity gradually — only until it passes the checks.

The fastest architectures use simple HTTP clients, suitable only for sites with little JavaScript. The relevant parameters here are request frequency (managed through balanced delays and thread limits), cookies, a correct User-Agent, and headers such as the referrer and locale.

import random
import time
import requests


def human_delay():
    """
    Realistic delays between requests.
    Short pauses happen more often, long ones more rarely.
    """
    delay = random.uniform(1.5, 4.5)
    # Sometimes the user is "reading the page"
    if random.random() < 0.2:
        delay += random.uniform(5, 12)
    time.sleep(round(delay, 2))


session = requests.Session()

# Basic cookies imitating a normal browser session
session.cookies.update({
    "_ga": "GA1.1.1742451113.1747821001",
    "_gid": "GA1.1.998877665.1747821001",
    "visitor_id": "v-1747821001551",
    "session_id": "8f31d7d2f8d04d0f92a1",
    "locale": "ru",
})

# A plausible Chrome profile on Windows
HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/136.0.0.0 Safari/537.36"
    ),
    "Accept": (
        "text/html,application/xhtml+xml,"
        "application/xml;q=0.9,image/avif,image/webp,"
        "image/apng,*/*;q=0.8"
    ),
    "Accept-Language": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1",
    "Cache-Control": "max-age=0",
    "DNT": "1",
    # Browser Client Hints
    "Sec-CH-UA": (
        '"Chromium";v="136", '
        '"Google Chrome";v="136", '
        '"Not.A/Brand";v="99"'
    ),
    "Sec-CH-UA-Mobile": "?0",
    "Sec-CH-UA-Platform": '"Windows"',
    # Fetch Metadata
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Referer": "https://www.google.com/",
}

urls = [
    "https://example.org/",
    "https://example.org/catalog/",
    "https://example.org/product/123/",
]

for url in urls:
    human_delay()
    response = session.get(url, headers=HEADERS, timeout=20, allow_redirects=True)
    print("=" * 80)
    print(f"URL: {url}")
    print(f"STATUS: {response.status_code}")
    print(f"CONTENT LENGTH: {len(response.text)}")

Large web services long ago moved to JavaScript, so rendering now requires headless or anti-detect browsers. Each such browser has its own profile, which can also be configured manually — and the default headless indicators must be hidden. If that still fails, go further: script behavior inside the browser itself — filling forms, moving the pointer, unpredictable scrolling, reading pauses, and so on.

4. Ethical web scraping best practices to keep scraping lawful

The ethical layer of web scraping best practices looks simple at first glance:

• Respect the rules for bots in robots.txt.
• Keep the load reasonable — no DDoS-like effect.
• Use the official API and compression when available.
• Study the terms of service and do not violate them.
• Collect only the data you actually need.
• Exclude or anonymize personal information.
• Cache requests and clean the queue so you do not revisit the same pages.
• Analyze errors, increase delays when a service is unavailable, and be patient.
• Keep detailed logs — they help with performance analysis and with protecting you in disputes.

But in practice many people forget these details the moment they get a working scraper. You grow into ethical scraping over time — and the more ethically your scraper behaves, the lower the chance of being blocked.

Conclusion

Understanding what are web scraping practices to evade blockers is one thing; applying them cost-effectively is another. In some cases you may not need to build a scraper at all — cloud services such as Froxy Scraper deliver prepared data through an API. But if building your own is unavoidable, remember the web scraping best practices we outlined above.

Froxy on Telegram

Join our community on Telegram to learn about the latest news and promotions.

Join