Businesses frequently resort to parsing competitors. They say: "everyone is doing that, but not everyone wants to admit that " and it’s really so. Parsing can be useful for comprehensive marketing research, for finding errors on their websites, for collecting data from search engines and for many other tasks.
Web scraping does not always run smoothly. Large internet platforms, and some smaller ones too, may protect themselves against manipulations, bots and automated traffic. This is how they save on hosting and free up resources to serve real (primary) users.
This material is about what to do in situations when your parser has been detected and blocked by an IP address. We will share tips and hacks on how to bypass IP ban.
Many CMS systems (website engines) as well as web server software (at the firewall level and other traffic filtering software) may use proprietary tools to protect against unwanted visitors. Primarily, these tools are aimed at spammers, but there may be other reasons for blocks. Notably, scraping and automatically generated traffic are also considered parasitic loads.
The simplest method of protection is to create a kind of blacklist where client IP addresses, which you do not want on your server/site, can be added.
When a client connects to the server, their IP address is checked against this blacklist. If it is found on the list, the client will be completely denied access to the server, usually resulting in a 403 error (access forbidden).
This is essentially what it means to have your IP banned.
Sometimes, an IP address ban functions a bit differently, only at the website engine level. In such cases, for example, you may be able to view general pages but cannot log into your account, send messages or access a restricted part of the site. This is the so-called "admin ban."
The main idea behind a protection system based on blacklists is that a regular user will not be able to quickly change their IP address. At most, the internet provider might assign a new dynamic IP upon reconnection, but even this is not infinite.
Modern network architectures of local providers are such that hundreds of users may operate under one IPv4 address, and their IP isn’t changed upon reconnection.
Now that we understand what an IP ban is, the remaining question is how to remove or bypass such a block.
When connecting to the Internet, all devices are assigned a unique identifier—an IP address. We've discussed what this is and how it's assigned in our article about dynamic IP addresses and how to hide your IP address. In brief, IP addresses are always registered to companies, either local or regional registrars. It's impossible to "lose" an IP address as they are all accounted for. The exception is special local IP addresses intended for building internal networks, but they cannot be used to connect to the global network.
So, how can you avoid an IP address block when scraping? We are eager to share some tips.
Large resources may specify server load limitations in their terms of use. This is rare, but such practices do exist. The main challenge lies in the fact that a user cannot calculate the load placed on the server, and therefore, it is impossible for them to know whether they are within the limits or not.
Typically, the "normal" user activity is specified. Anything that does not fall under this description may be blocked.
Alongside general and legal documents that stipulate the intended use of an internet resource, purely technical documents may also be applied. For example, these could be XML sitemaps (which gather links to pages that are to be indexed, i.e., crawled by search robots) or a robots.txt file (which specifies the rules for crawling pages and prohibitions). It is essential to study the directives from robots.txt and adhere to them. Do not attempt to parse sections that are clearly forbidden for crawling by robots through specific directives.
Violating these guidelines can easily result in a ban.
Major platforms understand that business audiences often require access to specific data. To reduce the load on the main website (the one accessed by regular users), resource owners set up an API.
Using an API may significantly automate the data collection process, as the information is provided by the service in a ready-to-use format (with markup). You can request specific data: individual pages, lists of pages, segments, statistics and more. All API capabilities are described in the documentation and depend on the specific resource (the technical aspects are determined by the site owners).
Access to the API is usually granted via a key, which serves both as an identifier and a security element.
However, APIs may have their own limits and restrictions. It is crucial to study and adhere to them.
It’s particularly worth noting: even if you "accidentally" exceed the API request limits per IP, you are unlikely to get banned. At most, the system will respond to your requests with an error. Once the limits are reset, access to the data will be restored.
Bypassing API restrictions is more challenging as the system does not tie to your IP addresses but to your API key.
One of the primary indicators used to identify bots and scrapers is the consistency of time intervals between requests. It's not just about how long the intervals are—whether one second or five minutes—but rather the pattern itself that gets detected.
If your scraper does not have the capability to randomize the time intervals between requests, this is an essential feature to implement (or it may be necessary to switch to another scraper).
Here are additional tips to ensure professional web scraping practices without getting blocked.
It doesn't make sense to send requests too often. The more requests are sent from one IP, the more attention from security systems is attracted. Real users are physically unable to open 100 pages per second.
If you want to speed up the data collection process, parallelize the scraper samples. It's crucial not to decrease the time between requests from a single IP but rather to distribute the load across multiple IPs.
Parallel request flows from the same IP is a huge problem. Your scraper is likely to get banned in this case.
How to Send Requests from New IP for Every Session
How can one reliably detect automated traffic and bots? That’s easy: by embedding a special link or form in the HTML code that is hidden from view using CSS styles.
The outcome is as follows: a bot or scraper during its crawl sees this link because it exists in the code (bots do not have real vision and read the code as regular data from a stream) and follows it.
Users cannot click such a link because the browser does not display it on the page.
Consequently, all IP addresses of clients who follow these trap links are immediately banned (blacklisted and blocked).
To avoid such traps, you can utilize computer vision technologies or train scrapers to ignore elements and links that are hidden from display.
Advanced protection systems analyze a multitude of user parameters: cursor movements, browser version, user-agent, cookies, support for various web technologies (JavaScript, HTML5 canvas, WebRTC, etc.), a list of system fonts, a set of installed browser plugins among others. These are the so-called “digital fingerprints”.
Furthermore, many web pages now load using Ajax and JavaScript frameworks (essentially full web applications). Direct HTML parsing might not work because the final content is generated directly in the browser, not on the server (normally, the server delivers a ready-to-render HTML page to the browser).
Headless browsers can save the situation. They allow for the full loading of dynamic pages and emulate user behavior: moving the cursor, waiting periods, typing text in forms etc.
However, if you need to simultaneously work with a large number of accounts, you might require more advanced browser versions—anti-detect browsers (to understand the principles of their work, consider the Dolphin{Anty} browser review).
Both headless and anti-detect browsers can interact with parsers via APIs or through special libraries.
Suppose, you did not follow our recommendations on how to avoid blocks and finally got banned. It may also turn out that website protection systems are quite complex and it’s impossible to circumvent them by using standard methods. Let’s discuss how to circumvent the ban of your IP address.
This actually seems ridiculous. It frequently happens that your network provider buys a new pool of IP addresses, some of which may already be blacklisted by specific internet resources due to their previous use. In such cases, you can reach out to the website owners, explain the situation and request an IP unblock. The site administration might be accommodating if they can verify the change of IP ownership.
However, contacting support might be challenging if your IP is already blocked. In this scenario, you could use mobile data or other methods to bypass the block as outlined below.
If the ban was a result of engaging in activities like scraping, asking for the unblocking does not make any sense. Most likely, the request will be denied because such activities typically violate the terms of normal resource use.
This advice sounds like the universal solution from the tech support of your network provider for any connection issues, but it holds merit. Nowadays, almost all wired ISPs use dynamic IP rotation among active users. If your provider uses public dynamic IPs (as opposed to private, internal ones given to network routers), there's a good chance you'll be assigned a new IP address upon reconnecting.
This tip won't help in networks with dynamic private IPs. In these cases, the IP address is assigned to the provider's network equipment, which routes traffic within the local network without distributing real IP addresses to users. Thus, you may receive a new local (technical) IP, but you won't be able to change the IP used for accessing the global network. Consequently, the ban will remain active.
VPN is an effective solution for circumventing regional blocks and for hiding your online sessions from prying eyes. All data between you and the VPN server is securely encrypted, creating a kind of virtual tunnel.
When you connect to the internet via a VPN, you appear under the IP address of the VPN server. It acts similarly to a proxy by sending your requests on your behalf.
However, there are several serious issues to consider:
In summary, VPNs are not proxies, they do not work for tasks like scraping.
Read more about the detailed comparison of VPNs with proxy servers.
This is the most practical and effective advice. Proxies are the ideal solution for parsing a large number of pages, parallelizing streams and circumventing specific IP bans.
Proxies come in various types. We strongly recommend using the following for scraping:
Proxies act as intermediaries. Thus, bypassing an IP ban is straightforward with their help. You change your IP to the proxy’s IP and the ban no longer affects you. If the target site bans your proxy's IP address, it can be easily changed to a new one either via an API command or based on a set time interval. You can even send each new request from a new address. In this case, getting you banned via blacklists becomes virtually impossible.
We have reviewed the major reasons for IP bans, discussed how to avoid blocks, but most importantly, explained what to do if your IP has been banned when scraping.
The best solution for circumventing IP banning are proxy servers. They are the only ones suitable for bulk tasks and provide quick IP rotation. All other methods are only suitable for one-time and personal purposes.
You can get high-quality residential and mobile proxies with rotation from us. The Froxy service can offer more than 10 million IPs, precise targeting, an API interface and reasonable prices. We have ready-to-use cloud scraping tools available. With these, you don't have to worry about blocks.