The advertising market is continuously growing, with the growth rate of online advertising significantly outpacing that of all other ad types (on TV, radio, billboards, etc.). This is a global trend. Just in case, here is a link to a study by Markets and Markets (2023).
Naturally, when outsourcing the display of their ads on various websites and in contextual advertising networks, advertisers want some explicit guarantees: to ensure adequate coverage, correct materials display (not just inflating statistics), and relevant targeting set up.
But here arises the problem: how do we control contractors? This material will discuss options for verifying online advertising and how this may be related to proxy rental.
What Is Ad Verification?
Ad verification is the process of qualitative and quantitative control of the display of ad messages to ensure compliance with the conditions of their display outlined in partnership agreements. For example, the verification may include checking the display region, targeting parameters like reaching the intended audience, timing of display, and other targeting criteria.
Controlling contractors responsible for displaying your ad messages helps address several related issues:
- Minimization and optimization of advertising expenses: you can verify the effectiveness of various ad strategies, develop new directions, or discontinue unprofitable ones.
- Creating evidence for dispute resolution: to prove the contractor's misconduct and demand proportional compensation or issue resolution.
- Verifying compliance with the conditions of actual ad display, with the terms outlined in contracts with advertising agencies or other partners.
- Identifying fraudulent traffic, low-quality platforms, manipulation attempts, and bot use allows you to discontinue working with them in the future.
- Prompt detection of errors: broken links, missing images, incorrect prices, etc.
In some cases, monitoring competitors may be required to analyze their advertising policies and marketing strategies.
Challenges of Ad Verification
To achieve the outlined tasks, a high-quality technical component is required—everything that allows the emulation of user behavior, virtualization of parameters of digital footprint and location (for verifying targeting conditions), screenshots, and evidence collection.
In fact, the advertising verification script differs from parsers. It focuses on blocks of ad messages, not the main content. Such a specialized parser can work on different target sites or on a specific one. In the former case, a separate page structure handler must be created for each target resource.
As with regular parsers, scripts for ad verification may encounter attempts at blocking:
- Due to repeated automatic requests with a specific timing interval.
- Due to incorrect or suspicious HTTP headers within HTTP requests and responses to the server (to the target site).
- Due to falling into forms or trap pages (URLs that ordinary users cannot see, yet parsers can see them due to sequential parsing of HTML code).
- Due to the lack of JavaScript and Ajax support (applies to many dynamic sites and PWA applications that are almost entirely composed of scripts executed on the browser side).
- Due to the CAPTCHA refusal.
- Etc.
Detailed problems with blocking parsers were described in the article "Scrape Like a Pro! Best Practices for Web Scraping Without Getting Blocked".
Why verify ad displays at all? Contractors or end sites that provide banner rotation services on their pages may:
- Manipulate statistics results (displays, clicks, and form submissions).
- Manipulate clicks and other activity using automation tools (bots and scripts).
- Hide ad messages or display them with violations - under a layer of other elements, in reduced frames, in the wrong place specified in the contract, etc.
Additionally, intermediary services may not control targeting conditions themselves: the location of customers, their interests, digital footprints etc.
All this ultimately results in the waste of your ad budget without producing the expected result: in the form of increased brand recognition, sales growth or promotion of new products/services.
Note that not just HTML parsing scripts may be required for the highest quality ad control, but also complex solutions with elements of computer vision (for screenshot recognition), tools for working with a large number of user profiles (anti-detection browsers), APIs for user behavior emulation (headless browsers) and external services for solving captcha (with computer vision or those that involve manual labor).
Proxies for Ads and Ad Verification
Now let's talk about how the process of ad display verification and large proxy databases are related.
As mentioned earlier, target sites (internet resources where your ads or competitors' ads are displayed) can defend themselves using available methods. They detect parasitic traffic and block all automated requests.
Naturally, many security checks are based on analyzing the structure of browser HTTP headers/requests and on analyzing the IP address.
Based on the IP address, you can determine the user's approximate location, network operator, IP pool owner, and the type of connection (mobile network, home, or corporate).
If the user's profile does not match the IP parameters, this is the first reason for blocking, or at least a reason to request re-authorization/verification of the user's "humanity" (captcha, solving logical tasks, confirmation of authorization by additional factors, etc.). For example, when a mobile app client tries to connect from a server IP, the site may have legitimate security concerns about the session.
How Proxies Are Used in Ad Verification Processes
Even if you have high-quality user profiles (digital fingerprints), they must work with equally high-quality IP addresses.
But where can you find IP addresses that meet your criteria? In proxy server networks.
As mentioned, the match between the IP address and user profile is crucial. Advertising networks can determine many parameters based on IP, including location, connection type, etc. This means that an ad verification proxy should not just be a mere "proxy." It must closely match real clients from specific regions or even specific cities.
To acquire proxies matching your criteria, you must be aware of their types and associated features.
Different Types of Proxies Used for Ad Verification
Proxies vary: they can be direct and reverse (back-connect proxies), transparent and anonymous, public (open) and private (exclusive), static and dynamic. However, the most critical division of proxies for ad verification is based on the type of device they belong to.
In particular, the following proxies are distinguished: mobile, residential, and datacenter proxies.
- Mobile Proxies: Operate on the basis of mobile operators' addresses. They have the highest trust level from security systems (minimum captcha and blocking), so their rental is often more expensive. However, they have weak stability (life span) and minimal bandwidth (many countries still use 3G networks). Read more details about mobile proxies, their advantages, and disadvantages.
- Residential Proxies: Operate on the basis of addresses from home internet providers (familiar wired networks). The trust level here is slightly lower, but stability and bandwidth (connection speeds) are higher. Providers of such services can offer a large pool of addresses, ensuring quick IP rotation. Read more details about residential proxies, their advantages and disadvantages.
- Datacenter Proxies: Operate on the basis of addresses owned by companies. These are usually hosting, IT infrastructure, or corporate data centers. Corporate addresses raise the most suspicion from security systems, making them less suitable for parsing and ad verification tasks. Their application lies in dealing with heavy content when unlimited or wide stable access channels are required. If the pool of corporate addresses is large, they may be used similarly to residential proxies. However, it is worth considering that the number of errors due to IP blocks will be higher in this case.
The convenience of selecting addresses and the method of connecting proxies to niche software is worth mentioning. Lists usually connect classic direct proxies, while the latter are exported in a specific format: IP:port@login:password etc.
Back-connect proxies can be connected with just one line or through multiple ports. Technically, these are the same proxies, in the same format (IP:port@login:password etc.), but incoming requests are routed within the network and can come out with completely different IPs. Consequently, the rotation of non-functional proxies can be automated as well as the selection of proxies based on your criteria: address type, location, retention time etc. The connection parameters remain the same because the entry point to the proxy network does not change.
Naturally, proxies with reverse feedback are more convenient and practical, letting you save much time on connection and automatic rotation.
Benefits and Limitations of Using Proxies for Ad Verification
Only proxy services with extensive coverage and precise targeting will help you emulate location so that security and ad rotation systems perceive the visitor as a real customer.
No other technical solution can achieve this. For example, VPN networks always have a limited choice of addresses and do not provide conditions for their selection in specific locations.
On the other hand, proxies alone are not enough. To ensure the ads proxy verification process is of the highest quality, other technical aspects need to be considered:
- Database of digital fingerprints, which will be used to analyze user profiles and display ads. For example, these need to be preliminary "warmed up" by visiting thematic websites and entering specific search queries in Google, Bing etc. Special software complexes like anti-detect browsers are used for profile rotation.
- List of target resources where ad display will be verified (+ determination of their bypass frequency).
- Captcha resolution in case of suspicion of automated traffic (captchas can be solved manually, but if we're talking about thousands of parallel requests, automation systems are required).
- JavaScript Execution. Standard HTML parsers cannot handle JS scripts. Therefore, the page needs to be rendered in a full-fledged browser before analyzing its content. Headless browsers and special libraries for API organization are required for this purpose.
- Depending on the complexity of security systems on target sites, bypass systems need to be devised (different delays, correct HTTP headers, JS support, bypassing traps, etc).
Be sure to calculate the approximate traffic and all associated expenses. Based on this data, you can obtain adequate figures for the future budget and assess the profitability of the ad verification process. In some cases, it may be cheaper to control the ads manually or not to control them at all (this also happens).
Conclusion and Recommendations
If you have significant expenses on internet advertising and work with a large number of contractors or partners, it makes sense to automate the process of verifying ad campaigns. This will help optimize expenses and avoid speculation from the contractors.
Automatic verification of ad campaigns is essentially done through special parsing programs aimed at analyzing the content of ad blocks. These can be simple custom scripts or complex software suites with elements of computer vision, integration of third-party APIs and web services. The complexity of the systems primarily depends on the algorithms of protection on target sites. The better the protection is, the more complex and high-quality the means of bypassing it should be.
Multi-threaded operation of any parser is impossible without the use of proxies. The best ad verification proxies are mobile or residential with prompt rotation and precise geotargeting.
You can rent these very proxies from us. Froxy offers over 10 million IPs with selection up to the city and telecom operator. The number of threads may vary. You only pay for the consumed traffic. A special trial package is provided for focused testing.
