The advertising market is continuously growing, with the growth rate of online advertising significantly outpacing that of all other ad types (on TV, radio, billboards etc.). This is a global trend. Just in case, here is a link to a study by Markets and Markets (2023).
Naturally, when outsourcing the display of their ads on various websites and in contextual advertising networks, advertisers want some clear guarantees: to ensure adequate coverage, correct materials display (not just inflating statistics) and relevant targeting set up.
But here arises the problem - how to control contractors? In this material, we will discuss options for verifying online advertising and how this may be related to proxy rental.
What Is Ad Verification?
Ad verification is the process of qualitative and quantitative control of the display of ad messages to ensure compliance with the conditions of their display outlined in partnership agreements. For example, the verification may include checking the display region, targeting parameters like reaching the intended audience, timing of display and other targeting criteria.
Controlling contractors responsible for displaying your ad messages helps address several related issues:
- Minimization and optimization of advertising expenses: you can verify the effectiveness of various ad strategies, develop new directions or discontinue unprofitable ones;
- Creating evidence for dispute resolution: to prove the contractor's misconduct and demand proportional compensation or issue resolution;
- Verifying compliance with the conditions of actual ad display with the terms outlined in contracts with advertising agencies or other partners;
- Identifying fraudulent traffic and low-quality platforms, attempts at manipulation and the use of bots: this allows you to discontinue working with them in future;
- Prompt detection of errors: broken links, missing images, incorrect prices etc.
In some cases, monitoring competitors may be required to analyze their advertising policies and marketing strategies.
Challenges of Ad Verification
To achieve the outlined tasks, a high-quality technical component is required - everything that allows the emulation of user behavior, virtualization of parameters of digital footprint and location (for targeting conditions verification), taking screenshots and collecting evidence.
In fact, the advertising verification script differs from parsers. It is just focused on blocks of ad messages not on the main content. Such a specialized parser can work on different target sites or on a specific one. In the former case, a separate handler for the page structure has to be created for each target resource.
As with regular parsers, scripts for ad verification may encounter attempts at blocking:
- Due to repeated automatic requests with a specific timing interval;
- Due to incorrect or suspicious HTTP headers within HTTP requests and responses to the server (to the target site);
- Due to falling into forms or trap pages (URLs that ordinary users cannot see yet they can be seen by parsers due to sequential parsing of HTML code);
- Due to the lack of JavaScript and Ajax support (applies to many dynamic sites and PWA applications that are almost entirely composed of scripts executed on the browser side);
- Due to captcha refusal;
- Etc.
Detailed problems with blocking parsers were described in the article "Scrape Like a Pro! Best Practices for Web Scraping Without Getting Blocked".
Why verify ad displays at all? Contractors or end sites that provide banner rotation services on their pages may:
- Manipulate statistics results (displays, clicks and form submissions);
- Manipulate clicks and other activity using automation tools (bots and scripts);
- Hide ad messages or display them with violations - under a layer of other elements, in reduced frames, in the wrong place specified in the contract etc.;
Additionally, intermediary services may not control targeting conditions themselves: the location of customers, their interests, digital footprints etc.
All this ultimately results in the waste of your ad budget without producing the expected result: in the form of increased brand recognition, sales growth or promotion of new products/services.
Note that not just HTML parsing scripts may be required for the highest quality ad control, but also complex solutions with elements of computer vision (for screenshot recognition), tools for working with a large number of user profiles (anti-detection browsers), APIs for user behavior emulation (headless browsers) and external services for solving captcha (with computer vision or those that involve manual labor).
Proxies for Ads and Ad Verification
Now let's talk about how the process of ad display verification and large proxy databases are related.
As mentioned earlier, target sites (internet resources where your ads or competitors' ads are displayed) can defend themselves using available methods - they detect parasitic traffic and block all automated requests.
Naturally, many security checks are based on the analysis of the structure of browser HTTP headers/requests as well as on the analysis of the IP address.
Based on the IP address, you can determine the approximate location of the user, their network operator, the IP pool owner as well as the type of connection (mobile network, home or corporate).
And if the user's profile does not match the IP parameters, this is the first reason for blocking, or at least a reason to request re-authorization/verification of the user's "humanity" (captcha, solving logical tasks, confirmation of authorization by additional factors etc.). For example, when a mobile app client tries to connect from a server IP, the site may have legitimate security concerns about the session.
How Proxies Are Used in Ad Verification Processes
Even if you have high-quality user profiles (digital fingerprints) at your disposal, they must work with equally high-quality IP addresses.
But where can you find IP addresses that meet your criteria? In proxy server networks.
As mentioned earlier, the match between the IP address and user profile is crucial. Advertising networks can determine many different parameters based on IP, including location, connection type etc. This means that an ad verification proxy should not just be a mere "proxy." It must closely match real clients from specific regions or even specific cities.
To acquire proxies that match your criteria, you have to be aware of their types and associated features.
Different Types of Proxies Used for Ad Verification
Proxies vary: they can be direct and reverse (back-connect proxies), transparent and anonymous, public (open) and private (exclusive), static and dynamic. However, the most critical division of proxies for ad verification is based on the type of device they belong to.
In particular, the following proxies are distinguished: mobile, residential and server (corporate) proxies.
- Mobile Proxies: Operate on the basis of addresses of mobile operators. They have the highest trust level from security systems (minimum captcha and blocking), so their rental is often more expensive. However, they have weak stability (life span) and minimal bandwidth (many countries still use 3G networks). Read more details about mobile proxies, their advantages and disadvantages.
- Residential Proxies: Operate on the basis of addresses from home internet providers (familiar wired networks). The trust level here is slightly lower, but stability and bandwidth (connection speeds) are higher. Providers of such services can offer a large pool of addresses, ensuring quick IP rotation. Read more details about residential proxies, their advantages and disadvantages.
- Server Proxies: Operate on the basis of addresses owned by companies. These are usually hosting, IT infrastructure or corporate data centers. Corporate addresses raise the most suspicion from security systems, making them less suitable for parsing and ad verification tasks. Their application lies in dealing with heavy content when unlimited or wide stable access channels are required. If the pool of corporate addresses is large, they may be used similarly to residential proxies. However, it is worth considering that the number of errors due to IP blocks will be higher in this case.
It's worth mentioning the convenience of selecting addresses and the method of connecting proxies to niche software. Classic direct proxies are usually connected by lists, while the latter are exported in a specific format: IP:port@login:password etc.
Back-connect proxies can be connected with just one line or through multiple ports. Technically, these are the same proxies, in the same format (IP:port@login:password etc.), but incoming requests are routed within the network and can come out with completely different IPs. Consequently, the rotation of non-functional proxies can be automated as well as the selection of proxies based on your criteria: address type, location, retention time etc. The connection parameters remain the same because the entry point to the proxy network does not change.
Naturally, proxies with reverse feedback are more convenient and practical, letting you save much time on connection and automatic rotation.
Benefits and Limitations of Using Proxies for Ad Verification
Only proxy services with extensive coverage and precise targeting will help you emulate location so that security and ad rotation systems perceive the visitor as a real customer.
No other technical solution can achieve this. For example, VPN networks always have a limited choice of addresses and do not provide conditions for their selection in specific locations.
On the other hand, proxies alone are not enough. To ensure the ads proxy verification process is of the highest quality, other technical aspects need to be considered:
- Database of digital fingerprints, which will be used to analyze user profiles and display ads. For example, these need to be preliminary "warmed up" by visiting thematic websites and entering specific search queries in Google, Bing etc. Special software complexes like anti-detect browsers are used for profile rotation;
- List of target resources where ad display will be verified (+ determination of their bypass frequency);
- Captcha resolution in case of suspicion of automated traffic (captchas can be solved manually, but if we're talking about thousands of parallel requests, automation systems are required);
- JavaScript Execution. Standard HTML parsers cannot handle JS scripts. Therefore, the page needs to be rendered in a full-fledged browser before analyzing its content. Headless browsers and special libraries for API organization are required for this purpose.
- Depending on the complexity of security systems on target sites, bypass systems need to be devised (different delays, correct HTTP headers, JS support, bypassing traps etc.).
Be sure to calculate the approximate traffic and all associated expenses. Based on this data, you can obtain adequate figures for the future budget and assess the profitability of the ad verification process. In some cases, it may be cheaper to manually control the ads or not to control them at all (this also happens).
Conclusion and Recommendations
If you have significant expenses on internet advertising and work with a large number of contractors or partners, it makes sense to automate the process of verifying ad campaigns. This will help optimize expenses and avoid speculation from the contractors.
Automatic verification of ad campaigns is essentially done through special parsing programs aimed at analyzing the content of ad blocks. These can be simple custom scripts or complex software suites with elements of computer vision, integration of third-party APIs and web services. The complexity of the systems primarily depends on the algorithms of protection on target sites. The better the protection is, the more complex and high-quality the means of bypassing it should be.
Multi-threaded operation of any parser is impossible without the use of proxies. The best ad verification proxies are mobile or residential with prompt rotation and precise geotargeting.
You can rent these very proxies from us. Froxy offers over 8 million IPs with selection up to the city and telecom operator. The number of threads may vary. You only pay for the consumed traffic. A special trial package is provided for focused testing.
