Let's imagine you've written a simple script using the popular JS library Axios and sent a request to a specific page on your target website. The server responded with the expected 200 status code – but instead of a full HTML document, it returned an empty page, an access error message, or content with no real value for your task (for example, without prices or without key page blocks). Sound familiar? This is a problem every second scraping developer runs into.
Most likely, you didn't dig into the HTTP headers – and as a result, your scraper was identified on the very first connection. Below, we'll help you configure Axios's default headers for scraping.
Technically, HTTP headers are service fields that are filled in and sent along with the request body to a web server, or returned together with the server's response. These fields contain important metadata about the connection, the client, and the request context. In particular, they help synchronize details such as language, encoding, current time, HTTP protocol version, supported content types, compression methods, and more. For example, cookies are also transmitted via HTTP headers.
The set of headers that are relevant "right now" keeps changing – there isn't one universal, fixed standard. On top of that, websites can define their own custom headers (they often start with the "X-" prefix). The most complete lists of headers are easiest to explore on Wikipedia or in the MDN developer documentation.
In practice, it looks roughly like this (each header passes its parameters after a colon):
GET / HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.9,ru;q=0.8
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Upgrade-Insecure-Requests: 1However, the full combination of headers can also be used as a signal source for fast and accurate detection of bots, scrapers, or automation systems. By analyzing a client's HTTP parameters, a server can often determine who is actually making the request.
The server compares your headers to the patterns real browsers typically send, and if they don't match, the connection may be blocked. This kind of check can be performed on the scraper's very first request.
HTTP headers can confidently be treated as digital fingerprints. But it's important to remember that these two concepts aren't identical: headers reveal only a small part of what anti-bot systems analyze in reality.
Axios is an isomorphic HTTP client for both the Node.js environment and real browsers. In JavaScript scrapers, it's installed like any other npm library:
npm install axiosIn code:
import axios from 'axios';But you can also use it directly on a web page:
<script src="https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js"></script>What I want to emphasize: Axios lets you set headers globally, so you can configure them once and have all subsequent requests send the same predefined header values.
At the same time, you can override those headers for a specific request if needed.
So, what does the minimal "gentleman's set" of common headers for Axios scraping look like:
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.9,en-US;q=0.8,en;q=0.7',
'Accept-Encoding': 'gzip, deflate, br, zstd',
'Referer': 'https://www.google.com/',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'cross-site',
'Sec-Fetch-User': '?1',
'Upgrade-Insecure-Requests': '1'
}Now let's bring in the "explanation crew" – what each header does and why it matters:
This is only the basic minimum. A specific site or web server may analyze many more headers coming from the client. The main goal of a scraper is to make its HTTP headers indistinguishable from those of a real browser.
Perfect proxies for accessing valuable data from around the world.
It all depends on a combination of factors: the type of website (many large platforms are complex web applications that don't work without JavaScript, so accessing them typically requires a headless browser or an anti-detect setup) and the level of protection in place (how deeply the site analyzes your digital fingerprints).
You can only determine whether you need to imitate browser headers in Axios scraping through testing. Open the site, study its architecture, try sending requests without headers, then with a minimal set, then with additional ones – and keep iterating until the scraper starts working.
Without the right header set, a ban is almost guaranteed if the site uses advanced protection systems such as Cloudflare, Akamai, PerimeterX, DataDome, Kasada, and similar. You can often spot them by the scripts the site loads and by specific attributes in the page code.
Also note: in some cases, even switching entirely from Axios to headless browsers won't solve the problem, because headless setups have their own default fingerprints as well – and they require careful tuning to hide those traces.
A cookie is a special piece of text data that a web server sends to the browser in the Set-Cookie header. The browser stores it and automatically sends it back on subsequent requests to the same site, but this time in the Cookie header. This is exactly how user sessions and authentication are maintained and managed.
Session cookies store a dedicated session identifier (session ID). The server links that identifier to a separate set of data: login, authentication state, shopping cart, and so on. Without the right cookies, a scraper looks like an unauthenticated guest – so it either gets an access error or receives empty/anonymized data.
Many web firewalls (WAF systems) use cookies as a mechanism to filter out bots. For example, they may set their own tokens and cookies that work like a unique access pass: __cf_bm and cf_clearance for Cloudflare; _px and _px3 for PerimeterX; and so on.
Here's the ethical bypass of WAF protections.
When scraping via Axios without a browser, cookies are not stored or sent by default. That's why you have to manage them manually:
You will definitely need cookies if you want access to data available only to logged-in users, or if the site uses advanced token-based protection.
A CSRF token (Cross-Site Request Forgery token) is a unique identifier generated by the server for each session, form, or request. The purpose of this token is to ensure that an incoming request (POST/PUT/DELETE) comes from a real user, not from a malicious website acting through the victim's browser. In practice, an automated script or an attacker could copy a user's cookies and send a request on their behalf. Such interception can happen, for example, when a user visits a site controlled by the attacker.
If you're dealing with an Axios-based scraper, a CSRF token typically blocks data submission via POST, PUT, and DELETE, since these methods are often tied to authentication. To get past this protection, you need to handle not only cookies, but also read tokens from the HTML (usually from special attributes) and work with dedicated CSRF-related headers. Token values must be refreshed on every interaction with the server.
Important: axios/requests-style bots often can't retrieve tokens if they are generated dynamically via JavaScript after the initial HTML is delivered.
Two-step (multi-step) requests are a classic bypass pattern:
Below are typical situations showing how to work with common and custom headers in Axios when scraping.
Setting headers globally means you configure the parameters once. They replace the default values and will be used in all subsequent requests within the script.
import axios from 'axios';
axios.defaults.headers.common = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36 Edg/133.0.0.0',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
'Accept-Language': 'ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7',
'Accept-Encoding': 'gzip, deflate, br, zstd',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1'
};
axios.get('https://example.com').then(r => console.log(res.data));This approach is convenient for large-scale scraping of pages from the same website – when you need a consistent digital fingerprint across requests.
Note: default values can be overridden by headers specified directly in an individual request.
This is the most common and flexible way to fine-tune Axios scraping for a particular call.
axios.get('https://shop.com/search?q=iphone', {
headers: {
'Referer': 'https://www.google.com/search?q=buy+iphone', // simulate coming from search
'Sec-Fetch-Site': 'cross-site', // came from a different domain
'Sec-Fetch-Mode': 'navigate', // full navigation
'Sec-Fetch-Dest': 'document', // target is an HTML document
'Sec-Fetch-User': '?1', // user-initiated (click)
'Priority': 'u=0, i' // newer Chrome priority header
}
});Let's look at interception using the example of automatically adding a CSRF token.
axios.interceptors.request.use(config => {
// before each request, check whether there is a fresh CSRF token
const token = localStorage.getItem('csrf_token') ||
// or take it from a global variable / a domain-based Map
globalCsrfTokens?.[new URL(config.url).origin];
if (token && ['post', 'put', 'delete', 'patch'].includes(config.method)) {
config.headers['X-CSRF-Token'] = token; // the most common header name, but it may be customized on a specific site
// Alternatives: 'X-XSRF-TOKEN', 'X-CSRFTOKEN', or '_csrf' in the body, etc.
}
return config;
}, err => Promise.reject(err));This is most useful when working with forms, adding items to a cart, submitting POST-based filters, and similar scenarios.
To automate cookie handling, you can plug in additional libraries.
import { wrapper } from 'axios-cookiejar-support';
import { CookieJar } from 'tough-cookie';
const jar = new CookieJar();
const client = wrapper(axios.create({
jar,
withCredentials: true, // required for cross-domain cookies
headers: { 'User-Agent': '...' } // headers can be set here globally
}));
// Login – cookies are stored automatically
await client.post('https://site.com/auth/login', {
email: 'user@example.com',
password: 'strongpass123'
}, {
headers: { 'Content-Type': 'application/x-www-form-urlencoded' }
});
// Now all subsequent requests include cookies
const profile = await client.get('https://site.com/account/profile');Important: if you don't set withCredentials to true, cookies from subdomains or different paths may not be saved.
axios.interceptors.response.use(
response => response,
async error => {
const { response, config } = error;
if (response?.status === 403 || response?.status === 429) {
console.warn(`Blocked (${response.status})! Switching headers`);
// Example: rotate User-Agent + Client Hints
config.headers['User-Agent'] = getNextUserAgent();
config.headers['Sec-CH-UA'] = getRandomCHUA();
// Retry with the same parameters (but new headers)
return axios(config);
}
if (response?.status === 401) {
// For this error type, you could implement re-auth under a different user, for example
}
return Promise.reject(error);
}
);const browsers = [
{ ua: 'Chrome 133 Win', chua: '"Google Chrome";v="133", ...' },
{ ua: 'Edge 133 Win', chua: '"Microsoft Edge";v="133", ...' },
{ ua: 'Firefox 124 Win', chua: '"Firefox";v="124", ...' }
];
function randomBrowser() {
return browsers[Math.floor(Math.random() * browsers.length)];
}
axios.get('https://target.com', {
headers: {
'User-Agent': randomBrowser().ua,
'Sec-CH-UA': randomBrowser().chua,
'Sec-CH-UA-Platform': '"Windows"',
'Sec-CH-UA-Mobile': '?0'
}
});axios.get('https://api.ipify.org?format=json', {
proxy: {
protocol: 'http',
host: 'proxy.example.com',
port: 3128,
auth: {
username: 'myuser123',
password: 'strongpass456'
}
}
})
.then(res => console.log(res.data));Axios (just like requests, httpx, curl, node-fetch, or any other HTTP client) is not a browser. Even if you spoof every header to match what a real browser sends, the library can still be detected as soon as JavaScript execution is involved. Axios works only with HTTP requests and responses. Anything that must be executed inside a browser is outside its capabilities.
Many advanced WAF systems analyze not only headers, but also how the browser behaves while rendering pages: what fonts are available, whether the user moves the mouse, clicks buttons, whether pixels end up where they should after real rendering, and so on.
Sometimes a single simple JS token check – where the token is validated against a corresponding header – is enough for Axios to fail the verification.
Axios performs quite well for scraping, but it has a limited scope of use. In practice, it makes the most sense when scraping websites that don't rely on advanced WAF-based protection.
Proper header management in Axios helps you achieve the highest possible level of trust. The key goal is to imitate a real browser as closely as possible: send natural values for the user agent, locale, accepted content types, and similar parameters; store and reuse cookies; and handle CSRF tokens.
Another important defensive layer for any scraper is high-quality proxies. Axios can send and receive requests only through HTTP(S) proxies. For other proxy protocols, you'll need third-party libraries such as socks-proxy-agent.
You can rent reliable residential and mobile proxies from us. Froxy offers a pool of over 10 million IPs with targeting down to city level and mobile carrier. Coverage includes 200+ countries, with rotation by time or on every request.