AI Web Scraping with ChatGPT: A Practical Guide

Large websites and services are increasingly switching to JavaScript, which results in unreadable markup and complicates navigation through the DOM structure. Moreover, some intentionally add layers of complexity by replacing consistent CSS classes and attributes with unique identifiers. As a result, traditional parsers become ineffective — you can't predict what a required element will look like on the next similar page, since the identifiers are randomly generated and new ones are specified each time. In such cases, AI web scraping combined with scraping with ChatGPT can help interpret and extract data from dynamic pages more flexibly.

Even the use of headless browsers with various methods for bypassing known types of protection doesn’t help much.

Technology, however, keeps evolving, and for every action, there is always a countermeasure. Everything can be parsed, especially with the help of AI. This article is about how to use ChatGPT for web scraping.

Basic Methods Using ChatGPT for AI Web Scraping

ChatGPT is a product developed by OpenAI. Initially, it was just an LLM (Large Language Model) — a neural network trained on massive volumes of textual data. Today, however, this LLM has been enhanced with a wide range of auxiliary capabilities: web browsing, working with uploaded files and documents (including images, using computer vision and object recognition), audio stream processing (you can talk to the neural network or use it for transcription and translation of spoken language), as well as generative technologies. These capabilities make ChatGPT web scraping not just possible but highly efficient for combining vision, language understanding, and structured output generation.

Currently, ChatGPT acts as a kind of universal access terminal for all OpenAI neural networks — a single entry point for all of its products.

For website scraping, the following features are especially relevant:

• API interface that allows direct interaction with the neural network from your own programs and scripts — for example, Python-based parsers;
• LLM code generation (the model can generate working scripts in various programming languages based on simple text prompts);
• Computer vision — to analyze screenshots and extract meaningful information from them.

Let’s start with web scraping using generative AI.

Using ChatGPT as a Code Generator for Scrapers

What can be easier than having a neural network write the scraper code for you? While GitHub and other public sources are full of examples with code, digging through them to find something that actually works can be tedious for a non-developer. ChatGPT, however, is already trained and can generate functional code in many languages — Python, Java, Go, JavaScript, etc. All you need is a well-structured request (called a prompt).

Using AI Web Scraping: Formulating Code Requirements

Firstly, you can just ask ChatGPT to write code for a specific AI web scraping task. It will likely provide a working result and often include instructions on how to run the script.

Secondly, you can keep asking ChatGPT to revise the code until it works or meets your expectations. It maintains context within the same chat.

Thirdly, you can clearly specify input variables, expected output format, and functionality — even build complex frameworks. But you’ll need solid programming skills. ChatGPT web scraping helps speed up the process (like a productive junior developer), but the main architecture is still on you.

Fourthly, you can ask AI to analyze your code, find errors, and suggest recommendations for optimizations.

Fifthly, scraping with ChatGPT includes its ability to browse the web to find relevant information, including code examples and ready-to-use scripts, often serving as a more efficient alternative to Google.

Here is a sample of a script for web scraping using AI — it was written by ChatGPT on the following request: “the simplest script on Python to collect data from 'People Also Ask' blocks in Google search results”:

You don't have to try too hard — the code won't work anyway (our material on People Also Ask parsing). But what's notable is that ChatGPT initially points out all possible issues, including the fact that Google has protections in place, and recommends using a proxy. Our respect.

Other ready-made services that provide search engine result data via API are mentioned separately. It's a pity that Froxy Scraper isn't mentioned there.

The code includes detailed comments in the language you use to write your request to the AI.

In other words, ChatGPT didn't just spit out code — it anticipated all possible nuances and pitfalls. Even if the code itself doesn't work. 😉

Example 2:

Prompt: Write a Python script to scrape product data from https://www.scrapingcourse.com/ecommerce/product/adrienne-trek-jacket/, export the data in JSON, and log all actions and errors using Selenium."

Listed below are ChatGPT recommendations:

Install the Selenium library using the «pip install selenium» command

The final script:

Here is how the extracted info looks in the console:

The formatted data has been extracted to the «product_data.json» file in the folder using a script.

It works!

More details about the Selenium library.

Screenshot Recognition (AI Web Scraping for Ultra-Complex Websites)

This is the kind of situation where you can't find any reliable patterns in the CSS code or attach to a specific layout element to extract data. If you're not sure what that means, take a look at the classic People Also Ask parsing algorithm.

ChatGPT has a computer vision system. The AI can quite reliably identify the type and position of elements on the page. Therefore:

1. We just need to take a screenshot of the page.
2. Send it to ChatGPT for analysis.
3. Ask it to return the formatted data.

Here is the screenshot sample:

Chat GPT data scraping prompt:

Return the information from the screenshot formatted like this:

{"name": "PRODUCT NAME HERE","price": "PRICE","description": "PRODUCT DESCRIPTION."}Real AI Answer:{"name": "Adrienne Trek Jacket","price": "$57.00","description": "You're ready for a cross-country jog or a coffee on the patio in the Adrienne Trek Jacket. Its style is unique with stand collar and drawstrings, and it fits like a jacket should.\n\n• gray 1/4 zip pullover.\n• Comfortable, relaxed fit.\n• Front zip for venting.\n• Spacious, kangaroo pockets.\n• 27\" body length.\n• 95% Organic Cotton / 5% Spandex."}

Impressive, right?

Now, let’s check the parsing of the well-known People Also Ask block:

Here is a screenshot:

Prompt: Return a formatted list of the contents from the "People also ask" block

Result:

Integrating ChatGPT into Scraping Pipeline

Almost all the operations mentioned can be executed via API. This means you can integrate AI at specific stages of your data parsing pipeline. For example:

• A Python script will sequentially navigate through website pages.
• It will render them in a headless browser.
• It will take screenshots.
• It will send them to ChatGPT for analysis.
• It will receive formatted data in return.
• It will save that data into tables or transform it.

A more complicated workflow:

• The script may include a headless browser.
• The browser will return the resulting code of a website’s start page.
• This code will be sent to GPT for web scraping (not converted to images or anything — just raw HTML).
• ChatGPT will return a list of links to additional pages on the site.
• These links will then be used to build the main parsing queue.
• The script will then visit each of these links.
• The page data will be sent again to ChatGPT.
• ChatGPT will return the extracted data in the desired format, + it will look for more links to crawl (all in one iteration).
• Formatted data will be saved to a file or table, and new links will be added to the queue.

A more specific use case for ChatGPT:

• You send a video link (e.g., YouTube, TikTok video) or upload a file (MP4, MOV, AVI) to ChatGPT.
• Ask ChatGPT to return subtitles (the neural network extracts the audio track and transcribes it to text).
• Pass those subtitles back to ChatGPT for analysis or to create a structured description — including formatted text, headers, bullet points, etc., instead.
• The ChatGPT output can then be added to a publishing queue for your website or channel.

As a result, you get not just extracted data but ready-to-publish content. You can even ask ChatGPT to generate a Title and Description.

Here is an example script to create a parser extracting text from images (don’t forget to install relevant libraries: «pip install --upgrade openai playwright» and «playwright install»):

The script operates using the Playwright library (a driver for controlling headless browsers). The parser will make requests to the Google search engine and wait for CAPTCHA resolution. Once the CAPTCHA is solved, a full-page screenshot will be created (if the page takes too long to load, stop the process manually). This screenshot is sent via API to ChatGPT for recognition. The response returns formatted data — JSON, containing only a list of queries from the “People also ask” section. If your language differs from English, change the section name in the script and update the on-screen text shown during the CAPTCHA. This technique works exceptionally well in AI web scraping workflows, where visual and structural parsing are combined.

Instead of screenshots, you can also send the entire source code of the page. However, it is still recommended to process it through a headless browser. Otherwise, some JavaScript scripts won’t execute, and the resulting HTML will be incomplete, reducing the accuracy of AI web scraping pipelines.

Here is an example of such AI parser code:

What Can Be Automated with GPT

In short, you can automate almost anything with AI. Just in case, here's a more detailed breakdown:

• Simple text extraction – from code (formatted or not, for example, if the resulting code is returned by Selenium, Puppeteer or Playwright) or from screenshots/images. The output format can be whatever you request (you can even set your own unique structuring scheme).
• Searching and filtering specific data, and answering questions based on that data.
• Data transformation: summarizing, shortening, and extracting main ideas.
• Generating descriptions and recognizing objects in photos and screenshots.
• Recognizing text in images.
• Translating into other languages.
• Finding and fixing errors in texts/data/formatting.

Limitations and Gotchas of AI Web Scraping

The first and most significant limitation is cost. If you use AI for web scraping, all operations are measured in tokens. This includes both input data (text or images) and output data, with output usually being more expensive. Each prompt also consumes tokens as part of the input.

To estimate the token cost of parsing a single page, you can use special calculators — but even better, track actual usage statistics as requests are processed, because token usage also depends on the specific model of the neural network used.

Here is an official token calculator for ChatGPT.

Image recognition is charged at a more or less fixed rate — about 85 tokens per image.

Text is not counted by characters, but by tokenized units. The ratio varies by language. It is most efficient for English: 1000 tokens ≈ 700–800 words in English.

As a result, processing one image will cost around 500–1000 tokens, or approximately $0.01. A hundred screenshots might cost about $1.

Text and HTML can be extensive, especially if the code is sent without cleaning or filtering. If each operation consumes ~8000 tokens, the cost of parsing a single page is roughly $0.01, and 100 pages ≈ $1 (with gpt-4o).

Keep in mind that token rates differ for input and output tokens.

Cost, however, is not the only challenge.

Long HTML Issues (Tokenization)

Current token limits per request (input + output combined) for each model:

• gpt-3.5-turbo — 16,385 tokens
• gpt-4 — 8,192 tokens
• gpt-4-turbo — 128,000 tokens (Equivalent to ~300 pages of English text)

HTML contains more tokens than regular text due to the volume of elements for analysis and syntax understanding. Large pages — especially those with CSS styles and JavaScript — can quickly hit the token limit.

Even worse, a full page's code may not fit into a single request.

There are two ways out here:

• Pre-clean or remove unnecessary elements like CSS and JS in the code or select content by root tags, IDs, or class names (e.g., only send the contents of a specific <div> for analysis).
• Split extended code into chunks.

More details on how to split the code into chunks inside the AI web scraping script:

Imagine your ChatGPT web scraping script failed to bypass a CAPTCHA and sent an error screen for processing, instead of a full page screenshot. However, we required the AI to return a specific structured response. AI can’t refuse, so it “makes up” a response and sends it back.

The best scenario is: the fields come back empty. In reality, however, different models can behave differently. They have no problem fabricating small or even major details. You won’t be able to check and verify where the parser went wrong, because this is a generative network.

Similarly, if GPT misunderstands your prompt or the data is poorly structured, you won’t even receive an error — you’ll just get a “plausible-sounding” response that it made up.

API Limitations: Request Frequency, Response Time

The request rate limit depends on the model used, access type (free or paid), and subscription plan:

• GPT-4 Turbo — 500–1000 RPM (requests per minute) or 300,000–600,000 TPM (tokens per minute)
• GPT-3.5 Turbo — 3500+ RPM or 3–5 million TPM

These limits are quite reachable for large-scale enterprise projects — for instance, real-time competitor parsing.

Asynchronous neural network responses are also critical — you need to wait for the AI to generate and return its answer.

We recommend adding deliberate delays between API requests (to stay within rate limits) and handling error codes properly. Also, you need to ensure asynchronous response handling is implemented.

Advanced AI Integration Ideas

Large language models have some demerits when used via API. It is not about message length restrictions and the need to chunk large requests only. The thing is, most LLMs don’t retain conversation history via API. So you have to spend extra tokens to "remind" the model of context or guide it in the right direction. ChatGPT developers have fixed the problem already, but other LLMs suitable for AI web scraping still have it.

Additionally, ChatGPT’s error handling for scraping isn't the most informative. For complex pipelines, you should also implement state tracking, variable management, detailed logging, token pre-counting, etc. Specialized libraries and frameworks can help here.

Using LangChain or LangGraph

LangChain is a framework designed to manage long call chains. It offers ready-to-use prompt templates and formatting helpers. A framework also simplifies memory management, session persistence, logging, and debugging.

LangGraph is a library that facilitates the execution of complex, branching interactions with LLMs, based on graphs (it even offers tools for graph visualization).

As a result, it is easier to centralize configuration and simplify the syntax when building an AI web scraping script.

Embedding GPT in Scrapy Pipelines

Scrapy is a robust framework for building complex scrapers. Setting up a crawler takes just a few lines of code here. How to run Scrapy with proxies.

What’s interesting is that Scrapy easily integrates with relevant tools and libraries. For example, you can connect headless browsers and libraries like Selenium or Playwright here. If there are no ready-made integrations for something, you can implement them with code. For example, you can embed scraping with ChatGPT directly into your Scrapy pipeline. Here is a code example:

Don’t forget to connect the pipeline in your Scrapy settings.py file.

Since GPT-powered parsing is expensive, it makes sense to use it only in complex cases. For example, when recognizing images (using Vision model) or when parsing content with unreadable layout.

Collect, Cleanup, Generate Descriptions and Meta Tags in One Loop

Traditional parsers first retrieve HTML, then analyze its structure and extract needed data based on syntax analysis. After that, the data should be formatted and saved to a file or database. Each step uses a different tool or library.

ChatGPT can handle most of these steps. It can analyze clean HTML and return all the necessary data in one request: titles, meta tags, product descriptions, prices, links, etc. Since the results are structured, they’re easy to export to JSON, CSV, MySQL, etc. This makes it a powerful component in modern AI web scraping workflows.

P.S. Don’t forget to estimate token consumption!

Conclusion and Recommendations

AI and neural networks can solve a wide range of practical tasks — from image recognition to analyzing raw HTML. With ChatGPT, you don’t need to worry about CSS classes, unique attributes, or random IDs in the code. This makes it a powerful tool for AI web scraping, especially when dealing with complex or dynamic websites.

However, ChatGPT can’t solve CAPTCHAs or interact directly with target websites. You are still responsible for retrieving the page code or screenshots. Many sites

Here is how the extracted info looks in the console:

are actively protected and can detect automated traffic. So, it’s crucial to use proxies and natural digital fingerprint emulation. Here is a manual on scraping without getting blocked.

To create and rotate digital fingerprints, you can use anti-detect browsers. You can get high-quality rotating proxies from us — Froxy offers access to over 10 million IPs, with targeting up to city and carrier levels.

If AI web scraping with ChatGPT turns out to be too costly because of API token use, consider either pre-cleaning code (removing unnecessary elements) or even running a local AI model like DeepSeek, Qwen, LLaMA, etc.

However, don’t forget about extra funds needed to buy high-performance hardware or rent dedicated hosting.