From Scraping to Signals: Automated Threat Detection with GPT and Proxies

Modern cyber threats have become more sophisticated and complex: viruses, botnets, trojans, DDoS, social engineering, AI-driven robocalls, phishing, etc. Attacks on websites and company IT infrastructures may occur in various forms, often involving highly advanced automation tools that attackers use. However, the more attacks, the harder it becomes to analyze event logs and to make timely decisions regarding cyber threat detection. And if a security breach cannot be identified, the system cannot complete its intended function - react and solve the problem on time.

Below is our attempt to present an effective pipeline algorithm for automated threat detection using neural networks like ChatGPT, DeepSeek, LLaMa, and proxy servers to work with Big Data.

Why Automate Threat Monitoring?

Suppose you have your own cybersecurity department. Its job is to monitor events in your IT infrastructure to detect breaches and prevent them in time. This is needed to minimize or avoid the consequences of exploitation or active attacks on the corporate network.

Technically, cyber threat monitoring involves analyzing log data from specialized security systems. Some of this happens in real time, while others may be reviewed after an incident has occurred.

Human resources, however, are minimal: employers simply cannot instantly analyze massive volumes of information. Now imagine you’re managing a website with thousands of visitors per day. Just reviewing access logs could take lots of hours. Therefore, you need to hire more specialists or automate the threat monitoring process to speed up the process.

Human eyes can easily "miss" recurring patterns and overlook security issues. Automated algorithms don't make such mistakes.

Most routine tasks can be automated using specialized systems:

• TIP platforms (Threat Intelligence Platforms) aggregate threat data from various sources.
• SIEM systems (Security Information and Event Management) — centralize event processing and threat assessment.
• IDS (Intrusion Detection Systems) — detect intrusions.
• IPS (Intrusion Prevention Systems) — prevent intrusions.

However, these tools are often expensive to buy and maintain, even when outsourced as cloud-based services. Also, they don’t cover all possible tasks - there are too many nuances in each specific project, whether a website, a service, or an IT infrastructure.

A more practical solution for small startups and companies is to develop their threat detection system, with real-time monitoring and alert system integration in appropriate formats (for example, through public or corporate messengers).

What can an AI-powered threat or attack detection system look like? Let’s try to explore a case study below.

Tools and Technologies

The most critical components of a threat monitoring and detection system:

• Specialized parser or a set of various niche-specific parsers. These are required to collect data from different sources, including your website, competitor websites as well as open sources and services related to cyber threats: signature repositories, IP blacklists, spam databases etc.
• Proxy service. Proxies are essential for quickly and easily bypassing the security systems of target resources. They also help parallelize data processing streams and allow you to build your architecture for a corporate information exchange network.
• AI. A locally deployed neural network or a GPT-based service with a ready-to-use API for data analysis (e.g., pattern recognition and threat assessment).
• Orchestration system. The IT infrastructure must be scalable when working with big data, so microservices are essential. Also, if there are multiple neural networks, you’ll need a specialized middleware layer to distribute requests among them.
• Databases or data storage. Everything is clear here - the detected data (cyber threat patterns and risk assessments) must be stored somewhere. However, they should be easy and quick to access.
• Alert notification system. If you monitor threats and detect a high-risk one, your script must notify all relevant parties. Communication channels may vary.
• Scheduled scan system. The more frequent the checks are, the quicker you can respond to a security incident. The ideal case is real-time threat monitoring, though it requires more computing resources and therefore higher costs. A more budget-friendly option is periodic threat detection at set intervals (e.g., every minute, hour, day, etc.). The frequency should align with your security policy and the potential impact (i.e., how much you may potentially lose even from a minor incident).

We’ll briefly go over each point below.

Python 3.10+

Python is attractive as a programming language not only for its simple syntax and interactive nature (code can be run without prior compilation) but also for its vast ecosystem of ready libraries, particularly those intended for parsing and data processing.

Sure, other languages like Go, JavaScript, and Java can compete with Python. However, this language has long been the go-to choice for beginners, thanks to the abundance of example code for nearly any task and goal. The built-in pip package manager can work with both the global library index and private corporate repositories (the task is more convenient to manage with a pip+proxy setup).

Just in case, here are a few recommendations to keep in mind: We recommend keeping API keys, passwords, and other sensitive data required to access external services in environment variables. If you're using public code repositories like GitHub, keys and passwords should not be uploaded there, even if a repository is closed (private). And don’t forget to disable debug mode when running scripts in production.

Scrapy + BeautifulSoup / Playwright

Yes, writing a parser in Python can be done in just a few lines of code. Here are some subject-specific examples: scraping Google Scholar, a simple TikTok scraper, extracting data from LinkedIn profiles, etc.

This, however, is not enough. To work effectively with data, you'll need specialized libraries and frameworks. Thus, web drivers like Playwright or Selenium are responsible for integrating with headless browsers or anti-detect browsers. Both are powerful, but each has its quirks. Headless browsers are essential for executing JavaScript and retrieving the final HTML output. They can also complete tasks like user behavior emulation and customization of digital fingerprints.

If you need to parse the HTML code structure to extract specific data, using a ready-made syntax analyzer like BeautifulSoup makes sense.

If you need a complex, multi-threaded scraper, consider using Scrapy as a ready-made foundation that supports pipelines and spiders. It easily integrates with other libraries and services, including proxies, web drivers, and neural networks.

Note: a Scrapy + Playwright or Selenium setup is not just for scraping websites (your own or competitors’). You can also extract data from other sources: files, APIs, databases, etc. However, the most likely use cases in cybersecurity and threat intelligence are scraping public threat signature repositories and monitoring changes to your websites/web apps that could signal malware injection or defacement.

Froxy

The more parsing threads you run, the higher the chance your connections will be flagged as suspicious traffic. To bypass such restrictions, proxies are essential. The most effective types for parsing are rotating proxies based on mobile or home IPs (residential proxies). In some cases, rotating datacenter proxies may also be sufficient.

To avoid writing your proxy rotation scripts and testing their quality/affordability, consider buying backconnect proxies, like those from Froxy.

Such proxies are configured once in your code (including via environment variables), while targeting and rotation settings are managed from a web dashboard. At any time, you can block or restrict a proxy filter based on traffic consumed (to prevent misuse by team members).

OpenAI GPT-4 or Mistral / LLaMA (via API or Locally)

Artificial Intelligence (AI) can indeed replace people in specific tasks. Neural networks are particularly effective at analyzing large volumes of data and identifying recurring patterns. This is precisely what's required of them in TIP systems.

Neural networks can run on a local PC, on your server (with a full-featured API), or as ready-to-use web services.

Neural networks you can install on your own hardware include LLaMA, DeepSeek, Mistral, Qwen, Grok (from Elon Musk’s xAI), etc. These can be deployed using pre-configured environments with unified APIs and even graphical interfaces. Examples of such solutions include GPT4All, LM Studio, etc. They can also act as intermediaries when working with cloud-based (SaaS) models.

Neural networks that work solely as ready-made SaaS infrastructure include: ChatGPT (from OpenAI), Gemini (from Google), Claude GPT (from Anthropic), etc.

Technically, interacting with AI from scripts is very straightforward: input instructions (preferences, tasks, prompts) are passed in one field, while the data is provided in another. If the data doesn’t fit in a single request (all models have different token or character limits), it is split into chains.

A neural network that can run locally consumes significant computing resources. Thus, powerful hardware is required to launch it. If you need constant availability, this kind of hardware may need to be rented as hosting. In some cases, it’s more logical and more straightforward to pay for a subscription to ready-made cloud-based GPT model APIs.

LangChain and Custom Orchestration Logic

Since an AI threat detection system might simultaneously utilize several independent GPT neural networks (local LLM, on your server, or remotely via API), a universal orchestration tool may be required.

This is where the LangChain framework comes into play. It makes it possible to "communicate" with all your neural networks in a unified language — LCEL. LangChain also makes splitting large data volumes into chunks (chains) easier. It has plugins for quick connection to models cached from HuggingFace (for local execution on your device). Additionally, LangChain can be tightly integrated with LangGraph (a framework for processing knowledge graphs used for rule-building and creating controlled agents based on third-party LLMs) and vector stores (VectorStore).

SQLite / JSON for Storage

This part is relatively simple: you must choose the data storage format that best suits your AI pipeline project.

SQL databases are better suited for real-time querying with structured requests. JSON is more convenient for API exchange/delivery.

Slack API / SMTP / WebHooks для Output

Once the data has been analyzed, the script may find a signature. But what to do with the detected threat? It makes sense to send notifications to those employees who are responsible for addressing the issue.

The simplest way to implement notifications on your server is via email (using the SMTP protocol).
However, emails are not always timely or convenient. The company is more likely to use a corporate or team messenger such as Slack, Telegram, etc. External messengers have their own API interfaces or WebHook handling mechanisms.

The script should be able to send alerts through your chosen notification channels.

CRON / Prefect / Airflow for Scheduling

CRON, Prefect, and Airflow are tools for automating and managing task execution. They differ in scale, capabilities, and levels of abstraction. CRON is a classic task scheduler that is pre-installed on most Linux servers. It can run tasks on recurring schedules. Its main drawback is the lack of task status tracking, making it less suitable for monitoring and restarting in case of failures.

Airflow is a powerful task orchestrator from Apache, a framework for building, scheduling, and monitoring complex tasks (workflows). Tasks are described using DAG graphs in Python and can have dependencies and branches. The framework has a web interface for task monitoring and supports external storage such as S3, PostgreSQL, BigQuery, etc.

Airflow can be overwhelming and cumbersome, but it’s the best choice for data engineering, ETL pipelines, and big data processing.

Prefect is an alternative to Airflow that offers a more modern API, cloud scalability support, async task handling, and convenient logging.

Pipeline Architecture Overview (AI Threat Detection Included)

Let's outline a sample workflow of an algorithm for monitoring cyber threats and sending alerts upon their detection.

Stage 1: Collecting Threat Signals

At this stage, TIP receives input data from various sources, such as GitHub (searching for specific keywords in repositories and issues), thematic RSS feeds, forums, Telegram channels, or specialized API-based threat repositories.

Obviously, each source typically requires a dedicated parser to extract only the relevant information needed for cyber threat detection.

Parsing can run on a simple CRON-based schedule.

Stage 2: Normalize and Preprocess Text

Raw data with signatures will almost certainly come in unstructured. Therefore, the data needs to be cleaned (removing unnecessary HTML/CSS/JS code), split into chains/chunks (according to tokenization limits), passed through a language filter, have key entities extracted, etc.
The goal is to prepare the text for practical AI analysis and to bring it into a unified format suitable for input into the model. Otherwise, threat detection will be impossible.

Stage 3: Analyze with GPT or Open-Source LLM

Once you have a knowledge base of signatures (knowledge about how to identify threats), you can provide instructions to the LLM and then feed it the data that needs to be analyzed for potential threats.

Actually, this is precisely where the "magic" of AI-powered cyber threat detection happens.

It makes sense to note that the data is sent via API regardless of whether the AI model is running locally or remotely.

One of the tasks for the AI model should be threat level classification, based on IoC (Indicators of Compromise) identifiers. Additionally, the AI can assess threat credibility and generate a brief summary for the operator.

Requests can be customized — for example, by using prompt chaining or few-shot examples.

Stage 4: Decision Engine

The AI analysis results are passed to a logic-based decision module. It may operate based on simple rules, such as:

• If the signature severity is high and technical artifacts (IoCs) are detected, the algorithm triggers an alert, and the incident data is then forwarded to the notification module.
• If the threat level is medium, the data may simply be stored in a database for trend tracking. Notifications may also be sent to relevant parties, but with lower reaction priority.
• If the threat level is low, the event may be logged or archived without further reaction.

If needed, additional heuristics or ML classifiers can be integrated.

Stage 5: Output and Notification

Here, reports and notifications are generated. For example:

• A detailed Markdown report is sent to a Slack channel or a SOC dashboard.
  
  
• A daily or weekly summary is sent via email to security analysts.
  
  
• All events are logged in SQLite or Elasticsearch for further analysis, dashboard generation, or audit purposes.

Conclusion and Recommendations

Carefully monitor access permissions to your TIP system. Run parsers, data collectors, and AI analyzers in isolated environments like Docker, Kubernetes, and virtual machines. Avoid giving excessive rights to external services (mainly neural networks).

Store proxy access parameters, tokens, and other authorization data separately from the main codebase. Never upload them to shared or public repositories.

If possible, separate your system’s components: data collection, analysis, and storage. They should run in different processes and on various ports (or even better, in separate virtual machines/isolated environments).

To save on tokens when processing data through AI, pre-filter and validate incoming data. Validation mechanisms prevent injections, XSS, and other manipulations from external sources.

Strictly control outbound calls. If your platform sends requests to GPT via the OpenAI API or accesses external repositories, log all requests as much as possible. You can also use a proxy to reduce the risk of data leaks or abuse through LLM integrations.

Regularly audit your system and review event logs. Logs should be stored separately and monitored for rotation. Although files may be large, they are essential for incident analysis. At least once a week, check for known CVEs in your modules and Python libraries (plus, keep all components up to date).

Do not forget about API rate limits. Don’t exceed the allowed request thresholds. When splitting content into chunks, make them slightly smaller than what the documentation allows - this gives you a buffer and helps avoid issues.

Never forget about backups! There are two types of developers: those who already make backups, and those who are just going to. Backups should include your IoC database, metadata, and configurations. Do not forget to test that recovery from backups works periodically.

That's it for now!