Protection against parsing (parsers and bots)

What this solution does

The Anti-Parser module provides comprehensive protection for 1C-Bitrix websites against data scraping, automated data collection, and aggressive bots. It prevents content theft, server overload, and the leakage of product and price databases to competitors. The solution operates on the PHP side and consolidates multi-layered security into a single interface: request rate limiting, behavioral analysis, browser verification, geo and IP filtering, API and content protection, Cloudflare integration, along with a notification and monitoring system.

The module is ready for immediate deployment with secure default settings. It includes gentle modes for testing without the risk of blocking legitimate visitors. The solution functions correctly in clustered environments and behind load balancers, with minimal impact on caching and search engine indexing.

The Kommutator.net anti-scraping and bot protection module provides traffic control and filtering. Request limits per IP are enforced using Redis, Memcache, or file storage, ensuring atomic counting and correct operation in clustered environments. When a threshold is exceeded, the system triggers a CAPTCHA challenge: standard Bitrix, Google reCAPTCHA v3, or Yandex SmartCaptcha.

Bot Detection and Blocking

The system identifies over 40 malicious User-Agent patterns and detects headless browsers. Threat scoring ranges from 0 to 100, blocking traffic based on a combination of indicators rather than a single weak signal to reduce false positives. Legitimate search engine bots (Google, Bing, Yandex, Apple, Baidu, DuckDuckGo, Facebook) are verified via forward-confirmed reverse DNS, preventing Googlebot spoofing. A soft mode is available for safe testing, logging events without blocking. Users can configure custom white and black lists for User-Agents.

Browser JS Verification and Behavioral Analysis

Friction-based JS verification filters out primitive parsers and automated browsers while leaving verified search engines and APIs unaffected. Behavioral analysis confirms genuine user interaction through signed cookies, further lowering false positive rates. The system also analyzes cold direct visits with selectable modes: off, monitoring, or blocking.

IP Blocking

White and black lists support individual IP addresses, CIDR subnets, masks, and IPv6 addresses.

The Kommutator anti-scraping module automatically blocks IP addresses when the hourly violation limit is exceeded, ensuring cluster consistency.

Geo-blocking

Implemented via black and white country lists. User location is determined through a chain of sources: the Cloudflare header, web server headers (nginx/apache GeoIP and GeoIP2), the built-in 1C-Bitrix GeoIP (SypexGeo, MaxMind, etc.), the Bitrix\Sale\Location\GeoIp component, the PHP geoip extension, or the MaxMind database. Only one active source is required for functionality.

Content protection

Includes three levels of protection, blocking text copying and the context menu, along with a configurable "Copying forbidden" notice. Forms are protected using honeypot traps, while email addresses and phone numbers are obfuscated.

API protection

Achieved through request rate limiting, X-API-Token validation, CORS policy management, and path-based security.

WAF integration

Enables synchronization of IP blocking rules with the Cloudflare service.

Notifications

Support sending messages via E-mail, Telegram, and Webhook with configurable thresholds, spam protection (cooldown), and retry mechanisms.

Monitoring and logs

Provide a dashboard with hourly and daily activity graphs, top lists of countries, IPs, and events, plus filtering capabilities. Data can be exported to CSV and Fail2Ban, with logs written to files and/or a database, including statistics distinguishing between human users and bots.

Multi-site support

The module supports deployment across multiple websites.

Behavioral settings are configured individually for each site with a fallback to global values. The infrastructure (CAPTCHA keys, IP lists, Redis, WAF, notifications) is shared.

The module consolidates all necessary protection features into a single solution, eliminating the need to assemble protection from multiple tools. Safe default values and gentle modes allow for deployment without the risk of blocking legitimate visitors. Verified search engines pass all checks, ensuring indexing is not affected.

The solution is ready for high load and cluster environments. It uses shared counters in Redis or Memcache, implements protection against X-Forwarded-For spoofing, and ensures that lazy sessions do not disrupt the composite cache. The module automatically integrates with existing infrastructure, including Bitrix's native GeoIP and Cloudflare.

Settings are accessible via the «Services → Anti-Parser Protection» section or through «Settings → Product Settings → Module Settings → Anti-Parser Protection».

Warning: Incorrect configuration may result in loss of access to the site. The module is designed to block unwanted traffic, but overly aggressive or erroneous settings can block real users and hinder access to the resource.

Risks of incorrect anti-parsing configuration

An excessively low request rate limit can block regular users and legitimate integrations. Geo-blocking in whitelist mode with an incorrect country list or with blocking enabled for undefined regions may render the site inaccessible to all visitors. Adding IP addresses or subnets used by clients, CDNs, payment gateways, or integrations to the blacklist can also result in traffic loss. Enabling aggressive modes, such as bot blocking or direct access blocking, without prior testing increases the risk of service disruption. Errors in the trusted proxy list behind a load balancer can cause all traffic to be treated as originating from a single IP address.

Important: by default, requests from the command line (CLI), cron jobs, and authorized administrators are not checked. This reduces the risk of completely blocking access to the administrative panel. However, with incorrect rules, the public part of the site may become unavailable to visitors, and an incorrect server configuration (proxies, cache, access rights) can exacerbate the consequences.

Mandatory backup creation

Before installing the module and before making any changes to its settings, you must create a backup of the site and the database. The backup must allow for a complete system restoration.

The module protects website files, including the module directory and configuration files, as well as the database, specifically the 'b_option' configuration tables and module logs.

Before installation, ensure you have a verified method for recovery and access to the server or hosting environment without using the web interface (SSH, hosting file manager, control panel). This is required to quickly disable the module or revert changes in case of a failure.

Recommendations for safe implementation:

1. Create a full system backup.

2. Activate protection mechanisms gradually, starting with soft modes: enable bot blocking in 'log only' mode and direct access analysis in 'monitoring' mode.

3. Monitor the Dashboard for several days: evaluate which users would be blocked and check for false positives.

4. Once correct operation is confirmed, switch the mechanisms to active blocking mode.

5. Pre-add trusted IP addresses (office networks, monitoring systems, integration gateways, payment gateways) and required subnets to the whitelist.

6. If the module is installed behind a load balancer, correctly configure the 'Trusted Proxies' section.

7. Know the emergency disable procedure: use the global switch in settings or delete/renaming the module directory on the server.

Emergency Disable

If the website becomes unavailable due to module settings:

To disable the module, use the global switch in the settings if access to the admin panel is available. Alternatively, rename or delete the module directory on the server. If necessary, unregister the module or restore the site from a backup.

General iT can install and configure this module and check how it works on your website.