Robots Meta Tag and X-Robots-Tag

What this solution does

The 'Meta Tag Robots and X-Robots-Tag' module controls page indexing and display in search engines via HTML robots meta tags and the X-Robots-Tag HTTP header. Configuration requires no changes to templates, components, or developer involvement.

Sites with catalogs, filters, and complex URL structures often generate technical pages and duplicates. These pages are indexed by Yandex, Google, and Bing, compete with important site pages, create duplicates in search results, and hinder SEO efforts. The module allows management not only of indexing but also of how search engines process and display pages: link crawling, snippet generation, cached copies, image indexing, and display duration.

An administrator or SEO specialist creates a rule via the admin panel, specifies a URL or condition, and selects the required directives. The module automatically applies these to matching pages using the robots meta tag or the X-Robots-Tag HTTP header.

Key Features

Management of robots directives for any page

You can set indexing rules for:

- specific pages;

- site sections;

- URL patterns, such as /catalog/*;

Rules are configured using regular expressions, such as #^/catalog/.*/filter/#. This enables precise control over the indexing of both standard pages and technical URLs that should not appear in search results.

The module supports two methods for transmitting directives to search engine crawlers: via the HTML robots meta tag or via the X-Robots-Tag HTTP header. These methods can be used simultaneously, or the appropriate method can be selected for each individual rule.

The system allows setting directives for specific search engines: general rules for all crawlers, as well as separate settings for Yandex (yandex), Google (googlebot), and Bing (bingbot).

The module implements core directives commonly used in technical SEO: index and noindex to control page indexing, and follow and nofollow to manage link traversal.

The module manages Robots meta tags and X-Robots-Tag headers to provide flexible control over page indexing. Available directives include:

noarchive — prevents the display of a cached page copy in search results;

nosnippet — disables the text snippet in search listings;

noimageindex — excludes images on the page from indexing;

notranslate — blocks automatic translation suggestions;

unavailable_after — allows setting a date after which the page should no longer appear in search results.

The module supports handling GET parameters when applying rules. This enables management of indexing for filtering, sorting, search, and pagination pages that often generate duplicate content. Rules can be configured for specific parameters, allowing some parameterized URLs to remain open for indexing while others are blocked from search engines.

Import and export functionality for rules is available in CSV and XLSX formats. This feature is beneficial for large-scale projects, SEO audits, and scenarios requiring rapid configuration of numerous rules. The import and export tools facilitate bulk rule creation, transfer of settings between sites, preparation of rules in spreadsheets, and efficient implementation of recommendations following an SEO audit.

The module enables creating backup copies of settings and analyzing indexing rules outside the administrative panel. SEO specialists can upload a table with prepared rules, eliminating the need to create each rule manually.

The module operates without modifying site templates, components, or server configuration files for standard scenarios, as it connects via 1C-Bitrix system events. Once installed, administrators or SEO specialists can immediately manage page indexing through the administrative panel without requiring developer intervention for every change, reducing costs on minor technical adjustments.

It supports the 1C-Bitrix multisite mode. Settings can be shared across all sites, configured individually for each site, or linked to specific sites. This is ideal for projects with multiple domains, regional versions, or various sites hosted on a single 1C-Bitrix installation.

Important: Unlike robots.txt, this module controls directives for the HTML robots meta tag and the X-Robots-Tag HTTP header.

The robots.txt file controls search engine bot access to URLs by specifying whether a page can be crawled. The robots meta tag and the X-Robots-Tag header manage the indexing and display of already crawled pages: they determine if a page can appear in search results, if links should be followed, if a snippet should be generated, if a cached copy should be stored, and apply other rules.

If a page is blocked in robots.txt, a search engine may not see the robots meta tag or X-Robots-Tag directives. Therefore, for these directives to take effect, the page must be accessible for crawling.

Support and Compatibility

Supports PHP 8.x.

Compatible with current versions of 1C-Bitrix.

Supports multisite configurations.

Compatible with composite mode.

Works correctly with ready-made templates and custom designs.

General iT can install and configure this module and check how it works on your website.