Sitemap.xml generator with page verification

What this solution does

Sitemap.xml generator without garbage: only pages that actually open are included, with address verification, robots.txt and access rights checks, reporting, and automatic regeneration.

The module builds a sitemap.xml file from information blocks and static site pages, verifying each address before adding it to the map. Search engines consider it an error to have addresses in the sitemap that return a 404 status, redirect, or are blocked in robots.txt. The module filters out such addresses and displays the reason for their exclusion in the report.

The module solves the problem of pages from deleted sections, empty lists, redirect links, and pages closed to guests appearing in the map. It verifies that the site actually serves the address: the presence of a file or urlrewrite rule with an existing handler, no blocking in the .access.php file, and a 200 response code.

If the information block URL template does not match how the component builds addresses on the site, the module detects this (addresses are not served or do not respond) and allows setting a custom URL template for lists, sections, and items.

The module eliminates the need for manual map regeneration: the process runs automatically on a schedule and after changes to products and articles.

The module identifies site configuration errors that hinder indexing: URL rewrite rules that intercept any address (such as #^# from a component with an empty SEF folder), and the return of a 200 status code for non-existent pages (soft 404). These issues are reported in the final report.

Profiles: each sitemap file corresponds to a separate profile containing settings for the site, domain, protocol, filename, sources, filters, and schedule. In multisite installations, an unlimited number of profiles can be created. Upon installation, a draft profile is automatically generated for each site with pre-selected information blocks.

Information blocks: individual change frequency (changefreq) and priority (priority) values can be set for the list page, sections, and items. Section filters are available for all sections, all except selected ones, or only selected ones (including sub-sections). The module supports skipping empty sections, generating only purchasable products, adding product and article images (image:image), and using a custom URL template.

Static pages: the module scans site files to locate folders containing index.php files that load the site template. Service scripts, AJAX handlers, redirects, folders closed to guests, and pages with the robots=noindex directive are ignored. Additionally, arbitrary addresses can be added using string format, for example /promo/ 0.8 weekly.

The module generates a sitemap.xml file with page verification. Page priority is determined by nesting depth: the homepage receives 1.0, the first level 0.8, and subsequent levels decrease by 0.1 steps. These values are configurable. A minimum priority can be set for specific sections, for example, /catalog/ must not be lower than 0.6. Custom update frequency and priority settings are available for any page.

Filtering uses exclusion masks in robots.txt syntax (characters *, $) and respects robots.txt rules for a selected search engine robot (all, Yandex, Googlebot).

The output format complies with the sitemaps.org protocol. It includes the last modification date (lastmod). Each file supports up to 50,000 URLs or 50 MB. If limits are exceeded or if the "file per source" setting is enabled, the system creates sitemap parts and an index file. Cyrillic domains are converted to punycode, and URLs use percent-encoding. Files are replaced atomically to ensure search engines never receive incomplete data.

Generation can be triggered from the admin panel with a progress indicator that operates in short steps without timeouts, even on large catalogs. Automatic generation is also supported via an agent: on a schedule (from 1 hour to 1 week) or 10 minutes after changes to profile infoblock elements and sections.

A report is generated after each run, detailing the number of URLs provided by each source, the count of skipped URLs, the reasons for skipping, and examples of those URLs.

Optionally, the module appends the Sitemap directive to the robots.txt file.

Access rights allow viewing profiles and reports or full configuration, managed separately for user groups.

Sitemap.xml generator with page verification

Environment requirements: PHP version 8.2 or higher, core module version 22.0 or higher, the Information Blocks module enabled, and the PHP dom extension. The dom extension is available in the BitrixVM environment and on most hosting providers. No external third-party libraries are used.

Server response verification requires the site to be accessible from the server itself via its domain, with HTTPS certificate validation performed. Requests are sent only to domains specified in the site settings. If the site is password-protected, automatic response verification is disabled, while other module functions remain operational.

Addresses handled by D7 routing or web server rules (as opposed to physical files and the urlrewrite.php script) do not pass the "site serves address" check. For such sites, this check can be disabled in the module profile.

It is recommended to run agents via cron. This allows large catalogs to be generated entirely in a single run, rather than processing 8 seconds per site visit.

The module does not transmit site data to third parties. The source code is fully open with no encrypted sections.

The module has passed verification using the marketplace-security-skills methodology for 1C-Bitrix: the security-journal.json file shows a status of ready_for_submission.

If “Sitemap.xml generator with page verification” is required, General iT can help with installation, configuration and a technical check after integration.