IndexNow: Accelerating Indexing in Yandex and Bing

What this solution does

IndexNow is a module for automatically notifying search engines Yandex and Bing about new, modified, or deleted site pages via the IndexNow protocol.

New products, sections, and articles regularly appear on the site, while prices, availability, and page addresses change. Without automation, search engines may learn about these changes only after several weeks. As a result, new pages do not appear in search results immediately, and information about existing pages remains outdated for a long time.

The module monitors changes in 1C-Bitrix and automatically sends the corresponding URLs via the IndexNow protocol. This accelerates the discovery of new pages and the re-crawling of modified ones, reducing the time until search data is updated.

Module benefits:

Faster change detection: Yandex and Bing receive timely notifications about new, modified, and deleted pages, speeding up their crawling and search index updates.

Automation: There is no need to manually send URLs after publishing or modifying pages. The module monitors events and generates notifications automatically.

Flexible configuration: You can select specific sites, information blocks, and events that require notifications to be sent.

Submission monitoring: request history, search engine responses, statistics, and diagnostic tools allow control over results and identification of failures.

Key features

1. Automatic change monitoring

The module tracks events in the 1C-Bitrix system and triggers IndexNow notifications when infoblock elements or sections are updated. Supported event types include:

creation and editing of products and sections;

URL changes;

activation or deactivation of elements;

deletion of elements and sections;

product price updates;

product stock changes.

For example, when a new product is created, its URL is automatically placed in the submission queue. Upon subsequent page changes, the module may resend the address according to current settings.

2. URL submission to Yandex and Bing

Two operating modes are available:

direct submission to search engines — separate notifications for Yandex and Bing;

submission via the shared IndexNow service — using a selected protocol service.

With direct submission, results and retry attempts are tracked separately for each search engine. Temporary unavailability of one service does not block processing of notifications for the other.

3. Content tracking configuration

Users select the sites, infoblocks, and change types subject to monitoring.

The IndexNow module by Webtaktika allows flexible configuration of notification delivery for specific elements and sections, enabling the selection of events that trigger adding URLs to the queue. For instance, notifications can be activated solely for the product catalog and the articles section, excluding service information blocks. A single setting applies to all matching objects, eliminating the need to create rules for each page manually.

When a section address changes, the system tracks the impact on nested sections and products. If the URLs of nested pages are generated based on the parent catalog path, their addresses are automatically updated. The module generates tasks for all affected pages, transmitting information about catalog structure changes along with the updated nested URLs to search engines.

To optimize load, URLs are accumulated in a queue and sent in batches of up to 10,000 addresses per request. This approach allows processing mass catalog changes without sending a separate HTTP request to the search engine service for every element save. The queue processing frequency can be configured, including individually for each site.

The module tools include viewing the sending history, analyzing statistics, and performing diagnostics.

The module records the history of notification submissions and allows detailed verification of the processing status for each URL. The administrative panel displays the page address and domain, the event that triggered the submission, the date and time of the action, the search engine name, the number of submission attempts, the HTTP response code and its outcome, the error description or reason for skipping, and the time scheduled for the next attempt. This enables users to confirm, for example, that a product URL was submitted after a price change, to identify the exact submission time, and to view the search engine's response. Diagnostic tools help verify the correctness of IndexNow settings, the status of the key, and the availability of required resources. Data regarding addresses and submission history can be exported to a CSV file for further analysis.

Manual URL Submission

Allows sending individual pages or lists of addresses manually without the need to modify infoblock content. This functionality is useful after conducting an SEO audit, updating static pages, or performing other tasks that require re-notifying search engines. Addresses undergo automatic validation before being added to the queue.

Exclusions and robots.txt Checking

The module enables excluding unnecessary URLs from submission and respecting restrictions defined in the robots.txt file. The robots.txt check can be enabled or disabled separately for each site.

When the check is enabled, the module adheres to access rules for search engine robots defined in the robots.txt file. If this file is temporarily unavailable, submission is deferred until the next check cycle.

Multisite support

The module operates within the 1C-Bitrix environment and supports multiple sites in a single installation. This includes resources hosted on different domains or in subdirectories. Common settings or individual parameters can be applied for each site.

If “IndexNow: Accelerating Indexing in Yandex and Bing” fits the task, General iT can install the module, configure it and verify the result in the project.