Top Man Digital: llms.txt

What this solution does

Search assistants and AI agents analyze sites differently than traditional crawlers: they require a concise machine-readable summary rather than full HTML. The llms.txt standard is a Markdown file placed in the site root, containing the company name, brief description, key facts, and a list of sections with links. This module automatically generates such a file from your Bitrix site data and keeps it up to date.

Without this file, assistants form an understanding of your store based on random pages in their index, which may result in displaying outdated prices and failing to grasp the product range.

The module exports a catalog with prices and stock availability. Sections are structured as a tree, with each node displaying the number of items, price range, and count of available positions. Data is drawn from the trade catalog, including prices for trade offers.

The functionality works without the Trade Catalog module. If prices and availability are stored in standard infoblock properties, you can specify them manually; the module will read and output these values correctly. Stock availability is recognized in three ways: as a numeric remainder value, as a filled field indicating presence, or as a match with a specified value, including list options.

Top Man Digital Module: llms.txt

The system aggregates data from six source types: information blocks (sections, elements, or both), static pages from the file system, menu items, sitemap.xml, the Organization block containing contact details and working hours, and a Q&A information block.

A ready-made file structure is generated, including sections for catalog, services, delivery and payment terms, frequently asked questions, company information, contacts, and an optional configurable section. Section order is adjustable, and custom line templates are supported.

Titles are stripped of SEO noise. Marketing suffixes such as "buy," "cheap," and "online store" are removed from headings so the assistant receives clean subject matter rather than promotional text. The dictionary can be extended with custom phrases.

Exclusion rules filter out service pages. Cart, personal account, parameterized pages, and other irrelevant content are removed using masks, regular expressions, or GET parameters. The rule set is created during installation and can be adjusted for the specific site.

The limit is defined by token count, not just link quantity. The file is maintained within a specified volume: when capacity is exceeded, the least significant links are discarded, while catalog sections are preserved longer than individual products.

Pre-publication validation of llms.txt. Performs 15 specification compliance checks: a single H1 heading, presence of a description, absolute links, absence of duplicates and external domains, and file size. A document failing validation is not written; a broken llms.txt is worse than its absence.

Preview and comparison. The file is built in memory and displayed line-by-line alongside the published version, showing exactly what will change before writing to disk.

Automatic updates without manual work. When elements or sections of connected information blocks are modified, the file is marked as outdated. An agent rebuilds it at the next run. Saving a product is not delayed because the rebuild does not occur during the edit.

Build log. For each run: the trigger source (admin panel, agent, content change, command line), result, whether the file changed, counts of links, tokens, and bytes, duration, and all messages. Clicking a log entry opens a card with pipeline counters and validation notes.

Ready-made block for robots.txt with permissions for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, YandexAdditional. The code is copied with one click. The module does not modify robots.txt itself.

The module automatically adds a link tag with the text/markdown type to website pages. The feature is enabled via a simple checkbox in settings without requiring any template modifications.

The generated file is always written in UTF-8 encoding, regardless of the site's current encoding, even if the site operates on cp1251. To ensure correct server delivery without explicit encoding specification, the BOM marker is enabled by default. The preview section includes ready-to-use configuration blocks for nginx and Apache servers.

The solution supports multi-site environments. Each website maintains its own separate profile of settings, its own file, and its own activity log.

The interface is available in Russian and English. Access rights are managed using standard Bitrix user groups. A dedicated menu section is provided within the administration panel.

This module is designed for online stores, wholesale suppliers, service companies, medical clinics, and construction developers. It ensures that AI assistants provide accurate responses regarding prices, product assortment, and working conditions by linking directly to current pages on the site, rather than relying on outdated data from their index.

It is important to note that the module is responsible solely for generating and publishing the file. The speed at which a specific assistant reads the file and how it utilizes the information is determined entirely by that assistant. No tool can guarantee indexing or usage by an assistant.

Top Man Digital module: llms.txt was verified on a catalog of 10,000 items. Full generation of the file with prices and stock levels takes approximately 1.5 seconds, using 17 SQL queries and 64 MB of memory. This allows the module to run on standard hosting without configuration changes.

The file is optimized for compactness: 10,000 links occupy 1.5 MB and approximately 760,000 tokens, exceeding the context window of most models. For large catalogs, the recommended strategy involves categorization plus a limited item list; the token limit helps maintain this structure.

If “Top Man Digital: llms.txt” fits the task, General iT can install the module, configure it and verify the result in the project.