Pantheon: AI Site Map. llms.txt and Markdown versions of pages

What this solution does

The module prepares websites built on the 1C-Bitrix platform for use by artificial intelligence and AI agents. It aggregates catalog data, prices, technical specifications, news, and page content to generate llms.txt and llms-full.txt files compliant with the llmstxt.org specification. For each designated page, a Markdown version is created, stripped of navigation menus, promotional banners, and scripts.

Buyers increasingly turn to AI agents such as ChatGPT, Perplexity, Claude, Yandex Neuro, DeepSeek, Kimi, and others instead of traditional search engines. They ask specific questions, for example: "where to buy a black dress under 5,000 rubles with delivery." The neural network forms its response solely based on information it was able to read and process. A standard online store page is overloaded with menus, filters, banners, and scripts, causing key data like price and product specifications to get lost in the layout structure.

The sitemap.xml file is designed to index page addresses for search engine crawlers. The llms.txt file performs a similar function for neural networks, but it transmits not only addresses but also structured titles, descriptions, prices, and specifications. This format is optimized for machine reading without extraneous data. The more accurate and complete the provided information, the higher the likelihood that an AI assistant will directly link to your site rather than paraphrasing data from other resources.

Pantheon: AI Site Map. llms.txt and Markdown page versions

The /llms.txt file serves as a structured site map designed for neural networks. It sequentially lists the site name and description, followed by main sections: catalog, news, articles, and individual pages. Each entry includes a link and a brief explanation. This structure allows artificial intelligence to assess the site content and determine optimal user routes within seconds.

The Pantheon: AI Site Map module generates llms.txt files and Markdown versions of pages. The /llms-full.txt file consolidates the complete text into a single document, including descriptions, specifications, prices, units of measurement, stock levels, SEO fields, and trade offers. Based on this data, the artificial intelligence formulates responses to queries regarding the availability of specific sizes, such as size L, and provides their pricing.

Pantheon: AI Site Map. llms.txt and Markdown page versions

A Markdown version of each catalog page is automatically generated. For example, alongside the catalog for nightwear dresses, a file named dress-nightlife.md is created. It contains the same content as the original page but without menus, templates, or scripts. The link to this file is automatically inserted into the head section of the source page, allowing AI agents to discover it without additional hints.

The Pantheon module generates llms.txt files and Markdown versions of site pages for site maps intended for artificial intelligence. Using a test clothing online store as an example, the system demonstrates that it creates these files entirely automatically based on flags set in the administrative panel. No manual editing of code or text is required.

The Pantheon module generates an llms.txt file that serves as a site index for artificial intelligence. This file contains links pointing to Markdown versions of the site pages. This structure enables AI agents to efficiently scan and index content in a standardized format.

The Pantheon: AI Site Map module generates llms.txt files and Markdown page versions for data structuring. The llms-full.txt file is a product card containing price, technical specifications, and available configuration options.

The Pantheon module generates a site map for artificial intelligence, providing data in llms.txt and Markdown formats. The Markdown page version is the structured text received by an AI agent instead of standard HTML code. This enables agents to efficiently analyze web page content without needing to parse markup.

Artificial intelligence processes the file sequentially from top to bottom and may not read the entire content if the file is too large. The order of blocks is determined by the information block number on the Data Source tab: the lower the number, the earlier the block appears in the file. Within each information block, sections are displayed first, followed by products, with sorting controlled by the Sort field. Website pages always occupy the final part of the file.

Pantheon: AI Site Map. llms.txt and Markdown page versions

Content management via visual checkboxes

The module enables selection from information blocks, sections, products, pages, and folders using checkboxes, eliminating the need to write code. Selecting an item automatically includes all nested objects. Individual items can be precisely removed from the final list without affecting other entries.

Pantheon: AI Site Map. llms.txt and Markdown versions of pages.

The section on prices, stock levels, and variants describes how cost, unit of measure, available quantity, and trade offers are displayed. These data points may be shown individually or as a summary line, such as "Size: from 40 to 46".

Pantheon: AI Site Map module generates llms.txt files and Markdown versions of pages. The system processes any properties, including lists, bindings, reference books, files, and SEO fields, using infoblock templates. The functionality allows generating a record preview based on data from a real product.

Pantheon: AI Site Map module generates llms.txt files and Markdown versions of pages. The module includes static sections titled About Us, Delivery, Warranty, and Contacts, along with connectable content areas.

Pantheon: AI Site Map. llms.txt and Markdown Page Versions

Scheduling Configuration

The module supports flexible task scheduling: daily execution, selection of specific weekdays, or monthly runs at a designated time. A dedicated "Run" button is provided for immediate process initiation.

Pantheon: AI Site Map. llms.txt and Markdown page versions

The module is designed for handling large catalogs. File generation runs in the background in batches, eliminating timeouts and preventing site slowdowns. The site map file is replaced entirely, ensuring that AI systems cannot read an incomplete or partially written document.

The Pantheon: AI Site Map module provides llms.txt and Markdown exports for multiple websites. Each export is unique: every site has its own set of files, a specific update schedule, and a selected label language—Russian or English.

Pantheon: AI Site Map. llms.txt and Markdown page versions

Statistics

The module tracks file status, sizes, and dates, run history, and the global processing queue for all sites.

Pantheon: AI Site Map. llms.txt and Markdown page versions

Files are always created in UTF-8 encoding. Even if the main site operates in windows-1251 encoding, files generated for artificial intelligence are automatically formatted in UTF-8.

Pantheon: AI Site Map. llms.txt and Markdown page versions

Access rights management

Administrators determine which user groups have view-only access and which can run and configure exports.

Pantheon: AI Site Map. llms.txt and Markdown Page Versions

For Developers

The module provides events that allow modifying the text of Markdown page versions or disabling their generation for specific pages.

By default, nothing is included in the files or their Markdown versions. Only the items you explicitly marked on the Data Source tab are included in the output.

Pantheon: AI Site Map. llms.txt and Markdown versions of pages.

The system ensures that hidden information is never included in the output. Inactive products, records outside activity dates, disabled sections, and materials restricted by permissions remain inaccessible.

Pantheon: AI Site Map. llms.txt and Markdown versions of pages

Search engines do not treat Markdown versions of pages as duplicate content. This is ensured by serving the files with a noindex directive and specifying a canonical URL pointing to the original page.

Pantheon: AI Site Map. llms.txt and Markdown Page Versions

Everything runs on your server. No external services, API keys, or paid subscriptions are required. The site generates the necessary files independently.

The Pantheon: AI Site Map module generates llms.txt files and Markdown versions of pages. It requires PHP 8.2 or later with the dom and mbstring extensions enabled. The module architecture is built on a core utilizing Drupal 7-style dependency injection. The installation process automatically verifies all required components and reports any missing dependencies.

The Information Blocks module implements the system's core functionality. Working with prices, stock levels, and trade offers requires the Trade Catalog module. Managing static pages requires the Structure Management module. Without these components, the corresponding capabilities are unavailable, while the remaining system functions continue to operate normally.

Scheduled execution via cron is optional. Without cron, tasks run on the next site visit. A ready-to-use crontab string is available on the Schedule tab.

Pantheon: AI Site Map. llms.txt and Markdown formats

Text is extracted directly from site data, including infoblock fields and properties, as well as page source files. Elements generated solely by component templates and absent from the underlying data are not included in the output files.

Detailed documentation with illustrations for each setting and a 15-minute video guide are available on the developer's website.

If “Pantheon: AI Site Map. llms.txt and Markdown versions of pages” fits the task, General iT can install the module, configure it and verify the result in the project.