Popular on TelAve
- Nextlink Internet Deploys netElastic to Modernize Its Hybrid Broadband Network - 200
- AeroNet Deploys netElastic vBNG and CGNAT with Tarana Wireless to Deliver Scalable, Fiber-Class Broadband Across Puerto Rico - 169
- One in three Americans may have lost track of retirement savings from a previous job - 143
- Commercial Manufacturing Validation Completed as November 10 FDA Milestone Approaches; Critical FDA Milestone Close: NRx Pharmaceuticals NAS DAQ: NRXP - 129
- 1569531 B.C. LTD. dba ELEOLUXE Connects Renovation Design and Project Coordination - 112
- AI Voice Agents Are Now Calling AI Receptionists, NextPhone 2026 Call Report Finds - 107
- Looop Officially Launches a Safety Notifications App for Families Who Care From a Distance - 104
- RAS AP Consulting Advances 1099 Education and Readiness Initiative Through IFOL Training and Finance Technology Showcase - 102
- Security Industry Veteran Anthony Rumore Launches AegisLead to Help Security Companies Find Business and Manage Growth
- KAIT to Demonstrate 400 Gbps 8K Video Processing via SRv6 In-Network Computing at SC26
Similar on TelAve
- Iglesia.com Positioned in Multimillion-Dollar Domain Market with $10 Million Asking Price
- ViewDNS.info Launches Monitors, Alerting Teams the Moment Something Changes on Their Domains
- Tampa Web Design Launches New Website for Entrepreneur and Capital Markets Advisor Martin Sumichrast
- AeroNet Deploys netElastic vBNG and CGNAT with Tarana Wireless to Deliver Scalable, Fiber-Class Broadband Across Puerto Rico
- Nextlink Internet Deploys netElastic to Modernize Its Hybrid Broadband Network
- BESTBUYSTORE opens its dropshipping model to independent merchants
- Clearnetworx Deploys netElastic vBNG, CGNAT And netVision To Power Fiber Expansion Across Colorado's Western Slope
- Coinfiliate Launches Platform to Helps Shoppers Earn Cashback and Creators Earn Referral Rewards
- netElastic Announces vBNG And CGNAT 2.0 General Availability
- 3ptechies Makes Its Articles Audio-Ready With Instaread Voice Technology
Robots.txt one line AI training. Anthropic, Microsoft, Google and OpenAI back the concept
TelAve News/10912160
How Lab451 Generator handles it: an opt-in training preference for robots.txt. No preference is the default; choose to opt out of the whole site or all but some paths. When you opt out, the per-crawler training groups are written to match.
VICTORIA, British Columbia - TelAve -- Lab451 Helps Website Owners Get Ready for the New AI-Training Signal in robots.txt Backed by Microsoft, OpenAI, Google and Anthropic
Free tool generates, verifies and monitors robots.txt, sitemap.xml, llms.txt and llms-full.txt — now with support for the IETF draft '__usage_ai-training' token
One line instead of a growing block list
Until now, a publisher who did not want their content used to train AI models had to add a separate robots.txt rule for every AI crawler, and keep that list current as new crawlers appeared. In late October, the four companies announced they would honor a single shared signal instead: a policy token, '__usage_ai-training', placed in the User-Agent line of an ordinary robots.txt group. The proposal was written by Kevin Kelley, Anthropic's technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an Internet-Draft.
More on TelAve News
A site that wants to opt out of AI training would add a group such as:
User-Agent: __usage_ai-training
Disallow: /
AI Training, Generate, Verify, Monitor
Availability
More on TelAve News
The Lab451 generator and checker are free at https://lab451.org. Monitoring is included in the Pro and Agency plans.
Media contact
Roger Remacle
hello@lab451.org
https://lab451.org
Free tool generates, verifies and monitors robots.txt, sitemap.xml, llms.txt and llms-full.txt — now with support for the IETF draft '__usage_ai-training' token
One line instead of a growing block list
Until now, a publisher who did not want their content used to train AI models had to add a separate robots.txt rule for every AI crawler, and keep that list current as new crawlers appeared. In late October, the four companies announced they would honor a single shared signal instead: a policy token, '__usage_ai-training', placed in the User-Agent line of an ordinary robots.txt group. The proposal was written by Kevin Kelley, Anthropic's technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an Internet-Draft.
More on TelAve News
- Emerald Williams Selected to Super Lawyers Rising Stars 2022-2026
- Massive Acquisition Into Defense With STS for Virtuix Holdings; $40 Million Revenue Business Will Accelerate Growth Transforming Market Position
- Orlando Real Estate Broker Bent Danholm Wins 2026 Goody Business Book Award for The Architecture of Demand
- Commercial Sales for U.S.-Made, Preservative-Free Ketamine Amid Long-Runnin Shortage Following FDA Approval by NRx Pharmaceuticals, Inc
- The King Returns: Satirical December 2026 Rematch Sees Challenger Delayed by a Christmas Light Installation
A site that wants to opt out of AI training would add a group such as:
User-Agent: __usage_ai-training
Disallow: /
AI Training, Generate, Verify, Monitor
- AI Training (new): Lab451's generator offers the __usage_ai-training group as an opt-in option clearly labelled "IETF draft". The checker recognizes and validates the token in existing robots.txt files, monitoring alerts when it is added, removed or malformed, and a short explainer page walks site owners through what it means.
- Generate: Build four files from one crawl of a site — robots.txt, sitemap.xml, llms.txt and llms-full.txt — with robots.txt as the entry point that points search engines and AI agents to the other three.
- Verify: Every generated file is checked for dead links and hostnames, unregistered package names, and text hidden from human readers before it is published. The free Checker runs the same validation on files a site already publishes.
- Monitor: Pro and Agency plans watch published files six times a day, send a weekly report by email, and notify the owner immediately when a critical problem appears — such as a robots.txt change that accidentally blocks search engines or drops the AI-training preference.
Availability
More on TelAve News
- LCH2 Awarded Additional $1 Million to Advance Lewis County Clean Energy Campus
- Security Planning Tips For Your Holiday Event From Access Patrol Services
- Contracting Resources Group Receives 2026 HIRE Vets Platinum Medallion Award
- Business News & Info Goes Inside Paragon FX Group's Hunt for Original Film Props
- Kitchen Table Economics: An Invitation to Think Differently
The Lab451 generator and checker are free at https://lab451.org. Monitoring is included in the Pro and Agency plans.
Media contact
Roger Remacle
hello@lab451.org
https://lab451.org
Source: Lab451
0 Comments
Latest on TelAve News
- New Six-Figure Media Agreement Builds Powerful Momentum, Expanding Government Contracts, FBI Access and Growing AI Recognition: Cyabra (NAS DAQ: CYAB)
- Christmas Beats Summer in Paris: Travel Photoshoots Peak on Dec. 30
- Qscription Technologies Welcomes Joanna Krause, MD, MRMD to Its Clinical Advisory Board
- Jamie Foxx's SKVLK Fest Sets the Record Straight on Stabbing Reports Following Star-Studded Celebration
- Andrew Balzano Releases One of My Life's Passions: Classics and Mysteries, a Fan's Guide to Great Mysteries Worth Rewatching
- The 4th Annual Newark Summit for Economic Development, Placemaking & Innovation Returns on December 8th
- Romanian Software Company ASSIST Software Ranks Second Nationally Among Large Enterprises in Its Category
- HIRO Ostomy Belt Launch by Richard O'Hamill
- New Omega-3 Fish Oil Under NIH Investigation: What Makes It Different From EPA and DHA?
- Technical Brief Examines Enterprise Integration Challenges as AI Agent Platforms Expand
- OncoBeta Introduces Rhenium-SCT for Non-Melanoma Skin Cancer at Theageneio Cancer Hospital of Thessaloniki
- Brian G. Miller Co., LLC Receives Tier 1 Ranking in the 2027 Edition of Best Law Firms®
- The Villari Firm, PLLC Named to the 2027 Edition of Best Law Firms
- Bacchus Wine & Spirits and Michelin Guide–Recommended Ethel's Fancy to Host Exclusive LOUIS XIII Limited Edition Diwali Decanter Celebration
- Arizona Pest Control Surpasses $200,000 in Contributions to Southern Arizona High School Sports and Education Programs
- ZenaTech (NAS DAQ: ZENA) $1 Million in New Utility Contracts- Potential $2.5 Million Follow-On Opportunity Puts Its Drone Business in the Spotlight!
- New Patient Data Show REQORSA® Activating Cellular Repair Pathways as Roche Collaboration Advances Precision Lung Cancer Strategy for Genprex, Inc
- Genuine Hospitality, LLC Selected to Operate Holiday Inn Express & Suites Jacksonville – Town Center
- Shiokly Launches Online Marketplace to Help Singapore Home-Based Businesses Sell More, With First Product Listed Free
- Freight Broker 911 and Milton Collier Introduce "What's The 911?"—A New Podcast Built on Real Talk, Real Freight, and Real Opportunity