Rate limiting: Constraints on the frequency or concurrency of automated
requests to prevent excessive server load.
Automation technology: Specific tools or frameworks used for automation,
such as headless browsers or browser automation protocols.
XHR/Fetch: XMLHttpRequest or Fetch API calls performed programmatically.
Session validation: Mechanisms to verify that automated requests are part
of a legitimate user session.
Asset annotation: Metadata embedded within HTML documents to specify
automation policies for individual content elements.
Extended Protocol SpecificationThis section defines additional directives that extend the automation-preferences.txt
protocol. These directives may be used alongside the core directives in any
group within the automation-preferences.txt file.Rate LimitingRate limiting directives specify constraints on the frequency and concurrency
of automated requests to prevent excessive server load. The following directives
are defined:
RequestLimit: Specifies the maximum number of requests allowed within a
time period, expressed as a count followed by a time unit (e.g., "60/minute").
Supported time units are "second", "minute", "hour", and "day".
ConcurrentLimit: Specifies the maximum number of concurrent connections
allowed from a single client.
Example:Rate limiting directives apply to all requests within the scope of the group,
regardless of HTTP method. If no rate limiting directives are specified,
clients SHOULD NOT assume any specific rate limits, but SHOULD implement
reasonable self-throttling to avoid overloading the server.Automation Technology RestrictionsAutomation technology directives specify whether specific automation tools or
frameworks are permitted. The following directives are defined:
AllowCDP: Boolean value indicating whether the use of Chrome DevTools
Protocol (CDP) is permitted.
AllowHeadless: Boolean value indicating whether the use of headless
browsers is permitted.
AllowSelenium: Boolean value indicating whether the use of Selenium
WebDriver is permitted.
AllowPuppeteer: Boolean value indicating whether the use of Puppeteer
is permitted.
AllowPlaywright: Boolean value indicating whether the use of Playwright
is permitted.
Example:If an automation technology directive is not specified, clients SHOULD NOT
assume that the use of that technology is permitted. Implementations SHOULD
respect these directives when applicable, even if the specific detection
methods may vary.API and XHR PermissionsAPI and XHR permission directives specify rules for API usage and automated
use of XMLHttpRequest, Fetch, or AJAX. The following directives are defined:
APIAutomation: Indicates how API endpoints may be accessed by automated
clients. Valid values are:
none: No API automation is permitted.
with-key-only: API automation is permitted only with proper authentication.
open: API automation is generally permitted.
AllowXHR: Indicates how XMLHttpRequest or Fetch API may be used by
automated clients. Valid values are:
none: No XHR/Fetch automation is permitted.
read-only: Only GET requests are permitted via XHR/Fetch.
open: XHR/Fetch automation is generally permitted.
DisallowFetchFrom: Comma-separated list of URL patterns from which
automated XHR/Fetch requests are prohibited. Wildcards MAY be used.
Example:If API and XHR permission directives are not specified, clients SHOULD assume
the most restrictive value (i.e., "none" for APIAutomation and AllowXHR).Session RequirementsSession requirement directives specify whether automated requests must be part
of a legitimate user session. The following directives are defined:
RequireHumanInitiatedSession: Boolean value indicating whether automated
requests must be part of a session that was initiated by a human user.
SessionValidation: Specifies the method used to validate sessions. Valid
values are:
cookie-based: Sessions are validated using HTTP cookies.
token-based: Sessions are validated using authentication tokens.
oauth: Sessions are validated using OAuth.
none: No session validation is required.
SessionTTL: Specifies the maximum time-to-live for a session, expressed
as a duration (e.g., "30m", "2h", "1d").
RequireUserAgent: Boolean value indicating whether automated requests
must include a valid User-Agent header.
Example:If session requirement directives are not specified, clients SHOULD NOT assume
any specific session requirements, but SHOULD include a valid User-Agent header
in all requests.HTML Asset AnnotationIn addition to a site-level automation-preferences.txt file, automation preferences
MAY be embedded directly within HTML documents to annotate individual
assets. This mechanism enables content creators to specify fine-grained
automation policies for particular content items.Authors SHOULD use structured data markup using JSON-LD embedded in a <script>
element. The JSON object SHOULD use a defined type (e.g., "AutomationPolicyAnnotation")
and include relevant fields that mirror those used in automation-preferences.txt.Note that unlike site-wide directives, asset-level annotations SHOULD NOT include
HTTP method restrictions, request limits, or concurrency limits, as these concepts
apply to endpoints and services rather than to individual content assets.Example:
{
"@context": "https://schema.org",
"@type": "AutomationPolicyAnnotation",
"automationPolicy": "limited",
"allowCDP": false,
"allowHeadless": false,
"automationPurpose": {
"require": true,
"allowed": [[PLACEHOLDER_PURPOSE1], [PLACEHOLDER_PURPOSE2]],
"disallowed": [[PLACEHOLDER_PURPOSE3]]
},
"contactEmail": "automation-policy@example.com"
}
]]>When both a automation-preferences.txt file and HTML asset annotations are present,
the more specific rule (typically the HTML annotation) SHALL be applied to
the corresponding content asset. Clients supporting HTML asset annotations
SHOULD parse and respect these annotations when present.The annotation schema MAY include any directives defined in the core or
extension specifications. Fields in the annotation SHOULD use camelCase naming
to align with JSON-LD conventions, while maintaining semantic equivalence to
the corresponding directives in the automation-preferences.txt file.Backward CompatibilityThe extensions defined in this document maintain backward compatibility with
implementations of the core specification. This compatibility is achieved
through the following mechanisms:
All directives defined in this document are OPTIONAL. Implementations that
support only the core specification can safely ignore these directives, as
specified in the core specification's extension mechanism.
The extensions do not modify or override the behavior of any directives
defined in the core specification.
Extended directives enhance but do not replace core functionality.
Implementations supporting these extensions SHOULD degrade gracefully when
interacting with servers or clients that support only the core specification:
Servers supporting extensions SHOULD still process all core directives
correctly, even if extended directives are also present.
Clients supporting extensions SHOULD still honor all core directives, even
if they do not recognize extended directives in a file.
When HTML asset annotations are not supported by a client, the client SHOULD
fall back to the site-level automation-preferences.txt file for guidance.
This approach ensures that the introduction of extensions does not break
existing implementations while providing a path for enhanced functionality.Implementation and EnforcementServers implementing the extensions defined in this document SHOULD:
Employ detection mechanisms (e.g., CDP fingerprinting, headless browser
detection) to identify automated clients using specific technologies.
Implement rate limiting according to the specified directives.
Validate sessions as required by the session requirement directives.
Process HTML asset annotations when interpreting automation policies for
specific content.
Respond with appropriate HTTP status codes for non-compliant requests,
such as:
429 Too Many Requests for rate limit violations.
403 Forbidden for unauthorized automation technology use.
401 Unauthorized for missing or invalid authentication.
Clients supporting these extensions SHOULD:
Honor rate limiting directives by self-throttling requests.
Respect automation technology restrictions by avoiding prohibited tools.
Adhere to API and XHR permissions as specified.
Establish and maintain valid sessions when required.
Parse and respect HTML asset annotations when present.
Both servers and clients MAY implement additional detection and enforcement
mechanisms beyond those explicitly described in this document, as long as they
maintain compatibility with the specified directives.Security ConsiderationsIn addition to the security considerations mentioned in the core specification,
the extensions defined in this document introduce the following considerations:
Rate Limiting: Implementations of rate limiting SHOULD use secure methods
to track request counts and prevent circumvention through IP spoofing or
other means.
Technology Detection: Methods used to detect specific automation
technologies MAY be circumvented by sophisticated clients. Servers SHOULD
employ multiple detection approaches and adapt to evolving evasion techniques.
Session Validation: Session validation mechanisms SHOULD be resistant to
replay attacks and session hijacking attempts.
HTML Asset Annotations: Parsing of JSON-LD annotations MUST be performed
securely to prevent injection attacks or denial-of-service through malformed
input.
The extensions provide more granular control over automated access, which can
enhance security, but they also introduce complexity that may lead to
misconfiguration. Implementers SHOULD carefully test and validate their
configurations to ensure they provide the intended protections.IANA ConsiderationsThis document has no IANA actions.Future WorkFuture enhancements to the automation-preferences.txt protocol MAY include:
Soliciting further feedback from browser vendors, content owners, AI model
and automation tool developers.
Developing reference implementations and comprehensive detection libraries.
Formalizing the protocol in collaboration with the IETF and W3C.
Expanding interoperability with related protocols for consistent content preference
signaling across the web.
Standardizing the HTML asset annotation schema through formal registration
with schema.org or similar organizations.