]> Universität Bremen TZI

Postfach 330440 Bremen D-28359 Germany +49-421-218-63921 cabo@tzi.org

Textuality

Canada tbray@textuality.com

Internet-Draft This document specifies I-Regexp, a flavor of regular expressions that is limited in scope with the goal of interoperation across many different regular-expression libraries. About This Document Status information for this document may be found at . Discussion of this document takes place on the JSONPath Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction This specification describes an interoperable regular expression ("regexp") flavor, I-Regexp. I-Regexp does not provide advanced regular expression features such as capture groups, lookahead, or backreferences. It supports only a Boolean matching capability, i.e., testing whether a given regular expression matches a given piece of text. I-Regexp supports the entire repertoire of Unicode characters (Unicode scalar values); both the I-Regexp strings themselves and the strings they are matched against are sequences of Unicode scalar values (often represented in UTF-8 encoding form for interchange). I-Regexp is a subset of XSD regular expressions . This document includes guidance for converting I-Regexps for use with several well-known regular expression idioms. The development of I-Regexp was motivated by the work of the JSONPath Working Group. The Working Group wanted to include in its specification support for the use of regular expressions in JSONPath filters, but was unable to find a useful specification for regular expressions which would be interoperable across the popular libraries.

Terminology This document uses the abbreviation "regexp" for what are usually called regular expressions in programming. "I-Regexp" is used as a noun meaning a character string (sequence of Unicode scalar values) that conforms to the requirements in this specification; the plural is "I-Regexps". This specification uses Unicode terminology. A good entry point into that is provided by . The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. The grammatical rules in this document are to be interpreted as ABNF, as described in and , where the "characters" of are Unicode scalar values.

Requirements I-Regexps should handle the vast majority of practical cases where a matching regexp is needed in a data model specification or a query language expression. The editors of this document conducted a survey of the regexp syntax used in published RFCs. All examples found there should be covered by I-Regexps, both syntactically and with their intended semantics. The exception is the use of multi-character escapes, for which workaround guidance is provided in .

I-Regexp Syntax An I-Regexp MUST conform to the ABNF specification in .

I-Regexp Syntax in ABNF ' / %x40-5A ; '@'-'Z' / %x5E-7A ; '^'-'z' / %x7E-10FFFF ) charClass = "." / SingleCharEsc / charClassEsc / charClassExpr SingleCharEsc = "\" ( %x28-2B ; '('-'+' / "-" / "." / "?" / %x5B-5E ; '['-'^' / %s"n" / %s"r" / %s"t" / %x7B-7D ; '{'-'}' ) charClassEsc = catEsc / complEsc charClassExpr = "[" [ "^" ] ( "-" / CCE1 ) *CCE1 [ "-" ] "]" CCE1 = ( CCchar [ "-" CCchar ] ) / charClassEsc CCchar = ( %x00-2C / %x2E-5A ; '.'-'Z' / %x5E-10FFFF ) / SingleCharEsc catEsc = %s"\p{" charProp "}" complEsc = %s"\P{" charProp "}" charProp = IsCategory IsCategory = Letters / Marks / Numbers / Punctuation / Separators / Symbols / Others Letters = %s"L" [ ( %s"l" / %s"m" / %s"o" / %s"t" / %s"u" ) ] Marks = %s"M" [ ( %s"c" / %s"e" / %s"n" ) ] Numbers = %s"N" [ ( %s"d" / %s"l" / %s"o" ) ] Punctuation = %s"P" [ ( %x63-66 ; 'c'-'f' / %s"i" / %s"o" / %s"s" ) ] Separators = %s"Z" [ ( %s"l" / %s"p" / %s"s" ) ] Symbols = %s"S" [ ( %s"c" / %s"k" / %s"m" / %s"o" ) ] Others = %s"C" [ ( %s"c" / %s"f" / %s"n" / %s"o" ) ] ]]>

As an additional restriction, charClassExpr is not allowed to match [^], which according to this grammar would parse as a positive character class containing the single character ^. This is essentially XSD regexp without character class subtraction, without multi-character escapes such as \s, \S, and \w, and without Unicode blocks. An I-Regexp implementation MUST be a complete implementation of this limited subset. In particular, full support for the Unicode functionality defined in this specification is REQUIRED; the implementation MUST NOT limit itself to 7- or 8-bit character sets such as ASCII and MUST support the Unicode character property set in character classes.

Checking Implementations A checking I-Regexp implementation is one that checks a supplied regexp for compliance with this specification and reports any problems. Checking implementations give their users confidence that they didn't accidentally insert non-interoperable syntax, so checking is RECOMMENDED. Exceptions to this rule may be made for low-effort implementations that map I-Regexp to another regexp library by simple steps such as performing the mapping operations discussed in ; here, the effort needed to do full checking may dwarf the rest of the implementation effort. Implementations SHOULD document whether they are checking or not. Specifications that employ I-Regexp may want to define in which cases their implementations can work with a non-checking I-Regexp implementation and when full checking is needed, possibly in the process of defining their own implementation classes.

I-Regexp Semantics This syntax is a subset of that of . Implementations which interpret I-Regexps MUST yield Boolean results as specified in . (See also .)

Mapping I-Regexp to Regexp Dialects The material in this section is non-normative, provided as guidance to developers who want to use I-Regexps in the context of other regular expression dialects.

Multi-Character Escapes Common multi-character escapes (MCEs), and character classes built around them, which are not supported in I-Regexp, can usually be replaced as shown for example in . Example substitutes for multi-character escapes

MCE/class	Replace with
\S	[^ \t\n\r]
[\S ]	[^\t\n\r]
\d	[0-9]

Note that the semantics of \d in XSD regular expressions is that of \p{Nd}; however, this would include all Unicode characters that are digits in various writing systems, which is almost certainly not what is required in IETF publications. The construct \p{IsBasicLatin} is essentially a reference to legacy ASCII, it can be replaced by the character class [\u0000-\u007f].

XSD Regexps Any I-Regexp also is an XSD Regexp , so the mapping is an identity function. Note that a few errata for have been fixed in , which is therefore also included as a normative reference. XSD 1.1 is less widely implemented than XSD 1.0, and implementations of XSD 1.0 are likely to include these bugfixes, so for the intents and purposes of this specification an implementation of XSD 1.0 regexps is equivalent to an implementation of XSD 1.1 regexps.

ECMAScript Regexps Perform the following steps on an I-Regexp to obtain an ECMAScript regexp :

• For any unescaped dots (.) outside character classes (first alternative of charClass production): replace dot by [^\n\r].
• Envelope the result in ^(?: and )$.

The ECMAScript regexp is to be interpreted as a Unicode pattern ("u" flag; see Section 21.2.2 "Pattern Semantics" of ). Note that where a regexp literal is required, the actual regexp needs to be enclosed in /.

PCRE, RE2, Ruby Regexps Perform the same steps as in to obtain a valid regexp in PCRE , the Go programming language , and the Ruby programming language, except that the last step is:

• Enclose the regexp in \A(?: and )\z.

Motivation and Background While regular expressions originally were intended to describe a formal language to support a Boolean matching function, they have been enhanced with parsing functions that support the extraction and replacement of arbitrary portions of the matched text. With this accretion of features, parsing regexp libraries have become more susceptible to bugs and surprising performance degradations which can be exploited in Denial of Service attacks by an attacker who controls the regexp submitted for processing. I-Regexp is designed to offer interoperability, and to be less vulnerable to such attacks, with the trade-off that its only function is to offer a boolean response as to whether a character sequence is matched by a regexp.

Implementing I-Regexp XSD regexps are relatively easy to implement or map to widely implemented parsing regexp dialects, with these notable exceptions:

• Character class subtraction. This is a very useful feature in many specifications, but it is unfortunately mostly absent from parsing regexp dialects. Thus, it is omitted from I-Regexp.
• Multi-character escapes. \d, \w, \s and their uppercase complement classes exhibit a large amount of variation between regexp flavors. Thus, they are omitted from I-Regexp.
• Not all regexp implementations support accesses to Unicode tables that enable executing constructs such as \p{Nd}, although the \p/\P feature in general is now quite widely available. While in principle it is possible to translate these into character-class matches, this also requires access to those tables. Thus, regexp libraries in severely constrained environments may not be able to support I-Regexp conformance.

IANA Considerations This document makes no requests of IANA.

Security considerations While technically out of scope of this specification, Section Security Considerations of applies to implementations. Particular note needs to be taken of the last paragraph of Section UTF-8 definition of ; an I-Regexp implementation may need to mitigate limitations of the platform implementation in this regard. As discussed in , more complex regexp libraries may contain exploitable bugs leading to crashes and remote code execution. There is also the problem that such libraries often have hard-to-predict performance characteristics, leading to attacks that overload an implementation by matching against an expensive attacker-controlled regexp. I-Regexps have been designed to allow implementation in a way that is resilient to both threats; this objective needs to be addressed throughout the implementation effort. Non-checking implementations (see ) are likely to expose security limitations of any regexp engine they use, which may be less problematic if that engine has been built with security considerations in mind (e.g., ); a checking implementation is still RECOMMENDED. Implementations that specifically implement the I-Regexp subset can, with care, be designed to generally run in linear time and space in the input, and to detect when that would not be the case (see below). Existing regexp engines should be able to easily handle most I-Regexps (after the adjustments discussed in ), but may consume excessive resources for some types of I-Regexps or outright reject them because they cannot guarantee efficient execution. Specifically, range quantifiers (as in a{2,4}) provide specific challenges for both existing and I-Regexp specific implementations. These may therefore limit range quantifiers in composability (disallowing nested range quantifiers such as (a{2,4}){2,4}) or range (disallowing very large ranges such as a{20,200000}), or detect and reject any excessive resource consumption caused by them. Note that different versions of the same regexp library may be more or less vulnerable to excessive resource consumption for these cases. I-Regexp implementations that are used to evaluate regexps from untrusted sources need to be robust to these cases. Implementers using existing regexp libraries are encouraged to check their documentation to see if mitigations are configurable, such as limits in resource consumption, and to document their own degree of robustness resulting from employing such mitigations.

References Normative References Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] This document extends the base definition of ABNF (Augmented Backus-Naur Form) to include a way to specify US-ASCII string literals that are matched in a case-sensitive manner. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References ISO/IEC 10646-1 defines a large character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. The originally proposed encodings of the UCS, however, were not compatible with many current applications and protocols, and this has led to the development of UTF-8, the object of this memo. UTF-8 has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo obsoletes and replaces RFC 2279. Fachhochschule Dortmund Universität Bremen TZI JSONPath defines a string syntax for selecting and extracting JSON (RFC 8259) values from a JSON value. Unicode, Inc. n.d. n.d. Ecma International I-JSON (short for "Internet JSON") is a restricted profile of JSON designed to maximize interoperability and increase confidence that software can process it successfully with predictable results.

Regexps and Similar Constructs in Recent Published RFCs This appendix contains a number of regular expressions that have been extracted from some recently published RFCs based on some ad-hoc matching. Multi-line constructions were not included. With the exception of some (often surprisingly dubious) usage of multi-character escapes and a reference to the IsBasicLatin Unicode block, all regular expressions validate against the ABNF in .

Example regular expressions extracted from RFCs

Acknowledgements This specification has been motivated by the discussion in the IETF JSONPATH WG about whether to include a regexp mechanism into the JSONPath query expression specification, as well as by previous discussions about the YANG pattern and CDDL .regexp features. The basic approach for this specification was inspired by The I-JSON Message Format.