]> Universität Bremen TZI

Postfach 330440 Bremen D-28359 Germany +49-421-218-63921 cabo@tzi.org

Internet-Draft The Concise Binary Object Representation (CBOR, RFC 8949 == STD 94) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. CBOR does not provide any forms of data compression. CBOR data items, in particular when generated from legacy data models, often allow considerable gains in compactness when applying data compression. While traditional data compression techniques such as DEFLATE (RFC 1951) can work well for CBOR encoded data items, their disadvantage is that the receiver needs to decompress the compressed form to make use of the data. This specification describes Packed CBOR, a simple transformation of a CBOR data item into another CBOR data item that is almost as easy to consume as the original CBOR data item. A separate decompression step is therefore often not required at the receiver. The present version (-06) includes changes that are intended for discussion at the CBOR Interim meeting 12 in 2022. These are intended to be the result of the WGLC handling discussions, but need more eyeballs. Note to Readers This is a working-group draft of the CBOR working group of the IETF, https://datatracker.ietf.org/wg/cbor/about/. Discussion takes places on the GitHub repository https://github.com/cbor-wg/cbor-packed and on the CBOR WG mailing list, https://www.ietf.org/mailman/listinfo/cbor.

Introduction The Concise Binary Object Representation (CBOR, ) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. CBOR does not provide any forms of data compression. CBOR data items, in particular when generated from legacy data models, often allow considerable gains in compactness when applying data compression. While traditional data compression techniques such as DEFLATE can work well for CBOR encoded data items, their disadvantage is that the receiver needs to decompress the compressed form to make use of the data. This specification describes Packed CBOR, a simple transformation of a CBOR data item into another CBOR data item that is almost as easy to consume as the original CBOR data item. A separate decompression step is therefore often not required at the receiver. This document defines the Packed CBOR format by specifying the transformation from a Packed CBOR data item to the original CBOR data item; it does not define an algorithm for a packer. Different packers can differ in the amount of effort they invest in arriving at a minimal packed form; often, they simply employ the sharing that is natural for a specific application. Packed CBOR can make use of two kinds of optimization:

• item sharing: substructures (data items) that occur repeatedly in the original CBOR data item can be collapsed to a simple reference to a common representation of that data item. The processing required during consumption is limited to following that reference.
• argument sharing: application of a function with two arguments, one of which is shared. Data items (strings, containers) that share a prefix or suffix (affix), or more generally data items that can be constructed from a function taking a shared argument and a rump data item, can be replaced by a reference to the shared argument plus a rump data item. For strings and the default "concatenation" function, the processing required during consumption is similar to following the argument reference plus that for an indefinite-length string.

A specific application protocol that employs Packed CBOR might allow both kinds of optimization or limit the representation to item sharing only. Packed CBOR is defined in two parts: Referencing packing tables () and setting up packing tables ().

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Packed reference:

A shared item reference or an argument reference.

Shared item reference:

A reference to a shared item as defined in .

Argument reference:

A reference that combines a shared argument with a rump item as defined in .

Affix:

Prefix or suffix, used as an argument in an argument reference employing the default function "concatenation".

Function reference:

An argument reference that uses a tag for argument, rump, or both, causing the application of a function to reconstruct the data item.

Packing tables:

The pair of a shared item table and an argument table.

Current set:

The packing tables in effect at the data item under consideration.

Expansion:

The result of applying a packed reference in the context of given Packing tables.

The definitions of apply. Specifically: The term "byte" is used in its now customary sense as a synonym for "octet"; "byte strings" are CBOR data items carrying a sequence of zero or more (binary) bytes, while "text strings" are CBOR data items carrying a sequence of zero or more Unicode code points, encoded in UTF-8 . Where bit arithmetic is explained, this document uses the notation familiar from the programming language C (including C++14's 0bnnn binary literals), except that, in the plain text form of this document, the operator "^" stands for exponentiation, and, in the HTML and PDF versions, subtraction and negation are rendered as a hyphen ("-", as are various dashes).

Packed CBOR This section describes the packing tables, their structure, and how they are referenced.

Packing Tables At any point within a data item making use of Packed CBOR, there is a Current Set of packing tables that applies. There are two packing tables in a Current Set:

• Shared item table
• Argument table

Without any table setup, these two tables are empty arrays. Table setup can cause these arrays to be non-empty, where the elements are (potentially themselves packed) data items. Each of the tables is indexed by an unsigned integer (starting from 0). Such an index may be derived from information in tags and their content as well as from CBOR simple values.

Referencing Shared Items Shared items are stored in the shared item table of the Current Set. The shared data items are referenced by using the reference data items in . When reconstructing the original data item, such a reference is replaced by the referenced data item, which is then recursively unpacked. Referencing Shared Values

reference	table index
Simple value 0-15	0-15
Tag 6(unsigned integer N)	16 + 2*N
Tag 6(negative integer N)	16 - 2*N - 1

As examples in CBOR diagnostic notation (), the first 22 elements of the shared item table are referenced by simple(0), simple(1), ... simple(15), 6(0), 6(-1), 6(1), 6(-2), 6(2), 6(-3). (The alternation between unsigned and negative integers for even/odd table index values -- "zigzag encoding" -- makes systematic use of shorter integer encodings first.) Taking into account the encoding of these referring data items, there are 16 one-byte references, 48 two-byte references, 512 three-byte references, 131072 four-byte references, etc. As CBOR integers can grow to very large (or very negative) values, there is no practical limit to how many shared items might be used in a Packed CBOR item. Note that the semantics of Tag 6 depend on its tag content: An integer turns the tag into a shared item reference, whereas a string or container (map or array) turns it into a straight (prefix) reference (see ). Note also that the tag content of Tag 6 may itself be packed, so it may need to be unpacked to make this determination.

Referencing Argument Items The argument table serves as a common table that can be used for argument references, i.e., for concatenation as well as references involving a function tag. When referencing an argument, a distinction is made between straight and inverted references; if no function tag is involved, a straight reference combines a prefix out of the argument table with the rump data item, and an inverted reference combines a rump data item with a suffix out of the argument table. Straight Referencing (e.g., Prefix) Arguments

straight reference	table index
Tag 6(straight rump)	0
Tag 224-255(straight rump)	0-31
Tag 28704-32767(straight rump)	32-4095
Tag 1879052288-2147483647(straight rump)	4096-268435455

Inverted Referencing (e.g., Suffix) Arguments

inverted reference	table index
Tag 216-223(inverted rump)	0-7
Tag 27647-28671(inverted rump)	8-1023
Tag 1811940352-1879048191(inverted rump)	1024-67108863

Argument data items are referenced by using the reference data items in and . The tag number of the reference indicates a table index (an unsigned integer) leading to the "argument"; the tag content of the reference is the "rump item". When reconstructing the original data item, such a reference is replaced by a data item constructed from the argument data item found in the table (argument, which might need to be recursively unpacked first) and the rump data item (rump, again possibly recursively unpacked). Separate from the tag used as a reference, a tag ("function tag") may be involved to supply a function to be used in resolving the reference. It is crucial not to confuse reference tag and, if present, function tag. A straight reference uses the argument as the provisional left hand side and the rump data item as the right hand side. An inverted reference uses the rump data item as the provisional left hand side and the argument as the right hand side. In both cases, the provisional left hand side is examined. If it is a tag ("function tag"), it is "unwrapped": The function tag's tag number is established as the function to be applied, and the tag content is kept as the unwrapped left hand side. If the provisional left hand side is not a tag, it is kept as the unwrapped left hand side, and the function to be applied is concatenation, as defined below. The right hand side is taken as is as the unwrapped right hand side. If a function tag was given, the reference is replaced by the result of applying the unpacking function to be computed to the left and right hand sides. The unpacking function is defined by the definition of the tag number supplied. If that definition does not define an unpacking function, the result of the unpacking is not valid. If no function tag was given, the reference is replaced by the left hand side "concatenated" with the right hand side, where concatenation is defined as in . As a contrived (but short) example, if the argument table is ["foobar", h'666f6f62', "fo"], each of the following straight (prefix) references will unpack to "foobart": 6("t"), 225("art"), 226("obart") (the byte string h'666f6f62' == 'foob' is concatenated into a text string, and the last example is not an optimization). Note that table index 0 of the argument table can be referenced both with tag 6 and tag 224, however tag 6 with an integer content is used for shared item references (see ), so to combine index 0 with an integer rump, tag 224 needs to be used. Taking into account the encoding and ignoring the less optimal tag 224, there is one single-byte straight (prefix) reference, 31 (2⁵-2⁰) two-byte references, 4064 (2¹²-2⁵) three-byte references, and 26843160 (2²⁸-2¹²) five-byte references for straight references. 268435455 (2²⁸) is an artificial limit, but should be high enough that there, again, is no practical limit to how many prefix items might be used in a Packed CBOR item. The numbers for inverted (suffix) references are one quarter of those, except that there is no single-byte reference and 8 two-byte references.

Concatenation The concatenation function is defined as follows:

• If both left hand side and right hand side are arrays, the result of the concatenation is an array with all elements of the left-hand-side array followed by the elements of the right-hand side array.
• If both left hand side and right hand side are maps, the result of the concatenation is a map that is initialized with a copy of the left-hand-side map, and then filled in with the members of the right-hand side map, replacing any existing members that have the same key.

• If both left hand side and right hand side are one of the string types (not necessarily the same), the bytes of the left hand side are concatenated with the bytes of the right hand side. Byte strings concatenated with text strings need to contain valid UTF-8 data. The result of the concatenation gets the type of the unwrapped rump data item; this way a single argument table entry can be used to build both byte and text strings, depending on what type of rump is being used.
• Other type combinations of left hand side and right hand side are not valid.

Discussion This specification uses up a large number of Simple Values and Tags, in particular one of the rare one-byte tags and two thirds of the one-byte simple values. Since the objective is compression, this is warranted only based on a consensus that this specific format could be useful for a wide area of applications, while maintaining reasonable simplicity in particular at the side of the consumer. A maliciously crafted Packed CBOR data item might contain a reference loop. A consumer/decompressor MUST protect against that.

Table Setup The packing references described in assume that packing tables have been set up. By default, both tables are empty (zero-length arrays). Table setup can happen in one of two ways:

• By the application environment, e.g., a media type. These can define tables that amount to a static dictionary that can be used in a CBOR data item for this application environment. Note that, without this information, a data item that uses such a static dictionary can be decoded at the CBOR level, but not fully unpacked. The table setup mechanisms provided by this document are defined in such a way that an unpacker can at least recognize if this is the case.
• By one or more tags enclosing the packed content. Each tag is usually defined to build an augmented table by adding to the packing tables that already apply to the tag, and to apply the resulting augmented table when unpacking the tag content. Usually, the semantics of the tag will be to prepend items to one or more of the tables. (The specific behavior of any such tag, in the presence of a table applying to it, needs to be carefully specified.) Note that it may be useful to leave a particular efficiency tier alone and only prepend to a higher tier; e.g., a tag could insert shared items at table index 16 and shift anything that was already there further down in the array while leaving index 0 to 15 alone. Explicit additions by tag can combine with application-environment supplied tables that apply to the entire CBOR data item. Packed item references in the newly constructed (low-numbered) parts of the table are usually interpreted in the number space of that table (which includes the, now higher-numbered, inherited parts), while references in any existing, inherited (higher-numbered) part continue to use the (more limited) number space of the inherited table.

For table setup, the present specification only defines a single tag, which operates by prepending to the (by default empty) tables.

Basic Packed CBOR A predefined tag for packing table setup is defined in CDDL as in :

Packed CBOR in CDDL

(This assumes the allocation of tag number 113 ('q') for this tag.) The arrays given as the first and second element of the content of the tag 113 are prepended to the tables for shared items and arguments that apply to the entire tag (by default empty tables). As discussed in the introduction to this section, references in the supplied new arrays use the new number space (where inherited items are shifted by the new items given), while the inherited items themselves use the inherited number space (so their semantics do not change by the mere action of inheritance). The original CBOR data item can be reconstructed by recursively replacing shared and argument references encountered in the rump by their expansions.

Function Tags Function tags that occur in an argument or a rump supply the semantics for reconstructing a data item from their tag content and the non-dominating rump or argument, respectively. The present specification defines a pair of function tags.

Join Function Tags Tag 106 ('j') defines the "join" unpacking function, based on the concatenation function (). The join function expects an item that can be concatenated as its left hand side, and an array of such items as its right hand side. Joining works by sequentially applying the concatenation function to the elements of the right-hand-side array, interspersing the left hand side as the "joiner". An example in functional notation: join(", ", ["a", "b", "c"]) returns "a, b, c". For a right hand side of one or more elements, the first element determines the type of the result when text strings and byte strings are mixed in the argument. For a right hand side of one element, the joiner is not used, and that element returned. For a right hand side of zero elements, a neutral element is generated based on the type of the joiner (empty text/byte string for a text/byte string, empty array for an array, empty map for a map). For an example, we assume this unpacked data item: A packed form of this using straight references could be: Tag 105 ('i') defines the "ijoin" unpacking function, which is exactly like that of tag 106, except that the left hand side and right hand side are interchanged ('i'). A packed form of the first example using inverted references and the ijoin tag could be: A packed form of an array with many URIs that reference SenML items from the same place could be:

IANA Considerations

CBOR Tags Registry In the registry "" , IANA is requested to allocate the tags defined in . Values for Tag Numbers

Tag	Data Item	Semantics	Reference
6	integer (for shared); text string, byte string, array, map, tag (for straight)	Packed CBOR: shared/straight	draft-ietf-cbor-packed
105	text string, byte string, array, map, tag	Packed CBOR: ijoin function	draft-ietf-cbor-packed
106	text string, byte string, array, map, tag	Packed CBOR: join function	draft-ietf-cbor-packed
113	array (shared-items, argument-items, rump)	Packed CBOR: table setup	draft-ietf-cbor-packed
224-255	text string, byte string, array, map, tag	Packed CBOR: straight	draft-ietf-cbor-packed
28704-32767	text string, byte string, array, map, tag	Packed CBOR: straight	draft-ietf-cbor-packed
1879052288-2147483647	text string, byte string, array, map, tag	Packed CBOR: straight	draft-ietf-cbor-packed
216-223	text string, byte string, array, map, tag	Packed CBOR: inverted	draft-ietf-cbor-packed
27647-28671	text string, byte string, array, map, tag	Packed CBOR: inverted	draft-ietf-cbor-packed
1811940352-1879048191	text string, byte string, array, map, tag	Packed CBOR: inverted	draft-ietf-cbor-packed

CBOR Simple Values Registry In the registry "" , IANA is requested to allocate the simple values defined in . Simple Values

Value	Semantics	Reference
0-15	Packed CBOR: shared	draft-ietf-cbor-packed

Security Considerations The security considerations of apply. Loops in the Packed CBOR can be used as a denial of service attack, see . As the unpacking is deterministic, packed forms can be used as signing inputs. (Note that if external dictionaries are added to cbor-packed, this requires additional consideration.)

References Normative References The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. These design goals make it different from earlier binary serializations such as ASN.1 and MessagePack. This document obsoletes RFC 7049, providing editorial improvements, new details, and errata fixes while keeping full compatibility with the interchange format of RFC 7049. It does not create a new version of the format. Internet Assigned Numbers Authority Internet Assigned Numbers Authority This document proposes a notational convention to express Concise Binary Object Representation (CBOR) data structures (RFC 7049). Its main goal is to provide an easy and unambiguous way to express structures for protocol messages and data formats that use CBOR or JSON. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References ISO/IEC 10646-1 defines a large character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. The originally proposed encodings of the UCS, however, were not compatible with many current applications and protocols, and this has led to the development of UTF-8, the object of this memo. UTF-8 has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo obsoletes and replaces RFC 2279. The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. These design goals make it different from earlier binary serializations such as ASN.1 and MessagePack. This document describes the Concise Binary Object Representation (CBOR) Sequence format and associated media type "application/cbor-seq". A CBOR Sequence consists of any number of encoded CBOR data items, simply concatenated in sequence. Structured syntax suffixes for media types allow other media types to build on them and make it explicit that they are built on an existing media type as their foundation. This specification defines and registers "+cbor-seq" as a structured syntax suffix for CBOR Sequences. This document defines a set of ways to identify a thing (a digital object in this case) using the output from a hash function. It specifies a new URI scheme for this purpose, a way to map these to HTTP URLs, and binary and human-speakable formats for these names. The various formats are designed to support, but not require, a strong link to the referenced object, such that the referenced object may be authenticated to the same degree as the reference to it. The reason for this work is to standardise current uses of hash outputs in URLs and to support new information-centric applications and other uses of hash outputs in protocols. This specification defines a lossless compressed data format that compresses data using a combination of the LZ77 algorithm and Huffman coding, with efficiency comparable to the best currently available general-purpose compression methods. This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind.

Examples The (JSON-compatible) CBOR data structure depicted in , 400 bytes of binary CBOR, could lead to a packed CBOR data item depicted in , ~309 bytes. Note that this particular example does not lend itself to prefix compression.

Example original CBOR data item

Example packed CBOR data item

The (JSON-compatible) CBOR data structure below has been packed with shared item and (partial) prefix compression only.

Example original CBOR data item

Example packed CBOR data item

Acknowledgements CBOR packing was originally invented with the rest of CBOR, but did not make it into , the predecessor of . Various attempts to come up with a specification over the years didn't proceed. In 2017, proposed investigating compact representations of W3C Thing Descriptions, which prompted the author to come up with what turned into the present design.