Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)
updated by
- rfc-5891 — Internationalized Domain Names in Applications (IDNA): Protocol
Extracted elements (25)
An alternative overflow-handling strategy for decoders used exclusively inside IDNA ToUnicode is to skip overflow checking entirely, since ToUnicode performs a higher-level re-encoding and comparison where a mismatch produces the same consequence as a decoder failure.
Because the decoder state advances monotonically and there is only one representation per delta, there is exactly one encoded string for any given sequence of code points. This eliminates the need to re-encode output to verify uniqueness — the only error conditions are invalid code points, unexpected end-of-input, overflow, and basic code points encoded as deltas.
Bootstring is designed to achieve six properties: completeness (every extended string representable), uniqueness (at most one basic string per extended string), reversibility, efficient encoding (small ratio of basic to extended length), simplicity, and readability (basic code points represented as themselves).
Insertion unsort coding processes non-basic code points in numerical order rather than order of appearance, which typically results in smaller deltas and thus more efficient encoding.
Little-endian ordering (least significant digit first) is chosen for generalized variable-length integers so that concatenated deltas can be separated starting with the first delta, which aligns with sequential decoding.
Punycode is designed exclusively for use with IDNA to convert domain labels to ASCII. It is explicitly not designed for processing arbitrary free text.
Because the Punycode delimiter is hyphen-minus and IDNA prepends an ACE prefix while forbidding pure-ASCII labels from being encoded, IDNA using Punycode conforms to the RFC 952 rule that host name labels neither begin nor end with a hyphen-minus.
Mixed-case annotation is not used by the ToASCII and ToUnicode operations specified in IDNA (RFC 3490); implementors of IDNA can disregard Appendix A entirely.
A Punycode decoder MUST recognize letters in both uppercase and lowercase forms, including mixtures of both forms.
A Punycode encoder SHOULD output only uppercase forms or only lowercase forms, unless it uses mixed-case annotation.
Basic code point segregation: all basic code points from the extended string are copied literally to the beginning of the basic string in their original order, followed by a delimiter if and only if the count of basic code points is nonzero. The decoder locates the end of the literal portion by scanning for the last delimiter.
Bias adaptation after each delta: (1) scale delta dividing by damp (first delta) or 2 (subsequent); (2) increase by delta/numpoints to compensate for longer string; (3) repeatedly divide by (base-tmin) until below threshold; (4) set bias = base*divisions + (base-tmin+1)*delta/(delta+skew).
Bootstring parameter constraints: 0 <= tmin <= tmax <= base-1; skew >= 1; damp >= 2; initial_bias mod base <= base - tmin. Parameters tmin, tmax, skew, damp, and initial_bias affect efficiency but not correctness, and are best chosen empirically.
Bootstring uses four techniques: basic code point segregation (literal copy), insertion unsort coding (deltas in numerical code point order), generalized variable-length integers (self-delimiting delta encoding), and bias adaptation (dynamic parameter adjustment for efficiency).
Decoding procedure: initialize n=initial_n, i=0, bias=initial_bias; copy literal portion (code points before last delimiter) to output; then loop consuming generalized variable-length integers to compute deltas, updating bias via adapt(), incrementing n and i, and inserting n at position i into the output string.
Encoding procedure: copy basic code points to output followed by a delimiter if any exist; then iterate finding the minimum non-basic code point >= n, computing the delta to advance the decoder state to that insertion point, and encoding the delta as a generalized variable-length integer, updating bias after each insertion.
Generalized variable-length integers use per-position thresholds t(j) to make integers self-delimiting: exactly one digit (the most significant) satisfies digit_j < t(j), enabling unambiguous concatenation. Each nonnegative integer has exactly one such representation. Bootstring uses little-endian ordering so deltas can be separated starting with the first.
Mixed-case annotation (Appendix A): when extended strings are case-folded before encoding, the last basic code point in each delta's encoding annotates the case preference (uppercase if the basic code point is uppercase, lowercase otherwise) for the corresponding non-basic code point. These annotations are returned separately and do not alter decoded code point values. This feature is not used by IDNA ToASCII/ToUnicode.
Overflow handling: overflow must be detected in delta computations during both encoding and decoding. For valid IDNA labels, 26-bit unsigned integers are sufficient (a 27-bit delta would exceed the Unicode code point limit 0..10FFFF or the 63-character label length limit), but inputs are not necessarily valid IDNA labels.
Punycode parameter values: base=36, tmin=1, tmax=26, skew=38, damp=700, initial_bias=72, initial_n=128 (0x80). These are chosen to work well with Unicode code points in the range 0..10FFFF.
The Punycode delimiter is U+002D (hyphen-minus). Digit values for basic code points: A-Z and a-z map to 0-25 respectively; 0-9 map to 26-35. Basic code points are the ASCII code points 0..7F.
Thresholds are defined as t(j) = base*(j+1) - bias, clamped to [tmin, tmax]. The bias state variable dynamically shifts which integer range is favored, updated after each delta via the adapt() function.
If a Unicode string could map to multiple ACE labels, an internationalized domain name could map to multiple ASCII domain names controlled by different authorities, enabling spoofing attacks that hijack service requests. Punycode is therefore designed so that each Unicode string has exactly one encoding.
Multiple Unicode representations of the 'same' text (e.g., different normalization forms) can still exist; this problem is addressed by Unicode canonicalization and leveraged for domain names by Nameprep (RFC 3491), not by Punycode itself.
The decoder uses a state machine with two variables: index i (position in extended string, 0 to current length) and counter n (code point value to insert). State <n,i> transitions to <n,i+1> if i < length, or <n+1,0> if i equals length, advancing monotonically with no way to return to an earlier state. Each delta represents a run of non-insertion states before an insertion.