On the Implementation of the TCP Urgent Mechanism
Extracted elements (20)
Despite the recommendation against using the urgent mechanism, TCP implementations are required to retain support for it. The rationale is to preserve backward compatibility with the few legacy applications (e.g., Telnet, FTP) that depend on urgent indications, even though those applications may face difficulties with middleboxes.
RFC 793, RFC 1011, and RFC 1122 are updated to define the Urgent Pointer as pointing to the sequence number of the octet following the urgent data (last+1), reversing the RFC 1122 clarification. The rationale is that when both endpoints use the same semantics there is no functional difference, and virtually all existing implementations already use this interpretation.
All widely deployed TCP implementations interpret the Urgent Pointer as pointing to the octet following the urgent data (RFC 793 section 3.1 semantics), not the last octet of urgent data (RFC 1122 semantics). The RFC 1122 clarification was never adopted in practice.
FreeBSD, Linux, NetBSD, OpenBSD, Cisco IOS, and all tested Microsoft Windows versions all implement the Urgent Pointer as pointing to the octet following the urgent data (RFC 793 section 3.1 semantics), and all support only one byte of urgent data in practice.
In practice, all implementations that treat TCP urgent indications as an OOB mechanism maintain only a single-byte buffer for urgent data. If a new urgent indication arrives before the application reads the pending OOB byte, the pending byte is discarded (overwritten). This contradicts the RFC 1122 requirement to support urgent data sequences of any length.
Linux's net.ipv4.tcp_stdurg sysctl, intended to switch to RFC 1122 urgent pointer semantics, applies only to incoming segments and not outgoing segments. This asymmetry means a host with this sysctl enabled cannot interoperate with itself when both sender and receiver run on the same machine.
RFC 793 section 3.1 states the Urgent Pointer points to the octet following urgent data, while section 3.9 of the same RFC implies it points to the last byte of urgent data (SND.NXT-1). RFC 1011 and RFC 1122 clarified it as pointing to the last octet of urgent data, but virtually all implementations have always used the RFC 793 section 3.1 semantics (last+1 interpretation).
Some middleboxes (e.g., Cisco PIX) clear URG flags and zero out the Urgent Pointer to counter NIDS evasion techniques. This transforms urgent data into in-line data for OOB-mode implementations, and means urgent indications cannot be relied upon in the current Internet.
Virtually all TCP implementations deliver the last byte of urgent data out-of-band via recv(2) with MSG_OOB, contrary to the IETF specification that urgent data be delivered in-line. Most implementations provide the SO_OOBINLINE socket option to override this broken default and restore in-line delivery.
Applications that still use the TCP urgent mechanism MUST set the SO_OOBINLINE socket option so that urgent data is delivered in-line as intended by the IETF specifications, rather than as OOB data.
Applications using the urgent mechanism must be designed to operate correctly even when the URG flag is cleared by middleboxes, since such clearing is a known behavior in deployed networks.
New applications SHOULD NOT employ the TCP urgent mechanism due to interoperability issues with current implementations and middlebox interference.
RFC 1122 section 4.2.2.4 requires that a TCP MUST support a sequence of urgent data of any length, since the urgent mechanism is simply a pointer into the data stream with no inherent length constraint.
TCP implementations MUST still include support for the urgent mechanism so that existing legacy applications can continue to use it.
The TCP urgent mechanism permits a point in the data stream to be designated as the end of urgent information. It is NOT a mechanism for sending out-of-band data; urgent data must be delivered in-line to the TCP user. The URG control flag indicates the Urgent Pointer field is meaningful and must be added to the segment sequence number to yield the urgent pointer.
Packet scrubbers can be configured to clear the URG bit and zero the Urgent Pointer, forcing urgent data in-line and mitigating NIDS evasion. However, this may cause interoperability problems or undesired behavior in applications (such as Telnet and FTP) that rely on the TCP urgent mechanism.
Some implementations that queue multiple urgent bytes fail to enforce any memory limit on the urgent data queue, making them vulnerable to trivial resource exhaustion (DoS) attacks. Implementations SHOULD enforce a limit on queued urgent data.
The ambiguity in urgent pointer semantics (RFC 793 section 3.1 vs. RFC 1122), combined with OOB vs. in-line delivery differences across implementations, makes it difficult for NIDS to track the application-layer data stream, enabling false positives and false negatives. This has been exploited as a NIDS evasion technique.
A TCP receiver enters 'urgent mode' whenever the urgent pointer is in advance of RCV.NXT; it returns to 'normal mode' when RCV.NXT catches up to the urgent pointer. Data received as normal data may retroactively become urgent data if an urgent indication arrives in a subsequent segment before that data is consumed.
The Urgent Pointer is a 16-bit field in the TCP header that is interpreted only when the URG control bit is set. Per this document (updating RFC 793, RFC 1011, and RFC 1122), the Urgent Pointer points to the sequence number of the octet immediately following the urgent data (i.e., last-urgent-byte + 1), matching de facto implementation practice.