Debugging RDP 0x609: The Protocol Bit a Proxy Should Never Toucheric

RDP error 0x609 — "An authentication error has occurred" — is a CredSSP failure. The standard advice is to check your NLA settings, patch your CredSSP version, or relax the encryption level. None of it helped us, because the error only appeared when connecting through a proxy. Direct connections to the same server worked perfectly every time.

That asymmetry is the only clue you need. If direct works and proxied doesn't, the proxy is transforming something it shouldn't be.

The setup

We were relaying RDP over a WebSocket tunnel — client connects to a local port, a relay forwards the raw TCP stream through a WebSocket connection to a gateway, gateway connects onward to the Windows machine. A standard CONNECT-style proxy, just over WebSocket instead of a raw TCP pipe.

The first phase of an RDP connection (X.224 → CredSSP → MCS → RDP) is negotiated in the clear before any screen data flows. Our relay passes bytes through untouched, so in theory the server should see exactly the same handshake whether or not a proxy is in the middle.

In theory.

What 0x609 actually tells you

The Windows error 0x609 maps to STATUS_LOGON_FAILURE inside the CredSSP layer, but that description is misleading. It doesn't necessarily mean the credentials are wrong. It can also mean the CredSSP negotiation completed but the session couldn't be established afterwards — the error surfaces at the CredSSP layer because that's where the TLS channel lives, and whatever went wrong downstream comes back through it.

The useless nature of this error code is part of what makes 0x609 hard to debug. You're essentially told "something authentication-related failed" without any pointer to where in the multi-phase handshake the failure occurred.

Reading the X.224 handshake

RDP starts with an X.224 Connection Request (CR) from the client. Buried inside the TPKT/X.224 framing is a requestedProtocols field — a bitmask that advertises which security layers the client supports:

• 0x01 — SSL/TLS
• 0x02 — NLA (CredSSP over NTLM/Kerberos), also called HYBRID
• 0x08 — HYBRID_EX (an extension to NLA used by modern Windows)

Modern mstsc sends 0x0b — all three bits set. The server responds with an X.224 Connection Confirm (CC) that includes a selectedProtocol field indicating which mode it chose.

Here is where we found the divergence.

Direct connection: the CC came back with selectedProtocol = 0x08 (HYBRID_EX).

Through the proxy: the CC came back with selectedProtocol = 0x02 (HYBRID, plain NLA).

Same client. Same server. Different selectedProtocol. The proxy was changing something in the CR before it reached the server.

The hidden transformation

The proxy had a small normalisation function applied to the first TCP frame on each new connection. Its purpose was compatibility: some older Windows environments support NLA but reject HYBRID_EX outright, returning an immediate RST. So the function detected the X.224 CR structure, read the requestedProtocols field, and cleared the 0x08 bit before forwarding.

The intent was conservative — strip the extended mode, keep plain NLA, everything still authenticates. It worked on the servers we originally tested it against.

What we hadn't accounted for: some Windows Server versions advertise both HYBRID and HYBRID_EX, but internally require the HYBRID_EX flow to complete a session correctly. When forced into HYBRID mode by the altered CR, they complete CredSSP — all five steps of the NTLM exchange succeed, credentials are verified — and then the MCS Connect-Initial fails. The session can't start.

The mechanism: HYBRID_EX adds a step that HYBRID omits. After the CredSSP exchange, the server sends an Early User Authorization Result PDU before the MCS layer begins. This PDU is part of the HYBRID_EX contract. When the server is put into HYBRID mode (because 0x08 was stripped from the CR), it doesn't send this PDU — but if the server's internal session setup expects it, or if the subsequent MCS framing depends on state that only HYBRID_EX establishes, the session fails. From the client's side it arrives as a 0x609.

In pseudo-code, what was happening:

Copy # CR goes out with 0x0b (SSL | NLA | HYBRID_EX)
# proxy strips 0x08 → 0x03 (SSL | NLA)
# server selects HYBRID (0x02) — the highest it can given the offer
# CredSSP runs normally and succeeds
# server tries to start MCS in HYBRID_EX mode internally
# HYBRID_EX Early User Authorization PDU never arrives
# MCS Connect-Initial fails
# error surfaces as 0x609 on the CredSSP channel

The fix

Remove the stripping entirely. Pass requestedProtocols through unchanged.

The original compatibility concern — older servers rejecting HYBRID_EX — turned out to be unwarranted in practice. Modern mstsc has been sending HYBRID_EX for years and servers handle it gracefully: if they don't support it they simply select a lower mode in the CC. The server's CC is the right place for that negotiation to happen, not a proxy in the middle.

What to check if you hit this

If you're seeing 0x609 only through a proxy and direct connections work:

1. Capture both sides of the relay. You want the raw bytes the server sees, not the bytes the client sent. Log the first frame in each direction.

2. Find the X.224 CC and read selectedProtocol. It's a 4-byte little-endian value a few bytes into the CC. If it differs between direct and proxied connections, the proxy is rewriting the CR.

3. Check for any transformation applied to the first message. Header parsers, encoding normalizers, protocol fixups — anything that touches the first frame of a new TCP connection is a candidate.

4. Compare requestedProtocols in the CR. If the proxy is changing the bitmask, restore it to pass through verbatim and retest.

The broader principle: a byte-stream proxy should be transparent. Any transformation that modifies the protocol negotiation — even a seemingly harmless bit-clear — can break assumptions downstream that neither the proxy nor its author can fully anticipate.

The server, not the proxy, decides which protocol variant to use. Let it.