The Connection Reset That Wasn't: Hunting a Protobuf Bug in IBKR's TWS APIeric

Our IBKR market data gateway had been streaming quotes for hours without a hitch. Then, with no code change and no configuration change on our end, every single connection attempt started failing the same way:

Copy IBKR error id=-1 code=-1: Connection reset by peer

Instantly. Every time. Regardless of client ID. Regardless of which host we connected from. It survived restarting TWS. It survived a full reboot of the machine. Settings were fine, the account was fully logged in, no prompt was waiting for a click. By every conventional signal, everything was correct — and it still reset in under a fifth of a second, every single attempt.

This is the story of chasing that down: five real bugs fixed along the way, none of which were the actual cause, and the one root cause that finally showed up when we stopped trusting the application layer and went straight to the wire.

The bugs that were real, but weren't it

Before the actual culprit, it's worth walking through what we did fix, because each of these was a genuine, confirmed bug — just not the one causing this particular symptom.

A reconnect storm. The gateway had two independent, uncoordinated reconnect paths — one inside the connector class, one in the process's own connection-status handler — that could both fire for the same disconnect event and both call the underlying socket's connect method concurrently. Under the right conditions this produced an exponential explosion: a systemctl status check showed the task count climbing past 120 and still rising. Fixed by making one path the sole owner of reconnect logic.

A stale timestamp defeating the backoff. Even after fixing the storm, retries were still hammering the server every 300 milliseconds instead of backing off. The backoff logic decided "fast retry vs. exponential backoff" based on how long the previous session had been alive — but that timestamp was only updated on a successful connect. On a run that never once succeeded, it stayed stale, making every immediate rejection look like "this was healthy for ages," which took the fast-retry branch forever. The fix: stamp the timestamp at the start of each attempt, not just on success.

A busy loop pegging an entire CPU core. This one had nothing to do with the reset at all, and we only found it because we happened to check ps aux while debugging something else and saw a process sitting at a sustained 100% CPU. A dotnet-dump capture and a managed stack trace showed the real story:

Copy System.IO.StreamReader.ReadLine()
Program+<>c__DisplayClass93_0.<Main>b__1(System.Object)
System.Threading.ThreadPoolWorkQueue.Dispatch()

The process had an interactive "press x to exit" input loop reading from stdin — completely reasonable for a console app run by hand. Under systemd, though, stdin is closed. Console.ReadLine() on a closed stream returns null immediately, forever, and the loop had no check for that — so it called ReadLine() again at full speed, permanently. This is the shared entry point every one of our broker connectors goes through, not something specific to this one gateway, so it had almost certainly been burning a full core on every systemd-managed instance in the fleet, for as long as any of them had been running. One if (null == line) break; and CPU usage on that process dropped from 100% to under 2%.

A double subscription with a silent side effect. After the actual fix (below) got the connection stable, we noticed every persisted symbol produced a Duplicate ticker id error on every connect. The connection-established handler fires synchronously, and the resubscribe loop inside the connector's own "restore market data after reconnect" logic ran after the process's separate, canonical resubscribe path had already resubscribed the same symbols moments earlier — so it fired again, reusing the same ticker IDs. Worse: that redundant loop called the market-less subscribe overload, meaning on a genuine reconnect it would have silently dropped market context and re-resolved exchange-specific symbols against the wrong venue. Removed entirely — the canonical path already covers it correctly.

Four real, verified fixes. None of them touched the reset.

Going to the wire

At this point we'd validated login, TWS settings, trusted-IP configuration, client ID uniqueness, and machine state, and restarted or rebooted at every layer available to us. The application-level signal — "Connection reset by peer" — kept pointing everywhere and nowhere. So we stopped trusting it and captured the actual packets:

Copy sudo tcpdump -i lo -w reset.pcap 'port 7496'

The capture told a different story than the error message did. TWS accepted the plaintext version handshake fine — negotiated server version 223 — and then, the instant it received the client's next message, it sent a graceful FIN. Not a RST. Our own error handling was mislabeling an orderly, voluntary close as a socket reset.

That next message was the payload of interest:

Copy 20:42:05.070228 ... length 11
0x0030:  76fa 62b5 0000 0007 0000 010f 08e8 07
20:42:05.070642 ... length 10
0x0030:  76fa 62b5 0000 0006 0000 0103 0803

That's not the classic IB API wire format — a null-byte-delimited text protocol we'd tested by hand with a raw socket and confirmed worked, every time, for the exact same connection sequence and client ID. This was binary, varint-shaped data. 0xe8 0x07 decodes to 1000 — our client ID, Protobuf-encoded.

The one-line fix

Our vendored IB API client library automatically switches specific outgoing messages to Protobuf encoding once the negotiated server version clears a threshold:

Copy public static bool useProtoBuf(int serverVersion, OutgoingMessages outgoingMessage)
{
    return Constants.PROTOBUF_MSG_IDS.TryGetValue(outgoingMessage, out int unifiedVersion)
        && unifiedVersion <= serverVersion;
}

StartApi — the message sent automatically, immediately after every connection — was mapped to require server version 213. Our negotiated version was 223. The threshold cleared, so the library always sent StartApi as Protobuf. And this particular TWS build, despite reporting a version that the client library's own table says should support it, evidently doesn't actually parse a Protobuf-encoded StartApi — it just closes the connection the instant it arrives.

We confirmed this the only way that actually proves anything: hand-crafting the identical connection sequence over a raw socket, with the plaintext format instead of Protobuf, same client ID, same host, same everything else. It connected immediately and stayed open.

The fix was one dictionary entry:

Copy // StartApi deliberately excluded: this TWS build doesn't support Protobuf
// for it despite reporting a version that implies it should.
{ OutgoingMessages.ChangeServerLog, MinServerVer.MIN_SERVER_VER_PROTOBUF_REST_MESSAGES_3 },

No other message at that version tier is used anywhere in our codebase, so removing just the StartApi entry was the entire change.

Why it "worked before"

If this is deterministic given a fixed server version, why did the same code stream data reliably for hours before this started? The most likely explanation is that it wasn't the same server version — TWS auto-updates itself, and its daily scheduled restart is exactly the mechanism that would pull in a new build mid-session. Before whatever update happened, the negotiated version was presumably below the Protobuf threshold, so the client used the classic format the whole time and never touched this code path. After the update, every connection attempt crossed the threshold and hit a gap in that specific build's Protobuf support.

We can't directly confirm TWS's build history — that detail lives inside an encrypted session log we don't have tooling to read — but it's the only explanation consistent with every observed fact: hours of stable streaming beforehand, a sudden onset with zero configuration changes on our side, and a packet capture proving the break point is version-gated behavior that only exists above a specific threshold.

What actually mattered

None of the individual debugging steps here were exotic. ps aux, dotnet-dump, tcpdump, a raw socket in twenty lines of Python. What mattered was the discipline of not stopping at the first plausible-looking bug. Four of the five things we fixed were real, and it would have been easy to declare victory after any one of them — the reconnect storm alone was dramatic enough (task count climbing past 120) to feel like the bug. It wasn't. The actual fix only showed up once we stopped reading the application's own description of the failure and looked at what was actually on the wire.

If you're chasing a "connection reset" that survives every fix that should have worked: check whether it's really a reset at the TCP level before you believe the error message, and check what your client is actually sending, not what you assume it sends.

If you're running the IB API client library and see a similarly unexplainable reset immediately after connecting, checking whether a Protobuf-eligible message is involved is worth five minutes before anything else.