{/* banner prompt: Clean isometric diorama on a white background: a miniature network cable plugs into a small server tower, but instead of connecting, a stylized red "RST" shockwave bounces off an invisible barrier. Beside it, a magnifying glass reveals tiny binary/protobuf-like blocks hidden inside the cable's data stream. Minimalist, crisp shadows, matte sage-green/cream/terracotta color palette, soft studio lighting, 16:9. */}

Our IBKR market data gateway had been streaming quotes for hours without a hitch. Then, with no code change and no configuration change on our end, every single connection attempt started failing the same way:

```
IBKR error id=-1 code=-1: Connection reset by peer
```

Instantly. Every time. Regardless of client ID. Regardless of which host we connected from. It survived restarting TWS. It survived a full reboot of the machine. Settings were fine, the account was fully logged in, no prompt was waiting for a click. By every conventional signal, everything was correct — and it still reset in under a fifth of a second, every single attempt.

This is the story of chasing that down: five real bugs fixed along the way, none of which were the actual cause, and the one root cause that finally showed up when we stopped trusting the application layer and went straight to the wire.

## The bugs that were real, but weren't it

Before the actual culprit, it's worth walking through what we *did* fix, because each of these was a genuine, confirmed bug — just not the one causing this particular symptom.

**A reconnect storm.** The gateway had two independent, uncoordinated reconnect paths — one inside the connector class, one in the process's own connection-status handler — that could both fire for the same disconnect event and both call the underlying socket's connect method concurrently. Under the right conditions this produced an exponential explosion: a `systemctl status` check showed the task count climbing past 120 and still rising. Fixed by making one path the sole owner of reconnect logic.

**A stale timestamp defeating the backoff.** Even after fixing the storm, retries were still hammering the server every 300 milliseconds instead of backing off. The backoff logic decided "fast retry vs. exponential backoff" based on how long the *previous* session had been alive — but that timestamp was only updated on a *successful* connect. On a run that never once succeeded, it stayed stale, making every immediate rejection look like "this was healthy for ages," which took the fast-retry branch forever. The fix: stamp the timestamp at the start of each attempt, not just on success.

**A busy loop pegging an entire CPU core.** This one had nothing to do with the reset at all, and we only found it because we happened to check `ps aux` while debugging something else and saw a process sitting at a sustained 100% CPU. A `dotnet-dump` capture and a managed stack trace showed the real story:

```
System.IO.StreamReader.ReadLine()
Program+<>c__DisplayClass93_0.<Main>b__1(System.Object)
System.Threading.ThreadPoolWorkQueue.Dispatch()
```

The process had an interactive "press x to exit" input loop reading from stdin — completely reasonable for a console app run by hand. Under systemd, though, stdin is closed. `Console.ReadLine()` on a closed stream returns `null` *immediately*, forever, and the loop had no check for that — so it called `ReadLine()` again at full speed, permanently. This is the shared entry point every one of our broker connectors goes through, not something specific to this one gateway, so it had almost certainly been burning a full core on every systemd-managed instance in the fleet, for as long as any of them had been running. One `if (null == line) break;` and CPU usage on that process dropped from 100% to under 2%.

**A double subscription with a silent side effect.** After the actual fix (below) got the connection stable, we noticed every persisted symbol produced a `Duplicate ticker id` error on every connect. The connection-established handler fires synchronously, and the resubscribe loop inside the connector's own "restore market data after reconnect" logic ran *after* the process's separate, canonical resubscribe path had already resubscribed the same symbols moments earlier — so it fired again, reusing the same ticker IDs. Worse: that redundant loop called the market-less subscribe overload, meaning on a genuine reconnect it would have silently dropped market context and re-resolved exchange-specific symbols against the wrong venue. Removed entirely — the canonical path already covers it correctly.

Four real, verified fixes. None of them touched the reset.

## Going to the wire

At this point we'd validated login, TWS settings, trusted-IP configuration, client ID uniqueness, and machine state, and restarted or rebooted at every layer available to us. The application-level signal — "Connection reset by peer" — kept pointing everywhere and nowhere. So we stopped trusting it and captured the actual packets:

```sh
sudo tcpdump -i lo -w reset.pcap 'port 7496'
```

The capture told a different story than the error message did. TWS accepted the plaintext version handshake fine — negotiated server version 223 — and then, the instant it received the client's *next* message, it sent a graceful `FIN`. Not a `RST`. Our own error handling was mislabeling an orderly, voluntary close as a socket reset.

That next message was the payload of interest:

```
20:42:05.070228 ... length 11
0x0030:  76fa 62b5 0000 0007 0000 010f 08e8 07
20:42:05.070642 ... length 10
0x0030:  76fa 62b5 0000 0006 0000 0103 0803
```

That's not the classic IB API wire format — a null-byte-delimited text protocol we'd tested by hand with a raw socket and confirmed worked, every time, for the exact same connection sequence and client ID. This was binary, varint-shaped data. `0xe8 0x07` decodes to 1000 — our client ID, Protobuf-encoded.

## The one-line fix

Our vendored IB API client library automatically switches specific outgoing messages to Protobuf encoding once the negotiated server version clears a threshold:

```csharp
public static bool useProtoBuf(int serverVersion, OutgoingMessages outgoingMessage)
{
    return Constants.PROTOBUF_MSG_IDS.TryGetValue(outgoingMessage, out int unifiedVersion)
        && unifiedVersion <= serverVersion;
}
```

`StartApi` — the message sent automatically, immediately after every connection — was mapped to require server version 213. Our negotiated version was 223. The threshold cleared, so the library always sent `StartApi` as Protobuf. And this particular TWS build, despite reporting a version that the client library's own table says should support it, evidently doesn't actually parse a Protobuf-encoded `StartApi` — it just closes the connection the instant it arrives.

We confirmed this the only way that actually proves anything: hand-crafting the identical connection sequence over a raw socket, with the plaintext format instead of Protobuf, same client ID, same host, same everything else. It connected immediately and stayed open.

The fix was one dictionary entry:

```csharp
// StartApi deliberately excluded: this TWS build doesn't support Protobuf
// for it despite reporting a version that implies it should.
{ OutgoingMessages.ChangeServerLog, MinServerVer.MIN_SERVER_VER_PROTOBUF_REST_MESSAGES_3 },
```

No other message at that version tier is used anywhere in our codebase, so removing just the `StartApi` entry was the entire change.

## Why it "worked before"

If this is deterministic given a fixed server version, why did the same code stream data reliably for hours before this started? The most likely explanation is that it wasn't the same server version — TWS auto-updates itself, and its daily scheduled restart is exactly the mechanism that would pull in a new build mid-session. Before whatever update happened, the negotiated version was presumably below the Protobuf threshold, so the client used the classic format the whole time and never touched this code path. After the update, every connection attempt crossed the threshold and hit a gap in that specific build's Protobuf support.

We can't directly confirm TWS's build history — that detail lives inside an encrypted session log we don't have tooling to read — but it's the only explanation consistent with every observed fact: hours of stable streaming beforehand, a sudden onset with zero configuration changes on our side, and a packet capture proving the break point is version-gated behavior that only exists above a specific threshold.

## What actually mattered

None of the individual debugging steps here were exotic. `ps aux`, `dotnet-dump`, `tcpdump`, a raw socket in twenty lines of Python. What mattered was the discipline of not stopping at the first plausible-looking bug. Four of the five things we fixed were real, and it would have been easy to declare victory after any one of them — the reconnect storm alone was dramatic enough (task count climbing past 120) to feel like *the* bug. It wasn't. The actual fix only showed up once we stopped reading the application's own description of the failure and looked at what was actually on the wire.

If you're chasing a "connection reset" that survives every fix that should have worked: check whether it's really a reset at the TCP level before you believe the error message, and check what your client is *actually* sending, not what you assume it sends.

*If you're running the IB API client library and see a similarly unexplainable reset immediately after connecting, checking whether a Protobuf-eligible message is involved is worth five minutes before anything else.*


710cb12b6a8a2248378b97525bbe585c1decc93be251377d5959c488532d90df

Root-causing an IBKR TWS API 'Connection reset by peer' bug with tcpdump and a raw socket client - the real cause was a Protobuf-encoded StartApi message the TWS build didn't support.

A market data gateway that worked fine for hours started rejecting every single connection with 'Connection reset by peer' - and stayed broken through TWS restarts, a full machine reboot, and every conventional fix. Here's how a raw packet capture and a hand-crafted socket client found the real bug: a version-gated Protobuf auto-switch that TWS silently didn't support.

The Connection Reset That Wasn't: Hunting a Protobuf Bug in IBKR's TWS API


{/* banner prompt: A playful but polished 3D clay-render, matte materials in sage green, cream and terracotta, of a stylized detective magnifying glass hovering over a small padlock icon that is cracking open, with a tangled network cable looping around a proxy-server box in the background, soft studio shadows, simple background, 16:9. */}

`yt-dlp` gave us the least helpful error message it has: `HTTP Error 410: Gone`. The standard fix is `yt-dlp -U`. We ran it. Already on the latest stable release. The video was not, in fact, gone — but proving that took three separate bugs stacked on top of each other, and the last one had nothing to do with `yt-dlp` at all.

We were debugging a small local downloader tool — a browser extension that hands a page URL and cookies to `yt-dlp` on the user's own machine, optionally routed through a local HTTP proxy for geo-unblocking. One specific site started failing every time, on a URL that worked fine a week earlier.

## The first clue was buried in `-v`

The plain error told us nothing. Adding `-v` for verbose output surfaced a line that's easy to scroll past:

```
WARNING: [SiteExtractor] The extractor is attempting impersonation, but no
impersonate target is available. If you encounter errors, then see
https://github.com/yt-dlp/yt-dlp#impersonation for information on
installing the required dependencies
```

That's the real signal. A growing number of sites fingerprint the TLS handshake and HTTP/2 frame ordering of incoming requests to tell a real browser apart from a scripted client — even when every header looks identical. `yt-dlp` can spoof a browser's TLS fingerprint via an optional dependency, `curl_cffi`, which wraps `libcurl` with impersonation profiles for Chrome, Safari, Firefox and so on. Without it, some extractors still try the request, get rejected at the TLS layer, and the site's edge returns a generic `410` instead of anything that explains what actually happened. The video was fine. `yt-dlp` just didn't look enough like a browser to be allowed to ask.

## The version window nobody tells you about

The fix looked trivial:

```sh
python3 -m pip install curl_cffi
```

It installed cleanly — `curl_cffi-0.16.0`, the latest release. `yt-dlp` still refused to use it, and `--list-impersonate-targets` came back with every target marked `(unavailable)`. The debug line was the giveaway:

```
[debug] Optional libraries: ... curl_cffi-0.16.0 (unsupported) ...
```

`yt-dlp` doesn't just check that `curl_cffi` is importable — it pins a narrow set of versions it actually trusts, because `curl_cffi`'s impersonation API has changed shape across releases. Digging through the installed `yt-dlp` build (it ships as a zip-app; `python -m zipfile` opens it like any other zip) turned up the exact check:

```python
curl_cffi_version = tuple(map(int, re.split(r'[^\d]+', curl_cffi.__version__)[:3]))

if curl_cffi_version != (0, 5, 10) and not (0, 10) <= curl_cffi_version < (0, 16):
    curl_cffi._yt_dlp__version = f'{curl_cffi.__version__} (unsupported)'
    raise ImportError('Only curl_cffi versions 0.5.10 and 0.10.x through 0.15.x are supported')
```

Latest-and-greatest was one minor version too new. The fix was to pin below the ceiling:

```sh
python3 -m pip install "curl_cffi>=0.10,<0.16"
```

**Lesson:** when a library depends on another native-extension package for an optional feature, don't assume "latest" is "supported" — check for a version gate specifically on the feature you're trying to turn on, not just on whether the import succeeds.

## Collateral damage from `--force-reinstall`

The first install attempt used `--force-reinstall` to make sure the correct version actually took. It did — but it also silently upgraded packages that were already correctly pinned for something else entirely: `certifi`, `cffi`, `markdown-it-py`, and `pygments` all jumped to newer versions than another tool in the same shared Python environment required. `pip` said so plainly, right at the end of the install:

```
ERROR: pip's dependency resolver does not currently take into account all
the packages that are installed. This behaviour is the source of the
following dependency conflicts.
some-other-tool 2.15.0 requires certifi==2026.4.22, but you have
certifi 2026.7.22 which is incompatible.
some-other-tool 2.15.0 requires cffi==2.0.0, but you have cffi 2.1.1 ...
```

None of those four packages actually needed upgrading — `curl_cffi`'s own requirement (`cffi>=2.0.0`, `certifi>=2024.2.2`) was already satisfied by the versions in place. `--force-reinstall` doesn't just reinstall the package you named; it re-resolves and reinstalls its entire dependency subtree, and when it does that it grabs the latest version satisfying each constraint rather than leaving an already-satisfied pin alone. In a shared base environment with several unrelated tools installed, that's a quiet way to break something you weren't even looking at.

The fix was to pin those four straight back:

```sh
python3 -m pip install "certifi==2026.4.22" "cffi==2.0.0" \
  "markdown-it-py==4.0.0" "pygments==2.20.0"
```

**Lesson:** avoid `--force-reinstall` in a shared or base Python environment. If you need a specific version of one package, ask for exactly that (`pip install "pkg==x.y.z"`) rather than forcing a full dependency re-resolve you don't control.

## A new error, and a wrong theory

With the right `curl_cffi` version in place, the impersonation warning was gone — replaced by a new one, appearing on every single attempt:

```
SSLError: Failed to perform, curl: (35) TLS connect error:
error:00000000:invalid library (0):OPENSSL_internal:invalid library (0)
```

This looks exactly like a known class of `curl_cffi` bug: its bundled BoringSSL build can clash with another OpenSSL-family library already loaded in the same Python process — commonly `cryptography`, pulled in transitively by something unrelated like browser-cookie decryption support. The natural fix is isolation: a clean virtual environment with only `yt-dlp` and `curl_cffi`, nothing that could drag in a conflicting crypto library.

We built that environment. Same error, 100% reproducible, in complete isolation — no `cryptography`, no `secretstorage`, nothing else installed at all.

That result was more useful than a fix would have been. A clean-environment reproduction rules out an entire category of theories in one shot. It wasn't a Python packaging conflict. It had to be something outside Python's import graph.

## Stripping back to the dumbest possible test

The next step was to remove every layer of abstraction and test the actual thing that was failing: a TLS handshake over the proxy. Not `curl_cffi`. Not even `yt-dlp`. Plain, unimpersonated `requests`:

```python
import requests
r = requests.get("https://www.example.com",
                  proxies={"https": "http://127.0.0.1:8082"})
```

Same SSL error. Against a domain with nothing to do with the original site. That was the moment the whole investigation flipped: this was never a `curl_cffi` bug, or even a Python bug. Something about the proxy itself was breaking HTTPS.

Raw `curl` confirmed it in the plainest terms available:

```
* CONNECT tunnel established, response 200
> Client hello (1)
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.com:443
```

The proxy's `CONNECT` tunnel came up cleanly — a `200 Connection Established` — and then the TLS handshake died immediately afterward, for every client, on every site. Nothing about `curl_cffi`'s TLS fingerprint mattered; the tunnel itself wasn't relaying bytes correctly once real TLS traffic started flowing through it.

The proxy client had several selectable exit gateways by region. Switching from one region to another and rerunning the exact same `curl` command fixed it instantly — clean `200`, valid response, every time. The original gateway node was in a broken state for HTTPS relaying specifically; nothing in our code, `yt-dlp`, or `curl_cffi` was ever going to fix that from the client side.

## What actually shipped

Three real problems, three real fixes, none of them where the first error message pointed:

<Table data={
    {
        headers: ['Symptom', 'Looked like', 'Actually was'],
        rows: [
            ['HTTP 410: Gone', 'Stale yt-dlp / dead video', 'Site requires TLS impersonation'],
            ['curl_cffi "(unsupported)"', 'Broken install', "Version above yt-dlp's pinned ceiling"],
            ['Unrelated tool broke after install', 'Unrelated bug', '--force-reinstall re-resolved shared deps'],
            ['SSL "invalid library" error', 'curl_cffi vs OpenSSL clash', 'Proxy gateway failing the TLS relay'],
        ]
    }
} />

The lasting fix was to isolate `curl_cffi`-enabled `yt-dlp` into its own virtual environment — so a future dependency bump anywhere else on the machine can never touch it again — and to symlink the `PATH`-resolved `yt-dlp` at that environment's build, so every caller picks it up without extra configuration.

## The general shape of this kind of bug

None of the individual fixes here are exotic. What made this one take longer than it should have was trusting each error message's own framing for too long:

- **"410 Gone" told us the video was gone.** It wasn't — it was an anti-bot response wearing the shape of a different error.
- **"unsupported" told us the install was broken.** It wasn't broken, it was one version too new for a pin we didn't know existed.
- **"invalid library" pointed straight at `curl_cffi` and OpenSSL.** It was neither — it was a proxy gateway with a bad TLS relay.

The pattern that actually broke each false lead was the same one every time: reproduce with the fewest possible moving parts. A clean virtual environment ruled out the packaging theory. Plain `requests` with no impersonation ruled out `curl_cffi` entirely. Raw `curl` against an unrelated domain ruled out the target site. Each step removed one whole category of suspects instead of guessing at the next fix — and the actual root cause, when it finally showed up, wasn't in any of the code we'd been staring at.


9fa2ef35cdb16bce3f05bf518227a3d6de0d42440772c0f4c3b4eaddebedb611

How we debugged a yt-dlp 410 Gone error that wasn't staleness: curl_cffi TLS impersonation, a strict version-pinning gotcha, a --force-reinstall dependency mishap in a shared Python environment, and a proxy gateway failure that mimicked a library bug.

A yt-dlp download kept failing with 'HTTP Error 410: Gone' even on the latest version. The real cause was three layered problems stacked on top of each other: missing TLS impersonation, a narrow curl_cffi version-compatibility window, and an unrelated proxy gateway fault that looked exactly like a library bug.

Debugging a Fake 410: yt-dlp, curl_cffi, and a Proxy That Lied


{/* banner prompt: A vivid infographic-style scene where a stream of glowing data blocks flows through a semi-transparent proxy filter that visibly plucks out a single red block, labelled "0x08", causing a connection icon downstream to display a red error symbol "0x609". Clean light-gray background, sharp isometric perspective, muted blue-and-red palette, technical but clear. */}

RDP error 0x609 — "An authentication error has occurred" — is a CredSSP failure. The standard advice is to check your NLA settings, patch your CredSSP version, or relax the encryption level. None of it helped us, because the error only appeared when connecting *through a proxy*. Direct connections to the same server worked perfectly every time.

That asymmetry is the only clue you need. If direct works and proxied doesn't, the proxy is transforming something it shouldn't be.

## The setup

We were relaying RDP over a WebSocket tunnel — client connects to a local port, a relay forwards the raw TCP stream through a WebSocket connection to a gateway, gateway connects onward to the Windows machine. A standard CONNECT-style proxy, just over WebSocket instead of a raw TCP pipe.

The first phase of an RDP connection (X.224 → CredSSP → MCS → RDP) is negotiated in the clear before any screen data flows. Our relay passes bytes through untouched, so in theory the server should see exactly the same handshake whether or not a proxy is in the middle.

In theory.

## What 0x609 actually tells you

The Windows error 0x609 maps to `STATUS_LOGON_FAILURE` inside the CredSSP layer, but that description is misleading. It doesn't necessarily mean the credentials are wrong. It can also mean the CredSSP negotiation *completed* but the session couldn't be established afterwards — the error surfaces at the CredSSP layer because that's where the TLS channel lives, and whatever went wrong downstream comes back through it.

The useless nature of this error code is part of what makes 0x609 hard to debug. You're essentially told "something authentication-related failed" without any pointer to where in the multi-phase handshake the failure occurred.

## Reading the X.224 handshake

RDP starts with an X.224 Connection Request (CR) from the client. Buried inside the TPKT/X.224 framing is a `requestedProtocols` field — a bitmask that advertises which security layers the client supports:

- `0x01` — SSL/TLS
- `0x02` — NLA (CredSSP over NTLM/Kerberos), also called HYBRID
- `0x08` — HYBRID_EX (an extension to NLA used by modern Windows)

Modern `mstsc` sends `0x0b` — all three bits set. The server responds with an X.224 Connection Confirm (CC) that includes a `selectedProtocol` field indicating which mode it chose.

Here is where we found the divergence.

**Direct connection:** the CC came back with `selectedProtocol = 0x08` (HYBRID_EX).

**Through the proxy:** the CC came back with `selectedProtocol = 0x02` (HYBRID, plain NLA).

Same client. Same server. Different `selectedProtocol`. The proxy was changing something in the CR before it reached the server.

## The hidden transformation

The proxy had a small normalisation function applied to the first TCP frame on each new connection. Its purpose was compatibility: some older Windows environments support NLA but reject HYBRID_EX outright, returning an immediate RST. So the function detected the X.224 CR structure, read the `requestedProtocols` field, and cleared the `0x08` bit before forwarding.

The intent was conservative — strip the extended mode, keep plain NLA, everything still authenticates. It worked on the servers we originally tested it against.

What we hadn't accounted for: some Windows Server versions advertise both HYBRID and HYBRID_EX, but internally require the HYBRID_EX flow to complete a session correctly. When forced into HYBRID mode by the altered CR, they complete CredSSP — all five steps of the NTLM exchange succeed, credentials are verified — and then the MCS Connect-Initial fails. The session can't start.

The mechanism: HYBRID_EX adds a step that HYBRID omits. After the CredSSP exchange, the server sends an Early User Authorization Result PDU before the MCS layer begins. This PDU is part of the HYBRID_EX contract. When the server is put into HYBRID mode (because `0x08` was stripped from the CR), it doesn't send this PDU — but if the server's internal session setup expects it, or if the subsequent MCS framing depends on state that only HYBRID_EX establishes, the session fails. From the client's side it arrives as a 0x609.

In pseudo-code, what was happening:

```
# CR goes out with 0x0b (SSL | NLA | HYBRID_EX)
# proxy strips 0x08 → 0x03 (SSL | NLA)
# server selects HYBRID (0x02) — the highest it can given the offer
# CredSSP runs normally and succeeds
# server tries to start MCS in HYBRID_EX mode internally
# HYBRID_EX Early User Authorization PDU never arrives
# MCS Connect-Initial fails
# error surfaces as 0x609 on the CredSSP channel
```

## The fix

Remove the stripping entirely. Pass `requestedProtocols` through unchanged.

The original compatibility concern — older servers rejecting HYBRID_EX — turned out to be unwarranted in practice. Modern `mstsc` has been sending HYBRID_EX for years and servers handle it gracefully: if they don't support it they simply select a lower mode in the CC. The server's CC is the right place for that negotiation to happen, not a proxy in the middle.

## What to check if you hit this

If you're seeing 0x609 only through a proxy and direct connections work:

1. **Capture both sides of the relay.** You want the raw bytes the server sees, not the bytes the client sent. Log the first frame in each direction.

2. **Find the X.224 CC and read `selectedProtocol`.** It's a 4-byte little-endian value a few bytes into the CC. If it differs between direct and proxied connections, the proxy is rewriting the CR.

3. **Check for any transformation applied to the first message.** Header parsers, encoding normalizers, protocol fixups — anything that touches the first frame of a new TCP connection is a candidate.

4. **Compare `requestedProtocols` in the CR.** If the proxy is changing the bitmask, restore it to pass through verbatim and retest.

The broader principle: a byte-stream proxy should be transparent. Any transformation that modifies the protocol negotiation — even a seemingly harmless bit-clear — can break assumptions downstream that neither the proxy nor its author can fully anticipate.

The server, not the proxy, decides which protocol variant to use. Let it.


6a01956bb020c50d9f63b34a3f6be7e108447b2e7a723fabe945abf36bfa1498

How we debugged RDP error 0x609 appearing only through a WebSocket proxy, and why stripping the HYBRID_EX bit from the protocol negotiation breaks the entire session.

A CredSSP 0x609 error appearing only through a proxy but not on direct connections — traced to a single HYBRID_EX bit being silently stripped from the RDP X.224 handshake.

Debugging RDP 0x609: The Protocol Bit a Proxy Should Never Touch

{/* banner prompt: Warm, realistic analog photography of a cluttered but organized workshop desk, where a long ticker tape of computer output unspools from a vintage telegraph machine, with sections circled in red ink and handwritten notes in the margins, representing an AI agent debugging code through text logs, all in soft, natural window light, 16:9. */}

We called the [previous post](/blog/2026/06/13-the-coding-agent-shootout-and-the-best-harness-we-could-not-read/) the last in the series. That was the telescope view — what agents *are*, what a [harness does](/blog/2026/06/13-what-an-ai-agent-harness-actually-does/), how they [learn](/blog/2026/06/13-skills-and-memory-how-ai-agents-actually-learn/), which one wins. But a developer doesn't experience an agent as an architecture diagram. You experience it one edit at a time: it changes three lines, runs the tests, reads the failure, tries again. So here is the microscope view — how a coding agent actually turns an instruction into a working, tested change, held up against how a human programmer does the same thing.

## The human inner loop

Picture your own setup. You have an editor where you type, a terminal where you compile and run, and some way to *see the result* — a test runner's green bar, a printed value, a web UI you reload in a browser. Around it sits the big lifecycle everyone draws on whiteboards: idea → plan → architecture → design → write. But that's the outer loop. The thing you actually do all day is the **inner loop**, and it's tight: write a little, run it, look at what happened, adjust. Seconds per cycle.

Two habits define how humans fill that loop. First, we mostly **build small and grow** — a few lines, run, a few more — keeping a working program at almost every step, rather than generating a whole finished file and hoping. Second, we choose a direction: **vertically** (one feature sliced end-to-end through every layer until it runs) or **horizontally** (one layer built out across features). Either way the program is *alive* in front of us the whole time. We glance at the screen and instantly know if it's broken.

That last point is the one that matters most for what follows, because it's exactly the sense an agent doesn't have.

## The agent inner loop

An agent runs the same shape of loop — instruction in, change out, observe, repeat — but through a fundamentally narrower keyhole. Its loop is: the harness sends the model the context, the model emits a **tool call** (edit this file, run this command), the harness executes it and feeds the **text result** back, and the model decides what to do next. That's it. The agent has no eyes. It cannot glance at a screen. **Its only senses are the text that tools return** — file contents, stdout, stderr, an exit code. Everything a human takes in visually — the rendered web page, the debugger paused on a line, the red squiggle under a typo — has to be converted into text and handed back, or it doesn't exist for the agent at all.

This single fact explains almost everything distinctive about how agents code. It's why output truncation matters (an 80,000-line log blinds it as surely as a wall of noise), why edit-acceptance is fiddly, why verification has to be deliberate, and why debugging — the most *visual*, interactive thing a human does — is the hardest part to translate.

It also shifts the economics of "small and grow." Each model round-trip costs real tokens and seconds, so an agent can't afford the human's keystroke-level incrementalism; it tends to write a whole function or file in one shot, then verify. The good ones pull back toward small steps anyway — write, run, read, fix — because a 200-line blob that fails gives the model almost nothing to reason about, whereas a 20-line change that fails points right at the problem. The tension between "big chunks are cheaper per round-trip" and "small chunks are debuggable" is the central rhythm of agentic coding.

## The crux: turning intent into a changed file

Here is the most under-appreciated micro-mechanic in the whole stack: how does the model's *intention* to change code become an actually-changed file on disk? This step — call it **output acceptance** — is where coding agents most visibly differ, and where they most often fail. The five harnesses we read take three distinct approaches:

- **Exact string replacement.** The model supplies the old text and the new text; the tool finds the old verbatim and swaps it. This is what Claude Code's `Edit` and Pi's `edit` do. It's surgical and the resulting diff is trivially reviewable, but it's brittle: if the model's remembered snippet is off by a space, the match fails. Pi softens this with a *fuzzy fallback* that normalises smart quotes, Unicode dashes and trailing whitespace before giving up.
- **A patch grammar.** The model emits a structured diff in a format it was trained on — Codex's `*** Begin Patch … *** End Patch` `apply_patch` syntax. This handles multi-hunk edits cleanly and reads like a real patch, at the cost of the model having to produce exactly-valid patch syntax.
- **Self-correcting replace.** Gemini CLI's `replace` tool, on a failed match, fires a follow-up model call to *repair its own edit* rather than just erroring — trading tokens for a higher landing rate.

A failed edit is the single most common micro-failure in agentic coding, and each of those designs is one harness's answer to it. It's also exactly where your "native programmer versus scripter" observation enters — but that deserves its own section below.

## Calling the tools: compile, run, build, spawn

Once a file is changed, the agent needs hands to *do* something with it, and those hands are a single tool: shell execution (`Bash` in Claude Code and Pi, `exec`/unified-exec in Codex, a PTY shell in Gemini and Kimi Code). Through that one door it compiles, runs the test suite, starts a build, installs a dependency, spawns a server.

The interesting wrinkles are all about state and blindness:

- **Statefulness.** A naive `subprocess.run` can't hold a REPL open or answer an interactive prompt. Codex's "unified exec" gives the model a persistent PTY it can write stdin to; the mature shells track background processes so the agent can start a dev server and keep working while it runs.
- **Truncation.** Every harness caps tool output (Kimi Code's read tool stops at 1,000 lines or 100 KB; Pi spills overflow to a temp file). Without it, one chatty build destroys the context budget.
- **The blind spot, literally.** The agent can `npm run dev` and start a web UI — but it cannot *look* at the page. So it `curl`s the endpoint, greps the logs, or, increasingly, drives a headless browser and feeds itself a screenshot through a vision model. Watching an agent start a server it fundamentally cannot see is the clearest reminder of the text-keyhole problem.

## Tests and verification

Because the agent can't eyeball correctness, **running things is its substitute for seeing** — which makes tests and verification not a virtue but a necessity.

On creating tests, agents are comfortable both ways: writing the test first and coding to green (some skills enforce test-first explicitly), or writing tests after to lock in behaviour. Their real edge is that they are tireless about *running* the full suite, the linter and the type-checker on every change — the chores humans skip when tired.

Their matching weakness is that a language model will cheerfully announce "done" without having run anything. This is why mature harnesses and skills bolt on a **verification discipline**: re-read the file you just edited, run the build, run the tests, and only then claim success — evidence before assertion. The agent's superpower (re-run the world twenty times without complaint) and its failure mode (declare victory unverified) are two sides of the same coin, and verification gates exist to keep it on the right side.

## Debugging: the widest gap of all

Now the part of the inner loop that translates worst. Here is the canonical human bug hunt, in \*nix muscle memory:

1. `grep` the error string across the tree.
2. Open the file at the offending line, read the surrounding context.
3. Set a **breakpoint**.
4. Run, and when execution pauses, **step** through it — step over, step into, step out — watching variables change.
5. Catch the moment the state goes wrong, and you've found the cause.

An agent does steps 1 and 2 almost exactly as a human does — grep the error, read the context around the line. Then the methods fork hard, because **step 3 is mostly unavailable to it.** Agents rarely drive an interactive debugger; they can't easily sit at a paused frame and inspect live state. (Some can speak the Debug Adapter Protocol through an MCP server, but it's the exception, not the daily path.) So instead of stepping, the agent debugs the way we all did before debuggers existed, just faster: it inserts `print`/log statements, re-runs, reads the output, reasons about it, removes them. Printf debugging at machine speed.

The deep difference is one of *substrate*. A human debugs a **live, paused, inspectable** program — you can poke it while it's frozen mid-thought. An agent debugs a **dead transcript** of what the program already emitted. It loses the ability to interrogate a running frame, and it compensates with two things humans can't match: it will happily re-run the failing case fifty times with different instrumentation, and it can hold the entire call path in context at once rather than stepping through it line by line. Slower per insight, but relentless and wide.

## The observation: native programmer vs. scripter

Which brings us to the thing you actually notice after watching these agents work for a while: **they have different programming temperaments, and Claude Code behaves the most like a native \*nix programmer.**

By "native" we mean it reaches, by default, for the same surgical tools a seasoned terminal user does — `grep` to locate, a targeted read of the exact line range, a precise string-replace edit, the right small command for the job — and it leaves behind a clean, reviewable trail of minimal diffs. Watching it is like watching someone who has `grep`, `sed`, `cut`, `head` and `tail` in their fingers and uses them with intent.

The contrast: when faced with a bulk change — rename this symbol across forty files, rewrite every import — some agents (Codex and Gemini among them, in our observation) are more inclined to **write a throwaway script to do it**: a `python3 -c "..."`, a `perl -pi -e`, a `sed -i` one-liner run through the shell, rather than a series of surgical edits. It works, and for genuinely mechanical mass-edits it's even the *right* call. But it has a cost: a thrown-away script's effect is harder to review than a diff — you have to reconstruct what it did — and a slightly wrong regex can silently mangle forty files at once, where a string-replace edit simply fails loudly on the one it couldn't match.

Two honest caveats, because this is the part most easily overstated. First, it's a *tendency*, not a wall: every one of these agents can use native tools, and every one can write a script; the difference is which it reaches for first, and that's shaped as much by the model's temperament as by the harness's tool design (a patch-grammar editor nudges toward batch thinking; a string-replace editor nudges toward surgical thinking). Second, these behaviours drift from model version to model version, so any snapshot like this one has a short shelf life. The durable point isn't "Claude Code wins the bug hunt" — it's that **there is a real, observable spectrum between surgical and scripted code-editing, and it changes how reviewable and how safe the agent's work is.** When you pick an agent, that temperament is something you can feel within an hour, and it matters more to the day-to-day than any benchmark.

There's a fitting irony to end on: this post about how agents wield `grep` and surgical edits was itself drafted by one, reaching for exactly those tools as it went. The microscope, it turns out, works on the thing holding it.

## Where micro meets macro

Across five posts the same theme keeps surfacing from different distances. The harness (macro) decides which tools exist and how their results come back; the model (micro) decides which tool to reach for and how big a bite to take. A great coding experience is when those align — a native-feeling toolset in the harness, and a model with the judgement to be surgical when a scalpel is called for and to script when a bulldozer is. The model sets the ceiling; the harness and the editing temperament decide how close to it you actually get to live — one edit, one test, one bug at a time.

*This is a micro-level companion to our four-part series on AI agent architecture. All six harnesses behind these observations are open source; the best way to see the difference between a surgical edit and a thrown-away script is to watch two agents solve the same task and read what each left behind.*

18d6cc6e657efd5cb01c1d637f0c5ba834370d18752e1843bccfa2b53ad6f7fe

A micro-level look at how AI coding agents write, run, test, verify and debug code: the edit loop, output acceptance, the exec tool, and why some agents feel like native command-line programmers while others feel like scripters.

The first four posts in this series took the telescope view of AI agents. This one takes the microscope: how a coding agent turns an instruction into a working, tested change — how it edits, compiles, runs, tests and debugs — and why Claude Code behaves like a native *nix programmer while others reach for a throwaway script.

Tag: Debugging

Tag: Debugging

The Connection Reset That Wasn't: Hunting a Protobuf Bug in IBKR's TWS API

Debugging a Fake 410: yt-dlp, curl_cffi, and a Proxy That Lied

Debugging RDP 0x609: The Protocol Bit a Proxy Should Never Touch

How a Coding Agent Actually Writes Code