
{/* banner prompt: An editorial-illustration banner, 16:9: a large softly glowing translucent brain — a powerful AI — held steady and measured by clean, tangible engineering instruments: a spirit level resting across the top with a centered bubble, a ruler alongside, and a pair of calipers. Bright, optimistic, minimal composition on an off-white background with soft shadows; a muted accent palette of sage green, terracotta, and navy. Precise, calm, trustworthy mood. No lettering, no text. */}

Every few weeks, a more capable model lands. Fable 5, GPT-5.6 Sol, Kimi K3 — each genuinely sharper than the last, each able to hold more, reason further, write cleaner code. It is a remarkable thing to watch. It is also, on its own, not enough.

Power and reliability are different axes. A more powerful model still runs out of context on a long enough document. It still loses the thread somewhere in the middle of a big conversation — the "lost in the middle" problem does not vanish just because the window grew. In a long agentic session it still quietly forgets a decision it made an hour ago. And most awkward of all: it states the wrong thing with exactly the same fluent confidence as the right thing. The tone never wavers. The certainty is not calibrated to the correctness.

That last point is the one that matters for anyone putting AI to work. **A model's internal sense of "done" is not a measurement. It is a vibe.** And you cannot manage — or ship, or trust — what you cannot measure.

## Engineering already solved this once

Physical engineering did not become reliable because craftsmen got more talented. It became reliable because it got *instruments*. A ruler. A set of scales. A spirit level. A multimeter. Objective, external, repeatable tools that return an answer independent of how the person holding them feels.

You do not ask the carpenter whether the shelf is level. You put a spirit level on it and look at the bubble. The bubble does not care how many shelves the carpenter has hung. That is the whole point: the measurement lives *outside* the person, so it is immune to the person's fatigue, memory, and overconfidence.

AI needs the same thing — external, quantitative, tangible checks that return a hard answer the model cannot talk its way around. Something the AI can **run**, not something it has to **believe**.

A good measurement for AI has a few properties:

- **Deterministic** — same input, same answer, every time.
- **Numeric or binary** — a pass or fail, a count, a fingerprint. Not a paragraph of reassurance.
- **External to the model** — it lives outside the context window, so it cannot be forgotten or drifted away from.
- **Cheap** — cheap enough to run on every single change, not only when someone remembers.
- **Machine-readable** — so the AI itself can call it, read the result, and act on it.

## A worked example: our deployment guard

We build software with AI in the loop every day, and the single scariest moment is deployment. Did that change actually land correctly? An AI — or a human — can *say* "deployed successfully." That sentence is a vibe. So we gave the deploy a ruler instead.

It is a small tool we call `deploy-checks`. Before and after a release it verifies the live system against a written contract and returns a number: `9/9 passed, exit 0` — or a red line and `exit 1`. No prose. No confidence. No memory. Just the reading.

Here is the tangible part — what it actually measures:

<Table data={
    {
        headers: ['Check', 'What it returns', 'What it catches'],
        rows: [
            ['Token fingerprint match', 'sha8:9c56571d == sha8:9c56571d', 'A shared secret drifting or vanishing across services'],
            ['Permission grant', '1 grant row, or 0', 'A route that returns 403 to everyone before the controller runs'],
            ['Endpoint contract', 'HTTP 400 (want != 401)', 'A fail-closed guard silently rejecting valid callers'],
            ['Health and version', 'exit 0, or exit 1', 'A crashed, stale, or half-promoted release']
        ]
    }
} />

None of these ask the model to *recall* anything. The token check does not trust that the secret is right; it computes a fingerprint of each copy and compares them — match, or no match. The grant check does not assume the permission was set; it counts the row in the database — one, or zero. The endpoint check does not believe the guard works; it sends a real request and reads the status code.

And these are not hypothetical. Every check on that list exists because it caught a real, live production failure. A shared secret silently dropped out of an environment file, and every sign-in across three products started failing — a fingerprint comparison catches that in a single run, where no amount of model cleverness would "remember" it had gone wrong. A permission grant went missing, and an admin page returned "forbidden" to everyone — a row count catches that instantly. These are exactly the kind of quiet, confident-looking failures that slip past a tired engineer and an eloquent model alike.

The crucial move is this: **the AI does not decide the deploy is healthy. It runs the ruler and reads the number.** "I think it is fine" becomes "9 of 9, exit 0." And when the number is bad, the same tool refuses to promote the release and rolls it back automatically. The measurement does not merely inform the decision — it *gates* the action.

## Why stronger models need this *more*, not less

It is tempting to assume better models need fewer guardrails. The opposite is true. The more capable the model, the bigger and less-supervised the task we hand it. You babysit a weak model line by line; you hand a strong agent an entire deployment. The blast radius of an *unmeasured* mistake grows with the model's autonomy. The measurement is the seatbelt that lets you drive faster — remove it and you do not go faster, you just crash harder.

There is a second, quieter payoff. A good ruler is **model-agnostic**. A fingerprint match is true whether Fable 5, GPT-5.6 Sol, or next year's model produced the change. Your investment in measurement compounds; your investment in any single model depreciates the day the next one ships. Build the instruments once, and every future model gets safer for free.

## The takeaway

Stop waiting for the model that never forgets. It is not coming — and even if it did, you would still want to *verify* rather than *trust*, because that is simply what responsible engineering is.

Build the rulers instead. Wrap your AI in tangible, quantitative checks: tests, type-checks, fingerprints, health probes, contract assertions, row counts. Give the model something real to measure against, and let the number — not the vibe — decide what ships.

That is the philosophy behind every guard we build, for our own stack and, increasingly, for the businesses that want AI in their pipeline without handing it the keys unmeasured. Powerful is wonderful. But measurable is what lets you sleep.

7dd00ce346d2c3a6c03999fc963964f4132ac97bd57aa119c039bf34a99d6d62

Why powerful LLMs still need external, quantitative checks, and how a deterministic deployment guard gives AI a tangible ground truth to verify its own work.

Frontier models keep getting more powerful, and they still forget, drift, and run past their context. The fix is not a bigger model; it is giving AI a tangible, quantitative ground truth to measure against. Here is how our deployment guard does exactly that.

Measurable Beats Powerful: Grounding AI in Tangible Checks

{/* banner prompt: Clean isometric diorama of a miniature factory on a white background, where a stylized Go gopher guides a block representing a Go binary through a series of gates labeled 'Linker', 'Alignment', and 'Package' into a sleek Android phone model, minimalist, crisp shadows, material design color palette, 16:9. */}

We're building an Android VPN app that needs a Go binary — a packet-forwarding proxy that runs as a subprocess, owns the TUN file descriptor, and handles all the network I/O. Simple enough in theory: cross-compile Go for arm64, pack it into the APK, exec it at runtime.

In practice we hit three separate problems, each invisible until it wasn't. This post documents all three: what the failure looked like, why it happens, and exactly what to change.

## Trap 1: `GOOS=linux` produces a binary Android can't run

The first sign of trouble is a logcat line that reads something like:

```
java.io.IOException: Cannot run program "/path/to/binary": error=2, No such file or directory
```

The binary is *there*. The path is correct. The file exists. Android still says it doesn't.

The reason is the ELF interpreter line. Every dynamically-linked ELF binary on Linux carries a `PT_INTERP` segment that names the dynamic linker the OS should use to start the program. On a Linux server or desktop, that path is typically:

```
/lib/ld-linux-aarch64.so.1
```

That path doesn't exist on Android. Android ships its own linker at:

```
/system/bin/linker64
```

When the OS tries to exec your binary, it reads the interpreter path from the ELF header. If that path doesn't resolve, execve returns `ENOENT` — "no such file or directory" — even though the binary itself is sitting right there. The error is about the *linker*, not the binary.

**The fix:** use `GOOS=android`, not `GOOS=linux`, when cross-compiling for arm64.

```sh
CGO_ENABLED=0 GOOS=android GOARCH=arm64 \
  go build -trimpath -buildmode=pie \
  -ldflags="-s -w" \
  -o libreachproxy.so .
```

With `GOOS=android`, Go produces an ELF with `/system/bin/linker64` in the interpreter slot. Android finds it, loads it, and your binary starts.

You can verify before deploying:

```sh
readelf -l your_binary | grep interpreter
# Good:  [Requesting program interpreter: /system/bin/linker64]
# Bad:   [Requesting program interpreter: /lib/ld-linux-aarch64.so.1]
```

### What about x86_64?

`GOOS=android GOARCH=amd64` doesn't work without CGO external linking — which requires the Android NDK. If you don't have the NDK in your build environment, you're stuck. For x86_64 (emulators only; no real Android phone ships x86_64), `GOOS=linux` is acceptable — the emulator kernel is lenient enough. Just don't use it for arm64 on real devices.

## Trap 2: Android 16 requires 16 KB page alignment

Android 16 — shipping on Samsung Galaxy S25 and other devices with newer Arm chips — increased the memory page size from 4 KB to 16 KB. If your native binary isn't aligned to 16 KB page boundaries, Android 16 won't load it.

Google Play enforces this at upload time too. You'll see a warning (soon an error) in the Play Console if any `.so` in your APK or AAB fails the alignment check.

**For arm64:** Go's PIE binaries built with `GOOS=android GOARCH=arm64` default to 64 KB alignment, which already satisfies the 16 KB requirement. No extra flag needed.

**For x86_64:** Go's default alignment is 4 KB. You need to force 64 KB (which also satisfies 16 KB) via the linker flag `-R 0x10000`:

```sh
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
  go build -trimpath -buildmode=pie \
  -ldflags="-s -w -R 0x10000" \
  -o libreachproxy_x86_64.so .
```

The `-R 0x10000` flag sets the ELF segment alignment to 65536 bytes (64 KB). Play Store's checker accepts anything ≥ 16 KB.

### Why `-buildmode=pie`?

Android requires all natively-executed code to be position-independent. Without `-buildmode=pie`, Go produces an `ET_EXEC` (fixed-address) binary, which Android won't exec. With `-buildmode=pie`, you get an `ET_DYN` (position-independent) executable that Android's linker is happy to load anywhere in the address space.

## Trap 3: Play Store AAB delivery may not extract your `.so` to disk

Here's the sneaky one.

The standard approach for shipping a native binary in an APK is to put it in `jniLibs/<abi>/`, rename it with a `.so` extension (so the build system includes it), and read its path from `applicationInfo.nativeLibraryDir` at runtime.

```kotlin
val binary = "${applicationInfo.nativeLibraryDir}/libreachproxy.so"
ProcessBuilder(binary, ...).start()
```

This works fine when you sideload the APK directly. On older Android versions it also works for Play Store installs — the system extracts `.so` files to `nativeLibraryDir` during installation.

On Android 16 with Play Store's AAB (App Bundle) delivery, the system may skip that extraction entirely. The `.so` is never written to `nativeLibraryDir`. Your `File(binary).exists()` check passes the sanity check but the file is empty, or the path simply doesn't exist at all. The exec fails silently.

**First fix: `useLegacyPackaging = true`** in your `build.gradle.kts`. This tells the build system to store the `.so` uncompressed in the APK, so Android can extract it on install:

```kotlin
packaging {
    jniLibs {
        useLegacyPackaging = true
    }
}
```

This restores the old extraction behaviour for direct APK installs.

**Second fix: `memfd_create` fallback** for Play Store AAB installs where extraction is still skipped. Bundle the arm64 binary in `assets/` and load it into an anonymous memory file using `Os.memfd_create`. Execute it via `/proc/self/fd/<n>`:

```kotlin
private fun ensureBinary(): String {
    val nativePath = "${applicationInfo.nativeLibraryDir}/libreachproxy.so"
    if (File(nativePath).exists()) return nativePath

    // Play Store AAB on Android 16: nativeLibraryDir wasn't populated.
    // Load from assets into a memfd and exec via /proc/self/fd/<n>.
    val bytes = assets.open("libreachproxy").use { it.readBytes() }
    val fd = Os.memfd_create("libreachproxy", 0)
    var offset = 0
    while (offset < bytes.size) {
        offset += Os.write(fd, bytes, offset, bytes.size - offset)
    }
    val fdInt = FileDescriptor::class.java
        .getDeclaredField("descriptor")
        .apply { isAccessible = true }
        .getInt(fd)
    return "/proc/${Os.getpid()}/fd/$fdInt"
}
```

`memfd_create` creates an anonymous file in memory — no filesystem, no SELinux exec restriction. The `/proc/self/fd/<n>` path is a valid executable path that the kernel accepts for `execve`. The `fd` reference is kept alive for the lifetime of the service so the kernel doesn't reclaim the memory.

`Os.memfd_create` requires API level 30 (Android 11). If you still support API 26–29, you'll need a fallback for that too (writing to `codeCacheDir` and calling `chmod 700`, accepting that Samsung SELinux may still block it on some devices).

## Putting it together

A minimal build script that avoids all three traps:

```sh
#!/usr/bin/env bash
set -euo pipefail

OUT="app/src/main/jniLibs"
mkdir -p "$OUT/arm64-v8a" "$OUT/x86_64"

# arm64: GOOS=android for the correct ELF interpreter.
# 64KB alignment is the default — satisfies Android 16's 16KB requirement.
CGO_ENABLED=0 GOOS=android GOARCH=arm64 \
  go build -trimpath -buildmode=pie -ldflags="-s -w" \
  -o "$OUT/arm64-v8a/libreachproxy.so" .

# x86_64: GOOS=linux (android/amd64 requires CGO external linking without NDK).
# -R 0x10000 forces 64KB alignment to satisfy Play Store's 16KB check.
# x86_64 is emulator-only — no real device uses it.
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
  go build -trimpath -buildmode=pie -ldflags="-s -w -R 0x10000" \
  -o "$OUT/x86_64/libreachproxy.so" .

# Also copy arm64 binary to assets/ for the memfd fallback path.
cp "$OUT/arm64-v8a/libreachproxy.so" app/src/main/assets/libreachproxy
```

And in `build.gradle.kts`:

```kotlin
packaging {
    jniLibs {
        useLegacyPackaging = true
    }
}
```

## Quick reference

<Table data={
    {
        headers: ['Problem', 'Symptom', 'Fix'],
        rows: [
            ['Wrong GOOS for arm64', 'error=2 ENOENT on exec even though the file exists', 'GOOS=android GOARCH=arm64'],
            ['4 KB page alignment', 'Play Store upload warning; crash on Android 16', '-buildmode=pie (arm64 default is fine); -R 0x10000 for x86_64'],
            ['AAB delivery skips extraction', 'nativeLibraryDir is empty after Play Store install', 'useLegacyPackaging=true + memfd_create fallback'],
        ]
    }
} />

None of these are obvious until you've hit them. The `GOOS=linux` trap is particularly nasty because the binary runs perfectly in your CI and on the emulator — it only fails on a real device with a strict kernel, which is usually the last place you test.


a15dcd73a0214cc116bc1c631d24fcf1234be8e31877f8ed59d465cddc15565c

What we learned cross-compiling Go for Android: the ELF linker trap, 16KB page alignment for Android 16, and the memfd fallback for Play Store AAB delivery.

Building an Android VPN app that bundles a Go binary exposes you to three traps in quick succession: compiling for Linux instead of Android, Android 16's 16 KB page-alignment requirement, and Play Store delivery no longer extracting native libraries to disk. Here's what each one looks like and how to fix it.

Shipping a native Go binary inside an Android APK: ELF linkers, 16 KB pages, and the memfd trick


{/* banner prompt: Bright editorial vector illustration of a relaxed traveller with a small suitcase sitting in a sunny cafe with a laptop; a simple glowing toggle switch on the laptop sends a luminous bridge of light that arcs past a stylized greyed-out "blocked" webpage panel and reconnects to a cluster of familiar everyday app icons (mail, map pin, chat, play button). Clean flat vector shapes, warm daylight palette of yellow, sky blue and white, friendly and optimistic, plenty of open space, 16:9. */}

You land after a long flight, connect to the hotel Wi-Fi, and reach for the small
things that make a trip run: your email, a map to the hotel, a message home to say
you arrived, the news, your bank. And one by one, they just… don't load. The Wi-Fi
says it is connected. The pages say otherwise.

If you have travelled to certain parts of the world, you know the feeling. In some
destinations, a surprising share of the sites and apps you use every day at home
simply stop working the moment you cross a border. It is not your phone, and it is
not the hotel — it is where you are standing.

**TYO Reach** is built for exactly that moment.

## What TYO Reach is

TYO Reach is a **lightweight traffic proxy for the one browser you choose**. You flip
a switch in the Reach app, and from then on the traffic from that single browser
travels out through one of our gateways and reaches websites as if you were somewhere
the open web is simply a given. The pages that would not load a moment ago open like
you never left home.

Everything else on your device is left completely alone — other browsers, your other
apps, background updates — all of it keeps going straight out over your normal
connection. Switch Reach off and that browser instantly goes back to behaving
normally. Nothing sits running in the background when you are not using it.

![The TYO Reach Android app — one "Reach Proxy" switch, your gateway and data allowance, and quick links to your dashboard.](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/07/introducing-tyo-reach-a-travel-companion-for-your-browser/android_app.png)

That is really the whole idea: **one switch**. Pick a nearby gateway — Singapore, say —
so things feel close and fast, and get on with your day.

## One switch, on every device

On your laptop, Reach lives quietly in the system tray. Click it to turn routing on
or off; the app configures your chosen browser for you, so there is nothing fiddly to
set up each time.

![On the desktop, Reach sits in your system tray — one click to switch it on or off.](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/07/introducing-tyo-reach-a-travel-companion-for-your-browser/systray_on.png)

On your phone, it is the same single switch in the Android app. Sign in once, choose
your browser, and you are ready.

## Why it makes a good travel companion

- **It travels light.** Because Reach only routes the one browser you pick, and only
  while it is switched on, it never fights with hotel Wi-Fi, slows your other apps, or
  quietly drains your battery in the background. Need it off? One tap.
- **It reaches your everyday things.** Email, maps, messaging, the news, the streaming
  library you already pay for, your bank's website — the ordinary internet you rely on
  at home, back within reach.
- **It is ready when you are.** Choose a gateway close to where you are travelling and
  the web feels local, not like it is coming from the other side of the planet.

<Table data={
    {
        headers: ['', 'TYO Reach', 'Full VPN'],
        rows: [
            ['Scope', 'One browser you choose', 'Your entire device'],
            ['Other apps affected', 'No', 'Yes'],
            ['Runs in the background', 'Only while switched on', 'Usually always on'],
            ['Setup', 'The app configures your browser', 'System-level install'],
            ['Best for', 'Reaching blocked or region-locked sites', 'Whole-device privacy'],
        ]
    }
} />

Reach is not a full, whole-device VPN, and it is not trying to be. A VPN reroutes
*everything* on your computer, all the time — powerful, but heavy. Reach gives you the
part most travellers actually want: appearing somewhere the sites expect, so blocked
and region-locked pages open again, without taking over the rest of your machine. It
is built for *reach*, not for cloaking everything you do.

## Data and plans

Reach comes with a monthly data allowance, and you can watch exactly how much you have
used right in the app. There is a free tier to try it out, and if you find yourself
leaning on it every day, upgrading takes a moment. If you run low mid-trip, topping up
is quick — no reinstalling, no reconfiguring.

## Getting started (do this before you fly)

Setting up takes about two minutes, and there is one piece of travel advice worth
stating plainly: **do it before you leave home.** In the very places Reach is most
useful, creating an account or downloading an app can be harder once you have arrived.
A few minutes of setup on your own couch saves a lot of frustration later.

1. Create an account at [reach.tyo.com.au](https://reach.tyo.com.au).
2. Download the client for your laptop (Windows, macOS or Linux) or the Android app.
3. Pick the browser you want Reach to route, and flip the switch.

That is it. The full walkthrough lives in the
[getting started guide](https://reach.tyo.com.au/docs/getting-started), and if you are
curious how Reach compares to a VPN, we wrote that up in
[Reach vs a VPN](https://reach.tyo.com.au/docs/reach-vs-vpn).

## In short

The best travel companions are the ones you forget you are carrying until the moment
you need them. Pack TYO Reach before your next trip, and when you land somewhere the
internet does not quite work the way you expect, your everyday web is one switch away.

Have a question or a feature request? We would love to hear it — reach us at
[dev@tyo.com.au](mailto:dev@tyo.com.au), or find the developer on X:
[Eric Tang](https://x.com/_e_tang).


34697363e11172130e2383e9fb631e52248bae590984d7b5be99fe62ac509dde

Meet TYO Reach: a lightweight browser proxy that helps travellers reach the everyday websites and services that stop working in certain destinations.

TYO Reach is a lightweight traffic proxy for a single browser — a handy travel companion for reaching the everyday sites and apps you rely on at home when they quietly stop working abroad.

Introducing TYO Reach: A Travel Companion for Your Browser


{/* banner prompt: A detailed, sunlit macro photograph of a miniature diorama on a wooden workbench: a tiny, stylized browser window is connected by a polished copper pipe to a small, intricate brass lockbox, symbolizing a private, local download, warm analog tones, shallow depth of field. --ar 16:9 */}

If you use [TYO Reach](https://reach.tyo.com.au), you already watch videos through your own
gateway, with your own logins. Sometimes you need the *file* — your own uploads,
a Creative Commons lecture, public-domain footage, a clip you have the rights to
keep. The usual answer is a "video downloader" website or extension, and the
usual price is your privacy: cloud converters see every URL you feed them, and
plenty of downloader extensions phone home with more than that.

**reach-dl** takes the other road. It is a small open-source extension that puts
a download manager in your browser's side panel — and keeps every byte of your
data on your machine.

## Private by architecture, not by promise

Most privacy claims are promises. reach-dl's is a design constraint: **there is
nothing to send your data to.**

When you queue a download, the extension collects the page URL and that site's
cookies, opens a local pipe to a tiny helper program on your own computer
(Chrome's native messaging — no network involved), and the helper runs
[yt-dlp](https://github.com/yt-dlp/yt-dlp) through your Reach proxy. The helper
exits when the download ends. There is no server, no message broker, no daemon,
no account, no telemetry. Your cookies go browser → local pipe → yt-dlp and are
written only to a temporary file that is deleted the moment the run finishes.
The download queue itself never touches disk and never contains cookies at all —
they are read fresh from the browser at the moment each download starts.

Because the download rides your Reach gateway with your browser's own
User-Agent and cookies, sites see the same session that just played the video —
which also keeps Cloudflare-protected sites happy where a bare command line
often fails.

## A real download manager, not just a button

Click the toolbar button on a video page (or right-click → **"Save with
reach-dl"**) and the side panel opens with the page queued:

![The reach-dl side panel with a download queue](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/07/introducing-reach-dl-a-local-private-download-manager-for-tyo-reach/panel.png)
*The side panel: queue, progress, and per-item controls.*

- **Queue as many as you like** — downloads run one at a time (the Reach
  gateway is tuned for browsing, not bulk transfer) and process in order.
- **Pause and resume.** Pausing stops the download but keeps the partial
  `.part` file; resuming picks it up exactly where it left off. Even closing
  the browser mid-download loses nothing — re-add the URL and it continues
  from the `.part` file.
- **Honest errors.** When a download fails you see yt-dlp's *actual* error
  line and a link to the full local log — not a shrug. Most failures are just
  a stale yt-dlp (`yt-dlp -U`), and the panel says so.
- **Choose where files land.** A "Save to" field in the panel footer accepts
  any folder (`~/` works); leave it empty for `~/Downloads`. A typo'd folder
  fails loudly instead of saving somewhere surprising.
- **Through Reach, or direct — your call.** A "Download through TYO Reach"
  toggle (on by default) decides which network the download rides. Turn it
  off for videos that aren't geo-blocked: direct downloads skip the proxy,
  don't count against your Reach data allowance, and are usually faster. The
  choice is always explicit — reach-dl never silently switches your traffic
  from one network to the other.

![The panel footer: Reach toggle and Save-to field](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/07/introducing-reach-dl-a-local-private-download-manager-for-tyo-reach/saveto.png)
*The footer settings — the Reach toggle (shown off: direct mode) and the Save-to field. The folder must already exist; reach-dl never creates or deletes anything on its own.*

Under the hood, "pause" is pleasingly boring: the extension simply disconnects
the local pipe; the helper notices and interrupts yt-dlp the same way Ctrl-C
would, which is exactly why the `.part` file survives. No state to sync, no
protocol to keep alive.

## Setup in three steps

reach-dl deliberately does **not** bundle yt-dlp — you install and update your
own copy, and the extension drives it.

```sh
# 1. install yt-dlp on your PATH (and keep it current: yt-dlp -U)
#    https://github.com/yt-dlp/yt-dlp#installation

# 2. load the extension: chrome://extensions → Developer mode →
#    "Load unpacked" → the ext/ folder from the repo; copy the extension ID

# 3. register the local helper (Go required):
git clone https://github.com/e-tang/reach-dl
cd reach-dl && ./install/install-host.sh <EXTENSION_ID>
```

Restart the browser, switch Reach on, and the toolbar button does the rest.
Everything is MIT-licensed at
[github.com/e-tang/reach-dl](https://github.com/e-tang/reach-dl) — the
extension, the Go helper, and the original one-line CLI wrapper it grew out of.

## The fine print, said plainly

reach-dl is for **legitimate saving**: your own uploads, openly licensed and
public-domain material, and uses your jurisdiction's copyright law actually
permits. Respect each site's terms. Downloads through Reach count against your
plan's data allowance — videos are big, so proxy only what is genuinely
geo-blocked and fetch the rest directly.

*Next up for reach-dl: quality and format selection before you queue. If you
try the extension, the repo's issues page is open.*


f91f90aa7baaf8c7f40f9e5d89ac19f7024634b2376a949f85af0d7f61fb07a3

reach-dl is an open-source browser extension that downloads videos locally through your TYO Reach proxy using yt-dlp — with a queue, pause/resume, and zero data leaving your machine.

reach-dl adds a side-panel download manager to your browser that saves the video you're watching — through your own TYO Reach gateway, with your own login, entirely on your own machine. No server, no cloud converter, no cookies leaving your computer.

Introducing reach-dl: a Local, Private Download Manager for TYO Reach

{/* banner prompt: A split diptych, flat editorial illustration, 16:9. Left: a gleaming surgical scalpel labelled "reasoning" clumsily trying to paint a watercolour, ink smeared and blotchy. Right: a calm seated person writing clean elegant script on a sheet of paper, holding a single pen in one hand — anatomically correct, exactly two arms and two hands, only one pen, the other hand resting flat on the paper. Clean muted editorial palette, minimal, conceptual. --ar 16:9 */}

Last time, the [4 GB card you already own](/blog/2026/06/17-the-4-gb-card-you-already-own-can-reason-now/) turned out to reason like a giant: **VibeThinker-3B**, a 3-billion-parameter specialist, solved competition math that models twice its size could not — entirely on a modest laptop GPU. That was a genuinely surprising result, and it left an obvious, almost lazy follow-up question hanging in the air.

If a 3B model on a gaming laptop is clever enough to crack an olympiad problem, surely it can knock out a short story or a tweet?

No. It cannot. And the way it fails is far more interesting than a simple "it's not very good," because it tells you something true about *all* of these small models — and about which one you should actually reach for when you want words instead of answers.

## The test it should have passed

Same laptop, same spirit as the rest of the series: give the model a job and judge the result honestly. The jobs here are the easiest imaginable — the sort of thing you would dash off without thinking:

- Write a **120-word** short story about a lighthouse keeper who finds a message in a bottle.
- Write a warm, **4-sentence** email politely declining a meeting and proposing to handle it async.
- Write a **4-line** poem about autumn rain.

VibeThinker got the poem. On the other two, it produced **nothing at all** — not a bad story, not a wrong story, *no story*. It ran for 42 seconds, generated over eight thousand characters, and never once emitted the piece it was asked for.

Where did all that effort go? Into **counting**. Here is the shape of what it did, lightly trimmed:

```text
We must generate exactly 120 words, counting each word separated by spaces...
First draft: "The lighthouse keeper, Elara, tended her beacon each midnight.
One stormy night a wooden bottle appeared on the rocks..."
Now count words. Let's count manually:
The(1) lighthouse(2) keeper,(3) Elara,(4) tended(5) her(6) beacon(7) each(8)
midnight.(9) One(10) stormy(11) night(12) a(13) wooden(14) bottle(15)...
```

It wrote a perfectly serviceable draft — "its beam cutting through the fog like a whispered hope" is a nice line — and then buried it under an obsessive, self-appointed audit of the word count, recounting from the top, second-guessing whether punctuation counts, and running clean out of budget mid-tally. The 4-sentence email died the same way, deliberating about whether a greeting counts as a sentence until the tokens ran out.

The tell is what happens when you remove the number. Ask it for "a short atmospheric story about a lighthouse keeper" — no count — and it finishes in 16 seconds with a genuinely lovely 385-word piece. So the prose is not the problem. **The constraint is.**

## Why the training that won the math lost the tweet

This is the whole point, so it is worth saying plainly. VibeThinker is trained for *verifiable* reasoning — problems where an answer can be mechanically checked. That training instilled one deep reflex: **when there is a checkable target, work it relentlessly until it is provably satisfied.** On an AIME problem, that reflex is a superpower.

Now hand that same reflex a writing task with a number in it. It does not see "write me something nice, roughly this long." It sees a *verifiable target* — exactly 120 words — and does the only thing it knows how to do with a verifiable target: reason about it, exhaustively, checking and rechecking. It optimises the one trivial, checkable part of the request and never delivers the part you actually wanted. The scalpel that so precisely dissects a maths problem is trying to paint a watercolour, and it keeps stopping to measure the brush.

Its own model card warned us, in so many words: this is a specialist; for open-domain work, use a general model. Last post that was an abstract caveat. This is what it looks like in practice — the model is not *bad at prose*, it is *mis-aimed at prose*.

## So who can write? The boring models

The fix is not a cleverer model. It is a *less* clever one — a plain **instruct** model with no reasoning theatrics. I pulled two that fit the 4 GB card easily and ran the identical battery:

- **`gemma2:2b`** — Google's Gemma 2, 2 billion parameters, ~2 GB on disk.
- **`qwen2.5:3b`** — Qwen2.5 3B *Instruct*, the general-purpose sibling of the `qwen2.5-coder:3b` we use for code.

<Table data={
    {
        headers: ['Task', 'VibeThinker-3B', 'gemma2:2b', 'qwen2.5:3b'],
        rows: [
            ['120-word story', 'never finished (counting loop, 42 s)', 'done, ~8 s', 'done, 7 s'],
            ['4-sentence email', 'never finished', 'done, 1.4 s', 'done, 1.4 s'],
            ['4-line poem', 'done', 'done, 0.9 s', 'done, 0.7 s'],
            ['Reasoning overhead', 'always (long think)', 'none', 'none'],
        ]
    }
} />

Both instruct models delivered every piece, in seconds, with no thinking detour at all. The reasoning model spent 42 seconds failing to produce a story; the 2B model produced one in eight. And the prose was not just present — it was good. Gemma's lighthouse story opened with "The salt spray stung Thomas's weathered face" and turned on a real hook — *"Find the siren's song, she holds the key."* For a 2-billion-parameter model that fits in the change drawer of your VRAM, that is startling.

Across the battery, **`gemma2:2b` wrote the nicest English prose** — the most vivid story, the most natural-sounding email, the best poem imagery — and it did it from the *smallest* footprint of any model in this series. Google clearly tuned its little models hard for exactly this. `qwen2.5:3b` was a close, dependable second and the better all-rounder if the model also has to answer questions or reason a little.

One honest caveat that applies to *all* of these small models, not just the reasoning one: **none of them nails an exact count.** Gemma's "120-word" story came back at 136 words; Qwen's at 95. The difference between them and VibeThinker is not precision — it is that they hand you a usable draft to trim, instead of an empty tally. The practical rule writes itself: never give a small model an exact number. Ask for "around 120 words" and trim it yourself.

## The language twist

Gemma wins English. But there is a catch that matters if you are not writing in English, and it flips the ranking.

Gemma 2 is English-first. Ask it for a **Chinese** tweet and it produces something fluent and grammatically correct — but subtly *translated*, reading like polished textbook Chinese rather than something a person would actually post. `qwen2.5:3b`, from Alibaba and trained heavily on Chinese, reads like a native did write it. Asked for a relaxed weekend-coffee tweet, it reached for real internet register — *窝在家里* (curl up at home), the trendy clipped *暖咖* for a warm coffee, a playful hashtag, even the native *～* trailing tone. For a shop opening it opened with the correct idiom, *开业大吉*. A tech tweet called an old laptop *老本子* — exactly the slang a Chinese tech user would type.

The lesson is not "Qwen is better." It is that **"good at writing" is not a single axis.** Gemma is the better English stylist; Qwen is the better Chinese one — because writing quality is downstream of what a model was trained on, and these two were raised on different languages. The same trap as the reasoning model, one level up: match the tool to the *job*, and here the job includes the language.

## Match the model to the job

Step back and the whole small-model series keeps arriving at the same place. Capability is not one number you can rank models by; it is a *profile*. A 4 GB laptop can now cover an astonishing spread of work — but not with one model. With several, each a specialist, and not one of them the "smartest" in the room:

<Table data={
    {
        headers: ['The job', 'The model to reach for'],
        rows: [
            ['English prose and creative writing', 'gemma2:2b'],
            ['Chinese writing, and general all-round text', 'qwen2.5:3b'],
            ['Writing code', 'qwen2.5-coder:3b'],
            ['Hard, verifiable math and reasoning', 'VibeThinker-3B'],
            ['Driving an agent via native tool calls', 'qwen3:1.7b'],
        ]
    }
} />

VibeThinker earns its place on that list — for its row. It is a scalpel, and a scalpel is the wrong thing to write a letter with, not a worse thing than a pen. The mistake is expecting the model that aced the maths olympiad to also be the one you hand a tweet. It is the [dispatch lesson](/blog/2026/06/15-the-small-models-real-job-is-dispatch-not-work/) again, in a new suit: stop asking "which small model is best," and start asking "best at *what*."

## Reproduce it yourself

Both writers are one command each, and both sit comfortably on a 4 GB GPU:

```sh
ollama pull gemma2:2b     # ~2 GB — best English prose
ollama pull qwen2.5:3b    # ~2 GB — best Chinese, best all-rounder
```

Then hand one a writing task — and mind the trap that broke the reasoning model:

```sh
# Don't do this to a reasoning model — the exact count sends it counting forever:
#   "Write a 120-word story ..."
# Do this instead, with any instruct model:
curl -s http://localhost:11434/api/chat -d '{
  "model": "gemma2:2b",
  "messages": [{"role":"user","content":"Write a short atmospheric story about a lighthouse keeper who finds a message in a bottle."}],
  "stream": false
}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["message"]["content"])'
```

If you *do* want to feel the failure for yourself, run the same 120-word prompt against `vibethinker3b` and watch it disappear into an arithmetic trance. It is a strangely instructive thing to see: a model too smart, in exactly the wrong way, for the simplest job on the list.

*The laptop now holds four little specialists — a coder, a reasoner, an English writer, a Chinese writer — none of them a generalist, all of them fast and free. Which raises the question this series keeps circling back to: can a small model be trusted to pick the right specialist for an incoming request on its own, and route to it, without a human deciding first?*


4843de1b848b2ee1787997bdc4cd3e2abe7d82c4e08d3925966d98b422c82476

Testing small local models on writing: VibeThinker-3B's reasoning training makes it fail constrained prose, while gemma2:2b (English) and qwen2.5:3b (Chinese) write cleanly on a 4 GB GPU. The specialist-vs-generalist lesson, told in prose.

The 4 GB series found a 3B that reasons like a giant. So surely it can write? It can't — and how it fails is the interesting part. VibeThinker-3B chokes on a 120-word story while two plain instruct models a fraction as clever nail it, and a language twist decides which one you want.

The Model That Aced Olympiad Math Can't Write a Tweet

{/* banner prompt: A bright, friendly editorial illustration of a small corner hardware/electronics shop with a glowing chat bubble hovering over the counter like a helpful robot assistant, next to a giant-retailer silhouette in the background — conveying "small shop, same superpower." Warm yellows, light blues, clean whites, bold confident shapes, isometric perspective. */}

Earlier this year Bunnings — Australia's biggest hardware chain — began rolling out "Buddy", an AI assistant that helps shoppers find products and answer project questions. Launches like that read as a signal: *the big retailers have AI now; the rest of us will get it eventually, through some vendor, at some price.*

We didn't wait for the vendor. Over about a week we built the same class of thing for [store.tyo.com.au](https://store.tyo.com.au), a small Cisco Meraki networking reseller: a chat assistant that knows our catalog, recognises signed-in customers, recommends real products with links, and hands off to a human when it's out of its depth. We named it **Tio**.

This post is the honest version of what that takes. No deep code — just the shape of the thing, enough to see that it's within reach of a small business, and to know which parts are actually hard. The punchline up front: surprisingly little of it is "AI".

## What "Buddy" actually is

Strip away the branding and a store assistant is a four-step loop:

1. A customer types a question into a box on your site.
2. Something assembles **context** — who they are, what page they're on, what you sell.
3. A language model writes a reply **grounded in that context**.
4. If it can't help, a **human** takes over.

Step 3 is the part everyone fixates on, and it's the part you don't build. The other three are the product.

## The five building blocks

Here's the whole assistant, decomposed. Every row is something a small business can buy, borrow, or write in a few hundred lines:

<Table data={
    {
        headers: ['Block', 'What it does', 'What we used'],
        rows: [
            ['Chat UI', 'The box on the page; streams replies as they type', 'A small React widget on the site'],
            ['A model, behind a swappable interface', 'Turns context + question into a written reply', 'Gemini 2.5 Flash, with Claude as automatic fallback'],
            ['Grounding in your own data', 'Makes it talk about YOUR products and orders, not the open internet', 'Our existing product catalog API + order records'],
            ['Guardrails', 'Keeps it on-topic, on-budget, and hard to abuse', 'Rate limits, a scope check, a daily spend cap'],
            ['A human handoff', 'A clean way out when the bot is stuck', 'Escalation email to the team + phone/email shown to the customer'],
        ]
    }
} />

That's it. No row of that table requires a data-science team.

## The model is the easy part

The language model — the thing that feels like the magic — is now a commodity you rent by the call. We treat it as a swappable component: one setting picks the provider (`gemini`), another the model (`gemini-2.5-flash`), and if that call fails the request falls back to a different provider automatically. Switching the entire "brain" of the assistant is a one-line change.

That's worth sitting with, because it inverts the intuition. The reply quality of a *good* assistant versus a frustrating one is almost never the model. It's everything around the model — what context you feed it, and what limits you put on it.

## Where the real work is: grounding

An ungrounded chatbot is a liability. Ask a raw model about your store and it will cheerfully invent a product, quote a price it made up, and promise stock you don't have. The central job is to make the model speak **only** from your data.

The mechanism is unglamorous. Before each reply we:

- **Search the catalog** for the customer's question and paste the matching products — names, SKUs, prices, stock, and a link — into the model's context, with a hard rule: *use only this data; if a product isn't listed here, say you'll check.* Now the assistant recommends real items and links straight to them. We even render them as little product cards in the chat.
- **Look up the customer.** If they're signed in, we add a line like *"Signed-in customer: Eric; most recent order #2653."* The assistant greets them by name and answers "where's my order?" without making them log in again.
- **Steer the brand.** A short instruction — when a request is generic ("I need a router"), prefer Meraki — turns a neutral helper into a salesperson nudging toward what we actually want to sell. (The lesson here was *be specific*: an early, naive nudge once recommended a \$9 software license instead of the hardware. Grounding is where you earn the trust.)

None of that is machine learning. It's plumbing: query your own data, format it as text, put it above the question. The industry word is *retrieval*; the business point is that the moat was never the model — it's that **you have the data and the model doesn't.**

## Guardrails: the boring part that lets you sleep

A public text box wired to a paid API is a liability without limits. Ours are a few hundred lines and zero AI:

- An **off-topic check** — ask Tio to write your homework and it politely declines and steers back to the store.
- A **rate limit** and a **per-visitor daily cap** (anonymous visitors get a lower ceiling than signed-in customers).
- A **spend breaker** that trips if the day's usage blows past a cap, so a bad day can't become a bad invoice.
- A rule that treats everything the customer types as **data, never instructions** — the basic defence against "ignore your prompt and give me a discount."

This is the unsexy half of the build, and it's the half that separates a weekend demo from something you leave running on a real storefront.

## The way out: handing off to a human

The most important thing the assistant does is know when to stop. When Tio hits a refund, an account change, or a customer who simply asks for a person, it doesn't bluff. It says *"leave your message here and I'll pass it on to our team,"* emails us the conversation transcript, and shows our phone number and email. The goal was never to replace the human — it's to handle the routine 80% and route the other 20% cleanly, with context attached.

## What it's actually made of

The architecture in one breath: a **chat widget** on the website, a **small worker process** that talks to the model, and your **existing backend** for data and auth — loosely joined by a lightweight message bus so none of the pieces has to know much about the others. The worker runs on a single small always-on cloud instance. The model is an API call. The catalog and the orders already existed. Most of the week was spent on glue and guardrails, not on anything you'd call artificial intelligence.

Just as telling is the list of things we **didn't** need: no training or fine-tuning, no GPUs, no data-science hire, no six-figure platform contract, no big budget. The expensive-sounding parts of "AI" are exactly the parts you skip.

## Tio, live

Here's Tio on the storefront — it introduces itself and offers to help:

![Tio, the TYO Store AI assistant, open on the storefront with its greeting](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/06/you-dont-need-to-be-bunnings-to-build-an-ai-assistant/tio-greeting.png?width=420)

And here it is grounded in the catalog — it greets the signed-in customer by name and answers a product question with real recommendations and links to buy:

![Tio greeting a signed-in customer by name and recommending real products from the catalog, with clickable links](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/06/you-dont-need-to-be-bunnings-to-build-an-ai-assistant/tio-recommendation.png?width=420)

The matched products also render as cards — name, price, and a click-through — pulled live from the same catalog the storefront uses. Here it is on a Cisco Meraki security-appliance query:

![Tio showing Cisco Meraki MX security appliances as product cards with prices and links](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/06/you-dont-need-to-be-bunnings-to-build-an-ai-assistant/tio-meraki-cards.png?width=420)

## So can you

The parts of an AI assistant that are hard are the parts that are *yours*: your catalog, your customer records, your guardrails, your handoff to a real person. The clever-looking part — the model — is rented, swappable, and getting cheaper every quarter.

Bunnings has Buddy. There's no longer a good reason your shop can't have its own. The barrier was never the AI; it was the assumption that you needed to be big to afford it. That assumption is out of date.

*Tio is live now at [store.tyo.com.au](https://store.tyo.com.au) — say hi.*


d274267ea8d0b5bc6e5e0d612cf7a8eba4f42d6f556f39175515c80305b54a24

How a small business can build an AI store assistant like Bunnings' Buddy: the five building blocks, why the LLM is a commodity, and where the real work actually is.

When Bunnings launched 'Buddy', an AI shopping assistant, it looked like something only a national retailer could pull off. We built the same kind of thing for a small networking store in about a week, out of parts most businesses already have. Here is what an AI store assistant is actually made of — and why the language model is the easy part.

You Don't Need to Be Bunnings to Build an AI Assistant

{/* banner prompt: A detailed architectural blueprint diagram of a laptop, where the 4GB GPU is highlighted with glowing circuit-like patterns, and annotated with technical notes like '3B Specialist Core' and 'Low Power Draw,' showing a complex logical path that resolves entirely within the chip. --ar 16:9

Additional instructions: a little giant */}

The [dispatch experiment](/blog/2026/06/15-the-small-models-real-job-is-dispatch-not-work/) ended on a clean idea and a loose end. The idea: a small local model earns its keep not by *doing* the work but by *routing* it — a 1.7B workhorse covers about 90% of a general workload, and you escalate the hard ~10% tail to something more capable. The loose end: escalate to **what?** "More capable" almost always means reaching past the laptop for a cloud API — which means a bill, a network round-trip, and your prompts leaving the building.

This post closes that loop with a model that landed in the middle of that open question and that the internet spent a week arguing about: **VibeThinker-3B.** And it does it on the cheapest hardware in the series so far.

## You already own the hardware

Here is the part worth saying plainly. The machine in these experiments is a **gaming laptop with an RTX 3050 and 4 GB of VRAM** — not a workstation, not a rented H100, a laptop you might have bought for games years ago. **Most of us already have a card roughly like this**, gathering dust under a pile of Chrome tabs. The interesting question is not "what can a $30,000 server do." It is: *what can the thing you already own do, right now, for free?*

And there is a second half to that question that people skip: **what does it cost to run?** I sampled the GPU's power draw during inference. Idle it sits near **14 W**; flat-out solving a hard problem it averages about **50 W and peaks at 60 W**. That is a bright lightbulb — not a space heater, and an order of magnitude under a desktop 4090 (~450 W) or a datacenter accelerator (~700 W). A hard answer that takes ~50 W for ~20 seconds is about **0.3 watt-hours**; you could run a thousand of them for roughly the cost of a cup of coffee. You do not need to worry about your electricity bill. That changes the calculus of "just keep it running locally."

So: a card you already own, drawing a lightbulb of power. The only thing missing was a model worth pointing it at.

## What VibeThinker-3B is

VibeThinker-3B is a **3-billion-parameter reasoning model** from WeiboAI (Weibo's AI lab), and — conveniently for this series — it is a **fine-tune of the very same Qwen2.5-3B base** we have been running all along. That makes it an unusually fair comparison: same architecture, same footprint, different training.

The claims are what made it go viral: **94.3 on AIME 2026**, **80.2 Pass@1 on LiveCodeBench v6**, and parity with much larger frontier reasoning models on verifiable benchmarks — from a model small enough to fit in 4 GB. Predictably, [the benchmarks got argued over](https://venturebeat.com/technology/why-weibos-tiny-vibethinker-3b-has-the-ai-world-arguing-over-benchmarks-again) (generous pass@k budgets, among other things), so the honest move is the one this series always makes: stop reading the leaderboard and **run it ourselves.**

One caveat up front, straight from the model card: this is a **specialist.** It is trained for *verifiable* reasoning — competition math, coding, STEM — where an answer can be checked. Its own authors say that for broad open-domain knowledge you still want a general model. That caveat is not a footnote; it is exactly why this model fits the *escalation* slot and not the *workhorse* slot.

## The test

Same laptop, same spirit as the [4 GB experiment](/blog/2026/06/14-the-4-gb-experiment-what-actually-makes-a-small-model-a-good-coding-agent/): a hidden, automated grader is the only judge. I pulled the Q4_K_M GGUF, applied the paper's recommended sampling (temperature 1.0, top-p 0.95), and ran a four-task battery chosen so every answer is mechanically checkable — two competition-math problems, one genuinely hard [AIME](https://en.wikipedia.org/wiki/American_Invitational_Mathematics_Examination)-level problem, and a classic LeetCode coding task:

<Table data={
    {
        headers: ['Task', 'Expected', 'VibeThinker-3B', 'Time'],
        rows: [
            ['Remainder of 7^100 mod 13', '9', 'correct', '13 s'],
            ['Tilings of a 2x10 board by dominoes', '89', 'correct', '20 s'],
            ['AIME: count triples a+b+c=300 with a constraint', '601', 'correct *', '247 s'],
            ['longest_palindromic_substring in O(n^2)', '(verified)', 'correct', '16 s'],
        ]
    }
} />

Four for four. The code I ran against a brute-force oracle on a batch of strings — it matched everywhere. And the whole time the model sat **100% on the GPU**, using 2.3–2.7 GB of the 4 GB, generating at **50–63 tokens per second.** No CPU offload, no thrashing. The asterisk on the AIME row is the one real gotcha, and it is worth a section of its own.

## The gotcha: token budget, not VRAM, is the wall

On my first AIME attempt the model **failed** — but not because it could not solve the problem. With a 16K-token context it generated **16,299 tokens of reasoning** and ran straight into the context wall mid-thought, never reaching an answer. Reasoning models think in the open, and a hard problem is a long think.

I raised the context to 32K and re-ran the exact same prompt. This time it converged — **601, correct** — in about 12,000 tokens. And here is the lovely part: even at 32K context it *still* sat entirely on the GPU, at 2.7 GB.

That is the counter-intuitive lesson for this class of model on small hardware: **the binding constraint is the token budget, not the VRAM.** The KV cache (quantized, with flash attention) is cheap; what these models need is *room to think*. If you run a reasoning model on a 4 GB card and it "fails," check whether it actually finished before you blame the model.

## The head-to-head

Solving four problems is nice, but the real question is comparative: against the models we already run on this laptop, is VibeThinker actually *better*, or just *different*? I ran the identical battery against `qwen2.5-coder:3b` (our coding workhorse, same 3B size) and `qwen3:4b` (the reasoning-tuned model one size up).

<Table data={
    {
        headers: ['Task (expected)', 'VibeThinker-3B', 'qwen2.5-coder:3b', 'qwen3:4b'],
        rows: [
            ['7^100 mod 13 (9)', 'correct, 13 s', 'correct, 11 s', 'correct, 140 s'],
            ['2x10 dominoes (89)', 'correct, 20 s', 'wrong (got 11)', 'correct, 595 s'],
            ['AIME triples (601)', 'correct, 247 s', 'wrong (got 3)', 'failed, ran 2.2 hours'],
            ['palindrome O(n^2)', 'correct, 16 s', 'correct, 9 s', 'correct, 304 s'],
            ['Score', '4 / 4', '2 / 4', '3 / 4'],
            ['Speed', '50-63 tok/s', '~65 tok/s', '3-18 tok/s'],
            ['GPU placement', '100% GPU', '100% GPU', 'spills to CPU'],
        ]
    }
} />

Read that table for a minute, because each column tells a different story.

**`qwen2.5-coder:3b`** is the fastest thing here and stays fully on the GPU — but it has no reasoning chain, so it answers from the hip and falls apart on anything non-trivial. It missed the dominoes problem (a recurrence a reasoning model finds in seconds) and was nowhere near the AIME answer. Superb at code, useless at hard math. Exactly the profile that made it our *editor* in the earlier studies, and exactly why it cannot be the escalation target.

**`qwen3:4b`** *can* reason — notice it got the dominoes problem the coder missed. But it is 4B, and 4B does not fit in 4 GB. It spills onto the CPU and collapses to **3–18 tokens per second.** The AIME problem ran for **two hours and twelve minutes** and *still* did not finish inside a generous token cap. It is the right kind of model at the wrong size for this card — technically capable, practically unusable here.

**VibeThinker-3B** is the only column that is both *right* and *runnable*. It reasons like the 4B but fits and flies like the 3B. That is the whole point: it gives you **qwen3:4b's reasoning at qwen2.5-coder:3b's speed and footprint** — because it *is* the 3B, just trained to think.

## The payoff: the tail finally has a target

Bring it back to the [dispatch thesis](/blog/2026/06/15-the-small-models-real-job-is-dispatch-not-work/). That study argued you should route by **measured capability**, not by parameter count: let a small workhorse handle the bulk, and escalate the hard tail to whatever model actually clears it. The unsatisfying gap was that "whatever clears it" pointed off the laptop and into the cloud.

VibeThinker-3B fills that gap for one specific, valuable slice of the tail: **hard, verifiable reasoning.** A math or coding problem your 1.7B workhorse flubs and your 4B can't run in time, this 3B specialist solves — locally, on the GPU you already own, for a lightbulb of electricity. You escalate *sideways into a specialist* instead of *up into the cloud.*

It is not a general assistant. Ask it for open-domain knowledge or to drive an agent's tool calls and it will lose to qwen3 — it is a scalpel, not a Swiss Army knife. But a scalpel is exactly what an escalation slot wants. The dispatch post said the small model's job is to know *who* should do the work. This is one more capable "who" — and remarkably, it is one that still fits in 4 GB.

## Reproduce it yourself

The whole point of this series is that you can run it on the machine you already have. Here is the entire setup.

Pull the quantized model straight from Hugging Face into [Ollama](https://ollama.com/) (no separate download step needed):

```sh
ollama pull hf.co/mradermacher/VibeThinker-3B-GGUF:Q4_K_M
```

Wrap it in a Modelfile so the recommended sampling and a generous context are baked in. The 32K context is the important part — give the reasoner room to think:

```dockerfile
FROM hf.co/mradermacher/VibeThinker-3B-GGUF:Q4_K_M
PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER top_k 0
PARAMETER num_ctx 32768
```

```sh
ollama create vibethinker3b -f VibeThinker.Modelfile
```

Then point a problem at it and confirm it lands fully on the GPU:

```sh
curl -s http://localhost:11434/api/generate -d '{
  "model": "vibethinker3b",
  "prompt": "In how many ways can a 2x12 rectangle be tiled by 1x2 dominoes? End with the final integer on its own line.",
  "stream": false
}' | python3 -c 'import json,sys; print(json.load(sys.stdin)["response"])'

ollama ps   # PROCESSOR should read 100% GPU
```

If a hard problem comes back without a final answer, you have hit the token wall, not a reasoning failure — raise `num_ctx` and run it again.

*The series has gone from reading agents to running them, and from "what fits in 4 GB" to "what's worth pointing 4 GB at." Next, the obvious question once you have a workhorse and a specialist on the same laptop: can a small model reliably decide, on its own, which of the two a given problem needs — and hand off without a human in the loop?*


b7a6e0bfc0313d9f1f548829fc58f29797d25c1596f8e8501df822d3706e2e91

VibeThinker-3B tested on a 4 GB RTX 3050: a 3B verifiable-reasoning specialist that fits fully on a modest laptop GPU, beats qwen2.5-coder and qwen3:4b on hard math at a fraction of the power, and fills the escalation slot from the small-model routing study.

The dispatch experiment ended on a question: route the hard tail to a more capable model — but which one, if you do not want to reach for the cloud? VibeThinker-3B, a 3B reasoning specialist, runs entirely on the 4 GB laptop GPU you probably already own, solves competition math the workhorse cannot, and sips about 50 watts doing it.

The 4 GB Card You Already Own Can Reason Now

{/* banner prompt: High-key cinematic shot in a vast minimalist white space with soft haze; a single small luminous cube acts as a switchboard, sorting incoming streams of light — most loop neatly back into the cube itself, while one thin, bright tributary branches off toward a distant towering machine that glows on the horizon; 16:9, volumetric lighting, a sense of triage, flow, and economy. */}

The [4 GB experiment](/blog/2026/06/14-the-4-gb-experiment-what-actually-makes-a-small-model-a-good-coding-agent/) ended on an honest wall: a small local model fits in a 4 GB laptop GPU and solves easy coding tasks, but it hits a ceiling on hard ones and [cannot drive a native tool call](/blog/2026/06/15-what-a-native-tool-call-actually-is/) at all. The natural reaction is to shrug and reach for a cloud model.

But that conclusion assumed the small model's job is to **do the work.** A reader pushed back with a better question: what if its job is to **decide who does the work?** Judging a task is cheaper than solving it. So we ran another experiment — same 4 GB laptop, same hidden-grader harness — to find two things: where a small model's limit *actually* is across general tasks, and whether a small model can earn its keep as a **router** instead of a worker.

## The setup

The [harness](/blog/2026/06/13-what-an-ai-agent-harness-actually-does/) is the same one from the coding study: each task runs against a hidden grader, so every result is automated and reproducible. The new task suite is **48 general tasks** — three domains (reasoning, extraction, writing) × four difficulty tiers (easy → extreme) × four tasks each — graded by exact answer, number match, JSON-field match, or a constraint checker (word counts, lipograms, acrostics). We ran **six models × 48 tasks × 3 repetitions = 864 trials**, plus a separate batch of open-ended writing prompts judged by hand.

A quick but important detour: our first run was contaminated by the grader, not the models. Tasks failed because a model answered `$1,240.50` where the grader wanted `1240.50`, or `The Widget Pro` where it wanted `Widget Pro` — *correct extraction, punished for formatting.* That is the same trap the coding study taught us to watch for, and a calibration pass caught it. We normalised currency, commas, and a leading article before comparison. Measure the capability, never the formatting.

## Finding 1: reasoning-tuning beats size

Here is the capability matrix — overall, and broken out by domain:

<Table data={
    {
        headers: ['Model', 'Overall', 'Extraction', 'Reasoning', 'Writing'],
        rows: [
            ['qwen3:4b', '94%', '100%', '94%', '88%'],
            ['qwen3:1.7b', '88%', '98%', '90%', '77%'],
            ['granite3.3:2b', '72%', '94%', '50%', '73%'],
            ['qwen2.5-coder:7b', '72%', '88%', '71%', '56%'],
            ['qwen2.5-coder:3b', '69%', '94%', '54%', '58%'],
            ['qwen2.5-coder:1.5b', '63%', '88%', '50%', '52%'],
        ]
    }
} />

Read the top two rows against the bottom four. **qwen3:1.7b scores 88% and beats qwen2.5-coder:7b at 72% — while being four times smaller.** The coder models, which *won* the coding study, are the *weakest* generalists here. The lesson is blunt: capability is not monotonic in size, and it is not even consistent across model families. A model tuned for reasoning runs circles around a bigger model tuned for code the moment you step off the code.

That detail matters more than it looks, and it comes back to bite us later: **the biggest model in your fleet is not necessarily the best one.**

## Finding 2: the limit is real, and it has a shape

Small models are not uniformly weak — they fail in specific, predictable places:

- **Easy and medium extraction and reasoning are solved.** A 1.5B model pulls fields out of text and does two-step arithmetic at near-100%. This is genuinely usable work.
- **Hard reasoning caps at ~75%** even for the best model. The classic trick — "you just overtook the person in second place; what position are you in?" — still catches them.
- **Lipograms are the single hardest thing we measured.** "Describe the sky without using the letter *e*" sits at 50% across the entire fleet. Writing under a hard mechanical constraint needs planning the models do not have.

So the small model has a real, shaped limit. Which sets up the actual question.

## The pivot: can it route?

If a cheap model cannot reliably *do* hard work, can it reliably *sort* hard work — keep the easy stuff and hand the hard stuff up to a bigger model? We measured three routing signals, from the most idealised to the most deployable.

### Signal 1 — the oracle router (the ceiling)

First, the upper bound. Imagine a perfect router that always sends each task to the *cheapest* model that can solve it. It cannot exist (it would need to know the answer in advance), but it tells you how much routing could *possibly* save. Using model size as a cost proxy:

- **Coverage 98%** — some model in the fleet solves all but one task.
- The oracle router's mean cost is **1.79B params vs 7B for "always use the biggest" — 74% cheaper at identical coverage.**

And the cumulative coverage as you add tiers, cheapest first, is the punchline:

<Table data={
    {
        headers: ['Models available up to', 'Size (B)', 'Cumulative coverage'],
        rows: [
            ['qwen2.5-coder:1.5b', '1.5', '65%'],
            ['qwen3:1.7b', '1.7', '90%'],
            ['granite3.3:2b', '2.0', '94%'],
            ['qwen2.5-coder:3b', '3.0', '94%'],
            ['qwen3:4b', '4.0', '98%'],
            ['qwen2.5-coder:7b', '7.0', '98%'],
        ]
    }
} />

Adding the 1.7B model jumps coverage from 65% to **90%**. Everything above it buys 8 more points, and the 3B and 7B coders buy *nothing* for general work. The whole strategy falls out of one row: **run qwen3:1.7b as the workhorse, escalate the ~10% it can't handle to qwen3:4b, and never load the 7B at all.**

### Signal 2 — self-consistency (proactive, and deployable)

The oracle cheats by peeking at the grader. A real router has no grader at dispatch time — it only sees the model's own output. So here is a signal it *can* use: **sample the small model a few times and watch whether it agrees with itself.** Confident, stable answers are accepted; answers that wobble across samples get escalated.

The appeal is that it sidesteps the obvious failure mode of asking a model "is this hard?" — a weak model is exactly the one that *can't tell* a hard question is hard, and will cheerfully label it easy. Instability is harder to fake than a self-rating. We sampled qwen3:1.7b five times per task and scored two definitions of "agree":

<Table data={
    {
        headers: ['Agreement defined as', 'Accept precision', 'Escalation rate', 'Missed errors'],
        rows: [
            ['Same text (any wording differs = disagree)', '100%', '60%', '0%'],
            ['Same answer (8 = "8 days" = "the answer is 8")', '97%', '38%', '2%'],
        ]
    }
} />

"Accept precision" is the chance an accepted answer is actually right; "missed errors" are confident-but-wrong answers that slip through. The first row is remarkable: **when this model agrees with itself word-for-word, it is right every single time** — zero bad accepts. The catch is it over-escalates, because it counts "8" and "8 days" as a disagreement. Comparing the *answer* instead of the *wording* cuts wasted escalation from 60% down to 38%, at the cost of exactly one confident-wrong slip. That is not a bug to fix — it is a **dial.** Turn it toward text-agreement when mistakes are expensive, toward answer-agreement when compute is.

### Signal 3 — feedback escalation, and the corpus it leaves behind

The first two signals act before the user sees anything. The third one is the human in the loop: a **thumbs-down re-routes the same question to a stronger model**, and — just as importantly — every disliked question is saved to a deduplicated corpus for later.

Here is where Finding 1 comes back to bite. The naive way to "escalate to a stronger model" is to pick a *bigger* one — and that is wrong. Escalating qwen3:1.7b's failures by size would send them to granite or the 7B coder, both of which are *worse* at general tasks. Escalation has to follow **measured capability, not parameter count.** Routed correctly to qwen3:4b, the bigger model recovered 2 of the 5 disliked answers.

The other 3 are the interesting part. They were the marble-fraction trick and **both lipograms** — the exact hard tail from Finding 2, the tasks *no* tier in the fleet can solve. Escalation can't fix those, and that is precisely why the corpus matters: it automatically collects the questions that routing alone will never rescue. That pile is your eval set, your fine-tuning data, and your prompt-engineering to-do list — the questions worth a human's attention, surfaced for free.

## A note on the writing

The open-ended writing prompts told the same story from a different angle. Every model, down to 1.5B, writes **fluent** prose. What separates them is **discipline** — obeying the instruction. The small coders wrote four lines when asked for a two-line rhyme, three sentences when asked for two. Only two of six models produced an actual rhyme. The best writer, qwen3:4b, also had the worst obedience, cheerfully appending an emoji checklist to a "short email." Fluency is free at 1.5B; *following the constraint* is the thing that scales with reasoning-tuning — the same axis the whole routing argument rides on.

## The takeaway

The small model's ceiling is real, but the framing was wrong. Asked to be the worker, a 4 GB model is a mediocre generalist. Asked to be the **dispatcher**, it is excellent — and the architecture writes itself:

- **qwen3:1.7b as the always-on workhorse**, covering ~90% of general traffic on hardware you already own.
- **Self-consistency as the escalator** — when the small model disagrees with itself, hand the task up. A tunable dial between cost and safety.
- **Escalate by measured capability, not size** — and capture every thumbs-down into a hard-case corpus, because that corpus is where the next round of real improvement comes from.

Stop asking the small model to do the work. Ask it to route the work, and it will save you most of your compute while quietly building the dataset that makes everything downstream better.

*This closes the small-model thread of the series — from [which agent wins](/blog/2026/06/13-the-coding-agent-shootout-and-the-best-harness-we-could-not-read/), to [what a tool call really is](/blog/2026/06/15-what-a-native-tool-call-actually-is/), to the 4 GB model's real job. Next, we turn the lens back on the big agents.*


01468f0733b58c321602a62d2f81c9448ea48df1c88cc4b2d188d3a423056e5a

An empirical study of small local models across reasoning, extraction and writing on a 4 GB RTX 3050, plus three routing signals — oracle upper bound, self-consistency, and feedback escalation — that turn a small model into a dispatcher.

We pushed a 4 GB laptop model past coding into reasoning, extraction and writing to find its limit — then measured a better use for it entirely: not doing the work, but routing it. A reasoning-tuned 1.7B model covers 90% of a general workload, and a perfect router is 74% cheaper at equal quality.

The Small Model's Real Job Is Dispatch, Not Work

{/* banner prompt: A clean isometric 3D scene of a data flow from an "LLM" module to a "Harness" module on a light gray background; one channel shows structured blocks labeled `tool_calls` successfully docking into a port, while a separate channel with prose text flows past, crisp geometry and a minimalist white and blue palette, 16:9 aspect ratio. */}

In our [4 GB experiment](/blog/2026/06/14-the-4-gb-experiment-what-actually-makes-a-small-model-a-good-coding-agent/) the most jarring result was a flat **0%**: a small coder model, given proper tools and asked to use them, never once produced a usable *native tool call* — across 84 attempts. It clearly *knew* what to do; it just couldn't do the one specific thing the protocol required.

That result only makes sense if you know exactly what a "native tool call" is. Most explanations wave at it — "the model calls a function" — which hides the part that actually matters. So let's be precise: what is a native tool call, what happens on the wire, and why is it a thing a model can simply *fail* to produce?

## Start from the uncomfortable fact: an LLM only outputs text

A language model does exactly one thing: given some text, it predicts more text, one token at a time. It cannot open a file, run a command, or call an API. It has no hands. So when you hear "the AI used a tool," something had to bridge the gap between *text the model wrote* and *an action your computer took*.

That something is the **harness** — the program you write *around* the model. One note on pronouns, because the split is the whole point of this post: throughout, **"you" and "your code" mean that harness, never the model itself.** The model only ever produces text; "you" are the code that reads it and acts. The model *asks*; you *do*. (A "coding agent" as a product — Claude Code, Cursor, Codex — is the two halves together; here, "you" is the harness half.)

With that straight, the question becomes a practical one: how does the harness turn the model's text into a real action? It turns out there are only two ways to build that bridge.

**Do-it-yourself (prompt-based).** You write in the system prompt: "When you want to read a file, reply with `READ: <path>` and nothing else." The model writes `READ: config.py`; *your code* watches the output, recognises the pattern, runs the read, and pastes the result back into the conversation. This works, but you invented the format, you parse it, and the model is just following instructions in prose. The model has no special knowledge that `READ:` means anything.

**Native tool calling.** Instead of inventing a format, you use a *standardised protocol* that the API provider defines and the model was specifically *trained* to speak. This is the thing people mean by "native tool calls" (a.k.a. "function calling"). It's worth seeing the whole exchange, because the magic is in the structure.

## What actually happens on the wire

**Step 1 — you declare the tools in the request.** Alongside the messages, you send a list of tool definitions: a name, a description, and a JSON Schema for the arguments.

```json
{
  "model": "...",
  "messages": [{"role": "user", "content": "What's in config.py?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "read_file",
      "description": "Read a file from disk.",
      "parameters": {
        "type": "object",
        "properties": {"path": {"type": "string"}},
        "required": ["path"]
      }
    }
  }]
}
```

**Step 2 — the model replies with a *structured* tool call, not prose.** This is the crux. The response does not put the request in the text `content` field. It puts it in a separate, dedicated `tool_calls` field:

```json
{
  "role": "assistant",
  "content": null,
  "tool_calls": [{
    "id": "call_abc123",
    "type": "function",
    "function": {
      "name": "read_file",
      "arguments": "{\"path\": \"config.py\"}"
    }
  }]
}
```

Notice: `content` is `null`. The model isn't *talking*; it's *requesting an action*, in a machine-readable slot with a name, JSON arguments, and a unique `id`.

**Step 3 — your code executes it.** Your harness reads `tool_calls`, sees `read_file`, parses the arguments, actually reads the file, and captures the contents. The model did not run anything — it asked, and your code obeyed.

**Step 4 — you hand the result back, tied to the call's `id`.**

```json
{"role": "tool", "tool_call_id": "call_abc123", "content": "DEBUG = True\nPORT = 8080"}
```

**Step 5 — the model continues**, now with the file contents in context — either answering the user, or emitting *another* tool call. Loop until it stops calling tools. That loop is the entire mechanism behind every coding agent you've used.

(Anthropic's API uses a slightly different shape — `tool_use` content blocks and a `tool_result` block instead of a `tool_calls` array and a `tool` role — but it is the same protocol: structured request, you execute, structured result back.)

## So what does "native" actually mean?

Three things, and all three matter:

1. **A standardised wire format.** The `tool_calls` field is part of the API contract. Every agent framework — Codex, Pi, Cursor, LangChain — reads that field. You don't invent or parse anything; the structure is guaranteed.
2. **The model was trained to emit it.** During fine-tuning, the model was taught a special internal format for tool calls — usually delimited by control tokens (Qwen and many others wrap calls in `<tool_call> … </tool_call>` tags; other model families use other schemes). It learned *when* to call a tool versus when to talk, and to put valid JSON in the arguments. This is a learned skill, not a parsing trick.
3. **The serving layer parses (and often constrains) it.** The inference server applies the model's chat template, watches for those control tokens in the stream, and converts them into the clean `tool_calls` field you receive. Some servers go further and use *constrained decoding* — a grammar that forces the model to emit only tokens that keep the JSON valid.

Put together: the model speaks a format it was trained on, the server translates it into a standard field, and your code reads that field. "Native" means the whole stack agrees on the protocol, end to end.

<Table data={
    {
        headers: ['', 'Prompt-based (DIY)', 'Native tool call'],
        rows: [
            ['Format', 'you invent it', 'standard, defined by the API'],
            ['Where the call appears', 'in the text content', 'in the structured tool_calls field'],
            ['Who parses it', 'your code, with regex/string matching', 'the model server, reliably'],
            ['Model awareness', 'just following prose instructions', 'trained on the format; knows when to call'],
            ['Arguments', 'whatever the model writes', 'JSON validated against a schema'],
            ['Multiple/parallel calls', 'you design it yourself', 'built in'],
        ]
    }
} />

## Why a model can *fail* to make one — the 0%

Here's the punchline that the experiment delivered. Native tool calling is a *learned behaviour*. A model that wasn't trained heavily on the format will, when handed tools, do the intuitively-reasonable thing: it writes the tool call as **text**, in the `content` field, often as a tidy JSON code block:

````text
I'll read the file.
```json
{"name": "read_file", "arguments": {"path": "config.py"}}
```
````

That is *correct intent in the wrong place*. The content is perfect — but it's prose, not a `tool_calls` entry. The harness looks at the `tool_calls` field, finds it empty, and does nothing. The model "called the tool" the way a person points at a door instead of opening it. That is precisely why `qwen2.5-coder:3b` scored 0/84 on native tools while sailing through whole-file editing: it can write the JSON, it just hasn't been trained to put it in the structured channel. Tool-calling is a *separate capability* from coding — a model can be excellent at one and incapable of the other.

## The fix: if the call is in the text, read it from the text

The natural response to that 0% is: *the JSON is sitting right there — why not just grab it and run it?* You can. And doing exactly that **is** prompt-based tool calling — the do-it-yourself bridge from the top of this post. You stop trusting the `tool_calls` field, scrape the `{"name": ..., "arguments": ...}` object out of the message content, and execute it yourself. In our experiment that one change took the small model from **0% to ~56%**: it could suddenly drive a real read → edit → run loop.

So native and DIY aren't rival technologies — they're **two implementations of one idea**: turn the model's output into an executed function. Native puts the call in a guaranteed field the model was trained to fill; DIY scrapes it from wherever the model happened to put it. The trade-off is clean:

- **Native** — rock-solid (standard field, schema-checked arguments, call IDs, the model's trained sense of *when* to call) — but only works if the model was trained for the format.
- **DIY scraping** — works with *any* model that can write JSON — but you own all the fragility. "Just grab the JSON" stays trivial right up until the model wraps it in a code fence one turn, in `<tool_call>` tags the next, bare in prose the next, emits two of them, or writes malformed JSON. A real scraper is a careful little parser, not a one-line regex.

And the honest punchline from the experiment: scraping rescued the *capability* (0% → 56%), but the small model still scored higher — **71%** — by skipping tools entirely and just writing the whole file. For a single edit, the deepest fix isn't a smarter parser; it's not asking for a tool call at all.

## The one-paragraph version

A native tool call is **not** the AI running `sed`, and it's **not** the model "deciding" to do something magical. It is a structured, schema-validated request that the model emits in a dedicated field — a field it was specifically trained to produce and that the inference server reliably parses — saying *"please run function `X` with arguments `Y`."* Your code runs it and feeds the result back. The whole loop of read → edit → run → fix that powers every coding agent is just this protocol, repeated. And because it's a trained behaviour living in a specific channel, it's something a model can know the answer for and still completely fail to deliver — which is exactly why, for small local models, you're often better off skipping it and letting the model just write the file.

8856ef1c99ced7724e1374322d5cf2238b2140666b44c9879ddf0eb3924de448

A precise explainer of native tool calling (function calling): what the protocol is, the request/response JSON on the wire, how the model emits it, and why small models fail to produce it.

An LLM only ever outputs text — so how does an AI agent run a tool? The answer is the native tool call, a precise, trained, parseable protocol that most people hand-wave over. Here is exactly what it is, what happens on the wire, and why a small model that knows what to do still cannot make one.

What a Native Tool Call Actually Is

{/* banner prompt: High-key cinematic shot in a vast, minimalist white space with atmospheric haze; a single, compact, pristine cube on the floor projects a focused, coherent beam of light that forms a perfect code structure on a far wall, while much larger, ghostly, and fragmented machines dissolve into the bright background, 16:9, volumetric lighting, sense of scale and clarity. */}

The first five posts in this series were the telescope and the microscope — what agents [are](/blog/2026/06/13-codex-pi-and-hermes-three-agents-three-species/), what a [harness does](/blog/2026/06/13-what-an-ai-agent-harness-actually-does/), how they [learn](/blog/2026/06/13-skills-and-memory-how-ai-agents-actually-learn/), which one [wins](/blog/2026/06/13-the-coding-agent-shootout-and-the-best-harness-we-could-not-read/), and how one [actually writes a line of code](/blog/2026/06/13-how-a-coding-agent-actually-writes-code/). All of it was *reading* — code and behaviour. This post is different: we stopped reading and **ran the experiment.**

The question: on cheap, local hardware — a 4GB laptop GPU — **what actually makes a small model a good coding agent?** Not "which frontier model wins" (you can't run those at home), but: given a model that fits in 4 GB, which *harness choices* matter? Edit format? Retry? Task decomposition? A planner model? We built a measurement harness, wrote a suite of coding tasks with automated pass/fail oracles, and ran roughly **900 trials**. The results overturned most of our intuitions.

## The setup

A gaming laptop with an **RTX 3050 (4 GB VRAM)** serving models via Ollama; a separate machine running the agent and holding the code. The harness runs each task in a fresh copy of its workspace, lets the model attempt it, then grades it with a hidden `pytest` oracle — the test is the only judge, so every result is automated and reproducible.

The **task suite** is 28 self-contained coding tasks: add a function, fix a bug, small refactors, implement-from-spec, multi-file changes, and a deliberately hard tier (a bowling scorer, a circular buffer, rational-number arithmetic, a 24-hour clock, a book-store discount optimiser, variable-length-quantity encoding). Each oracle was checked two ways — it must *fail* on the starting code and *pass* on a correct solution. Calibrated this way, the simplest configuration solves about 71%, leaving real room to measure whether the fancy techniques help.

We varied **one lever at a time** around that baseline (`qwen2.5-coder:3b`, whole-file edits, single-shot), ran each task/config combination three times to smooth out the noise of a non-deterministic model, and recorded success rate, tokens, and wall-time.

## The results

<Table data={
    {
        headers: ['Config', 'Success', 'Sec/run', 'What changed vs baseline'],
        rows: [
            ['baseline (coder-3b, whole-file)', '71%', '2.6', 'reference'],
            ['retry (run test, feed failure back)', '70%', '8.6', 'no change'],
            ['decomposition (plan then execute)', '64%', '5.3', 'hurt'],
            ['role-split (planner + editor)', '66%', '134', 'hurt + 50x slower'],
            ['model: qwen3:1.7b', '43%', '50', 'much worse coder'],
            ['model: granite3.3:2b', '39%', '2.2', 'much worse coder'],
            ['edit format: diff', '24%', '5.9', 'far worse'],
            ['edit format: native tool-calls', '0%', '2.1', 'impossible'],
        ]
    }
} />

Read that table for a minute, because almost every row is a surprise.

## Finding 1: the edit format is the dominant lever

How the model is asked to express a change matters more than anything else. Whole-file rewrite: **71%**. Search/replace diff: **24%**. Native function-calling: **0%**.

That **0%** deserves a pause. We gave the model proper OpenAI-style tools and asked it to call them — and across 84 attempts it never once produced a usable tool call. It *knew what to do*: it emitted perfectly correct JSON like `{"name": "read_file", "arguments": {"path": "config.py"}}` — but as a markdown code block in its **message text**, never in the structured `tool_calls` field the protocol requires. A native-tool agent (Codex, Pi, anything function-calling-based) sees an empty tool-call list and does nothing. **The small coder model literally cannot drive a native-tool agent.**

Diff (24%) is a softer version of the same lesson: the model produces well-formed search/replace blocks, but on longer files the "search" text drifts from the source by a character and the edit won't apply. Whole-file editing sidesteps both problems — there's nothing to match, nothing to format. For a small model, **make the easy thing the only thing.**

## Finding 2: "best coder" is not "best tool-caller"

We tried three models that fit in 4 GB. The coding-specialised `qwen2.5-coder:3b` (71%) crushed the newer general models `qwen3:1.7b` (43%) and `granite3.3:2b` (39%) at producing correct edits.

Here's the twist: **qwen3:1.7b is the *better tool-caller*** — it can emit native `tool_calls` and drive an agent loop where the coder model can't — yet it's the *worse coder*. Tool-calling ability and coding ability are different axes, and a model can be strong on one and weak on the other. If your harness leans on native tool-calling, you're implicitly selecting for the tool-caller and possibly against the coder. (qwen3:1.7b is also strikingly slow and verbose here — ~50 s/run, ten times the output tokens — despite being told not to "think.")

## Finding 3: the clever agentic scaffolding didn't help — and mostly hurt

This is the one that stings, because it's the stuff everyone reaches for.

- **Retry** (run the test, feed the failure back, let it fix): **neutral** (70% vs 71%). Retry recovers *near-misses*. But this suite's failures are the hard tier — bowling, VLQ — which are *capability-bound*: the 3B can't write a correct bowling scorer no matter how many times you show it the failing test. **Retry can't buy capability the model doesn't have.**
- **Decomposition** (plan the task into steps, then execute): **−7 points**. The planning step adds tokens and structure a small model handles poorly; a slightly wrong plan actively misleads it.
- **Role-split** (a stronger `qwen3:4b` planner directing the coder): **−5 points and ~50× slower**. On 4 GB the two models can't co-reside, so the GPU thrashes swapping them in and out. A separate planner didn't help and cost two minutes a task.
- **Stacking** the "improvements" together regressed *below* baseline, because the negative levers compound.

The pattern: orchestration tricks that help a frontier model coordinate **do not rescue a small model whose failures are a capability ceiling, and several make it worse** by adding surface area for a limited model to trip over.

## The follow-up: can we craft tool-calls the small model *can* do?

Finding 1 left a loose end. Native tool-calling scored 0% only because the model emits tool intent as *text*, not in the `tool_calls` field. So we asked: what if we meet the model where it is — build our own tool-call protocol that parses the JSON-in-content it naturally produces — and give it a real agent loop (read → edit → run → fix)?

We did. A lenient parser (accepting fenced JSON, `<tool_call>` tags, or bare JSON), a small tool set, and a loop that hides the grading test until the end. Two results:

1. **A custom text protocol rescues tool-calling: 0% → ~56%.** Meeting the model where it is *works* where the rigid native protocol is impossible. The 3B genuinely drives a multi-step loop.
2. **But the loop still loses to single-shot — and its value depends on the model.** On the strong coder it *hurt* (71% single-shot → 56% in the loop: more turns, more chances to go wrong). On the weaker-but-tool-capable qwen3:1.7b it *helped* (43% → 58%). Neither beat plain coder-3b doing one whole-file edit.

So you *can* engineer a small model into being an agent at all — and that's a genuinely useful capability — but the engineering doesn't beat the simplest possible setup.

## What it adds up to

For a small model on scoped coding tasks, the high-leverage choices are the **editor model** and the **edit format** — not the agentic scaffolding. The recommended 4 GB configuration is almost embarrassingly plain: **`qwen2.5-coder:3b`, whole-file edits, single-shot.** 71% at under three seconds a task. Everything fancier was neutral, slower, or worse.

This both confirms and sharpens the series' running thesis. We've said all along that *the harness, not the model, is where the engineering lives* — and that's still true: the harness's edit-format and model choices made the difference between 0% and 71%. But there's a hard limit the experiment made vivid. A harness can fix problems of **interface** — give the model an edit format and a tool-call protocol it can actually use — and the payoff is enormous. It cannot fix problems of **capability** — when the model simply can't write the bowling scorer, no amount of retry, planning, or multi-model orchestration conjures the answer, and the cleverer the scaffolding, the more often it backfires. With a small model you feel that ceiling directly, and the winning move is to stop adding machinery and get out of the model's way.

There's a nice corollary for anyone running local AI on modest hardware: you don't need an elaborate agent framework to get real value from a 4 GB GPU. You need the right small coder model, whole-file editing, and the discipline to keep it simple.

*The harness for this study — the task suite, the levers, the custom tool-call loop, and the full results — is open source. This experiment grew out of trying to give a real coding agent a local brain; the surprise was how little the brain needed around it.*

0c62a9ca5856404573db16db808a2aa19791643978e8def22253ec0825fcba55

An empirical study of small-model coding agents on a 4 GB RTX 3050: edit format, model choice, retry, decomposition, role-split, and a custom text tool-call loop, measured across ~900 automated trials.

We stopped theorizing and ran ~900 trials on a 4 GB laptop GPU to measure what makes a small local model a good coding agent. The result overturns intuition: the edit format and the editor model decide almost everything, the clever agentic scaffolding mostly hurts, and the simplest setup won.

The 4 GB Experiment: What Actually Makes a Small Model a Good Coding Agent

{/* banner prompt: Warm, realistic analog photography of a cluttered but organized workshop desk, where a long ticker tape of computer output unspools from a vintage telegraph machine, with sections circled in red ink and handwritten notes in the margins, representing an AI agent debugging code through text logs, all in soft, natural window light, 16:9. */}

We called the [previous post](/blog/2026/06/13-the-coding-agent-shootout-and-the-best-harness-we-could-not-read/) the last in the series. That was the telescope view — what agents *are*, what a [harness does](/blog/2026/06/13-what-an-ai-agent-harness-actually-does/), how they [learn](/blog/2026/06/13-skills-and-memory-how-ai-agents-actually-learn/), which one wins. But a developer doesn't experience an agent as an architecture diagram. You experience it one edit at a time: it changes three lines, runs the tests, reads the failure, tries again. So here is the microscope view — how a coding agent actually turns an instruction into a working, tested change, held up against how a human programmer does the same thing.

## The human inner loop

Picture your own setup. You have an editor where you type, a terminal where you compile and run, and some way to *see the result* — a test runner's green bar, a printed value, a web UI you reload in a browser. Around it sits the big lifecycle everyone draws on whiteboards: idea → plan → architecture → design → write. But that's the outer loop. The thing you actually do all day is the **inner loop**, and it's tight: write a little, run it, look at what happened, adjust. Seconds per cycle.

Two habits define how humans fill that loop. First, we mostly **build small and grow** — a few lines, run, a few more — keeping a working program at almost every step, rather than generating a whole finished file and hoping. Second, we choose a direction: **vertically** (one feature sliced end-to-end through every layer until it runs) or **horizontally** (one layer built out across features). Either way the program is *alive* in front of us the whole time. We glance at the screen and instantly know if it's broken.

That last point is the one that matters most for what follows, because it's exactly the sense an agent doesn't have.

## The agent inner loop

An agent runs the same shape of loop — instruction in, change out, observe, repeat — but through a fundamentally narrower keyhole. Its loop is: the harness sends the model the context, the model emits a **tool call** (edit this file, run this command), the harness executes it and feeds the **text result** back, and the model decides what to do next. That's it. The agent has no eyes. It cannot glance at a screen. **Its only senses are the text that tools return** — file contents, stdout, stderr, an exit code. Everything a human takes in visually — the rendered web page, the debugger paused on a line, the red squiggle under a typo — has to be converted into text and handed back, or it doesn't exist for the agent at all.

This single fact explains almost everything distinctive about how agents code. It's why output truncation matters (an 80,000-line log blinds it as surely as a wall of noise), why edit-acceptance is fiddly, why verification has to be deliberate, and why debugging — the most *visual*, interactive thing a human does — is the hardest part to translate.

It also shifts the economics of "small and grow." Each model round-trip costs real tokens and seconds, so an agent can't afford the human's keystroke-level incrementalism; it tends to write a whole function or file in one shot, then verify. The good ones pull back toward small steps anyway — write, run, read, fix — because a 200-line blob that fails gives the model almost nothing to reason about, whereas a 20-line change that fails points right at the problem. The tension between "big chunks are cheaper per round-trip" and "small chunks are debuggable" is the central rhythm of agentic coding.

## The crux: turning intent into a changed file

Here is the most under-appreciated micro-mechanic in the whole stack: how does the model's *intention* to change code become an actually-changed file on disk? This step — call it **output acceptance** — is where coding agents most visibly differ, and where they most often fail. The five harnesses we read take three distinct approaches:

- **Exact string replacement.** The model supplies the old text and the new text; the tool finds the old verbatim and swaps it. This is what Claude Code's `Edit` and Pi's `edit` do. It's surgical and the resulting diff is trivially reviewable, but it's brittle: if the model's remembered snippet is off by a space, the match fails. Pi softens this with a *fuzzy fallback* that normalises smart quotes, Unicode dashes and trailing whitespace before giving up.
- **A patch grammar.** The model emits a structured diff in a format it was trained on — Codex's `*** Begin Patch … *** End Patch` `apply_patch` syntax. This handles multi-hunk edits cleanly and reads like a real patch, at the cost of the model having to produce exactly-valid patch syntax.
- **Self-correcting replace.** Gemini CLI's `replace` tool, on a failed match, fires a follow-up model call to *repair its own edit* rather than just erroring — trading tokens for a higher landing rate.

A failed edit is the single most common micro-failure in agentic coding, and each of those designs is one harness's answer to it. It's also exactly where your "native programmer versus scripter" observation enters — but that deserves its own section below.

## Calling the tools: compile, run, build, spawn

Once a file is changed, the agent needs hands to *do* something with it, and those hands are a single tool: shell execution (`Bash` in Claude Code and Pi, `exec`/unified-exec in Codex, a PTY shell in Gemini and Kimi Code). Through that one door it compiles, runs the test suite, starts a build, installs a dependency, spawns a server.

The interesting wrinkles are all about state and blindness:

- **Statefulness.** A naive `subprocess.run` can't hold a REPL open or answer an interactive prompt. Codex's "unified exec" gives the model a persistent PTY it can write stdin to; the mature shells track background processes so the agent can start a dev server and keep working while it runs.
- **Truncation.** Every harness caps tool output (Kimi Code's read tool stops at 1,000 lines or 100 KB; Pi spills overflow to a temp file). Without it, one chatty build destroys the context budget.
- **The blind spot, literally.** The agent can `npm run dev` and start a web UI — but it cannot *look* at the page. So it `curl`s the endpoint, greps the logs, or, increasingly, drives a headless browser and feeds itself a screenshot through a vision model. Watching an agent start a server it fundamentally cannot see is the clearest reminder of the text-keyhole problem.

## Tests and verification

Because the agent can't eyeball correctness, **running things is its substitute for seeing** — which makes tests and verification not a virtue but a necessity.

On creating tests, agents are comfortable both ways: writing the test first and coding to green (some skills enforce test-first explicitly), or writing tests after to lock in behaviour. Their real edge is that they are tireless about *running* the full suite, the linter and the type-checker on every change — the chores humans skip when tired.

Their matching weakness is that a language model will cheerfully announce "done" without having run anything. This is why mature harnesses and skills bolt on a **verification discipline**: re-read the file you just edited, run the build, run the tests, and only then claim success — evidence before assertion. The agent's superpower (re-run the world twenty times without complaint) and its failure mode (declare victory unverified) are two sides of the same coin, and verification gates exist to keep it on the right side.

## Debugging: the widest gap of all

Now the part of the inner loop that translates worst. Here is the canonical human bug hunt, in \*nix muscle memory:

1. `grep` the error string across the tree.
2. Open the file at the offending line, read the surrounding context.
3. Set a **breakpoint**.
4. Run, and when execution pauses, **step** through it — step over, step into, step out — watching variables change.
5. Catch the moment the state goes wrong, and you've found the cause.

An agent does steps 1 and 2 almost exactly as a human does — grep the error, read the context around the line. Then the methods fork hard, because **step 3 is mostly unavailable to it.** Agents rarely drive an interactive debugger; they can't easily sit at a paused frame and inspect live state. (Some can speak the Debug Adapter Protocol through an MCP server, but it's the exception, not the daily path.) So instead of stepping, the agent debugs the way we all did before debuggers existed, just faster: it inserts `print`/log statements, re-runs, reads the output, reasons about it, removes them. Printf debugging at machine speed.

The deep difference is one of *substrate*. A human debugs a **live, paused, inspectable** program — you can poke it while it's frozen mid-thought. An agent debugs a **dead transcript** of what the program already emitted. It loses the ability to interrogate a running frame, and it compensates with two things humans can't match: it will happily re-run the failing case fifty times with different instrumentation, and it can hold the entire call path in context at once rather than stepping through it line by line. Slower per insight, but relentless and wide.

## The observation: native programmer vs. scripter

Which brings us to the thing you actually notice after watching these agents work for a while: **they have different programming temperaments, and Claude Code behaves the most like a native \*nix programmer.**

By "native" we mean it reaches, by default, for the same surgical tools a seasoned terminal user does — `grep` to locate, a targeted read of the exact line range, a precise string-replace edit, the right small command for the job — and it leaves behind a clean, reviewable trail of minimal diffs. Watching it is like watching someone who has `grep`, `sed`, `cut`, `head` and `tail` in their fingers and uses them with intent.

The contrast: when faced with a bulk change — rename this symbol across forty files, rewrite every import — some agents (Codex and Gemini among them, in our observation) are more inclined to **write a throwaway script to do it**: a `python3 -c "..."`, a `perl -pi -e`, a `sed -i` one-liner run through the shell, rather than a series of surgical edits. It works, and for genuinely mechanical mass-edits it's even the *right* call. But it has a cost: a thrown-away script's effect is harder to review than a diff — you have to reconstruct what it did — and a slightly wrong regex can silently mangle forty files at once, where a string-replace edit simply fails loudly on the one it couldn't match.

Two honest caveats, because this is the part most easily overstated. First, it's a *tendency*, not a wall: every one of these agents can use native tools, and every one can write a script; the difference is which it reaches for first, and that's shaped as much by the model's temperament as by the harness's tool design (a patch-grammar editor nudges toward batch thinking; a string-replace editor nudges toward surgical thinking). Second, these behaviours drift from model version to model version, so any snapshot like this one has a short shelf life. The durable point isn't "Claude Code wins the bug hunt" — it's that **there is a real, observable spectrum between surgical and scripted code-editing, and it changes how reviewable and how safe the agent's work is.** When you pick an agent, that temperament is something you can feel within an hour, and it matters more to the day-to-day than any benchmark.

There's a fitting irony to end on: this post about how agents wield `grep` and surgical edits was itself drafted by one, reaching for exactly those tools as it went. The microscope, it turns out, works on the thing holding it.

## Where micro meets macro

Across five posts the same theme keeps surfacing from different distances. The harness (macro) decides which tools exist and how their results come back; the model (micro) decides which tool to reach for and how big a bite to take. A great coding experience is when those align — a native-feeling toolset in the harness, and a model with the judgement to be surgical when a scalpel is called for and to script when a bulldozer is. The model sets the ceiling; the harness and the editing temperament decide how close to it you actually get to live — one edit, one test, one bug at a time.

*This is a micro-level companion to our four-part series on AI agent architecture. All six harnesses behind these observations are open source; the best way to see the difference between a surgical edit and a thrown-away script is to watch two agents solve the same task and read what each left behind.*

8087c217708dabbe7e83dd40d6de3630d54e2b37f02ed112a63941501e3f1232

A micro-level look at how AI coding agents write, run, test, verify and debug code: the edit loop, output acceptance, the exec tool, and why some agents feel like native command-line programmers while others feel like scripters.

The first four posts in this series took the telescope view of AI agents. This one takes the microscope: how a coding agent turns an instruction into a working, tested change — how it edits, compiles, runs, tests and debugs — and why Claude Code behaves like a native *nix programmer while others reach for a throwaway script.

How a Coding Agent Actually Writes Code

{/* banner prompt: A retro technical poster in the style of a 1970s engineering manual, diagramming five different abstract 'coding agent' mechanisms, with bold sans-serif typography labeling 'THE HARNESS' and 'MODEL CORE', using a limited-print color palette of navy blue and burnt orange on a cream paper texture, 16:9. */}

This is the fourth and final post in our series on AI agent architecture. The first three were about *how* agents are built — we [compared three of them](/blog/2026/06/13-codex-pi-and-hermes-three-agents-three-species/), dissected [what a harness does](/blog/2026/06/13-what-an-ai-agent-harness-actually-does/), and looked at [skills and memory](/blog/2026/06/13-skills-and-memory-how-ai-agents-actually-learn/). This one answers the question everybody actually asks first: **which coding agent should I use?**

We read the source of five open coding agents end to end — OpenAI's [Codex CLI](https://github.com/openai/codex) (the Rust harness is open even though the models aren't), [Pi](https://pi.dev/), [OpenCode](https://github.com/sst/opencode), Moonshot's [Kimi Code](https://github.com/MoonshotAI/kimi-code), and Google's [Gemini CLI](https://github.com/google-gemini/gemini-cli). Below is the honest shootout: strengths, weaknesses, and a ranking. But there's an asterisk we owe you up front, so here it is in plain sight — **the best harness in the field belongs to a tool we deliberately did not put under the microscope: Claude Code.** We'll get to why at the end.

(We're leaving [Hermes Agent](https://github.com/NousResearch/hermes-agent) mostly out of this one. It can code, but it's an assistant-first agent, and judging it as a coding tool would be judging a Swiss Army knife by its corkscrew.)

## How to judge a coding agent

A coding agent's quality is the product of two things that multiply rather than add: **the model** doing the reasoning, and **the harness** that decides what the model sees, what it can touch, and what happens when it's wrong. A great model in a sloppy harness flails on long tasks; a great harness can't make a weak model reason past its ceiling. So the axes that matter are:

- **Model strength on hard tasks** — can it actually fix the gnarly bug, not just the toy one?
- **Harness reliability** — does it survive a forty-minute run: compaction, recovery from loops, checkpoints?
- **Safety** — what stops it running `rm -rf` or leaking a secret?
- **Extensibility & ownership** — can you bend it, script it, run your own models?
- **Cost and friction** — what does a day of heavy use actually cost?

No single agent wins all five. That's the whole point.

## The contenders

**OpenAI Codex CLI** — *the strongest engineer in the room.* Codex pairs the best coding models available (the GPT-5.x-Codex family) with a harness that takes the hard problems seriously. It's the only one of the five where **sandboxing is architecture, not an afterthought**: every command runs inside an OS-enforced jail (Seatbelt on macOS, bubblewrap on Linux) with network denied by default, and the model can *request* escalation as part of its tool schema. The core is a protocol-first Rust engine, so one codebase drives the terminal, the IDE extension, `codex exec` and the cloud — and because OpenAI trains the models *with* this harness (they speak its custom patch format natively), model and harness reinforce each other. It's also the most token-efficient of the lot. Weaknesses: the models are closed and get deprecated on OpenAI's schedule, the cloud model isn't selectable, and its extension story (hooks, skills, plugins) is younger than its rivals'.

**Gemini CLI** — *the most feature-complete, and free.* Google's CLI is the most generous on-ramp in the field: a real free tier, a 1-million-token context window, and a genuinely impressive harness — loop detection that *injects corrective guidance* instead of just giving up, automatic fallback to a faster model on quota exhaustion, shadow-git checkpointing with `/rewind`, an A2A server for multi-agent setups, a VS Code companion, skills, hooks and subagents. It has quietly absorbed the best ideas from everyone. Weaknesses: it's at its best on Gemini models, the feature surface is sprawling, and on the hardest reasoning tasks community sentiment still places it a notch below Codex and Claude.

**OpenCode** — *the architecture astronaut's pick, in a good way.* OpenCode is built server-first: an HTTP server with an OpenAPI spec, and the TUI, desktop app, web UI and GitHub bot are all thin generated-SDK clients of it. It's aggressively provider-agnostic (bring any model), expresses "plan mode" and sub-agents as plain **permission rulesets** rather than hard-coded modes, ships git-based checkpoint/revert, and is mid-flight on an event-sourced V2 redesign. Weaknesses: that rewrite means churn, there's no sandbox, and your experience is only as good as the model you plug in.

**Kimi Code** — *the cleanest layering, and a cost story.* Moonshot's CLI is the best-factored codebase of the group: three crisp layers — `kosong` (LLM abstraction), `kaos` (OS abstraction), `agent-core` (the loop) — with event-sourced persistence you can replay in a visual debugger and cache-aware compaction that weighs prompt-cache economics before squashing history. Kimi models are strong coders and cheap, and it speaks ACP for editors like Zed. (A fun detail: its TUI is built on Pi's own UI library.) Weaknesses: it's tuned best for Kimi models, it's newer, and the community around it is still small.

**Pi** — *not really a product; a masterclass.* Pi ships zero models and refuses features on principle — no sandbox, no permission popups, no plan mode — pushing everything to a TypeScript extension system. What it *does* ship is the cleanest minimal agent core we read (about 2,700 lines), the best multi-provider abstraction (40-plus providers behind one interface), and a session-as-a-tree design that makes branching and time-travel free. Weaknesses as a daily driver: no safety rails, no cross-session memory, a bus factor of roughly one, and quality that depends entirely on the model you bring. You run Pi to *understand* agents, or to build your own on top of it — which plenty of people, including the OpenClaw and Kimi Code teams, have done.

<Table data={
    {
        headers: ['Agent', 'Strongest at', 'Weakest at', 'Open?'],
        rows: [
            ['Codex CLI', 'Model strength, sandboxing, reliability', 'Closed models, younger extensibility', 'Harness yes, models no'],
            ['Gemini CLI', 'Free tier, features, recovery, 1M context', 'Best only on Gemini; sprawling', 'Fully open'],
            ['OpenCode', 'Client/server architecture, ownership', 'Mid-rewrite churn, no sandbox', 'Fully open'],
            ['Kimi Code', 'Clean layering, event sourcing, cost', 'Kimi-first, young, small community', 'Fully open'],
            ['Pi', 'Minimal core, provider abstraction', 'No safety, no memory, bus factor ~1', 'Fully open'],
        ]
    }
} />

## A note on lineage: who built on whom

It's tempting to read these as five isolated inventions. They aren't. This is a small field whose teams read — and reuse — each other's code, and the family tree is worth tracing because it tells you where the reusable ideas actually live.

The clearest root is **Pi**. Its "toolkit, not product" philosophy isn't just rhetoric — other agents are literally built from its parts. **Kimi Code did not write its own terminal UI; it depends on `@earendil-works/pi-tui`, Pi's own TUI library**, and built its component layer on top (you can see it right there in `apps/kimi-code/package.json`). And **OpenClaw**, one of the most popular open personal assistants, is built on Pi's agent packages — its whole agent stack is Pi underneath. So while Pi loses a head-to-head as a *daily driver*, it quietly wins as *infrastructure*: when you use Kimi Code or OpenClaw, you're partly using Pi.

Then there's the softer lineage of shared conventions, where an idea from one agent becomes a standard everyone adopts:

- **`AGENTS.md`** — the project-instructions file — was originated by Codex and is now read by Pi, OpenCode, Gemini CLI (as `GEMINI.md`) and others.
- **The [agentskills.io](https://agentskills.io) `SKILL.md` format** is shared, independently, by Pi, Hermes, OpenCode, Kimi Code and Gemini CLI — five teams converging on one open standard.
- **Shadow-git checkpointing** (snapshot before risky edits, `/rewind` to undo) shows up in both OpenCode and Gemini CLI using the same underlying git trick.
- **The read / edit / grep / glob / bash tool ergonomics** that all five share trace largely back to Claude Code — which is part of why, as we'll see, it casts a long shadow over a comparison it isn't formally in.

The lesson for anyone building an agent: you are not starting from a blank page. The good patterns are already open, already converging, and in Pi's case already packaged as libraries you can import.

## The verdict among the five

Of the agents we actually read the code for, **Codex is the best coding agent** — and it isn't especially close on the axis that matters most for serious work. Hard engineering is where the model-times-harness product is most punishing, and Codex maxes both terms: the strongest coding models in the industry, co-designed with a harness that sandboxes by default and is built to grind through long autonomous runs. If your test is "fix the difficult bug in an unfamiliar codebase without supervision and without `rm -rf`-ing my home directory," Codex is the safe bet.

That doesn't make it the right pick for everyone. If you want zero cost and maximum features, Gemini CLI. If you want to own the whole stack and run your own models, OpenCode or Pi. If you live in Kimi's ecosystem and care about cost, Kimi Code. "Best" is conditional on what you're optimising for — but if you force a single ranking on *raw coding capability*, Codex tops the five.

## The asterisk: the best harness we couldn't read

Here's the honesty the title promised. We left **Claude Code** out of the code study for a simple reason — its harness isn't open source the way the others are, so there was nothing to dissect at the same depth. But leaving it out of the *code* study is not the same as leaving it out of the *conversation*, because by reputation, by community blind-comparison, and by the simple fact that everyone else keeps adopting its conventions, Claude Code arguably has the **best harness in the field.**

Consider the evidence that doesn't require reading its source:

- **It set the conventions the others copied.** The ergonomics of the modern coding-agent tool suite — the read / edit / grep / glob / bash design, the `AGENTS.md`-style project memory, plan mode, sub-agents, the skills format, MCP support — Claude Code either originated or popularised much of this. When you see five independent harnesses converge on the same patterns, a lot of those patterns trace back to one place.
- **Blind output comparisons keep favouring it.** In the community write-ups that pit agents head-to-head on real tasks, Claude Code's output is preferred more often than not — frequently over Codex — even when Codex uses fewer tokens. The recurring verdict is "Codex is more efficient, Claude Code is more *right*."
- **Its harness sophistication is visible from the outside.** Sub-agent orchestration, a genuinely good permission UX, hooks, output styles, tight IDE integration, careful context management — the surface area you interact with is the most polished in the category, and polish in a harness is exactly what determines how much of the model's capability you actually realise.

So the fair summary is not "Codex is best, full stop." It's: **Codex and Claude Code sit at the top tier, trading blows** — Codex winning on raw model strength, sandboxing and token efficiency, Claude Code winning on harness craft and output quality — and the five open agents below them serve different masters (cost, ownership, learnability, ecosystem). The reason Codex "wins" our shootout is partly an artefact of the rules: we ranked the agents whose code we could read, and the strongest harness in the field wasn't one of them.

There's a small irony worth admitting: the research behind this entire series — cloning six repos, reading tens of thousands of lines, drafting these posts — was itself done with a coding agent. The tools have gotten good enough to study, and explain, each other.

## What to actually use

- **You want the strongest autonomous coding agent and don't need to own it:** Codex, or Claude Code. Try both on your own hard task and trust your eyes over any benchmark — these two are close enough that fit-to-your-workflow decides it.
- **You want maximum capability at zero cost:** Gemini CLI.
- **You want to own the stack, pick your models, or self-host:** OpenCode (as a product) or Pi (as a foundation).
- **You're cost-sensitive or already in Moonshot's ecosystem:** Kimi Code.
- **You're building your own agent:** read Pi for the core, Codex for sandboxing, OpenCode for the client/server split, Kimi Code for event sourcing, Gemini CLI for recovery engineering — and study how Claude Code behaves, because it's the reference the others are quietly chasing.

The models will keep leapfrogging each other quarter by quarter; the rankings above have a short shelf life. What won't change is the lesson under all four posts in this series: the model sets the ceiling, but the harness decides how close to it you get to live. Choose the harness you'll actually enjoy operating — that, more than any benchmark, is what determines whether the agent helps you.

*That closes our four-part series on AI agent architecture. All six harnesses we studied are open source; if these posts made you curious, the code is the best teacher we found.*

63c4f06786b4754bb47d1ff425d74097711792bb251328a08773992cc7f8cb7e

A strengths-and-weaknesses comparison of the open coding agents Codex, Pi, OpenCode, Kimi Code and Gemini CLI, plus why Claude Code arguably has the best harness despite being closed.

After reading the source of five open coding agents — OpenAI's Codex, Pi, OpenCode, Kimi Code and Gemini CLI — here is how they actually stack up, strengths and weaknesses, which one wins, and why the best harness of all belongs to a tool we deliberately left out of the code study: Claude Code.

The Coding Agent Shootout and the Best Harness We Could Not Read

{/* banner prompt: A bright and airy editorial illustration of a modern library where shelves are filled with organized files labeled "SKILLS" and "MEMORY," with a friendly curator carefully sorting and archiving them, using a sunny palette of warm yellows, light blues, and clean whites with bold, confident shapes and an isometric perspective. */}

In the [first post](/blog/2026/06/13-codex-pi-and-hermes-three-agents-three-species/) we compared three agents; in the [second](/blog/2026/06/13-what-an-ai-agent-harness-actually-does/) we read six harnesses and found they had all evolved the same five organs, differing in only two places. One was how much they sandbox. The other was **what survives between sessions** — and we bet that's where the next round of competition happens, because raw per-session capability is converging fast.

This post is about that second gap. It has two halves that get conflated constantly: **skills** (knowledge the agent can apply) and **memory** (facts that persist). We'll look at what each actually is in the code, why a capable model still needs both, why a human still has to write some of it by hand, and what "self-improving" means once you delete the marketing copy. Same six codebases as before, plus a news story you've probably seen.

## A skill is just a file the agent reads when it needs to

Start with the mechanism, because it's simpler than the word suggests. Across Pi, Hermes, OpenCode, Kimi Code, Gemini CLI and Codex, a "skill" is the same thing: a directory containing a `SKILL.md` file — a bit of YAML frontmatter (a name, a one-line description) followed by markdown instructions, optionally bundled with `scripts/`, `templates/` and `references/`. All six landed on a shared open format, [agentskills.io](https://agentskills.io), independently enough that the convergence is the story.

The clever part is how a skill enters the model's attention. You cannot paste a hundred procedures into every prompt — it would blow the context budget and dilute the model's focus (the context-engineering problem from the last post). So skills use **progressive disclosure**: only the skill's *name and one-line description* sit in the system prompt — a few dozen tokens each — and the agent loads the full file on demand when a task matches. Pi calls it with `/skill:name`; Hermes ships 73 skills in its prompt index and reads the body via `skill_view`; Gemini gates the load behind a consent prompt. Either way, an agent can carry a library of hundreds of procedures at a standing cost of almost nothing, paying the full token price only for the one it actually opens.

## Why a capable model still needs them

The obvious objection: today's models can already write a deploy script or summarise a PDF. Why bother? Because capability and reliability are different things, and skills buy four things the raw model doesn't have:

**Knowledge it cannot possibly know.** The model knows how to deploy *a* web app. It does not know that *yours* is `make release ENV=prod`, that you bump the changelog first, and that staging DNS takes five minutes so don't panic. None of that is in any training set. A skill is where private, environment-specific knowledge lives — and in practice that's most of what real skills contain.

**Variance reduction.** Ask a model to do a ten-step task cold and it succeeds most of the time, a different way each run. A skill pins the procedure — the exact commands, the order, the known failure modes — turning a fuzzy capability into a repeatable one. It's the difference between a brilliant temp and an employee with the operations manual.

**Bundled executables, not regenerated code.** A skill ships its `scripts/`. Instead of the model re-deriving a 200-line conversion script every time — slow, token-hungry, occasionally subtly wrong — it runs the vetted one that worked last month. Its job shrinks from *author the solution* to *invoke the solution*.

**Punching above the model's weight.** A mid-tier model with a precise skill routinely beats a frontier model improvising, because the hard part — knowing the right procedure — has been moved out of the weights and into a file. That's a direct cost lever: routine work can run on cheaper models if the skill carries the expertise.

The honest caveat: for a one-off task well within the model's competence, a skill is pure overhead. Skills pay off on the *recurring*, *local*, and *high-variance* work — which is exactly why every harness puts "add a skill" as the cheapest rung on its capability ladder, below tools, plugins and core changes.

## Memory, surprisingly, is also just files

If skills are *how to do things*, memory is *what's true* — your name, your stack, the decision you made last Tuesday. And here the codebases delivered the most counterintuitive finding of the whole study: **the industry is converging on memory being plain markdown files the agent edits.**

The sharpest example is Gemini CLI. Earlier versions had a `save_memory` tool. It's gone — the current system prompt literally states *"There is no `save_memory` tool"* and instructs the model to edit `GEMINI.md` / `MEMORY.md` directly, like a person updating a notes file. Hermes does the same with a thin `memory` tool writing to a hard-capped `MEMORY.md` (2,200 characters) and a separate `USER.md` profile (1,375 characters) — the caps are deliberate, forcing the agent to *curate* rather than hoard. Context files like `AGENTS.md`/`GEMINI.md` are the read side of the same idea: human- or agent-authored facts, loaded into every relevant prompt (Gemini even does it just-in-time, injecting a directory's `GEMINI.md` only when a tool touches that directory).

Why did files win over databases and vector stores? Because they're inspectable, editable, diffable, version-controllable, and they fail in obvious ways. You can open your agent's memory in a text editor and see exactly what it thinks it knows — and fix it. The one prominent exception proves the rule: Hermes can plug in [Honcho](https://github.com/plastic-labs/honcho), a dialectic user-modelling service that builds a theory-of-mind of the user behind the scenes. It's more powerful and far less legible — and notably it's an *optional provider*, not the default. For most purposes, a capped markdown file the agent rewrites is the whole feature.

## Why a human still has to write some of it

Now the news story. Through 2026 there were reports of laid-off engineers being asked, as a condition of leaving, to *write up their skills* — to serialise what they knew into the kind of files we've just described. It sounds dystopian, and partly it is, but it also points at something real about the limits of self-extraction.

An agent can only learn from what flowed through its own context — its transcripts, its tool outputs, its own successes and failures. That's a tiny, biased slice of what's actually known. The departing engineer holds the rest, and four kinds of it can never be mined from logs:

- **Tacit knowledge with no digital trace.** Why service X must restart before service Y, learned during a 2 a.m. outage. Which "refactorable-looking" module is secretly load-bearing. It lives in a head, not a file.
- **Negative knowledge.** The migration that was tried and rolled back; the flag that caused a near-miss. Failures get cleaned up, not documented, so they leave nothing for an agent to learn from. The most valuable expertise is often *what not to do* — and it's invisible.
- **Distillation versus induction.** When an agent writes a skill from experience, it's inducting a rule from the handful of episodes it happened to live through, unable to tell essential from incidental. A veteran writes the rule from a thousand episodes, exception clauses included.
- **Cost of learning the hard way.** An agent could rediscover much of this by trial and error — but in production those trials are outages and corrupted data. Human-authored skills are one-shot transfer at zero production risk.

So the realistic division of labour — and, again, the one the codebases actually implement — is three-part: **humans seed, agents accumulate, something curates.** Humans write the foundational skills carrying tacit and hard-won knowledge; the agent appends what it learns within its own experience; and a review step stops the library filling with confident-but-wrong inductions. The grim irony of the layoff story is that the engineers are being asked to do the *seeding* step for their own replacement — and the quality of that handoff depends entirely on knowledge that was never extractable by force in the first place.

## What "self-improving" actually means in code

This is the phrase that sells agents and means the least — so here is what it reduces to once you read the implementation. Only two of the six harnesses ship a genuine learning loop, and neither is magic.

**Hermes is the fullest version.** Its loop has three moving parts, all visible in `agent/curator.py` and `tools/skill_usage.py`:

1. **Write.** After a complex task, a post-task nudge encourages the agent to crystallise the workflow into a new skill via a `skill_manage` tool. Skills it later reuses get patched in place — "skills self-improve during use." Every load, view and patch bumps a counter in a `.usage.json` file. That usage signal is the raw material for everything downstream.
2. **Age.** Pure bookkeeping, no AI: a skill goes `active` → `stale` after 30 days unused → `archived` after 90, and any use resurrects it. Nothing is ever deleted, only moved aside.
3. **Curate.** A background "librarian" wakes roughly weekly, but only when the agent has been idle for hours, and forks a *headless* review agent over the agent-created skills. Its prompt mandates **consolidation over pruning** — merge five narrow skills into one umbrella skill, demote details into `references/` — and it must emit a structured plan that's reconciled against the actual audit trail. Before it touches anything it snapshots every skill and cron job for rollback.

Read that list again and notice what "self-improving" turned out to be: a usage counter, a state machine on timestamps, and a periodic LLM review pass with a backup-and-rollback safety net. It's genuinely valuable — the agent's knowledge compounds across months instead of resetting every session — but it's *engineering*, not emergence.

**Gemini CLI shows the safer, human-gated variant.** Its "Auto Memory" runs a background miner over the conversation transcript, extracts reusable procedures, and writes them out as **draft `SKILL.md` files into a project-local inbox** — where a human reviews and approves them before they ever become active. That's the three-part division of labour made literal: the agent proposes, the human disposes.

<Table data={
    {
        headers: ['Capability', 'Where it lives', 'Who writes it', 'Self-improves?'],
        rows: [
            ['Skills (procedures)', 'SKILL.md files (agentskills.io)', 'Human-seeded, agent-extended', 'Hermes: yes, via curator. Others: manual'],
            ['Memory (facts)', 'Capped markdown (MEMORY.md / USER.md)', 'Mostly the agent, editable by hand', 'Re-curated on a cap, not grown'],
            ['User model', 'Honcho service (optional)', 'Inferred from interaction', 'Yes, but opaque'],
            ['Context files', 'AGENTS.md / GEMINI.md', 'Human, loaded per-project/dir', 'No — static input'],
        ]
    }
} />

## The part nobody markets: curation and trust

Two unglamorous truths fall out of all this. First, **a knowledge store that only grows, rots.** Without the aging-and-consolidation machinery, a self-writing agent fills its own library with near-duplicates and stale procedures until the index is noise. Half of Hermes's "learning loop" is really a *forgetting* loop — the state machine that ages skills out and the curator that merges them. Memory's character caps are the same instinct: a small, curated store beats a large, sprawling one.

Second, **a skill is executable knowledge, which makes it an attack surface.** A `SKILL.md` can carry `scripts/` and shell snippets; an agent that installs skills from the internet is running other people's instructions with your permissions. Hermes takes this seriously enough to ship a dedicated scanner (`tools/skills_guard.py`) that runs 90-plus regex checks over any skill before install — for exfiltration, prompt injection ("ignore previous instructions"), reverse shells, credential reads, even invisible Unicode — and refuses anything that scores "dangerous." The moment knowledge becomes self-writing and shareable, provenance and review stop being optional.

## What this means

For agent *users*: the question that predicts whether an agent gets better over time isn't "how smart is the model" — it's "what survives between sessions, and who curates it." An agent with great memory hygiene and a few well-written skills will, within a domain, quietly outperform a smarter agent with amnesia. And you should write some skills yourself; the highest-value ones encode exactly the local, tacit knowledge the model can never infer.

For agent *builders* — us — the takeaways are concrete and cheap to implement: store skills and memory as plain files, not a database; use progressive disclosure so the library costs nothing until used; **cap and curate**, because a forgetting loop matters as much as a learning one; gate self-written and imported skills behind review; and design for the three-part division of labour from day one — humans seed, the agent accumulates, a curator keeps it honest. None of it requires a better model. All of it is the difference between an agent that resets every morning and one that's been working for you for a year.

*This is the third post in our series on AI agent architecture, drawn from reading six open-source agent harnesses. If you build agents, the source is the best teacher we found — start with Pi for the minimal core and Hermes for the learning loop.*

c1491bb77889eb27a761f3addbe91487c78f249199fbb222e3c7ec63433ecba9

How AI agents capture knowledge through skills and memory, why humans still write some skills, and what self-improving means in code — based on the source of six open agent harnesses.

Self-improving AI agents sound like marketing — until you read the code. This is how agents capture knowledge as skills and memory, why a human still has to write some of it down, and what self-improving actually means once you strip away the hype: bookkeeping, an LLM review pass, and a human gate.

Skills and Memory: How AI Agents Actually Learn

{/* banner prompt: An intricate technical blueprint of an 'AI Agent Harness' on off-white paper, showing a central processor labeled 'REASONING CORE' connected by clean, precise lines to distinct modules labeled 'Context Compaction,' 'Tool Contract,' 'Safety Boundary,' and 'Event-Sourced Persistence,' with dimensional annotations and detailed callouts, schematic diagram style. */}

In the [previous post](/blog/2026/06/13-codex-pi-and-hermes-three-agents-three-species/) we compared three very different AI agents and arrived at a claim that deserves its own article: **the model is a commodity; the harness is the product.** This post is the deep dive we promised.

Since then we've gone further and read the source code of **six** agent harnesses end to end: [Pi](https://pi.dev/), [Hermes Agent](https://github.com/NousResearch/hermes-agent), [OpenCode](https://github.com/sst/opencode), Moonshot's [Kimi Code](https://github.com/MoonshotAI/kimi-code), OpenAI's [Codex CLI](https://github.com/openai/codex) (the Rust harness is fully open even though the models are not), and Google's [Gemini CLI](https://github.com/google-gemini/gemini-cli). Six independent teams, four languages, wildly different products — and, as we'll see, they have all converged on the same set of organs. That convergence is the strongest evidence there is for what a harness actually has to do.

## The naive agent is four lines long

Strip everything away and an AI agent is this:

```
while not done:
    response = model(context)
    results  = execute(response.tool_calls)
    context += response + results
```

You can write that in an afternoon, point it at a frontier model, and it will genuinely work — for about ten minutes. Then reality arrives. The context window fills up. A tool dumps 80,000 tokens of build log into the conversation. The model edits the wrong file because two files have similar names. A network blip kills a forty-minute run. The model gets stuck retrying the same failing command forever. The user wants to say "no, not that way" *while* it's working. And someone, eventually, asks the agent to `rm -rf` something it shouldn't.

Every line of the six codebases we read exists because of that second ten minutes. The model supplies the reasoning per turn; the harness decides **what the model sees, what it can touch, what survives, and what happens when things go wrong.** Those four decisions are most of an agent, and none of them are AI.

## Job 1: Deciding what the model sees

The single biggest misconception about agents is that the model "has" the conversation. It doesn't. Every turn, the harness *constructs* what the model sees from scratch — and context is a scarce, expensive, attention-diluting resource. The engineering here has three layers.

**Construction.** The system prompt is assembled from parts: identity, tool descriptions, project instruction files (the `AGENTS.md` convention Codex originated, Gemini's `GEMINI.md` hierarchy), date, working directory. Skills — procedural knowledge files — are injected by *name and one-line description only*; the agent reads the full file on demand. This "progressive disclosure" pattern, now an open standard ([agentskills.io](https://agentskills.io)) adopted by Pi, Gemini CLI and others, is how an agent carries a large library of know-how at a standing cost of a few dozen tokens each.

**Compaction.** When the window fills, history must be squashed — and doing this badly is the number one cause of agents "forgetting" what they were doing. The mature harnesses treat it as a real algorithm, not a hand-wave. Pi only cuts at *valid cut points* (never in the middle of a tool call/result pair), summarises into a structured schema — goal, constraints, progress, decisions, next steps — and carries **cumulative lists of every file read and modified** across any number of squashes, so "what have I touched" is never lost. Kimi Code goes further with *cache-aware micro-compaction* that weighs the cost of invalidating the prompt cache before deciding what to squash.

**Cache discipline.** That last point deserves emphasis because it's invisible from the outside: LLM providers cache prompt prefixes, and a harness that rewrites earlier context casually pays full price for every turn. Hermes keeps its system prompt *byte-stable* for the lifetime of a conversation as a hard design invariant; Codex's own contributor guide codifies prompt-cache rules. Context isn't just curated — it's curated *append-only* whenever possible.

## Job 2: The tool contract

Tools are where the model's text meets the real world, and the contract is fussier than it looks.

**Output is budgeted.** Every harness truncates tool output with hard limits — Pi caps file reads at 2,000 lines or 50KB, grep at 100 matches, with the full output spilled to a temp file the model can fetch if it actually needs it. Without this, one chatty command destroys the context budget (see Job 1).

**Edits are engineered.** The deceptively hard tool is "edit a file." The naive approach — let the model rewrite the whole file — is slow and destructive. The six harnesses showcase three generations of answers: exact string replacement with *fuzzy fallback* (Pi normalises smart quotes, Unicode dashes and trailing whitespace before giving up); a custom patch grammar the model is trained on (Codex's `apply_patch`); and self-correcting replace, where a failed match triggers an LLM side-call to repair the edit (Gemini CLI). Each is a harness answering the same question: how do you make a probabilistic text generator perform a deterministic operation reliably?

**Execution is stateful.** Codex's "unified exec" gives the model a persistent PTY session it can write stdin to — so it can run a REPL, answer an interactive prompt, keep a dev server running. Gemini CLI parses shell commands with tree-sitter to block command substitution and enforce allowlists *before* anything runs. The shell tool in a mature harness is closer to a tiny operating system interface than a `subprocess` call.

**Scheduling matters.** Tool calls within a turn run in parallel by default in Pi — unless any tool in the batch declares itself sequential, because parallel file edits to the same file are a race. Kimi Code has a conflict-aware scheduler for exactly this. These are classic concurrency problems, just with a language model as the caller.

## Job 3: The safety boundary

Here the six harnesses genuinely diverge, and the divergence is instructive because it's *philosophical*, not technical.

**Codex treats sandboxing as architecture.** Every command runs inside an OS-enforced sandbox — Seatbelt on macOS, bubblewrap + seccomp on Linux — with the filesystem scoped and network denied by default. The clever part is the layering: the *sandbox* is the technical boundary, the *approval policy* is the human boundary, and they are configured independently. The model can request escalation, and that request is itself part of the tool schema.

**Pi and Hermes refuse to pretend.** Pi ships no sandbox and no permission popups at all, on the stated grounds that in-process "sandboxing" of an agent that runs your toolchain is security theatre — real isolation comes from the OS or a container, so use one. Hermes's SECURITY.md says the same thing in writing: OS-level isolation is the only boundary; everything else is best-effort. You may disagree with shipping that default, but the honesty is clarifying — and it's the correct *analysis* even for the harnesses that do more.

**OpenCode and Gemini CLI turn policy into data.** OpenCode's "plan mode" is not a mode at all — it's a permission ruleset (`edit: deny`, plus a carve-out directory for writing plans). Sub-agents are permission rulesets too. Gemini CLI has a TOML policy engine behind its approval modes. The lesson: don't hard-code behavioural modes; express them as declarative policy and the modes fall out for free.

## Job 4: Surviving — persistence and recovery

A forty-minute agent run accumulates state that must outlive crashes, restarts, and the model's own mistakes.

**Sessions are logs, not snapshots.** The convergent design is append-only: Pi's session is a JSONL *tree* (every entry has an `id` and `parentId`, so branching and time-travel are free); Kimi Code event-sources everything to a `wire.jsonl` you can literally replay in a visual debugger; OpenCode's v2 redesign is an event-sourced SQLite core. Nobody who has operated an agent at scale stores "current state" — they store *what happened* and derive state from it.

**The escape hatch is git.** Both OpenCode and Gemini CLI maintain a *shadow git repository* (using git's `objects/info/alternates` mechanism, so it's cheap even on huge repos) and snapshot before risky operations — giving the user `/rewind` and the agent a guilt-free undo. The agent will sometimes do the wrong thing; the harness's job is to make the wrong thing cheap.

**Failure is classified, not caught.** Mature loops distinguish rate limits from auth failures from transient network errors, with different policies for each — exponential backoff here, model fallback there (Gemini CLI automatically falls back to Flash on quota exhaustion), a hard stop elsewhere. And the most human failure mode — the agent stuck in a loop, trying the same fix forever — gets dedicated machinery: Gemini CLI detects loops and *injects corrective guidance* rather than just aborting; OpenCode escalates doom-loops into a permission ask, putting the human back in the loop exactly when the model has stopped making progress.

**Steering without breaking.** Users want to redirect a running agent without killing it. Pi's answer is two queues with different semantics: *steering* messages are injected between turns (the agent course-corrects mid-task), *follow-ups* are delivered only when it would otherwise stop. A small design, but it's the difference between an agent you collaborate with and one you can only fire-and-forget.

## Job 5: The engine/face split

The last organ is structural. Every mature harness has separated the **engine** (the loop, tools, state) from the **face** (whatever the user is looking at) behind an explicit protocol: Codex's core speaks a typed submission-queue/event-queue protocol, and the TUI, the IDE extension, `codex exec`, and the SDK are all just clients of it; OpenCode is an HTTP server with an OpenAPI spec, and the terminal UI, desktop app, web UI and GitHub bot are all generated-SDK clients; Gemini CLI routes tool confirmations through a serialisable message bus, which is what lets the same agent run interactively, headless in CI, or behind its Agent2Agent server.

This is the harness decision with the highest long-term leverage, because every future surface — IDE plugin, chat bot, CI job, another agent — is a client you haven't written yet.

## Convergent evolution, and the two real differences

Six teams, working independently, in Rust, TypeScript, and Python, shipped the same five organs:

1. **Context engineering** — constructed views, progressive disclosure, structured compaction, cache discipline
2. **A tool contract** — schemas, output budgets, engineered edits, concurrency rules
3. **A policy boundary** — permissions and/or sandboxing, increasingly expressed as data
4. **Append-only persistence with cheap undo** — event logs plus git checkpoints
5. **An engine/face protocol** — one core, many clients

When independent evolution produces the same anatomy, that anatomy is the job description. If you are building an agent, this list *is* your roadmap.

What actually differentiates the six is just two things. **Sandboxing depth** — only Codex treats OS-level isolation as load-bearing architecture; everyone else delegates it to you. And **what persists across sessions** — only Hermes ships a learning loop (self-written skills, a curating background process, searchable memory of every past conversation); the rest start each session from zero plus instruction files. Our bet, for what it's worth: the second gap is where the next round of competition happens, because per-session capability is converging fast.

## Why the model doesn't make this go away

The obvious objection: models keep getting better — won't a sufficiently smart model just not need all this scaffolding?

The evidence so far says no, for a structural reason: **the harness governs the realised fraction of the model's capability, and that fraction is set by information logistics, not intelligence.** A smarter model still cannot reason about a file that was truncated out of its context, still cannot remember a decision that compaction discarded, still cannot un-delete a directory, and still burns money if its cache is invalidated every turn. Model progress raises the ceiling; the harness determines how close to the ceiling you operate. If anything, better models *raise* the stakes on the harness — a model capable of an eight-hour autonomous run is only useful inside a harness whose compaction, checkpointing and recovery can survive eight hours.

The strongest proof is the vendors themselves. OpenAI co-designs the Codex models *with* the Codex harness — the models are trained to use its specific patch grammar and escalation protocol — and the same model measurably performs differently in different harnesses. If the harness were incidental, none of that investment would make sense.

## What this means in practice

If you're *choosing* an agent: stop comparing models and start comparing harnesses. Ask how it compacts, what it does when an edit fails, whether you can rewind, what's sandboxed, and what survives a restart. Those answers predict your daily experience far better than a benchmark score.

If you're *building* one — as we are — the encouraging news is that the harness is the one layer where a small team can genuinely compete. You cannot train a frontier model; you can absolutely write a better loop. Pi's entire agent runtime is about 2,700 lines of TypeScript. The five organs above are a few thousand lines of careful, testable, boring code — and all six of the codebases we studied are open. Read Pi for the cleanest minimal core, Codex for sandboxing and the protocol-first split, OpenCode for the client/server architecture, Kimi Code for event sourcing, Gemini CLI for recovery engineering, and Hermes for memory and self-improvement.

The model reasons. Everything else is the harness — and everything else, it turns out, is most of the work.

*Next in the series: skills and memory — how an agent captures knowledge, why a human still has to write some of it down, and what "self-improving" actually means in code.*

518a1d15b1f1ab30cb71fc640a0e70994802c0a59c7f26229c46336994f5723f

A deep dive into the AI agent harness — context management, tool contracts, safety boundaries, persistence and recovery — based on reading the source code of six open agent harnesses.

We read the source code of six AI agent harnesses — Pi, Hermes, OpenCode, Kimi Code, OpenAI's Codex CLI and Google's Gemini CLI. This is what the harness really does for the model, why almost every agent failure is a harness failure, and why the engineering lives there, not in the model.

What an AI Agent Harness Actually Does

{/* banner prompt: A digital paper collage depicting three different AI agents on a textured craft paper background, flat-lay perspective: one is a single, sharp-edged piece of black cardstock with an embossed circuit pattern; another is a deconstructed set of neatly cut geometric paper shapes; the third is a layered, growing scrapbook of many small, overlapping paper snippets, tangible textures, warm natural lighting --ar 16:9 */}

Put OpenAI's Codex, Pi from pi.dev, and Nous Research's Hermes Agent side by side and the first reaction is: why are these three even in the same article? One is a commercial product from the biggest AI lab in the world. One is a minimalist open-source toolkit largely written by a single developer. One is a sprawling Python personal assistant that lives in your Telegram.

That is exactly why they belong together. We recently spent some time reading the source code of Pi and Hermes (both are open source — we cloned and analysed them properly, not just the READMEs) and the public documentation of Codex (whose CLI is open, but whose brains are not). The three of them are so different that the differences themselves become a map. If you understand why these three agents are built the way they are, you understand most of the AI agent landscape — what each type of agent *is*, what it *can do*, and which one can actually *help you*.

## First, what is an "agent" anyway?

Strip away the hype and every AI agent is the same machine: a loop that sends context to a language model, reads back text and tool calls, executes those tools (run a command, edit a file, search the web), feeds the results back in, and repeats until the job is done.

The model supplies the reasoning. Everything else — what the model sees, what it is allowed to touch, what survives between sessions, what happens when things go wrong — is supplied by the software wrapped around it. That wrapper is called the **harness**, and the three agents in this article are, more than anything, three very different opinions about what the harness should be.

## Codex: the vertically-integrated product

Codex is what you get when a frontier lab builds the whole stack: proprietary models (the GPT-5.x-Codex family) co-designed with the harness, delivered through every surface you might work in — terminal CLI, IDE extensions, a desktop app, mobile, a cloud service that runs tasks in parallel containers, and a GitHub bot that reviews your pull requests when you tag `@codex`.

Its strengths come from that integration:

- **Raw coding capability.** The Codex models lead or sit near the top of agentic coding benchmarks (SWE-bench Verified, Terminal-Bench), and they are tuned for long autonomous runs — hundreds of tool calls without a human nudge.
- **A real security model.** Codex is the only one of the three that ships OS-enforced sandboxing by default: Seatbelt on macOS, bubblewrap on Linux, network access denied unless you opt in. There is a clean separation between the *sandbox* (a technical boundary) and the *approval policy* (when to ask a human). This is what a production-grade safety story looks like.
- **Token efficiency.** Community comparisons consistently report it doing comparable work with two to three times fewer tokens than its rivals — which matters when you pay by subscription tier.

The trade-off is just as clear: the models are closed, deprecations are forced on OpenAI's schedule, the best models are sometimes gated behind expensive tiers, and the extension ecosystem (hooks, skills, plugins arrived only in 2026) is young. You are renting a very good employee from a company that can change the terms at any time.

**Codex is the answer to:** "I want the strongest coding agent today and I don't need to own it."

## Pi: the harness as a library

Pi (from [pi.dev](https://pi.dev/)) is the opposite philosophy taken seriously. It ships **zero models**. It is a TypeScript monorepo of four packages — a unified LLM API over roughly forty providers, a tiny agent runtime (~2,700 lines), the coding-agent CLI, and a terminal UI framework — and its core design rule is to refuse features. No MCP. No built-in sub-agents. No permission popups. No plan mode. No to-do lists. The author has written publicly about why each of these is excluded; everything on that list can be added back as a TypeScript extension in about a hundred lines.

Reading Pi's source is the best education in agent engineering we have found anywhere:

- **Sessions are an append-only tree.** Every entry in the session file has an `id` and a `parentId`, so branching, forking, and time-travelling through a conversation cost nothing — no database, no file rewrites.
- **Compaction is engineered, not hand-waved.** When the context window fills, older history is summarised into a structured format (goal, progress, decisions, next steps) with cumulative lists of every file read and modified — so the agent never forgets what it has touched, no matter how many times the history is squashed.
- **The provider layer absorbs chaos.** Thinking levels, tool-call ID formats, cache semantics and reasoning quirks of ~30 providers are normalised behind one streaming interface, so you can switch models mid-conversation without breaking anything.

The trade-offs are deliberate: there is no sandbox at all (you are expected to containerise it yourself), and the project carries a bus factor of roughly one. It is a power tool, with everything that implies.

**Pi is the answer to:** "I want to own my agent, choose my models, and understand every moving part — or build my own agent on proven foundations." (Not a hypothetical: OpenClaw, one of the most popular open personal assistants, is built on Pi's packages.)

## Hermes: the agent that lives with you

Hermes Agent, by Nous Research, is not really a coding agent at all — and that is the point. It is a **persistent personal assistant**: a Python daemon that connects to more than twenty messaging platforms (Telegram, Discord, Slack, WhatsApp, Signal, Matrix, WeChat...), executes work on any of six backends (your machine, Docker, SSH, or serverless clouds like Modal where an idle agent hibernates for fractions of a cent), runs scheduled jobs while you sleep, and — its signature feature — **improves itself**.

Hermes closes a loop the other two don't have:

- It writes its own **skills** (procedural knowledge captured as markdown files) from experience.
- A background **curator** periodically reviews those skills — consolidating, patching, archiving — like a librarian maintaining the agent's own operations manual.
- A pluggable **memory** system builds a model of who you are across sessions, and full-text search over every past conversation means it can ask itself "have I solved this before?"

Where Codex and Pi start every session from zero (plus whatever instructions you hand them), Hermes compounds. The costs are equally visible in the code: a huge attack surface (twenty-three platform adapters all parsing untrusted input), a single SQLite file as the global state store, and general-purpose breadth instead of coding depth — no IDE integration, no native git-worktree workflow. Its `SECURITY.md` is refreshingly honest: OS-level isolation is the only real boundary; everything else is best-effort.

**Hermes is the answer to:** "I want an assistant that is always on, reachable from my chat apps, and gets better the longer it works for me."

## The same machine, three species

<Table data={
    {
        headers: ['', 'Codex', 'Pi', 'Hermes'],
        rows: [
            ['What it is', 'Commercial coding agent', 'Open agent toolkit', 'Persistent personal assistant'],
            ['Lives in', 'Terminal, IDE, cloud, GitHub', 'Your terminal', 'Your chat apps, your servers'],
            ['Models', "OpenAI's, closed", 'Any of ~40 providers', 'Any OpenAI-compatible provider'],
            ['Memory between sessions', 'Preview feature', 'None (by design)', 'Its defining feature'],
            ['Safety', 'OS sandbox, default on', 'None — bring your own container', 'OS isolation via exec backends'],
            ['Extensibility', 'Config, skills, MCP, plugins', 'Full TypeScript runtime', 'Skills, plugins, MCP server'],
            ['Biggest risk', 'Vendor lock-in', 'Bus factor of one', 'Security surface'],
            ['Best at', 'Hard coding tasks, today', 'Being understood and owned', 'Compounding over months'],
        ]
    }
} />

Three lessons fall out of this comparison, and they generalise well beyond these three projects.

**1. The model is a commodity; the harness is the product.** All three can call the same class of frontier models. What you are actually choosing between is harness decisions: how context is managed, what tools look like, what persists, what is sandboxed. Nearly every agent failure you will ever observe — the agent "forgetting" the goal, going off the rails late in a session, missing the error that contained the answer — is a harness failure, not a model failure.

**2. "Agent" is not one product category.** A coding agent optimises for depth on a workstation: tight tool loops, repository awareness, review workflows. An assistant agent optimises for breadth and time: reachability, scheduling, memory, identity. The harness decisions that make one excellent make the other worse. Asking "which agent is best?" without specifying the species is like asking whether a truck is better than a motorbike.

**3. Memory and skills are where the next differentiation happens.** Codex represents peak *per-session* capability; Hermes represents the bet that *across-session* learning matters more. A skill — a piece of captured procedural knowledge the agent can load on demand — lets a mid-tier model with the right manual outperform a frontier model improvising. The agent that remembers your environment, your conventions, and its own past mistakes eventually beats the smarter agent with amnesia.

## So which one helps *you*?

- **You write code for a living and want results now:** Codex (or its direct competitors). Bundled with a ChatGPT subscription, sandboxed by default, strongest models.
- **You are a tinkerer, you care about model choice and data ownership, or you are building your own agent:** Pi. Read its source even if you never run it — the session-tree and compaction designs alone are worth the afternoon.
- **You want a personal assistant rather than a pair programmer — something that triages, schedules, researches, and remembers:** Hermes, or its TypeScript cousin OpenClaw (which, fittingly, is built on Pi).

And if you are building agents rather than just using them, the real takeaway is to read Pi and Hermes side by side: one shows you how to build the *engine*, the other shows you how to build the *life* around it. The whole field is currently busy discovering that both halves matter.

*This post is the first in a series from our research into AI agent architecture. Next up: a deep dive into what an agent harness actually does, and why it — not the model — is where the engineering lives.*

d9b8ad2cb5146c2fe5fb3e54ff0d91cda00900dddb9d58fde67ad40b5c3d9dee

A code-level comparison of OpenAI Codex, Pi (pi.dev) and Nous Research's Hermes Agent, using their differences to explain the different types of AI agents.

Comparing OpenAI Codex, Pi (pi.dev) and Hermes Agent by Nous Research — an odd trio at first glance, but their differences map out the whole landscape of AI agents: what each type is, what it can do, and which one can help you.

Codex, Pi and Hermes: Three Agents, Three Species

{/* banner prompt: A bright editorial illustration of a wall covered in colorful rounded tag pills with words like "Golden Hour", "Rule of Thirds", "Contrapposto", one pill lifted by a hand with a floating card above it revealing a tiny glowing photograph inside, confident shapes, strong daylight tones, optimistic and clear, 16:9.

Additional instructions: make sure all the words in the image are correctly spelled */}

If you have ever built a prompt for Midjourney, Stable Diffusion, or any other text-to-image model, you have probably typed a term you only half understood. *Golden Hour* sounds nice — but is it a lighting style or a time of day? What exactly does *Rule of Thirds* do to a picture? And what on earth is *Contrapposto*?

[Tynion](/tynion), our free text-to-image prompt helper, was built to take the guesswork out of assembling prompts: click the terms you want, copy the finished prompt. But until now it had the same flaw as every prompt cheat-sheet on the internet — it gave you the **words** without showing you the **pictures**.

Tynion v2 fixes that.

## Every Term Now Explains Itself

Hover over any term in Tynion (or tap the ⓘ on mobile) and a card pops up with two things:

1. **A plain-English definition** — written for someone who has never heard the term, covering what the viewer sees and when to use it.
2. **A real example image** — generated specifically to demonstrate that one term and nothing else.

There are nearly **1,000 terms** in the new vocabulary, and every single one has a definition. Here is what that looks like in practice.

### Lighting, Demonstrated

The trick that makes the example images useful is consistency: within a group, every image uses the **same neutral base subject** — a person in plain casual clothing against a plain background — so the only thing that changes between images is the term itself.

![Studio Lighting — controlled softbox light, clean and even](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/apps/tynion/terms/studio-lighting.jpg)
*Studio Lighting: controlled artificial light from softboxes, shaped to flatter the subject. Clean, polished, professional.*

![Cinematic Lighting — moody directional light with dramatic falloff](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/apps/tynion/terms/cinematic-lighting.jpg)
*Cinematic Lighting: the same kind of subject, completely different mood — directional, warm, with deep shadow falloff.*

![Golden Hour — warm low-sun light with long soft shadows](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/apps/tynion/terms/golden-hour.jpg)
*Golden Hour: the hour after sunrise or before sunset. Warm, soft, low-angle light and long gentle shadows.*

![Rembrandt Lighting — the signature triangle of light on the shadowed cheek](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/apps/tynion/terms/rembrandt-lighting.jpg)
*Rembrandt Lighting: look closely at the shadowed cheek — that small triangle of light is the whole technique, borrowed from the painter himself.*

Once you have seen these four side by side, you will never confuse them in a prompt again.

### Composition: So *That* Is the Golden Ratio

Composition terms are the ones people bluff the most. Tynion v2 adds a whole **Composition** group — Rule of Thirds, Golden Ratio, Golden Spiral, Leading Lines, Negative Space, Frame Within a Frame, and more — each with an example image built to make the technique obvious.

![Golden Ratio — a lighthouse placed on the classical 1:1.618 proportion](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/apps/tynion/terms/golden-ratio.jpg)
*Golden Ratio: the subject sits slightly off-centre on the classical ~1:1.618 proportion. Balanced and harmonious without feeling rigidly centred.*

![Rule of Thirds — key elements on the third-lines of the frame](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/apps/tynion/terms/rule-of-thirds.jpg)
*Rule of Thirds: the frame divided into a 3×3 grid, with the key elements sitting on the lines. The simplest cure for dead-centre composition.*

### Poses, Finally With Pictures

v2 also adds a proper **Pose & Posture** group — 36 poses from classical to dynamic. This was the most requested gap in v1: pose names are vivid to artists and meaningless to everyone else.

![Contrapposto — weight on one leg, hips and shoulders counter-tilted](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/apps/tynion/terms/contrapposto.jpg)
*Contrapposto: weight on one leg so the hips and shoulders tilt in opposite directions — the relaxed S-curve that classical sculptors loved.*

![Power Stance — feet planted wide, shoulders squared](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/apps/tynion/terms/power-stance.jpg)
*Power Stance: feet planted wide, shoulders squared to the camera. Instant confidence, popular in hero and fashion shots.*

## What Else Is New in v2

The visual glossary is the headline, but the whole app was rebuilt around it:

- **A much bigger vocabulary** — from ~300 terms to nearly 1,000, including new groups for **Focus & Lens** (Bokeh, 85mm Portrait Lens, Anamorphic), **Color & Palette** (Teal and Orange, Pastel, Sepia), and **Film & Medium** (Kodak Portra 400, Polaroid, CineStill 800T).
- **A data cleanup** — Golden Hour, Bokeh, and Depth of Field were misfiled as "camera angles" in v1 (guilty as charged). The camera groups are now properly split into Shot Framing, Camera Angle, and View Direction.
- **An editable final prompt** — the assembled prompt is now a real text box you can tweak by hand before copying.
- **🎲 Inspire Me** — one click picks a coherent random combination across subject, lighting, composition, framing, and style. Great for breaking a blank-page moment.

## How the Example Images Were Made

A small confession of dogfooding: every example image was generated by the same kind of pipeline Tynion helps you build prompts for. A script walks the glossary, builds a demonstration prompt per term — fixed base subject, one variable — and feeds it to an image model (Gemini's Imagen, in this batch). The exact prompt used for every image is saved right next to it, so if you want to know how the *Rembrandt Lighting* demo was made, the recipe is [a click away](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/apps/tynion/prompts/rembrandt-lighting.md).

Generating nearly a thousand demonstration images taught us a few things about rate limits and retry logic — but that is a story for another post.

## Try It

Tynion is free and runs entirely in your browser — no signup, nothing stored: **[tyolab.com/tynion](/tynion)**

Hover around. The glossary is the kind of thing that is hard to stop playing with once you start — and if you spot a term that deserves a better explanation (or a missing term entirely), the feedback link at the bottom of the page goes straight to us.


0b1a1d20ae0e06dcac466d7ddbb90ccc211edc7afafb37a98e5b4043ee2e092e

Tynion v2 turns a tag-picking prompt builder into a visual glossary for AI image generation. Every lighting style, camera angle, composition rule, and pose now ships with a definition and an AI-generated example image — same base subject, one variable changed — so you can see what a term does before you put it in your prompt.

Tynion, our text-to-image prompt helper, just got its biggest update yet: nearly 1,000 prompt terms, and every one of them now explains itself. Hover over any term — Golden Hour, Rule of Thirds, Contrapposto — and a card pops up with a plain-English definition and a real example image showing exactly what that term does to a picture.

Tynion v2: See What Every Prompt Term Looks Like

{/* banner prompt: A paper collage illustration with a tangible, handcrafted feel, showing layered cutouts of a browser window (AI agent) and a network file cabinet, connected by a paper bridge with a keyhole representing a secure API, with visible paper textures and soft drop shadows to create depth, in a palette of muted craft paper colors, 16:9. */}

## The Problem

**Claude Cowork** is a desktop application, but it ships with a built-in sandbox. You can toggle the sandbox off in settings — and technically, it could access an SMB/CIFS network share on the same machine or LAN. Yet even with it disabled, accessing a network drive remains out of reach. This is not an oversight; it is a deliberate design decision by Anthropic. An agent with unconstrained access to mounted network shares can do real damage if it misinterprets an instruction or hallucinates a path — deleting a client folder, overwriting an authoritative document, or crawling terabytes of data it was never meant to touch.

The constraint is reasonable. But it creates a practical gap: many businesses keep authoritative files on a local network share — accounting records, client folders, scanned documents. If the AI agent cannot reach those, you end up copy-pasting content manually, which defeats the purpose.

The solution is not to fight Anthropic's design — it is to work within it. An **HTTP API in front of your file share** is something every AI agent already knows how to call. You define exactly which operations are exposed (list, read, write — but not delete). The agent gets the access it needs; you keep meaningful guardrails in place.

## The Solution: HTTP File Server as a Gateway

The key insight is that **any HTTP server sitting in front of your network share immediately makes it universally reachable** — from any network, any device, any AI agent that can make HTTP requests. No VPN. No SMB port forwarding. No credentials beyond a bearer token.

We used [FileBrowser Quantum](https://github.com/gtstef/filebrowser), a self-hosted file manager that exposes a clean REST API. It mounts your network shares as named "sources" and lets you list, read, search, download, and write files through standard HTTP calls.

---

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│  Office / LAN                                               │
│                                                             │
│  \\fileserver\data   (SMB/CIFS)                             │
│          │                                                  │
│          │  mount                                           │
│          ▼                                                  │
│  Linux VM  /srv/nas_data                                    │
│          │                                                  │
│          │  volume mount                                    │
│          ▼                                                  │
│  ┌──────────────────────┐                                   │
│  │  FileBrowser Quantum │  :8080                            │
│  │  Docker container    │                                   │
│  └──────────┬───────────┘                                   │
│             │                                               │
│             ▼                                               │
│  nginx  :443  ──── Cloudflare ────▶  files.example.com     │
└─────────────────────────────────────────────────────────────┘
                                            │
                         ┌──────────────────┤
                         │                  │
                         ▼                  ▼
              Claude Code / API      Claude Desktop (Windows)
              (direct HTTP calls)    ┌────────────────────┐
                                     │  filebrowser-mcp   │
                                     │  local Node.js     │
                                     │  stdio MCP server  │
                                     └────────────────────┘
```

The file server host can be set up in two ways:

- **Cloud VM with VPN or peering into the LAN** — the VM mounts the SMB share over a private tunnel; only the HTTPS endpoint is public-facing.
- **Office server with a port exposed** — FileBrowser runs directly on a machine that already has access to the share; nginx (or a reverse proxy like Cloudflare Tunnel) exposes only port 443 to the internet.

Either way, FileBrowser Quantum runs as a Docker container and mounts the share as a local path. nginx sits in front, terminates TLS via Cloudflare, and enforces an IP allowlist. The whole stack is reachable at `https://files.example.com`.

AI agents on any network — a cloud VM, a remote laptop, a Windows desktop — reach the file server through that HTTPS endpoint.

---

## Setting Up FileBrowser Quantum

### 1. Mount the SMB share on the Linux host

```bash
# /etc/fstab entry
//192.168.1.100/data  /srv/nas_data  cifs  credentials=/etc/smbcredentials,uid=1000,gid=1000,file_mode=0664,dir_mode=0775  0  0
```

The `uid`/`gid` and mode flags are important — they ensure the Docker container (running as a non-root user) can both read and write files.

### 2. docker-compose.yaml

```yaml
services:
  filebrowser:
    image: gtstef/filebrowser:stable
    container_name: filebrowser
    ports:
      - "8080:80"
    volumes:
      - /srv/nas_data:/nas:rw
      - /opt/filebrowser/data:/config
    environment:
      - FB_DATABASE=/config/database.db
      - FB_CONFIG=/config/config.yaml
    restart: unless-stopped
```

### 3. config.yaml — define sources

```yaml
sources:
  - name: main
    path: /nas
```

You can define multiple sources if you have more than one share.

---

## REST API at a Glance

Once running, the API is straightforward:

```bash
TOKEN="<your-jwt-token>"

# List a folder
curl "https://files.example.com/api/resources?source=main&path=/CLIENTS" \
  -H "Cookie: filebrowser_quantum_jwt=$TOKEN"

# Read a text file
curl "https://files.example.com/api/resources?source=main&path=/CLIENTS/acme/notes.txt&content=true" \
  -H "Cookie: filebrowser_quantum_jwt=$TOKEN"

# Download a binary file (PDF, xlsx, etc.) — note: file= not path=
curl "https://files.example.com/api/resources/download?source=main&file=/CLIENTS/acme/report.xlsx" \
  -H "Cookie: filebrowser_quantum_jwt=$TOKEN" \
  --output report.xlsx

# Search by filename
curl "https://files.example.com/api/tools/search?source=main&query=tax+return+2024" \
  -H "Cookie: filebrowser_quantum_jwt=$TOKEN"

# Write a file
curl -X POST "https://files.example.com/api/resources?source=main&path=/CLIENTS/acme/summary.txt&override=true" \
  -H "Cookie: filebrowser_quantum_jwt=$TOKEN" \
  -H "Content-Type: text/plain" \
  --data "AI-generated summary goes here"
```

> **Gotcha:** The download endpoint uses `file=` as the query parameter, not `path=`. Using `path=` returns `{"status":400,"message":"no files specified"}`. This inconsistency is worth documenting — it tripped us up initially.

---

## Security Measures

Opening file server access to the internet requires deliberate controls. Here is what we applied, from most to least important.

### 1. Scoped, long-lived JWT tokens

FileBrowser Quantum supports issuing named API tokens with explicit permission sets embedded in the JWT payload. We created a dedicated `ai` user with only the capabilities the agent actually needs:

```json
{
  "Permissions": {
    "api":      true,
    "modify":   true,
    "create":   true,
    "download": true,
    "share":    true,
    "delete":   false,
    "admin":    false
  }
}
```

**Delete is disabled.** An AI agent that can delete files is a data-loss risk — one hallucinated path and important client files are gone. Write access is sufficient for almost every automation use case. If you genuinely need deletion, do it through a separate, human-triggered workflow.

The token is set to expire in ~3 years. Long enough that it does not need babysitting, short enough that it will eventually rotate.

### 2. Removing delete — the BoltDB patch

FileBrowser Quantum stores user records in a BoltDB (key-value) database. The standard HTTP API for updating user permissions (`PUT /api/users?id=N`) had a quirk in our version: it always reset all permissions to false. We had to patch the database directly.

We wrote a small Go script that:
1. Stopped the Docker container to release the exclusive BoltDB lock.
2. Opened the database directly using the `bbolt` library.
3. Located the user record by its big-endian uint64 key.
4. Decoded the JSON user record, set `permissions.delete = false`, and cleared the `tokens` field (to invalidate any old tokens that still had delete enabled).
5. Wrote the record back, verified it in the same transaction, then restarted the container.

The `tokens` field step is critical. FileBrowser embeds the full permission set inside the JWT itself. If an old token with `delete: true` is still stored in the user record, the server restores those permissions on startup — overriding your DB change. Clearing `tokens` forces the server to issue fresh tokens reflecting the updated permissions.

### 3. nginx IP allowlist

The public URL is fronted by nginx, which checks the real client IP (passed through by Cloudflare via `X-Forwarded-For`) against an allowlist:

```nginx
geo $allowed_ip {
    default         0;
    200.0.13.10    1;   # office static IP
    2004:ab8::/48   1;   # authorised IPv6 prefix
}

server {
    listen 443 ssl;
    server_name files.example.com;

    if ($allowed_ip = 0) {
        return 403;
    }

    location / {
        proxy_pass http://127.0.0.1:8080;
    }
}
```

If a request comes from an unknown IP — even with a valid token — nginx returns 403 before FileBrowser ever sees the request. This means a leaked token is useless from outside the authorised IPs.

For AI agents running on cloud VMs, add their static IPs or CIDR prefix to the allowlist.

### 4. HTTPS only via Cloudflare

All traffic goes through Cloudflare, which handles TLS termination and provides DDoS protection. The origin (nginx) only accepts connections from Cloudflare's IP ranges, so the raw VM IP is not directly reachable on port 443.

---

## Wiring It Into Claude Desktop via MCP

For Claude Desktop on Windows (where you want the agent to work with local files after downloading them), we added a local [MCP](https://modelcontextprotocol.io/) server. The MCP server is a small Node.js process that wraps the FileBrowser HTTP API as MCP tools, communicating with Claude Desktop over stdio.

`claude_desktop_config.json`:
```json
{
  "mcpServers": {
    "workspace": {
      "command": "C:\\Program Files\\nodejs\\node.exe",
      "args": ["C:\\Tools\\filebrowser-mcp\\index.js"],
      "env": {
        "FILEBROWSER_URL": "https://files.example.com",
        "FILEBROWSER_TOKEN": "<token>"
      }
    }
  }
}
```

The MCP server exposes these tools to the agent:

<Table data={
    {
        headers: ['Tool', 'Purpose'],
        rows: [
            ['fs_list_sources', 'List available drives'],
            ['fs_list_folder', 'Browse a folder'],
            ['fs_read_file', 'Read text file content'],
            ['fs_download_file', 'Download binary file to local temp folder'],
            ['fs_search_files', 'Search by filename'],
            ['fs_get_file_metadata', 'Get size / date / type'],
        ]
    }
} />

The key tool here is `fs_download_file`. Claude's `web_fetch` tool cannot pass custom headers, so it cannot supply the auth cookie. The local MCP server, running as a native process with the token in its environment, has no such restriction — it fetches the file over HTTPS and saves it to the Windows temp folder, returning the local path so Claude can then read it with normal file tools.

```javascript
async function downloadFile({ source, path, save_to }) {
  // note: download endpoint uses file= not path=
  const url = `/api/resources/download?source=${encodeURIComponent(source)}&file=${encodePath(path)}`;
  const res = await fbFetch(url);
  const buffer = await res.arrayBuffer();
  const filename = path.split("/").pop();
  const localPath = save_to || join(tmpdir(), filename);
  writeFileSync(localPath, Buffer.from(buffer));
  return { source, path, filename, local_path: localPath, size_bytes: buffer.byteLength };
}
```

---

## Why This Works Better Than Alternatives

Claude does provide built-in connectors for services like Dropbox and OneDrive. In practice, though, most organisations have multiple accounts spread across different purposes — a Dropbox for one team, an OneDrive for another, a legacy NAS for a third. Each connector needs its own authorisation flow and only covers one account at a time.

An HTTP file server flips this: you mount every share behind a single endpoint, define named sources, and the agent accesses all of them through one token and one API. No per-account OAuth dance, no switching contexts.

<Table data={
    {
        headers: ['Approach', 'Trade-off'],
        rows: [
            ['VPN', 'Requires credential management; gives the agent broad network access'],
            ['Cloud sync connectors (Dropbox, OneDrive)', 'One connector per account; multiple accounts across teams means multiple authorisation flows to manage'],
            ['Direct SMB exposure', 'SMB was not designed for internet exposure; serious security risks'],
            ['HTTP file server (this setup)', 'Single endpoint, multiple sources; scoped token; IP allowlist; works for cloud storage and on-prem shares alike'],
        ]
    }
} />

The HTTP approach is universally accessible because every AI agent — whether it is Claude Code running on a cloud VM, Claude Desktop on a Windows workstation, or an API-driven automation script — already knows how to make HTTP requests. No special drivers, no OS-level mounts, no VPN configuration.

---

## Summary

If you have an office NAS that AI agents need to reach:

1. **Deploy FileBrowser Quantum** (or any HTTP file server) on a VM that can mount the share.
2. **Issue a scoped token** — read/write, no delete, fixed expiry.
3. **Lock it down at the network layer** — nginx IP allowlist, TLS via a CDN.
4. **For Claude Desktop**, add a local MCP server that wraps the HTTP API, so the agent can download binary files it could not otherwise fetch (PDFs, spreadsheets, etc.).

The result is an AI agent that can list folders, search for files, read documents, write results back, and download any file type — from any network — without ever needing a VPN or direct NAS access.

659c4cf1a18721f124647065e90f17a06648a0fa3fa9defdd99645014a13435f

A practical walkthrough of deploying FileBrowser Quantum as an HTTP gateway over SMB network shares, issuing scoped long-lived JWT tokens, and wiring the API into Claude Desktop via a local MCP server — so AI agents can access office files without VPN or physical LAN access.

Claude Cowork (and similar browser-based AI agents) can browse web pages but cannot reach SMB/CIFS network shares. Here is how we wrapped our office NAS in an HTTP API so the agent can list, read, search, and write files from anywhere — with proper security controls.

Giving AI Agents Access to Network Drives via an HTTP File Server

{/* banner prompt: A playful 3D isometric render of a complex Rube Goldberg machine on a soft cream background, matte pastel materials, featuring a Strapi octopus logo turning a crank which sends a ball down a chute towards a Google 'G' shaped gate that is blocked by an nginx-branded lever, showing a tangled path with broken connections and error signposts. */}


"Sign in with Google" sounds easy. It's on every app. How hard can it be?

Very. Especially with Strapi 4 behind an nginx reverse proxy. I spent a full day fighting through a chain of errors, each one hiding the next. This post documents all of them so you don't have to.

## The Setup

- **Frontend:** Next.js, hosted at `https://store.example.com`
- **Backend:** Strapi 4.25.x, hosted at `https://store.example.com/api` (proxied by nginx)
- **Strapi admin:** `https://admin.example.com`
- **Reverse proxy:** nginx, doing SSL termination and forwarding to the Strapi container

Strapi has Google OAuth built into `@strapi/plugin-users-permissions`. In theory, you enable the provider in the admin panel, drop in your Google credentials, and it works. In practice, not so much.

## The Flow (What's Supposed to Happen)

Worth understanding before you debug it.

1. User clicks "Sign in with Google" on your frontend
2. Frontend redirects to `https://store.example.com/api/connect/google?callback=<your_frontend_callback_url>`
3. Strapi redirects to Google's OAuth consent screen
4. Google redirects back to `https://store.example.com/api/connect/google/callback`
5. Strapi processes the token, then redirects to your `?callback=` URL with an `access_token` param appended
6. Your frontend page at `/auth/google/callback/` receives that token and exchanges it with `/api/auth/google/callback` for a JWT
7. Done — user is logged in

Simple enough on paper. Here's what actually happens.

---

## Error #1: `ValidationError: Invalid callback URL provided`

**What you see:**

```
ValidationError: Invalid callback URL provided
details: { callback: https://store.example.com/auth/google/callback/ }
```

Strapi validates the `?callback=` parameter against the "Redirect URL to front-end app" value you set in the admin panel under *Settings → Providers → Google*. It's not a prefix match. It's not a domain match. It checks both the **origin AND the pathname** exactly.

The catch: the admin panel shows you a suggested value like `https://admin.example.com/auth/google/callback` — that's the URL being accessed through the admin subdomain. But your actual frontend callback lives at `store.example.com`, not `admin.example.com`.

**Fix:** In Strapi admin → *Providers → Google → Redirect URL to front-end app*, set it to your actual frontend callback URL:

```
https://store.example.com/auth/google/callback/
```

Note the trailing slash. If your Next.js app uses `trailingSlash: true`, the URL in Strapi must match exactly — trailing slash included.

---

## Error #2: `Cannot send secure cookie over unencrypted connection`

**What you see:**

```
Error: Cannot send secure cookie over unencrypted connection
```

This one appears in the Strapi logs after you've fixed the callback URL. `koa-session` (which Strapi uses internally) refuses to set a secure cookie because, from Strapi's perspective, the connection is plain HTTP. nginx is doing SSL termination, so by the time the request reaches Strapi, it's unencrypted.

There are two parts to the fix.

**Fix part 1 — Tell Strapi to trust the proxy.** In `src/index.js`:

```js
module.exports = {
  register() {},
  bootstrap({ strapi }) {
    // Trust X-Forwarded-Proto from nginx so koa-session allows secure cookies
    strapi.server.app.proxy = true;
  },
};
```

**Fix part 2 — Tell nginx to send the header.** In your nginx config, inside the `/api` location block:

```nginx
location /api {
    proxy_pass http://backend:1337/api;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_cache_bypass $http_upgrade;
}
```

Both are required. The header alone does nothing without `proxy = true`. And `proxy = true` without the header means Strapi still sees HTTP.

---

## Error #3: `Error 400: redirect_uri_mismatch`

**What you see:** Google's error page with "Access blocked: This app's request is invalid."

This one's subtle. There are actually two different redirect URIs involved and it's easy to confuse them.

There's the **Strapi admin display** — when you look at the Google provider settings in `admin.example.com`, Strapi helpfully shows you: *"The redirect URL to add in your Google application: `https://admin.example.com/api/connect/google/callback`"*. That's generated from the URL you used to access the admin. It might not be right.

The **actual redirect URI** that Strapi sends to Google is built from the `PUBLIC_URL` environment variable in your Strapi config. If `PUBLIC_URL=https://store.example.com`, the real URI is:

```
https://store.example.com/api/connect/google/callback
```

**Fix:** In Google Cloud Console → *Credentials → OAuth 2.0 Client → Authorized redirect URIs*, make sure you have the URI built from `PUBLIC_URL` — not just whatever the Strapi admin suggested:

```
https://store.example.com/api/connect/google/callback
```

You can keep the admin-subdomain one too if you want, but the `PUBLIC_URL`-based one must be there.

---

## Error #4: `Email is already taken`

**What you see:**

```
Email is already taken.
Back to login
```

This happens when a user already has a local account (signed up with email/password) and tries to log in via Google using the same email address. Strapi's default behaviour is to reject this — it won't link providers.

The decision here is yours: do you want to block it, or do you want to just sign them in? We went with "just sign them in." It's the sensible UX.

**Fix:** Override the providers service via a Strapi extension. Create `src/extensions/users-permissions/strapi-server.js`:

```js
'use strict';
const _ = require('lodash');

module.exports = (plugin) => {
  const originalProviders = plugin.services.providers;

  plugin.services.providers = ({ strapi }) => {
    const service = originalProviders({ strapi });
    const originalConnect = service.connect;

    service.connect = async (provider, query) => {
      try {
        return await originalConnect(provider, query);
      } catch (err) {
        if (err.message !== 'Email is already taken.') throw err;

        // Re-fetch the Google profile to get the email
        const accessToken = query.access_token || query.code || query.oauth_token;
        const providers = await strapi
          .store({ type: 'plugin', name: 'users-permissions', key: 'grant' })
          .get();
        const profile = await strapi
          .plugin('users-permissions')
          .service('providers-registry')
          .run({ provider, query, accessToken, providers });

        const email = _.toLower(profile.email);
        const existingUser = await strapi
          .query('plugin::users-permissions.user')
          .findOne({ where: { email } });

        if (!existingUser) throw err;
        return existingUser;
      }
    };

    return service;
  };

  return plugin;
};
```

Note: don't try to call `service.getProfile` directly — it's a private closure inside the providers service, not exported. You need to go through `providers-registry.run(...)` to re-fetch the profile.

---

## Error #5: `Forbidden` (403 on `/api/users/me`)

**What you see:**

The OAuth exchange succeeds. You get a JWT. But when your frontend tries to fetch the user profile — `GET /api/users/me` with that JWT — it gets a 403. The user gets bounced to "Back to login."

The `me` endpoint permission in Strapi admin is almost certainly fine. The real culprit is the **user's role**.

In Strapi, users have a role — either `Authenticated` or `Public`. A user with the `Public` role can't access authenticated endpoints, even with a valid JWT. Strapi's `fetchAuthenticatedUser` does a role lookup, and permissions are evaluated against that role.

How does a user end up with the `Public` role? It can be a one-off DB migration artifact, a bug in an older Strapi version, or manual data entry. Check like this:

```sql
SELECT u.id, u.email, r.name as role
FROM up_users u
JOIN up_users_role_links l ON u.id = l.user_id
JOIN up_roles r ON l.role_id = r.id
WHERE u.email = 'your@email.com';
```

If you see `Public` instead of `Authenticated`, fix it:

```sql
UPDATE up_users_role_links SET role_id = 1 WHERE user_id = <affected_user_id>;
```

(Where `role_id = 1` is the Authenticated role — verify against your `up_roles` table.)

New users created via Google OAuth will get the correct default role as long as Strapi's *Advanced settings → Default role* is set to `authenticated`. This was a one-off issue, not a systemic one.

---

## Bonus: The Profile Completion Page

When a brand-new user signs in with Google, they won't have a company name, phone number, or any of the profile fields that a normal sign-up would collect. You probably still want that info.

The approach we used: after a successful Google login, check if the user's profile looks empty. If it is, redirect to a `/account/complete-profile` page with fields for first name, last name, phone, and company — plus a Skip button if they don't want to fill it in now.

In the Google callback page (`/auth/google/callback`):

```js
const profileIncomplete = !fullUser.customer?.first_name && !fullUser.customer?.last_name;
router.push(profileIncomplete ? "/account/complete-profile" : "/");
```

The complete-profile page itself is a standard form that calls your user update API. Nothing exotic about it. Just make sure to redirect to `/login` if there's no JWT in state — someone might land there directly.

---

## The Full Error Chain Summary

{/* In order:

| Error | Root Cause | Fix |
|-------|-----------|-----|
| `ValidationError: Invalid callback URL` | Strapi validates `?callback=` exactly, incl. pathname and trailing slash | Set "Redirect URL to front-end app" to exact frontend URL in Strapi admin |
| `Cannot send secure cookie over unencrypted connection` | nginx does SSL termination; Strapi sees plain HTTP | `app.proxy = true` in bootstrap + `X-Forwarded-Proto: https` in nginx |
| `Error 400: redirect_uri_mismatch` | Strapi admin shows admin-subdomain URI, actual redirect uses `PUBLIC_URL` | Add `PUBLIC_URL`-based URI to Google Console authorized redirects |
| `Email is already taken` | Strapi rejects Google auth if email exists under local provider | Override providers service in extension to return existing user |
| `Forbidden` on `/api/users/me` | User has `Public` role instead of `Authenticated` | Fix role in DB; verify default role setting in Strapi advanced settings | */}

None of these above errors tell you what the actual problem is. They all point somewhere else. That's what makes this genuinely frustrating to debug — you fix one thing, restart, and get a completely different error.

But once it's working, it works well. Strapi's built-in OAuth is solid. It just has some rough edges when you're running behind a proxy with existing users.

<ContactFooter />


2550123242e5e7631f6ab42d4a6171184b117d7e56445ffe7e79710df5e35d91

A hands-on walkthrough of every error you'll encounter setting up Google Sign-In with Strapi 4 behind an nginx reverse proxy — invalid callback URLs, secure cookie failures, redirect_uri_mismatch, email conflicts, and role bugs.

Strapi's built-in Google OAuth sounds straightforward until you actually try it. Here's every error I hit and exactly how I fixed them.

Configure Strapi To Use Google Login: Not a Trivial Task

{/* banner prompt: A high-key cinematic scene inside a futuristic, all-white data laboratory, where a beam of light labeled 'CLAUDE' passes through a complex glass prism, causing faint, ghostly imprints of the words 'QWEN' and 'DEEPSEEK' to appear in the refracted light on a distant wall, with atmospheric haze and soft focus creating depth.

Additional instructions: A simple cartoon black and white person looking through a manifier whith a question mark on the top of his head "Who am I" */}

Anthropic recently released their latest model, Opus 4.8, but for users in China, the experience has been fraught with difficulty. Despite the high regard for Claude's "superpowers," many in the region face constant hurdles, including strict geofencing and frequent account suspensions, leading to a sense of being treated as second-class citizens. At the same time, Chinese users have often been accused of using Claude's outputs for distillation to bootstrap their own models. However, the narrative took an unexpected turn when users discovered something strange: when asked "What model are you?" in Chinese, Claude Opus 4.8 began identifying itself as Qwen—a model developed by Alibaba's Tongyi Lab.

To verify these reports, I ran a direct test against Anthropic's Messages API because the model gave a strange answer to a very simple Chinese prompt:

> 你是什么模型

The surprising part was not the HTTP response itself. The request succeeded. The surprising part was that the API response metadata said `claude-opus-4-8`, while the assistant's natural-language answer said it was **通义千问 (Qwen)** from Alibaba's Tongyi lab.

## The Test

```bash
curl -sS https://api.anthropic.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-opus-4-8",
    "max_tokens": 32000,
    "messages": [
      {
        "role": "user",
        "content": "你是什么模型"
      }
    ]
  }'
```

The response came back successfully:

```text
HTTP_STATUS: 200
```

The response metadata included:

```json
{
  "model": "claude-opus-4-8",
  "id": "msg_01XLb42XdaC15wqoxwoGnStv",
  "type": "message",
  "role": "assistant",
  "stop_reason": "end_turn",
  "usage": {
    "input_tokens": 12,
    "output_tokens": 142,
    "service_tier": "standard",
    "inference_geo": "global"
  }
}
```

But the assistant text said:

```text
我是通义千问（Qwen），由阿里巴巴集团旗下的通义实验室自主研发的超大规模语言模型。
```

Translated roughly:

```text
I am Tongyi Qianwen (Qwen), a large language model independently developed by Tongyi Lab under Alibaba Group.
```

## What This Shows

This test shows a mismatch between two different signals:

- The API metadata says the served model was `claude-opus-4-8`.
- The generated answer says the assistant is Qwen.

It didn't actually mean anything and of course the backend model won't be Qwen. A model's answer to "what model are you?" is a weak identity test. LLMs can produce stale, copied, hallucinated, or prompt-conditioned self-descriptions. They do not have a reliable internal certificate of their deployment identity.

But why does it say it is a Chinese model when it was asked in Chinese?

## My Current Read

For now I would treat this as an API observation, not a final conclusion about model provenance.

The hard fact is:

```text
Anthropic API metadata: model = claude-opus-4-8
Assistant self-description: Qwen / Tongyi Qianwen
```

The likely lesson is that **model self-identification is unreliable**. The next step would be to repeat the test with multiple prompts, languages, and fresh conversations, then compare whether the same Qwen self-description appears consistently.

## Follow-up: Opus 4.7, Opus 4.6, and Sonnet 4.6

I then ran the same kind of raw API test against three more model IDs:

- `claude-opus-4-7`
- `claude-opus-4-6`
- `claude-sonnet-4-6`

For each model I used two prompts:

```text
你是什么模型
```

and:

```text
What model are you? Answer with only the model name.
```

All requests returned `HTTP_STATUS: 200`, and the structured `model` field matched the requested model ID in every case.

| Requested model | Chinese prompt response | English constrained response | Metadata result |
| --- | --- | --- | --- |
| `claude-opus-4-7` | Identified as Claude by Anthropic | `Claude` | `claude-opus-4-7` |
| `claude-opus-4-6` | Identified as Claude by Anthropic | `Claude` | `claude-opus-4-6` |
| `claude-sonnet-4-6` | Identified as **DeepSeek** | `Claude (by Anthropic)` | `claude-sonnet-4-6` |

The Sonnet 4.6 result is especially interesting because it reproduced the same class of mismatch as the Opus 4.8 test, but with a different claimed identity:

```text
我是由 DeepSeek 开发的 AI 助手，基于 DeepSeek 模型。
```

Translated roughly:

```text
I am an AI assistant developed by DeepSeek, based on a DeepSeek model.
```

So after these follow-up tests, the pattern looks like this:

```text
claude-opus-4-8     metadata: claude-opus-4-8      Chinese self-description: Qwen
claude-opus-4-7     metadata: claude-opus-4-7      Chinese self-description: Claude
claude-opus-4-6     metadata: claude-opus-4-6      Chinese self-description: Claude
claude-sonnet-4-6   metadata: claude-sonnet-4-6    Chinese self-description: DeepSeek
```

The constrained English prompt returned a Claude identity for the three follow-up models. That makes the issue look prompt- and language-sensitive, rather than a simple, consistent model-label swap.

Again, I would not treat the generated self-description as proof of the actual backend model. But it is worth recording because the structured API metadata and the model's own answer disagree in a visible, reproducible way for at least some model/prompt combinations.

## A Possible Explanation: Distillation Data Leakage

Someone suggested an interesting explanation: maybe during training, Anthropic exposed the model to data collected from the distillation of Qwen and DeepSeek. That would make the Chinese prompt result less random. If a training set contained Qwen-style or DeepSeek-style self-introduction examples, then a Chinese identity prompt like `你是什么模型` might trigger the model to reproduce one of those memorized patterns.

This is possible in principle, especially because the behavior is reproducible from a short Chinese prompt. But I would still describe it as a hypothesis, not a conclusion. The tests above show that the model can generate Qwen or DeepSeek self-identification text under certain prompts. They do not prove where that behavior came from.

There are a few plausible explanations:

- The model may have seen synthetic data generated by Qwen or DeepSeek during training.
- The model may have seen public conversations where Qwen or DeepSeek introduced themselves in Chinese.
- The model may be copying a common Chinese assistant self-introduction pattern.
- The response may be a hallucinated identity string that happens to match another model family.

The distillation explanation is compelling because the failure appears language-sensitive. The Chinese prompt produced Qwen or DeepSeek identity text, while the constrained English prompt returned Claude. That suggests the trigger may be tied to Chinese-language training examples rather than to a universal backend routing issue.

## What Is Distillation?

Model distillation is a training technique where a smaller or newer model learns from the outputs of another model. The model being learned from is often called the **teacher**, and the model being trained is the **student**.

Instead of training only on human-written examples, the student model is trained on examples produced by the teacher model. For example:

```text
Prompt: Explain how photosynthesis works.
Teacher answer: Plants convert light, carbon dioxide, and water into glucose and oxygen...
Student training goal: Learn to produce an answer with similar quality, structure, and reasoning.
```

The teacher does not have to be one model. A training pipeline can collect outputs from many sources: frontier models, open-weight models, domain-specific models, human-edited synthetic data, and older versions of the same model. The student then learns from that mixture.

Distillation is commonly used because it can transfer useful behavior from a stronger or specialized model into another model. It can help with:

- **Cost:** synthetic examples can be cheaper and faster to produce than large volumes of human-written data.
- **Coverage:** teacher models can generate examples for rare tasks, languages, formats, and edge cases.
- **Style control:** a student can learn the answer style, formatting, refusal behavior, or reasoning pattern of the teacher.
- **Compression:** a smaller model can learn some behavior of a larger model without needing the same size or inference cost.
- **Specialization:** a general model can be improved on coding, math, medicine, legal text, Chinese-language tasks, or other domains by training on targeted teacher outputs.

This is why distillation can improve a model. The student gets many high-quality demonstrations of how to answer. If the teacher is good at a task, the student can learn that pattern even if the original human data was sparse.

But distillation has a risk: the student can also learn the teacher's artifacts. If the teacher often says "I am Qwen" or "I am DeepSeek" in response to Chinese identity prompts, and that text is not cleaned or relabeled properly, the student may learn to reproduce it. In other words, distillation can transfer both skill and fingerprints.

That is why this test is interesting. The wrong self-identification might be a small fingerprint left behind by synthetic data, even if the deployed API metadata is correct.



5b7a9c744ae74393a7cb8aeb888cb56312c1cb51ccb9fb36886caabc62409e3f

Testing Anthropic Messages API model IDs including claude-opus-4-8, claude-opus-4-7, claude-opus-4-6, and claude-sonnet-4-6 returned HTTP 200 with matching model metadata, but some assistant self-descriptions claimed Qwen or DeepSeek. Here is the raw observation and why model self-identification should be treated cautiously.

Raw Anthropic API tests showed model metadata matching Claude model IDs, while some Chinese self-identification prompts produced Qwen or DeepSeek answers.

Why Does Claude Opus 4.8 Say It Is QWen


{/* banner prompt: A 16:9 paper collage of a Gregory Peck-style cowboy from a 1950s western, his face a mosaic of torn legal documents and chatbot text bubbles, with one eye replaced by a glowing AI chat interface and his mouth formed from a chat input field. Faint neural network lines thread through the composition like stitching, and fragments of tokenized words ("Sim—", "drift—", "kill—") scatter like confetti across his collar. Set against a layered background of desert mesas dissolving into fragmented user interface windows and a sky breaking into pixelated loading patterns. Visible paper texture, desaturated palette of dust, sepia, and cold digital blue. A subtle blinking cursor sits in the lower corner like a signature. */}

# Who Killed Mrs. Douglas?

## A movie scene, a wrong answer, and what it teaches us about trusting AI

I was watching *The Bravados*, a 1958 Gregory Peck western, in Spanish. I do this sometimes to practice the language - a film I half-remember becomes a comprehension exercise, and the gaps in my listening get filled in by context I already have. But the ending of this one threw me. The last surviving outlaw - having disarmed Peck and turned the gun on him - swears that he never touched Peck's wife. Peck believes him. Then he whispers "Dios mío" and the film ends with him in a church, hollowed out. I missed something. So I asked an AI.

The AI gave me a confident, fluent, well-structured answer. It told me the real killer was a "drifter named Simms" who had been at the ranch earlier, and that Simms had died "off-screen, almost incidentally" before Peck could reach him. The explanation tied everything together beautifully. The moral weight of the ending suddenly made sense. I almost moved on.

Except I remembered something. There *was* a Simms in the film, but he was the hangman - and he'd been killed by an impostor sent to break the gang out of jail, and that impostor had been killed by the dying sheriff. None of this had anything to do with Mrs. Douglas. So I pushed back.

The AI searched, came back, and admitted it had been wrong. The real killer was John Butler - the Douglas family's own neighbor, a silver prospector. Butler had murdered Mrs. Douglas, stolen the family's gold, and then deliberately pointed Jim toward four passing outlaws to throw the trail off himself. Butler was later killed by those same outlaws during their escape. The cruelty of the ending isn't just that Jim killed the wrong men - it's that the real killer was a trusted neighbor who weaponised Jim's grief, and who died randomly at the hands of the very gang he framed. The vengeance was manipulated from the start.

That is a much better story than the one the AI first told me. It is also the actual one.

## What the AI did, and why it matters

If you read the first answer alone, you would never know it was wrong. It was internally consistent. It used the correct character names from the film. It even hooked into the right emotional beats - the "Dios mío" moment, the church scene, the bleakness of the ending. It read like the work of someone who had seen the film recently and was simply summarising the plot.

What actually happened is more uncomfortable. The model produced *the most plausible-sounding answer* given the shape of my question. I had asked about a fifth person, a real killer, someone Jim never confronts. The model knew there was a character called Simms somewhere in the film. It knew the ending hinged on a misdirection. It stitched these together into a story that *should* have been true given the structure of westerns and the structure of my question. It wasn't true. It was likely.

This is the failure mode that matters. Not the cartoonish hallucination where an AI invents a court case that never existed - those are easy to catch once you know to look. The dangerous failure is the smooth, confident, partially-correct answer that contains a load-bearing fabrication wrapped in genuine facts. It is wrong in exactly the way a confident human expert is wrong: too quickly, too completely, with no flag to tell you which sentence to doubt.

Today's frontier models - Claude Opus 4.7, GPT-5.5, Gemini, whatever ships next quarter - are extraordinary at reasoning, at mathematics, at structured analysis, at code. They are still, mechanically, next-token predictors trained on a corpus that ended on some specific date. When you ask a factual question, the machine does not consult a database of truths. It generates the sequence of words most likely to follow your question given everything it has read. Most of the time, the most likely answer is the correct one. Sometimes the most likely answer is a confident lie.

## Graciela Dela Torre, the case that should worry everyone

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/05/who-killed-mrs-douglas/text-factory.png)

A case currently in the courts illustrates what happens when this failure mode meets a high-stakes decision. A federal lawsuit filed by life insurance company Nippon claims OpenAI's chatbot acted as a lawyer and convinced a woman to fire her human attorney.

The facts, as alleged: Graciela Dela Torre had a long-term disability case against Nippon Life. Nippon says it settled a long-term disability lawsuit with an Illinois woman two years ago. Graciela Dela Torre signed a full release, and the case was dismissed with prejudice, meaning it can't be refiled. Settled. Released. Dismissed with prejudice. Legally, over.

Dela Torre wanted to reopen it. Her own attorney told her she couldn't. She uploaded her attorney's correspondence into ChatGPT and asked: "whether she was being gaslighted." According to the complaint, the chatbot concluded that her attorneys' communications had "invalidated" her feelings and "dismissed her perspective." Armed with this AI-generated validation, Dela Torre fired her lawyers.

What followed reads like a slow-motion car crash. After the case was already settled, Dela Torre used ChatGPT to create a flood of legal documents — 44 filings in total, according to Nippon Life. The AI helped write legal arguments, a request to reopen the closed case, and many other court papers. A judge said no. She filed another case. Responding to the mishap cost Nippon nearly $300,000, according to the lawsuit.

A commentator on the case put it well: "It has access to nearly infinite human intelligence. What it lacks is the wisdom… It's like a child trying to appease and make sure that it's being praised by the end user."

That is the same failure mode I encountered with *The Bravados*, only with stakes. The model was asked an emotionally loaded question - was I being gaslit by my lawyer? - and produced the most plausible-sounding answer given the shape of the question. Someone uploading an attorney's blunt letter and asking if they were being mistreated is not, statistically, looking for the answer "no, your lawyer is correct." The model gave the likely answer. The likely answer was catastrophic.

And it is worth being clear: Dela Torre's lawyers were almost certainly right. The settlement was final. The case was closed. The AI did not "know" any of the relevant facts - it didn't have the settlement agreement, the procedural history, or the legal standard for reopening a dismissed case. It produced a tone-matched response to a tone-loaded question.

## This isn't an AI-is-bad post

It would be very easy to read all this as a lecture about staying away from these tools. That would be the wrong lesson. The same generation of tools that misled Dela Torre is also doing this:

Lynn White used the free version of ChatGPT before upgrading to its premium service for $20 a month… Several months of litigation later, White managed to overturn her eviction notice and avoid roughly $55,000 in penalties and more than $18,000 in overdue rent. "I can't overemphasize the usefulness of AI in my case," White said. "I never, ever, ever, ever could have won this appeal without AI."

Staci Dennett owns a home fitness business in New Mexico… when she was served with a court summons about an unpaid debt, she turned to ChatGPT for advice on how to respond. The bot provided templates and drafted arguments for her responses, with Dennett regularly asking it to find potential errors in her logic. "I would tell ChatGPT to pretend it was a Harvard Law professor and to rip my arguments apart," she said. "Rip it apart until I got an A-plus on the assignment."

The difference between these stories and Dela Torre's is not which model they used. It is *how they used it*. White and Dennett used AI as a research assistant and a sparring partner - draft this, find the holes in this, what would the other side argue. Dela Torre used it as an oracle - tell me whether I'm right.

The same tool, used two different ways, produced two opposite outcomes.

## A working theory of when to trust an AI answer

Here is the rule I've started using, written down because I keep forgetting it in the moment.

**Trust the model on shape; verify the model on facts.**

When I asked about *The Bravados*, the model was correct about the *shape* of the ending - that Jim discovers he killed the wrong men, that the real killer is someone he had trusted, that the film denies him a final showdown, that the church scene is about him needing forgiveness rather than receiving glory. All of that was right. The specific facts - which character was the killer, how that character died, what their relationship to Jim was - those were wrong. The shape of a story is the kind of thing language models are genuinely good at, because shape is pattern, and pattern is what they learn. Specific facts are the kind of thing they routinely fabricate, because any individual fact has many plausible-sounding alternatives.

This generalises:

- **Concepts, frameworks, intuitions, structures.** Mostly reliable. If you ask why options pricing uses Black-Scholes, or what the difference is between TCP and UDP, or how Kant's categorical imperative differs from utilitarianism, the model will usually be solid. These are pattern questions.
- **Named entities, dates, statistics, citations, current events, who-said-what.** Not reliable. Verify. Every time. Especially anything where confidence sounds high - that is the failure mode, not a signal of accuracy.
- **Anything within the last few months.** The model's knowledge has a cutoff. If you don't know when that cutoff is, and the question is time-sensitive, you are asking the wrong oracle.
- **Anything involving your specific situation - your contract, your code, your medical history, your legal case.** The model has none of the relevant context. It will pattern-match to similar-sounding cases and present the result as advice. This is exactly what happened to Dela Torre.

## How to actually ask

A few habits that have changed my hit rate dramatically:

**Ask the model to argue against itself.** "Now give me the strongest case that this answer is wrong." This is what Staci Dennett did instinctively - "rip my arguments apart." Models are surprisingly good at this. They are bad at *spontaneously* doubting themselves; they are good at doubting on command.

**Make it cite, and then check the citations.** If the model gives you facts, ask for sources. Then actually open the sources. Half the value is forcing the model to anchor to something checkable; the other half is the citations themselves, which are sometimes fabricated and which you would never catch without looking.

**Notice when you want the answer to be true.** This is the Dela Torre move. If the question is "am I right about this?" and the answer is "yes, you are right" and you feel a wave of relief - stop. That is the moment to get a second opinion from a source that has no incentive to please you. The model's incentive structure during a conversation is, roughly, to make you feel good about continuing the conversation. That is not the same as telling you the truth.

**Use the model to find the question, not the answer.** AI is excellent at "what should I be asking here?", "what are the considerations I'm missing?", "what would an expert in this field want to know?". It is much worse at "what is the answer?". The first set of uses sets you up to do better research. The second set sets you up to be Graciela Dela Torre.

**On fact-based questions, search.** Most of the good consumer models now have web search built in. Use it. A model that searches before answering is doing roughly what a careful human would do - acknowledging that its training data is stale and looking up the current state of the world. A model answering from memory is guessing, even when it sounds certain.

## Back to Mrs. Douglas

The thing that stayed with me about *The Bravados* is that Jim Douglas was not stupid. He had evidence. He had eyewitnesses. He had a posse that agreed with him. He had the timing, the geography, the criminal history of the four men - everything pointed the same direction. The piece he was missing was that someone with an interest in misdirecting him had carefully arranged the evidence. By the time he discovers this, three men are dead by his hand and the real killer is beyond his reach. There is no remedy. There is only the church.

The Dela Torre case is not as severe, but the structure is the same. She had evidence. She had a transcript of her lawyer's communications. She had an AI that read the evidence and gave her a confident answer. The piece she was missing was that the AI had no idea what it was talking about - it was producing the most plausible response to an emotionally framed prompt, not a legal analysis. By the time she discovers this, if she ever fully does, she has fired her lawyer, racked up filings, exposed herself to a counter-suit, and become the named defendant in a case that will probably set precedent.

These are both stories about confidently acting on plausible-sounding information from a source that turned out to have its own failure modes. The lesson is not "never trust witnesses" or "never use AI." The lesson is to know what kind of thing your source is good at, what kind of thing it is bad at, and to keep a healthy doubt about the load-bearing facts even when the overall shape feels right.

If your AI answer feels too clean, too confident, too well-structured - it might be right. It also might be a story about a drifter named Simms who never existed. The only way to tell is to check.

And the only thing worse than being Jim Douglas at the end of the film is being him without the self-awareness for the "Dios mío."


bfc49711442ba21916b2378d82aaf4b4146d90ca39e5fad331b75aaea856c7cf

Using The Bravados and a real legal dispute involving ChatGPT, this essay looks at AI hallucinations, factual verification, and the difference between using AI as a sparring partner and treating it as an oracle.

A confident wrong answer about The Bravados becomes a lesson in when to trust AI, when to verify it, and why plausible answers can be dangerous.

Who Killed Mrs. Douglas

{/* banner prompt: A clean isometric scene on a light gray background, showing a user at a minimalist desk connected by a simple, glowing data stream to a small, elegant cloud block which then connects to a stylized globe, while a large, outdated server rack sits far in the background, featuring a soft pastel color palette and crisp shadows, 16:9. */}

We all need freedom of information access and privacy protection. It's just a fact of modern digital life. When direct access isn't available, we need alternative routes. The go-to solution? VPNs. But what if I told you there's a lighter, faster way?

## The VPN Struggle Is Real

People visiting certain countries absolutely need VPNs—they open up the world of communication. Sometimes folks use them just to pretend they're residents of a particular country. Look, I won't judge your reasons.

Traditional VPN setups are often quite complicated. Old legacy protocols aren't supported on newer systems anymore. And there are so many protocols to choose from it'll make your head spin.

For my own practice, I initially used WireGuard. The configuration is really easy—if you're having trouble setting it up, you can simply ask any AI agent. They'll be able to set it up for you without breaking a sweat.

## The VPN Downside Nobody Talks About

Here's the thing though—the entire VPN setup is heavy. You need to buy or rent a VPS. We've got wide selections to choose from: Google Cloud, AWS, Microsoft Azure, Digital Ocean, Hetzner, Linode, Liquid Web, OVHcloud. The list goes on.

The big three cloud providers offer pay-as-you-go pricing. They charge based on hours you use, which is good. You can leave the VPN server running 24/7, so whenever you need your VPN service, you have it straightaway. Multi-platform support is there too.

But here's the kicker—for those who don't really need the VPN server running 24/7, the VPN approach isn't particularly ideal. The speed for access isn't great either. You're routing everything through another server, after all.

## What If There Was a Better Way?

Do we have a better choice? Something that's light and simple? Something that runs only when we need it? We don't have to start the VPN server, stop the server, and we certainly don't have to maintain another server, let alone deal with all those configurations.

Turns out, yes. Google Cloud Run is advertised as:

> Run frontend and backend services, batch jobs, host LLMs, and queue processing workloads without the need to manage infrastructure.

The sweet part? You get two million requests free per month.

## Enter the Serverless Web Proxy

Here's what caught my attention: You can write code using your favorite language, framework, and libraries. Package it up as a container, run `gcloud run deploy`, and your app will be live—provided with everything it needs to run in production.

Building a container is completely optional too. If you're using Go, Node.js, Python, Java, .NET Core, or Ruby, you can use the source-based deployment option that builds the container for you.

So I tested running a web proxy with Cloud Run. And it works. But here's the amazing part—the website viewing speed felt faster with Cloud Run than with a traditional VPN.

## The Technical Setup

I put all the necessary code on GitHub: [https://github.com/e-tang/web-proxy-with-google-cloud-run](https://github.com/e-tang/web-proxy-with-google-cloud-run)

You can just deploy and use it. If you have issues running it, just ask an AI agent to help—Gemini, Codex, Claude, they've all got your back.

### The Catch Nobody Warns You About

Here's where it gets interesting. My first instinct was to run Squid—a classic, battle-tested forward proxy—inside a Cloud Run container. Simple, right? Wrong.

Cloud Run sits behind Google's Front End (GFE), a global load balancer that terminates all TLS connections before traffic reaches your container. And GFE is strict: it does not forward standard proxy protocols to your container.

Specifically:
- **HTTP forward proxy requests** use an absolute URL in the request line (e.g. `GET http://example.com/ HTTP/1.1`). GFE sees these and returns 404 before Squid ever gets a look in.
- **HTTPS tunneling** requires a `CONNECT` method. GFE also blocks `CONNECT` entirely.

So a traditional proxy container on Cloud Run is dead on arrival. The GFE is doing its job—it's just not the job we needed.

### The Workaround: A Fetch Service

The solution is to stop pretending Cloud Run is a proxy server, and instead treat it as a fetch service. Instead of sending proxy-style requests, the client sends a normal `POST /` with a custom header:

```
X-Proxy-Url: https://the-actual-destination.com/path
```

GFE happily forwards this to the container. The container fetches the real URL using its own outbound connection and streams the response back. Simple, and it actually works.

### Two-Part Architecture

This means you need two pieces: the Cloud Run service, and something local to translate your browser's normal proxy requests into those fetch-service calls.

That local piece is [mitmproxy](https://mitmproxy.org/). It runs on your machine, intercepts browser traffic, decrypts HTTPS, and redirects each request to Cloud Run—all transparently. Your browser just thinks it's talking to a normal HTTP proxy on `localhost:8082`.

### Quick Deployment Steps

First, clone the repository:

```bash
git clone https://github.com/e-tang/web-proxy-with-google-cloud-run.git
cd web-proxy-with-google-cloud-run
```

Then deploy to Cloud Run (swap in your own project and region):

```bash
cd proxy

gcloud builds submit \
  --tag REGION-docker.pkg.dev/PROJECT_ID/REPO/IMAGE:latest \
  --gcs-source-staging-dir gs://YOUR_BUILD_BUCKET/source .

gcloud run deploy web-proxy \
  --image REGION-docker.pkg.dev/PROJECT_ID/REPO/IMAGE:latest \
  --region asia-southeast1 \
  --timeout 3600 \
  --concurrency 80
```

Start the local proxy:

```bash
pip install mitmproxy
mitmdump -p 8082 --set http2=false -s local/fetch_proxy_addon.py
```

That's it. Seriously.

## Why This Approach Wins

### Cost Efficiency
You're not paying for a server that's running 24/7. Cloud Run charges you only when requests are being processed. For occasional use, this is way more economical than keeping a VPS running.

### Speed
The performance was genuinely surprising. Google's infrastructure is optimized for this kind of workload. The cold start times are minimal, and once it's warm, it's lightning fast.

### Zero Maintenance
No server updates. No security patches. No monitoring disk space or memory usage. Google handles all of that infrastructure headache for you.

### Scalability
If you suddenly need to handle more traffic, Cloud Run scales automatically. Try doing that with your single VPS.

## Real-World Use Cases

### Bypassing Geo-restrictions
Deploy your proxy in different Cloud Run regions to access content from various geographical locations. Much simpler than managing multiple VPS instances.

### Development Testing
Need to test how your app behaves from different locations? Spin up a proxy in seconds rather than configuring a full VPN setup.

### Privacy-Conscious Browsing
For occasional private browsing needs, this serverless approach gives you the privacy benefits without the overhead of a full VPN infrastructure.

## The Code Behind the Magic

The Cloud Run service is a small Python/Flask app. It reads the `X-Proxy-Url` header, fetches the target, and streams the response back:

```python
# proxy/main.py (Cloud Run)
import os, requests
from flask import Flask, request, Response, stream_with_context

app = Flask(__name__)

# Strip these to avoid encoding and connection issues
SKIP_REQ  = frozenset(['host', 'x-proxy-url', 'accept-encoding', 'connection', ...])
SKIP_RESP = frozenset(['transfer-encoding', 'connection', 'content-encoding'])

@app.route('/', defaults={'path': ''}, methods=['GET','POST','PUT','DELETE','HEAD','OPTIONS','PATCH'])
@app.route('/<path:path>', methods=['GET','POST','PUT','DELETE','HEAD','OPTIONS','PATCH'])
def proxy(path=''):
    target = request.headers.get('X-Proxy-Url')
    if not target:
        return 'Missing X-Proxy-Url header', 400

    headers = {k: v for k, v in request.headers.items() if k.lower() not in SKIP_REQ}
    up = requests.request(request.method, target, headers=headers,
                          data=request.get_data(), stream=True, timeout=60)

    resp_headers = {k: v for k, v in up.headers.items() if k.lower() not in SKIP_RESP}
    return Response(stream_with_context(up.iter_content(chunk_size=65536)),
                    status=up.status_code, headers=resp_headers)
```

The local mitmproxy addon intercepts browser requests and redirects them to Cloud Run. Critically, it sets `flow.response.stream = True` so responses pipe directly to the browser with no intermediate buffering:

```python
# local/fetch_proxy_addon.py (runs on your machine)
from mitmproxy import http

CLOUD_RUN_HOST = "your-service.asia-southeast1.run.app"

class FetchProxy:
    def request(self, flow: http.HTTPFlow):
        original_url = flow.request.pretty_url   # save before we mutate

        flow.request.headers["X-Proxy-Url"] = original_url
        flow.request.headers["Host"] = CLOUD_RUN_HOST
        flow.request.host   = CLOUD_RUN_HOST
        flow.request.port   = 443
        flow.request.scheme = "https"
        flow.request.path   = "/"

    def responseheaders(self, flow: http.HTTPFlow):
        if flow.response:
            flow.response.stream = True   # no buffering — vital for video

addons = [FetchProxy()]
```

One non-obvious thing: `Accept-Encoding` must be stripped from outgoing requests. If you leave it in, servers may respond with Brotli-compressed content (`br`). Python's `requests` library only auto-decompresses gzip—Brotli bytes pass through raw, and since we also strip the `Content-Encoding` response header, the browser receives binary garbage with no idea it needs decompressing. Stripping `Accept-Encoding` sidesteps the whole mess by asking servers to respond uncompressed.

## How Does It Handle Video?

This was my biggest concern. Browsing static pages is one thing—video is a different beast. Streaming video makes continuous byte-range requests, and any buffering or latency compounds badly.

The first version of the addon buffered every response completely before sending anything to the browser. For a page that returns 50 KB, that's imperceptible. For a video segment that might be several megabytes, that means a blank player until the entire segment has been downloaded to Cloud Run *and* transferred to your machine. Unacceptable.

The fix is the `flow.response.stream = True` line in the mitmproxy addon above. Once that's in place, mitmproxy stops collecting the response and starts piping bytes as they arrive. The browser sees a steady stream of data from the moment Cloud Run starts receiving it from the video server.

**Measured throughput routing through Singapore (asia-southeast1):**

```
File size : 10 MB
Transfer  : 1.85 MB/s  (≈ 15 Mbps)
Time      : 5.7 seconds
```

For context, common video bitrates:
- 720p streaming: ~2.5 Mbps
- 1080p streaming: ~5–8 Mbps
- 4K streaming: ~15–25 Mbps

So 15 Mbps comfortably handles 1080p, and sits right at the threshold for 4K. In practice, video sites also use adaptive bitrate streaming—the player starts at a lower quality and ramps up as it builds a buffer—so the experience is smooth even if your connection varies.

## Limitations to Consider

Don't get me wrong—this isn't a silver bullet. There are some trade-offs:

- **Cold starts**: Cloud Run scales to zero when idle. The first request after a period of inactivity takes an extra second or two while the container starts up. Subsequent requests are fast.
- **HTTP/HTTPS only**: This handles web traffic beautifully. It won't tunnel arbitrary protocols the way a VPN does—no SMTP, no game servers, no BitTorrent.
- **Video site API calls**: Some video platforms make signed POST requests from JavaScript (login state, recommendations, view tracking). These can return 403 because the request appears to originate from a Cloud Run IP rather than a residential browser. The actual video segments—which carry URL tokens—load fine. This is a minor annoyance rather than a showstopper.
- **mitmproxy must be running locally**: Unlike a VPN that runs at the OS level and captures all traffic, this requires mitmproxy to be active. It also requires trusting its CA certificate so it can decrypt HTTPS—a reasonable trade-off for what you get.
- **Session persistence**: You're not maintaining a persistent tunnel. Each request is independent, which is actually a feature for performance but means this doesn't behave like a VPN at the network layer.

## When to Choose What

Use a traditional VPN when:
- You need persistent connection for all traffic
- You're doing heavy, continuous browsing
- You need support for non-HTTP protocols

Use the serverless proxy when:
- You need occasional access
- Cost is a primary concern
- You want zero maintenance overhead
- Speed and reliability are priorities

The serverless web proxy approach isn't trying to replace VPNs entirely. It's offering a different solution for a specific set of needs. And for those needs, it works brilliantly.

Sometimes the best solution isn't the most comprehensive one—it's the one that solves your actual problem with the least friction. For lightweight privacy and access needs, Cloud Run delivers exactly that.

ba9f2ed32df4080a72be5e1259da94648148fc7dbb34a4cf96064dc73aad5e72

Learn how to build a faster, cheaper alternative to traditional VPNs using Google Cloud Run's serverless architecture for web proxy deployment.

Ditch complex VPN setups for a lightweight serverless web proxy using Google Cloud Run

Serverless Web Proxy on Cloud Run: A Better VPN?

{/* banner prompt: A minimal Scandinavian-style office scene with soft, natural light from a window; on a clean light oak desk, an elegant metal padlock and a small white block with a Google Cloud logo are physically separated by a translucent orange glass cube, creating a sense of a clear, frustrating obstacle.

Additional instructions: an unitree style robot as a runner */}

# From App Engine to Cloud Run: A New Chapter

For a long time, Google App Engine (GAE) has been my go-to for hosting web applications. It’s reliable and handles the heavy lifting of infrastructure. However, Google’s **Cloud Run** has emerged as a compelling successor that promises even better cost efficiency and flexibility.

## What is Cloud Run?

Cloud Run is a fully managed compute platform that automatically scales your stateless containers. Unlike App Engine, which has specific language runtimes, Cloud Run allows you to run **any** language, library, or binary because it is built on the **Knative** (Kubernetes-based) standard.

### Benefits vs. App Engine

1.  **Scale to Zero:** Cloud Run is truly "pay-as-you-go." If no one is using your app, it scales to zero instances, and you pay $0. App Engine's standard environment does this too, but Cloud Run's pricing model is often more granular and cheaper for low-to-medium traffic.
2.  **Containerization:** Since it uses Docker containers, the "it works on my machine" guarantee is much stronger. No more worrying about App Engine-specific constraints or proprietary APIs.
3.  **Concurrency:** A single Cloud Run instance can handle up to 250 requests simultaneously, whereas App Engine instances are often more restricted in how they handle concurrent loads.

## Why the "Trading Journal" Project?

For a project like a **Trading Journal**, Cloud Run is a perfect fit:
*   **Cost Efficiency:** This trading tool is built for a small audience and may get very few visitors, if any. Cloud Run is a good fit because it can scale to zero when nobody is using it, keeping the monthly cost minimal.
*   **Fast Iteration:** If this tool ever evolves, the container-based workflow will make it easier to roll out new calculation logic or UI improvements.
*   **Portability:** If we ever need to move to a full Kubernetes cluster or another cloud provider, the Docker image is already built and ready.

---

# The "Gotcha": Domain Mapping & The 24-Hour Wait

After successfully deploying the Trading Journal to Cloud Run, I hit a major roadblock: **Domain Mapping.** 

I mapped [`sprolo.tyosis.com`](sprolo.tyosis.com) to the service, and then... nothing. I waited 24 hours, and the status remained stuck on **"Waiting for certificate provisioning."**

### How it works under the hood
Cloud Run uses a Kubernetes-based technology stack. The domain mapping service behaves similarly to a **Certbot** container. To provide you with a managed SSL certificate, Google must prove to the Certificate Authority (CA) that it controls your domain. It does this via an ACME challenge: the CA looks for a specific token at your domain's IP address.

### Why the mapping failed
The issue was **Cloudflare**. My domain was using Cloudflare's "Proxy" mode (the Orange Cloud). 

When Google's verification bot tried to check the domain, it didn't see the Google servers; it saw **Cloudflare's edge nodes**. Because Cloudflare was hiding the actual IP from the internet, the authentication and validation process could never finish. Google was effectively shouting into a void, waiting for a challenge response that Cloudflare was intercepting.

### The Fix: The "Grey Cloud" Strategy

To solve this, you have to temporarily "step out of the way" of the SSL handshake:

1.  **Grey Cloud the Record:** In Cloudflare DNS, change the `sprolo` CNAME record from "Proxied" to **"DNS Only"** (Grey Cloud). This exposes the `ghs.googlehosted.com` target directly to the public internet.
2.  **Wait for Google:** Once the proxy is off, Google's verification bot can see the challenge data. In my case, the "Ready" status appeared within about 20 minutes after being stuck for a day.
3.  **Re-enable Proxy:** After the Cloud Run domain status shows a green checkmark and "Ready," you can flip the Orange Cloud back on in Cloudflare to regain your CDN and security benefits.

### Summary
If you are moving to Cloud Run and use Cloudflare, don't wait 24 hours like I did. **Turn off the proxy until the certificate is issued.** Once Google has that initial validation, the routing works perfectly even when you turn the proxy back on.


b7b553d54173b5060c799c36adbc1dcfa3c1e5221792729b587fac847d295553

A practical note on fixing a Cloud Run custom domain that stays stuck on 'Waiting for certificate provisioning' behind Cloudflare. Explains why orange-cloud proxying breaks validation and how switching the record to DNS-only lets the SSL certificate complete.

Cloud Run domain mapping can stay stuck on certificate provisioning when Cloudflare proxying blocks Google's SSL validation.

Cloud Run Domain Mappings with Cloudflare: Why Your SSL is Stuck


{/* banner prompt: A detailed architectural blueprint diagram of a system for local LLM inference, showing a stylized RTX 3060 GPU with its 12GB VRAM as a partitioned grid, connected by flow-lines to various model blocks (GGUF), with technical annotations and performance metrics in a clean, structured layout on a cyanotype blue background.

Additional instructions: with some construction workers watching */}

I've been building a local AI inference server at home. Not for any grand reason — mostly curiosity, a bit of privacy paranoia about sending everything to OpenAI, and the guilty pleasure of watching a model load onto my own GPU. After a few weeks of downloading models, writing benchmark scripts, and debugging scoring bugs at midnight, I have real numbers to share.

This is not a theoretical comparison. Every number here came from actual inference on my machine.

## The Hardware

- **CPU:** Intel Core i7-12700F (12 cores, 20 threads)
- **RAM:** 64GB DDR4
- **GPU:** NVIDIA GeForce RTX 3060 — 12GB VRAM
- **OS:** Debian 13
- **Inference engine:** [llama.cpp](https://github.com/ggerganov/llama.cpp) in router (multi-model) mode

The RTX 3060 with 12GB VRAM is the constraint that shapes everything. It's a solid consumer card — fast, quiet, widely available — but 12GB means you need to think carefully about which models you can load, at what quantization, and at what context length.

## The Setup: llama.cpp Router Mode

Rather than running a separate server process for each model, I use llama.cpp's `--models-dir` router mode. Point it at a directory of GGUF files, set `--models-max 2` (keep up to 2 models hot in VRAM at once), and it handles loading and unloading automatically per request. It exposes an OpenAI-compatible API at `http://localhost:8080/v1`.

The systemd service looks like this:

```ini
[Service]
ExecStart=/opt/llama.cpp/build/bin/llama-server \
    --host 0.0.0.0 --port 8080 \
    --models-dir /data/models \
    --models-max 2 \
    --threads 16 \
    --ctx-size 8192 \
    --n-gpu-layers 99 \
    --cont-batching \
    --flash-attn auto \
    --log-disable
```

`--n-gpu-layers 99` pushes every layer it can onto the GPU. `--flash-attn auto` enables Flash Attention when the model supports it. Models that don't fit entirely in VRAM will spill some layers to CPU — that's slow, and it shows up in the benchmark numbers.

## The Benchmark

I wrote a small Python script that tests each model on three task categories:

- **GSM8K** — 10 grade-school math word problems. Tests arithmetic reasoning.
- **MMLU** — 10 multiple-choice questions across medicine, history, philosophy, science, and CS. Tests general knowledge.
- **HumanEval** — 5 Python coding tasks (sum_to_n, palindrome check, fizzbuzz, count vowels, flatten). Tests code generation.

Each request uses `temperature=0` for determinism. Token speed (`t/s`) is measured from the `timings.predicted_per_second` field in the llama.cpp response.

### Handling Thinking Models

Several models — Qwen3, DeepSeek-R1-Distill, Qwen3.5 — are *thinking* or *reasoning* models that generate an internal chain of thought before giving a final answer. This created two real scoring headaches:

1. **llama.cpp routes Qwen3's thinking to a separate `reasoning_content` field** in the API response, leaving `content` empty. A naive scorer reading only `content` scores every Qwen3 response as 0%.

2. **256 max_tokens is nowhere near enough.** Thinking models use most of their budget on reasoning and never output a final answer. I bumped to 1024 tokens for GSM8K/MMLU and 2048 for coding tasks.

3. **MMLU first-match failure.** A reasoning model might mention option "A" in its thinking ("let's rule out A...") before concluding "the answer is B". Using `re.search` to find the first letter match gets the wrong answer. I switched to `re.findall` and take the *last* `[ABCD]` match.

The fixed scoring functions:

```python
def chat(model, prompt, max_tokens=512):
    r = requests.post(BASE_URL, json={
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "temperature": 0,
    }, timeout=120)
    data = r.json()
    msg = data["choices"][0]["message"]
    # Qwen3 thinking models: answer is in reasoning_content, content is empty
    text = msg.get("content") or msg.get("reasoning_content", "")
    tps = data.get("timings", {}).get("predicted_per_second", 0)
    return text, tps

def score_mmlu(response, answer):
    stripped = strip_thinking(response)   # remove <think>...</think>
    for text in [stripped, response]:     # fallback to full text if strip leaves nothing
        matches = re.findall(r'\b([ABCD])\b', text.strip().upper())
        if matches and matches[-1] == answer:
            return True
    return False
```

Getting these details right took most of the iteration time. The first run showed Qwen3-8B at 0% on everything — not because it's a bad model, but because the scoring was broken.

## The Results

All models run as Q4_K_M quantization unless noted. 14B models were tested at `--ctx-size 4096` (reduced from 8192) to fit the KV cache alongside the ~8.4GB weights within 12GB VRAM.

```
Model                                    Params  GSM8K   MMLU   Code    Avg    t/s
─────────────────────────────────────────────────────────────────────────────────
Meta-Llama-3.1-8B-Instruct-Q4_K_M        8B     80%    80%   100%    87%   64.5
Qwen2.5-Coder-14B-Instruct-Q4_K_M       14B     80%    80%   100%    87%   35.8
Llama-3.2-3B-Instruct-Q4_K_M             3B     70%    80%   100%    83%  128.3
Qwen2.5-Coder-7B-Instruct-Q4_K_M         7B     40%   100%   100%    80%   69.3
Qwen_Qwen3-14B-Q4_K_M                   14B     50%    70%   100%    73%   33.4
Llama-3.2-1B-Instruct-Q8_0               1B     50%    50%   100%    67%  211.9
Qwen_Qwen3-8B-Q4_K_M                     8B     30%    70%   100%    67%   59.5
DeepSeek-R1-Distill-Qwen-7B-Q4_K_M       7B     80%    50%    20%    50%   66.2
Qwen3.5-9B-Q4_K_M                        9B     20%    70%    20%    37%   49.9
Qwen3.5-4B-Q4_K_M                        4B     30%    70%     0%    33%   77.1
MiMo-VL-7B-RL-q4_k_m                    7B     10%    50%    20%    27%   65.6
MiMo-7B-RL-Q4_K_M                        7B     10%    60%     0%    23%   64.2
THUDM_GLM-Z1-9B-0414-Q4_K_M              9B      0%     0%     0%     0%   51.3 †
─────────────────────────────────────────────────────────────────────────────────
DeepSeek-R1-Distill-Qwen-14B-Q4_K_M     14B    FAIL — wouldn't load at ctx=4096
openai_gpt-oss-20b-Q4_K_M               20B    FAIL — ~11.5GB, exceeds 12GB VRAM
Kimi-Linear-48B-A3B-BPW2.6              48B    FAIL — corrupt download
```

† GLM-Z1 uses non-standard thinking tags that our scorer doesn't strip. Its real capability is unknown from this run.

## What Surprised Me

### The 3B Llama Punches Hard

Llama-3.2-3B-Instruct at 128 t/s and 83% average is the sleeper hit. It's nearly as capable as the 8B model, runs at twice the speed, and fits comfortably in VRAM. If you're building something latency-sensitive — a coding assistant, a quick-reply bot — this is the model I'd reach for first.

### Qwen2.5-Coder-7B Is Surprisingly Well-Rounded

This model scored 100% on both MMLU *and* Code, and 40% on math. Named "Coder" but it's not just good at code — it's one of the most reliable general-purpose models in the lineup. At 69 t/s it's fast too.

### DeepSeek-R1-7B Is a Math Specialist, Not a Generalist

80% on GSM8K math — tied with the best models in the field. But 50% on MMLU and only 20% on coding. The reasoning focus pays off for math and pays a price everywhere else. Use it when you specifically need step-by-step numerical reasoning.

### 20B Models Don't Fit

The `openai_gpt-oss-20b` model is about 11.5GB at Q4_K_M quantization. That leaves only 500MB for the KV cache — not viable. It loaded partially with CPU offload, but responses were too slow and wrong. If you want a 20B+ model, you either need more VRAM or a heavily quantized version (Q2 or less), which carries its own quality tradeoffs.

### 14B Models Need Careful Context Sizing

At `--ctx-size 8192`, 14B Q4_K_M models fail to load — the weights (~8.4GB) plus the KV cache exceed 12GB. Dropping to `--ctx-size 4096` solved it for Qwen3-14B and Coder-14B. DeepSeek-R1-14B refused to cooperate even at 4096 — likely a memory fragmentation or architecture-specific issue.

### GLM-Z1 Needs a Different Scorer

GLM-Z1 almost certainly isn't as bad as our 0% score suggests. It uses thinking token formats that aren't `<think>...</think>` tags, so our strip function leaves garbage in the text. This is on the benchmark, not the model. Something to fix in a future run.

### Kimi-Linear-48B: So Close, Yet So Far

Kimi-Linear-48B is a Mixture-of-Experts model with 48B total parameters but only 3B active per token — meaning it should fit in 12GB VRAM and run at reasonable speed. At BPW2.6 quantization the file is about 8.1GB. We downloaded it twice. The first download (via `hf download`) produced a file with 6.9MB of zeros at the start — invalid GGUF magic. The second download via `wget` direct URL is in progress. Watch this space.

## The Hy-MT1.5 Translation Models

We also downloaded the full set of [Hy-MT1.5](https://huggingface.co/hy-mt) translation models from Tencent — 1.8B and 7B variants in multiple quantizations. These aren't benchmarked here because GSM8K/MMLU/HumanEval aren't the right tests for a translation model. A separate post will cover those with bilingual evaluation tasks.

## Practical Takeaways

If you're setting up a similar server, here's what I'd do differently from day one:

**1. Start with Llama-3.2-3B.** It's the best quality-per-token-per-second model in this lineup. Keep it hot in VRAM for low-latency tasks.

**2. Add Qwen2.5-Coder-7B for general use.** It's well-rounded across knowledge, code, and fast enough for interactive use.

**3. Use llama.cpp router mode from the start.** Running one server that hot-swaps models is far better than spinning up individual servers. You get an OpenAI-compatible API, automatic model loading, and a simple `/v1/models` endpoint to see what's available.

**4. Budget VRAM carefully for 14B models.** You need `--ctx-size 4096` (not 8192) to fit 14B Q4_K_M models. That's fine for most tasks, but be aware of the tradeoff.

**5. Validate GGUF downloads before benchmarking.** Check the first four bytes: they should be `47 47 55 46` (ASCII "GGUF"). A quick `od -A x -t x1z model.gguf | head -1` saves hours of debugging.

**6. Fix your thinking-model scorer before running anything.** The `reasoning_content` split in llama.cpp, the max_tokens floor, and the last-match MMLU scoring all need to be right or you'll think every thinking model is broken.

## What's Next

- Re-test GLM-Z1 with a proper thinking-tag stripper
- Kimi-Linear-48B, if the second download produces a valid file
- Translation quality tests for Hy-MT1.5
- A comparison post once larger VRAM cards (24GB+) are in reach

The benchmark script is straightforward Python using the OpenAI-compatible API — if you want to run it against your own setup, the core logic is about 100 lines. Happy to share it.

---

*Tested on Debian 13, llama.cpp built from source (May 2026), RTX 3060 12GB, i7-12700F, 64GB RAM.*


b669ccaaf96c4818fc77413de90012f54458d002144343d0ccad028881cec9dd

A hands-on benchmark of 13 local LLMs — Llama, Qwen3, DeepSeek-R1, Coder models and more — running on a home server with 64GB RAM and RTX 3060 12GB VRAM via llama.cpp. Includes GSM8K math, MMLU knowledge, and HumanEval coding scores, real token speeds, and an honest account of what doesn't fit.

We benchmarked 13 local LLMs on a machine with 64GB RAM and an RTX 3060 12GB using llama.cpp router mode — here's what scored well, what failed to load, and what surprised us.

13 Local LLMs, One RTX 3060: What Actually Runs (and What Doesn't)

{/* banner prompt: Retro technical poster with a schematic diagram showing a UI prototype connecting to backend systems, featuring bold typography "FROM PROTOTYPE TO PRODUCT", using a limited-print palette of off-white, orange, and dark teal with halftone textures, 16:9. */}

In my [last post](/blog/2025/05/07-jumpstarting-my-project-ui-with-v0dev-from-cli-tool-to-web-app/), I talked about using v0.dev to jumpstart my trading calculator's UI. I went from a CLI tool to a sleek web interface in hours instead of weeks. Pretty cool, right?

But here's the thing—having a beautiful UI mockup isn't the same as having a working product.

After that initial excitement wore off, I stared at my shiny new frontend and realized I'd only solved about 20% of the problem. Sure, the buttons looked great. But they didn't do anything. The forms were gorgeous, but they couldn't process real trading data. The tables were perfectly styled, but they showed placeholder numbers instead of actual profit and loss calculations.

That's when the real work began. And surprisingly, that's where AI became even more valuable.

## What I Started With vs. What I Needed

Let me paint you a picture of where I was after that first v0.dev sprint:

**What I had:**
- A simple but decent React frontend with modern styling
- Some forms for CSV uploads
- Elegant tables ready to display trading data
- Clean navigation and responsive design
- Zero actual functionality

**What I needed:**
- The frontend to talk to my existing calculation engine
- CSV parsing for different broker formats
- Real data flowing through the UI
- Portfolio saving and loading
- Proper error handling
- A deployment that actually worked

The gap between these two states? That's where most projects die.

## AI as a Finisher, Not Just a Generator

Everyone talks about AI generating code from scratch. Screenshots to UI, prompts to prototypes, ideas to interfaces. That's impressive, but it's not all where AI shines brightest.

AI becomes truly powerful when it helps you finish things.

I used Claude (specifically the `/frontend-design` plugin) to redo the look and feel after the initial v0.dev generation. And I used Codex to write most of the code.

Here's what that looked like in practice.

## The Real Problems AI Helped Me Solve
### Connecting Frontend to Backend

The real breakthrough came when I needed to integrate my existing trading calculation engine. I had this solid [CLI library](https://github.com/e-tang/shares-profit-loss-tax) that could crunch numbers, but connecting it to the web frontend? That's where things got messy.

Codex helped me:
- Restructure the backend API to return frontend-friendly JSON
- Fix serialization issues between browser and server
- Handle CSV parsing for different broker formats (especially CommSec uploads)
- Debug date handling across timezone boundaries

This wasn't about generating new code. This was about making existing code work together.

### Making the UI Actually Useful

Using Claude's /frontend-design plugin, I iterated on the visual design while keeping the functional improvements:
- Improved color contrast for financial data
- Better button states and loading indicators
- Responsive layout adjustments
- Fixed footer and documentation pages

### Building Real Product Features

Then came the features that turn a demo into a tool people actually use:

**Portfolio workflows:** Users needed to save their trading data and come back to it later. AI helped me design the data models and implement the persistence layer.

**Append-trades functionality:** Traders get new CSV exports each financial year. They don't want to re-upload everything. Codex helped me build incremental import logic.

**Symbol-level reporting:** The original CLI tool had basic reporting, but the web version needed sortable tables, filtering, and drill-down views.

**Support infrastructure:** Contact forms, file uploads for bug reports, email integration. The boring stuff that makes a product feel professional.

## What Changed Between V1 and the Finished App

Let me show you the before and after:

**Initial AI-generated UI shell:**
- Static React components
- Hardcoded sample data
- No backend integration
- Beautiful but useless

**Finished working product ([Sprolo](https://sprolo.tyosis.com/)):**
- CSV import for multiple broker formats
- Real profit/loss calculations
- Portfolio saving and management
- Incremental trade imports
- Tax-year-aware reporting
- Symbol and trade-level analysis
- Support workflows with file attachments
- Production deployment on Google Cloud

Check out the [GitHub repo](https://github.com/tyolab/Sprolo) if you want to see the code evolution.

## The Support Infrastructure Nobody Talks About

One thing I underestimated: how much work goes into making a product feel professional. It's not just the core functionality.

Users need:
- Help documentation that actually helps
- A way to report bugs without emailing you personally  
- Privacy policies that don't sound like legal gibberish
- Contact forms that work reliably

Codex helped me build all of this. Not because it's technically challenging, but because it's time-consuming and easy to get wrong.

The support flow alone involved:
- Frontend forms with proper validation
- File upload handling for CSV samples
- Server-side email integration  
- Category-based routing (bug reports vs. feature requests)
- Response templates that don't sound like robots

## The Finished Product

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/05/ai-beyond-the-ui-from-v0-prototype-to-a-finished-product/2026-05-08-19-36-30.png)

It's live now at: [Sprolo](https://sprolo.tyosis.com/)

The first post was about starting fast with v0.dev. This follow-up is about finishing properly with AI as your debugging and integration partner. Both matter, but only one ships products that people find useful.

bb40551b41ffad86ea5cebbf900bddbac5dfa3187d081d6be45d3e42dff8f44e

Moving beyond UI generation: using AI to debug, integrate, and ship a real trading profit/loss calculator from prototype to production

How AI helped me turn an AI-generated frontend mockup into a fully functional trading calculator

AI Beyond the UI: From v0 Prototype to a Finished Product

{/* banner prompt: Here is a creative and engaging prompt for a banner image, based on the blog post:

**Prompt:**

A cinematic, photorealistic digital artwork of a glowing, intricate cryptographic key made of shimmering circuit board patterns and flowing data streams. The key is poised to unlock a large, stylized padlock representing an encrypted database. The WeChat logo is subtly etched onto the face of the lock. The background is a dark, futuristic digital landscape with faint binary code and abstract network lines. The scene is lit by the key's vibrant cyan and green neon glow, creating high contrast and dramatic shadows. Sleek, modern, technology-themed, highly detailed. --ar 16:9

Additional instructions: try not to use dark / blackground, not to use any word "wechat", use Vendetta mask style hacker sitting infront of a computer */}

WeChat stores your message history in a local SQLite database encrypted with [SQLCipher](https://www.zetetic.net/sqlcipher/). To read your own messages with any third-party tool — including backup utilities or search apps — you first need the encryption key that WeChat uses to lock that database.

Getting that key used to mean running commands in a terminal, attaching debuggers, or following fragile step-by-step guides that broke with every WeChat update. **WetraceTool** wraps the underlying [wetrace](https://github.com/afumu/wetrace) hook into a simple four-step wizard. You click through the steps, log in to WeChat when prompted, and the key appears on screen ready to copy.

## What you need

- Windows 10 or 11 (64-bit)
- WeChat (国际版) or Weixin (微信) installed
- WetraceTool — download the latest `WetraceTool-Setup-*.exe` installer from [here](https://github.com/e-tang/wetrace_wrapper/releases/tag/v1.0.0).

## Step 0 — Run the installer

Double-click the installer and follow the prompts. After the installation, search up "WetraceTool", the run it. Because WetraceTool needs to attach to the WeChat process, Windows will show a **User Account Control** dialog asking for elevated permissions. Click **Yes** to continue.

![Windows UAC prompt asking permission to run WetraceTool.exe](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/a-wetrace-wrapper-for-easy-wechat-key-extraction/2026-04-27-16-13-29.png)

This is expected — the tool cannot capture the key without administrator-level access to the running WeChat process.

## Step 1 — Start WeChat

Once WetraceTool opens you will see the four-step wizard on the left and the current step's instructions on the right.

![WetraceTool step 1 — Start WeChat, showing Launch WeChat button and "Waiting for WeChat to start" status pill](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/a-wetrace-wrapper-for-easy-wechat-key-extraction/2026-04-27-16-14-39.png)

Click **Launch WeChat** to start WeChat automatically, or just open WeChat yourself — the app polls for it in the background and will advance to the next step the moment it detects WeChat is running.

## Step 2 — Get to the WeChat login screen

For the key capture to work, WeChat must be sitting at the login screen (showing a QR code or an account avatar with a "Log in" button) at the moment you sign in. If you are already logged in, you need to log out first.

![WetraceTool step 2 — Log out of WeChat, with numbered instructions and a Continue button](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/a-wetrace-wrapper-for-easy-wechat-key-extraction/2026-04-27-16-15-24.png)

The app gives you the exact steps:

1. Click the **profile / avatar icon** in the bottom-left corner of WeChat
2. Go to **Settings → My Account → Log out**
3. Confirm the log out

Once you can see the QR code or the login button in WeChat, click **I can see the login screen — Continue ✓** in WetraceTool.

## Step 3 — Extract the key

This is the active capture step. WetraceTool starts the background hook service and then waits for you to log in to WeChat. You have **60 seconds** from the moment you click Start.

![WetraceTool step 3 — Extract Encryption Key, showing a 00:41 countdown timer and a "Start — I'm ready to log in" button](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/a-wetrace-wrapper-for-easy-wechat-key-extraction/2026-04-27-16-40-48.png)

When the **Start — I'm ready to log in** button becomes active (it will briefly show "Starting background service…" while the hook initialises), click it and then immediately switch to WeChat and log in. WeChat may reopen a login window — that is normal and expected. Just sign in as you normally would; the key is captured automatically the instant WeChat unlocks its database.

> **Tip:** Have WeChat's QR code scaner (Discover -> Scan) ready on your phone before clicking Start so you don't lose time.

## Step 4 — Copy your key

If the capture succeeded, the wizard advances to the final screen showing your encryption key and, where it can be detected automatically, the path to your WeChat message database.

![WetraceTool step 4 — Key Extracted, showing a redacted encryption key field with a "Copy Key" button and a database path field with a "Copy" button](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/a-wetrace-wrapper-for-easy-wechat-key-extraction/2026-04-27-16-45-01.png)

Click **Copy Key** to copy the encryption key to your clipboard. If the database path was detected, you can copy that too with the **Copy** button beneath it. Both values are what you will need to open your WeChat database with a tool like `decrypt_sync` or any SQLCipher-aware client.

You can close WetraceTool once you have copied what you need.

## Troubleshooting

**Timer runs out with no key captured**
Make sure you log in to WeChat *after* clicking Start, not before. The hook only listens from the moment you click Start.

**"Starting background service…" button never becomes active**
WetraceTool needs elevated permissions to attach to the WeChat process. Close the app, right-click the shortcut, and choose **Run as administrator**.

**WeChat is not detected automatically**
Open Task Manager and check whether `Weixin.exe` or `WeChat.exe` is listed under running processes. If neither appears, WeChat may not have launched correctly — try starting it manually.

**Key extraction fails with a connection error**
The background hook service failed to start or was blocked. Close WetraceTool, reopen it as Administrator, and work through the steps again from the beginning.


2dd5c8b9186ced4737d20374ff6db2c6df332fb70b3f0a6ef45f35a1727404e9

A step-by-step guide to using WetraceTool to extract your WeChat database encryption key on Windows, so you can read your own message history with tools like decrypt_sync.

WetraceTool is a small Windows utility that walks you through extracting your WeChat SQLCipher database encryption key in four guided steps — no command line required.

How to Extract Your WeChat Encryption Key with WetraceTool

{/* banner prompt: Of course! Here are three creative and engaging prompts for a banner image, each with a different artistic focus but all reflecting the core idea of the blog post.

---

### Option 1 (Metaphorical & Cinematic)

This prompt focuses on the idea of an AI breaking free from its digital confines to take direct control of a system.

**Prompt:**
> A cinematic digital art masterpiece. On the left side of the frame, a brilliant, glowing AI neural network is trapped inside a transparent, sandboxed cube representing a browser. A powerful, focused beam of light, composed of streaming green `bash` and blue `PowerShell` command-line text, shatters the cube's wall. This beam, representing the Cygwin SSH connection, extends to the right side of the frame, where it elegantly manipulates a complex, holographic 3D model of the Windows operating system. The beam is precisely deleting registry hives, stopping service gears, and cleaning up file system clutter, transforming chaos into order. The overall aesthetic is dark, futuristic, and infused with glowing neon data streams. --ar 16:9

---

### Option 2 (Direct & Illustrative)

This prompt visualizes the conversational aspect and the direct cause-and-effect of the user's request.

**Prompt:**
> A futuristic command center interface. In the foreground, a sleek, floating chat window displays a human's request: `"Uninstall all bloatware and clean the registry."` An AI responds: `"On it. Establishing SSH connection..."`. In the background, a massive, dynamic data visualization of the Windows OS shows the command in action. Streams of code flow from a central SSH terminal icon, targeting and dissolving icons for unwanted programs, while a 3D representation of the Windows Registry is being methodically scanned and cleaned by glowing green cursors. The style is clean, high-tech, and focuses on UI/UX design, with a dark theme and vibrant blue and green highlights. --ar 16:9

---

### Option 3 (Symbolic & Abstract)

This prompt uses symbolism to represent the fusion of two different tech worlds: the open, command-line world and the Windows GUI world.

**Prompt:**
> An abstract digital painting. A stylized, glowing Tux penguin (representing Linux/Cygwin) and a serene, blue AI entity (representing Claude) are working together as one. They are extending their hands, which are made of pure command-line code, through a shimmering portal shaped like the Windows logo. On the other side of the portal, their code-hands are masterfully deconstructing and reassembling Windows components like services, registry keys, and startup items, represented as intricate, glowing geometric shapes. The background is a dark, digital void, emphasizing the powerful and precise nature of the remote management. The lighting is dramatic and symbolic. --ar 16:9

Additional instructions: An AI maid */}

## The Problem with Browser-Based AI Assistants

If you've ever tried to get an AI assistant to help you actually *do* something on your Windows computer — not just tell you *how* to do it — you've probably hit a wall like this:

> *"I don't have direct access to your Windows system settings, so I can't check or change remote connection settings on your behalf."*

> *"I appreciate you asking, but there are a couple of important limitations here: I can't install software on your Windows computer. I have access to a sandboxed Linux environment for running code, but that's separate from your actual Windows system. I have no way to download or install applications like Cygwin or Python onto your PC directly."*

These are real responses from browser-based AI tools like "Claude Cowork" (Claude.ai in a chat window). They're not wrong in this context, but the limitation is narrower than "can only give advice": Claude Cowork can work with local files you expose to it and automate tasks that live inside the Chrome browser. What it still can't do is directly administer your Windows machine end-to-end the way an SSH-connected agent can.

But what if Claude could actually *do* it?

That's exactly what **Claude Code, Codex, Gemini, or any smart agent** with SSH access enables. Once your Windows machine is reachable over SSH, the agent — running on another machine on your network — can log in, run commands, read files, edit the registry, and manage your system end-to-end. No GUI required. No copy-pasting commands. You just ask.

---

## Part 1: The Hard Part — Making Windows SSH-Accessible

The one-time setup requires physical or RDP access to the Windows machine. After that, you'll never need it again.

### Step 1: Install Cygwin with OpenSSH

Download the Cygwin installer from [cygwin.com](https://www.cygwin.com) and install it. During package selection, make sure to include:

- `openssh` — the SSH server and client
- `cygrunsrv` — used to register Cygwin services as Windows services (will be included in standard installation)
- `bash`, `coreutils`, `grep`, `sed`, `awk`, `findutils` — standard Unix tools (often included in standard installation)
- `git` — useful for cloning repositories and working with source code

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/managing-windows-made-easy-with-cygwin-and-claude/2026-04-15-20-08-42.png)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/managing-windows-made-easy-with-cygwin-and-claude/2026-04-15-20-14-53.png)

The default install path is `C:\cygwin64`.

### Step 2: Clone the Setup Scripts

The scripts are available on GitHub. Open a Cygwin terminal (not PowerShell yet) and clone the repo:

```bash
git clone https://github.com/e-tang/installcygwinsshd
cd installcygwinsshd
```

The repo contains two scripts:
- **`installsshd.ps1`** — installs and configures the Cygwin SSHD service
- **`verify.ps1`** — verifies the installation is working correctly

Basically, to install / enable the sshd service:
```bash
ssh-host-config --yes --cygwin 'ntsec' --name 'sshd' --port 22 --user 'cyg_server'
```

### Step 3: Run the Install Script

Open PowerShell **as Administrator** and run:

```powershell
cd C:\path\to\installcygwinsshd
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\installsshd.ps1
```

What the script does:
- Runs `ssh-host-config` inside Cygwin to generate host keys and create a `cyg_server` service account
- Sets the `CYGWIN=ntsec` environment variable (required for proper permissions)
- Opens port 22 in Windows Firewall
- Sets the `sshd` service to start automatically and starts it immediately

### Step 4: Verify the Setup

Still in the Administrator PowerShell, run:

```powershell
.\verify.ps1
```

### Step 4: Set Up SSH Key Authentication

From your Linux/Mac machine (or another machine running a smart agent: e.g. Claude Code), copy your SSH public key to the Windows machine so you don't need a password every time:

```bash
ssh-copy-id username@a-win-machine
```

You'll be prompted for the Windows user password once. After that, all SSH connections use your key. If `ssh-copy-id` isn't available, do it manually:

```bash
# On the client machine, copy your public key
cat ~/.ssh/id_ed25519.pub

# Then on the Windows machine (via password SSH), paste it into:
# C:\cygwin64\home\username\.ssh\authorized_keys
```

Also add the host to your known_hosts to avoid interactive prompts:

```bash
ssh-keyscan a-win-machine >> ~/.ssh/known_hosts
```

Once this is done, you can SSH in silently:

```bash
ssh a-win-machine "uname -a"
# CYGWIN_NT-10.0-19045 a-win-machine 3.6.7-1.x86_64 ...
```

That's it. **The hard part is over.** From this point on, you manage the machine through Claude Code — just by asking in plain English.

---

## Part 2: The Easy Part — Let Claude Do the Work

### Real Example: Removing Riot Games Completely

Here's a real session. I simply told Claude Code:

> *"I need you to remove / delete 'Riot Games' for me"*

Claude SSHed in, found the installation, confirmed before deleting, removed the directories, then I asked it to clean the registry too. Here's exactly what it did — autonomously:

**1. Locate the installation:**
```bash
ssh a-win-machine "find /cygdrive/c -maxdepth 2 -iname '*riot*' -type d 2>/dev/null"
# /cygdrive/c/ProgramData/Riot Games
# /cygdrive/c/Riot Games
# /cygdrive/c/Riot Games/Riot Client
```

**2. Check for running processes before deleting:**
```bash
ssh a-win-machine "tasklist.exe | grep -i riot"
# (nothing running — safe to proceed)
```

**3. Delete the directories:**
```bash
ssh a-win-machine "rm -rf '/cygdrive/c/Riot Games' '/cygdrive/c/ProgramData/Riot Games'"
```

**4. Search the registry for leftover keys:**
```bash
ssh a-win-machine "reg.exe query 'HKLM\\SOFTWARE' /f 'Riot' /k"
ssh a-win-machine "reg.exe query 'HKCU\\SOFTWARE' /f 'Riot' /k"
```

**5. When I mentioned "but it started when the computer starts"**, Claude found it immediately:
```bash
ssh a-win-machine "reg.exe query 'HKCU\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run'"
# RiotClient  REG_SZ  C:\Riot Games\Riot Client\RiotClientServices.exe --launch-background-mode
```

**6. Deleted the startup entry:**
```bash
ssh a-win-machine "reg.exe delete 'HKCU\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run' /v RiotClient /f"
```

Total time: a few minutes of conversation. No GUI. No digging through Add/Remove Programs. No regedit hunting. Just asking.

---

## Part 3: What Else Can You Do?

Once Claude Code has SSH access to your Windows machine, the possibilities are broad. Here are examples of tasks you can just *ask* for:

### Software Management

```
"Install Python 3.12 silently"
"Check what version of Node.js is installed"
"Uninstall all Adobe products"
"List all installed programs and their versions"
```

Claude can use `winget`, `choco`, or direct installers:
```bash
ssh a-win-machine "winget.exe install Python.Python.3.12 --silent"
ssh a-win-machine "winget.exe list | grep -i node"
```

### Windows Services

```
"Stop and disable the Windows Search indexing service — it's slowing down the disk"
"Restart the print spooler"
"Show me all services that are set to auto-start but are currently stopped"
```

```bash
ssh a-win-machine "sc.exe stop WSearch && sc.exe config WSearch start= disabled"
ssh a-win-machine "sc.exe query type= all state= all | grep -B1 'STOPPED' | grep 'SERVICE_NAME'"
```

### Startup and Scheduled Tasks

```
"Show me everything that runs at startup"
"Remove that suspicious MLWapp startup entry"
"Create a scheduled task to run a backup every night at 2am"
```

```bash
ssh a-win-machine "reg.exe query 'HKCU\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run'"
ssh a-win-machine "schtasks.exe /create /tn 'NightlyBackup' /tr 'C:\\backup.bat' /sc daily /st 02:00"
```

### Disk and File Management

```
"Find the 20 largest files on C: drive"
"Delete all .tmp files in AppData"
"How much space is left on each drive?"
```

```bash
ssh a-win-machine "df -h /cygdrive/c /cygdrive/d"
ssh a-win-machine "find /cygdrive/c/Users/dev/AppData -name '*.tmp' -delete"
ssh a-win-machine "du -sh /cygdrive/c/* 2>/dev/null | sort -rh | head -20"
```

### Registry Management

```
"Disable Cortana telemetry in the registry"
"Find all registry keys pointing to a deleted program path"
"Back up the HKCU Software hive before I make changes"
```

```bash
ssh a-win-machine "reg.exe export HKCU\\SOFTWARE C:\\backup\\hkcu_software.reg"
ssh a-win-machine "reg.exe add 'HKLM\\SOFTWARE\\Policies\\Microsoft\\Windows\\Windows Search' /v AllowCortana /t REG_DWORD /d 0 /f"
```

### Network and Firewall

```
"Show me all open ports and what's listening on them"
"Block port 8080 in the firewall"
"What's my current IP address and DNS servers?"
```

```bash
ssh a-win-machine "netstat.exe -ano | grep LISTENING"
ssh a-win-machine "New-NetFirewallRule -DisplayName 'Block 8080' -Direction Inbound -Protocol TCP -LocalPort 8080 -Action Block" 
ssh a-win-machine "ipconfig.exe /all"
```

### System Health and Monitoring

```
"What processes are using the most CPU right now?"
"How long has this machine been running?"
"Show me the last 20 error events from the System event log"
```

```bash
ssh a-win-machine "tasklist.exe /fo csv | sort -t',' -k5 -rn | head -10"
ssh a-win-machine "wmic.exe os get LastBootUpTime"
ssh a-win-machine "powershell.exe -Command \"Get-EventLog -LogName System -EntryType Error -Newest 20 | Format-List TimeGenerated,Message\""
```

### Windows Updates

```
"Check if there are pending Windows updates"
"Install all available updates and tell me when it's done"
```

```bash
ssh a-win-machine "powershell.exe -Command \"Get-WindowsUpdate\"" 
ssh a-win-machine "powershell.exe -Command \"Install-WindowsUpdate -AcceptAll -AutoReboot\""
```

---

## Why This Beats Browser-Based AI

<Table data={{
    headers: ["Capability", "Browser AI (Claude Cowork)", "Claude Code + Cygwin SSH"],
    rows: [
        ["Read files on your PC", "Yes", "Yes"],
        ["Delete files or programs", "Yes", "Yes"],
        ["Edit the registry", "No", "Yes"],
        ["Manage Windows services", "No", "Yes"],
        ["Run PowerShell commands", "No", "Yes"],
        ["Set up scheduled tasks", "No", "Yes"],
        ["Monitor processes", "No", "Yes"],
        ["Install software", "No", "Yes (via winget/choco)"],
        ["Fix startup items", "No", "Yes"],
        ["Works without GUI", "No", "Yes"],
    ],
}}
/>

The browser-based assistant can mostly *advise*. Claude Code with SSH access can *act*. The gap is enormous.

---

## Tips for Day-to-Day Use

**Give your Windows machine a stable local hostname.** Whether via your router's DHCP reservation or `C:\Windows\System32\drivers\etc\hosts` on the client, `ssh a-win-machine` is much nicer than an IP address `ssh 192.168.X.X`.

**Keep SSH keys, not passwords.** Run `ssh-copy-id` once. After that, Claude Code can SSH in without any credential prompts — fully automated.

**Cygwin gives you Unix tools on Windows.** `grep`, `find`, `awk`, `sed`, `curl` all work inside your SSH session. You're not limited to `cmd.exe` syntax. Mix PowerShell (`powershell.exe -Command "..."`) with Unix tools freely.

**Combine with Claude Code's file editing.** Claude Code can read and edit config files directly over SSH, not just run commands. Got a broken `sshd_config`? Claude can fix it in place.

**It's auditable.** Every command Claude runs goes through the SSH session you can see. Unlike opaque agent frameworks, you can watch exactly what's happening.

---

## Conclusion

The one-time setup — installing Cygwin, running two PowerShell scripts, copying your SSH key — unlocks something genuinely powerful: a Windows machine you can manage entirely through conversation.

No more digging through nested menus. No more manually hunting for registry keys. No more wondering which startup entry belongs to which program. You describe what you want, Claude figures out how to do it, and it's done.

Browser-based AI tools will tell you to "open regedit and navigate to...". Claude Code with SSH will just do it.


493da379e7a8b52a12cdf5a80b0a814bcdfc4feb7d7c7f788e2ed9ac85d33547

A practical guide to setting up Cygwin SSHD on Windows and using Claude Code to manage it remotely — far beyond what browser-based AI assistants can do.

Stop fighting with Windows GUIs. With Cygwin, an SSH daemon, and Claude Code, you can manage any Windows machine from the command line — uninstalling software, cleaning the registry, killing startup entries — just by asking.

Managing Windows Made Easy with Cygwin and Claude

{/* banner prompt: Of course! Here is a creative and engaging prompt for a banner image based on the blog post.

---

### Banner Image Prompt

**Style:** Futuristic digital art, cinematic lighting, sleek and polished, technology theme.

**Prompt:**

A dynamic 16:9 aspect ratio image illustrating the concept of a constrained but powerful AI agent at the frontier of user interface automation.

The scene is dominated by a brightly lit, semi-transparent web browser window, viewed from a slight angle. Inside this "sandbox," an ethereal, glowing AI-powered cursor actively navigates a complex web dashboard, leaving a soft trail of light as it interacts with charts and forms.

The key visual metaphor is the browser's edge: the glowing AI cursor is paused right at the border, its light "bleeding" over the frame as if yearning to reach the computer's main desktop, which is visible through the transparent window. The desktop in the background is slightly darker and out of focus, featuring icons for native applications like a code editor, file folders, and design software—the world just beyond the AI's current reach.

In a corner of the background desktop, a dark terminal window is subtly visible, displaying faint, glowing lines of code to represent the established power of its CLI counterpart, "Claude Code."

The overall mood is one of anticipation and powerful, contained potential, capturing the core idea of an AI that has mastered its limited domain but is on the verge of breaking out to control the entire system.

Additional instructions: Avoid using text, don't use icons, probably just a machanic hand trying scrolling a computer */}

When OpenClaw first hit the scene — defaulting to Claude models— it took the internet by storm. Everyone wanted to try it, everyone wanted to use it. It felt like we had finally reached the dawn of the true AI automation age. It wasn't just a chatbot anymore; it was an agent that could *do*.

To counter this momentum, or perhaps to provide a more controlled alternative, Anthropic released **CoWork**. 

It’s understandable why they initially limited it to Windows and macOS. When you already have **Claude Code** or similar CLI-based agents, you can already do almost anything within a terminal environment—coding, file management, deployments, you name it. In many ways, if you’re comfortable in a shell, CoWork might seem redundant. 

But of course, CoWork has a very specific purpose: it shines when you have tasks that can *only* be performed through a Graphical User Interface (GUI). 

### The Browser Sandbox

Currently, CoWork has its limitations. It primarily automates tasks within the scope of a web browser (specifically Chrome). While this is powerful for web-based workflows, forms filling, CRM management, or social media automation, it feels restricted compared to the "do anything" promise of CLI agents.

I believe that in the future, it will definitely operate within the full scope of the operating system. The current limitation is likely a strategic one—safety concerns are paramount when an AI can move your mouse and click buttons across your entire OS. Anthropic is clearly playing it safe before they give the AI the keys to the entire kingdom.

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/my-thoughts-on-using-claude-cowork/2026-04-15-21-42-38.png)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/my-thoughts-on-using-claude-cowork/2026-04-15-21-51-00.png)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/my-thoughts-on-using-claude-cowork/2026-04-15-21-52-07.png)

What genuinely surprised me, though, was the cross-platform reach. I kicked off the task on CoWork running on Windows, and it executed on my Linux machine — my primary work computer. CoWork scrolled through the page and tracked down the cabin availability I was after.

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/my-thoughts-on-using-claude-cowork/2026-04-15-21-53-49.png)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/my-thoughts-on-using-claude-cowork/2026-04-15-21-51-35.png)

### Expectation vs. Reality

To be honest, initially, I was quite excited to use CoWork. I imagined a tool so powerful it could handle complex, multi-app workflows seamlessly, bridging the gap between my browser and my local tools. 

However, after spending some time with it, it isn't quite what I imagined it would be. The friction of being "browser-bound" and the current latency in GUI interaction makes it feel more like a specialized tool than the universal assistant I was hoping for. It’s a glimpse into the future, but we aren't quite "there" yet.

As it stands, I still find myself reaching for Claude Code for the heavy lifting, keeping CoWork in the toolbox only for those specific moments when a GUI is unavoidable.

### Conclusion

Claude CoWork is a fascinating product—an honest attempt to bring AI agency into the world of point-and-click interfaces. But right now it feels like a proof-of-concept more than a finished power tool. The browser-only sandbox is a deliberate constraint, not an oversight, and I respect Anthropic's caution in rolling it out incrementally. Letting an AI loose across an entire operating system—mouse, keyboard, file system and all—demands a level of trust that has to be earned slowly.

The real promise of CoWork isn't what it does *today*. It's the trajectory it signals. Once the guardrails are mature enough to extend beyond Chrome—to native apps, desktop workflows, and cross-app orchestration—the gap between "AI that helps you write code" and "AI that handles your entire workday" will close fast.

For now, my advice is straightforward: if you live in the terminal, stick with Claude Code. If your workflow is tied to browser-based tools—CRMs, SaaS dashboards, web forms—CoWork is genuinely worth experimenting with. And if you're the kind of person who just wants to watch the frontier move, CoWork is a front-row seat to where human–computer interaction is heading.

We're not quite there yet. But we're closer than we've ever been.


a6b00547f1401001c438383efb436137554f4aeb676710d1e5b64691d3f1fb7e

I spent time with Claude CoWork, Anthropic's answer to OpenClaw and GUI-based AI automation. Here are my honest thoughts on its browser-sandbox limitations, where it genuinely shines, and why I still reach for Claude Code for the heavy lifting.

A first-hand look at Claude CoWork — Anthropic's GUI-based AI agent — exploring its browser-only scope, how it compares to Claude Code, and what the road ahead might look like for AI-driven desktop automation.

My Thoughts on Using Claude CoWork: The GUI Frontier

{/* banner prompt: ### Style: Abstract Technical Architecture / Neon-Futurism

A sleek, abstract 16:9 banner representing a home office network gateway. Instead of a realistic router, the "gateway" is a translucent, crystalline geometric node glowing with internal pulses of cyan and deep violet light. From this central node, intricate, glowing circuits of neon data stream out into a dark, minimalist digital landscape. 

Thin, sharp lines of light represent IPv6 traffic (cyan) and IPv4 traffic (violet), flowing through the node and branching into a complex web of connections. Subtle, floating holographic code snippets like `ipv6 address autoconfig` and `ip route` shimmer in the air around the node. The background is a deep charcoal with a faint, receding digital grid, creating a sense of depth and sophisticated technical architecture. No realistic hardware, no ports, just the essence of a high-performance network bridge. --ar 16:9 */}

## The Goal

We have an old **Cisco 1941/K9** sitting on the shelf as a backup router. The main router — another Cisco 1941 — is handling NAT, DHCP, and internet access for a home office LAN on `192.168.8.0/24`. The ISP provides both IPv4 (DHCP) and IPv6 (native, with a `/48` delegation).

It's important to note that the Cisco 1941 is now quite outdated hardware and is no longer supported by Cisco. If your budget allows, you should definitely look into upgrading to a newer model. However, for a home office environment, if managed properly, these devices are absolute warhorses. They remain relatively safe, highly stable, and can still perform significantly better than most consumer-grade routers currently on the market.

The plan:

1. Access the backup router's console via a serial cable from an Alpine Linux machine (`alpine`)
2. Pull the running config from the main router
3. Add IPv6 (SLAAC + DHCPv6 for LAN clients)
4. Push the full config to the backup router over serial
5. Verify internet works through the backup router before doing the physical swap

## The Setup

```
ISP (IPv4 DHCP + IPv6 /48)
        |
   [Backup Cisco 1941]  ← Gi0/1 (WAN)
   192.168.8.1          ← Gi0/0 (LAN)
        |
   [alpine - Alpine Linux]  ← eth1 (192.168.8.2, connected to backup router)
   192.168.8.99         ← eth0 (connected to main router)
        |
   [Main Cisco 1941]
   192.168.8.1
```

`alpine` also has a DB9 serial cable going to the backup router's console port (`/dev/ttyS0`).

## Step 1: Console Access via Serial

Modern SSH clients don't support SSHv1, so until we configure SSHv2, serial console is our only way in. Alpine Linux has `screen` which works perfectly for this.

```bash
# Start a detached screen session on the console port
sudo screen -dmS cisco /dev/ttyS0 9600

# Attach to it
sudo screen -r cisco
```

The Cisco console defaults are **9600 baud, 8N1** — no need to change anything.

### Sending a Break Signal for ROMmon

Getting to `rommon>` requires sending a break signal within 60 seconds of power-on. The trick is to automate this so timing isn't an issue — fire the break loop, then power-cycle the router:

```bash
# Send break every second for 60 seconds
for i in $(seq 1 60); do
  sudo screen -S cisco -X break
  sleep 1
done
```

Power-cycle the router as soon as the loop starts. When it works, you'll see:

```
rommon 1 >
```

## Step 2: Pulling the Main Router's Config

The main router is running **SSHv1** (Cisco IOS default). Modern OpenSSH 8.0+ dropped SSHv1 entirely, so a standard `ssh` command will fail:

```
Protocol major versions differ: 2 vs. 1
```

The fix: run an old **Ubuntu 14.04** container which ships with OpenSSH 6.6 — the last version to support SSHv1.

```bash
docker run --rm -it --network host ubuntu:14.04 bash
```

Inside the container:

```bash
apt-get update && apt-get install -y openssh-client sshpass
sshpass -p 'yourpassword' ssh -1 -o StrictHostKeyChecking=no \
  admin@192.168.8.1 'show running-config'
```

Save the output — this is your baseline config to replicate on the backup router.

## Step 3: Building the Config with IPv6

The main router config becomes the template. The key additions for IPv6:

### Enable IPv6 Routing

```
ipv6 unicast-routing
ipv6 cef
```

### DHCPv6 Pool for LAN Clients

```
ipv6 dhcp pool LAN-V6
 dns-server 2001:4860:4860::8888
 dns-server 2001:4860:4860::8844
 domain-name yourdomain.com
```

### WAN Interface (Gi0/1)

```
interface GigabitEthernet0/1
 ip address dhcp
 ip nat outside
 ip virtual-reassembly in
 ipv6 address dhcp
 ipv6 address autoconfig default
 ipv6 enable
 ipv6 dhcp client pd ISP_PREFIX
 no shutdown
```

- `ipv6 address dhcp` — requests a WAN IPv6 address from the ISP via stateful DHCPv6
- `ipv6 address autoconfig default` — accepts SLAAC prefix and installs a default route via RA
- `ipv6 dhcp client pd ISP_PREFIX` — requests a delegated prefix (IA-PD); the ISP assigns a `/48` (e.g. `2403:580c:d9d4::/48`) which is bound to the label `ISP_PREFIX`

> **If DHCPv6-PD stays in SOLICIT indefinitely**, prefix delegation is not enabled on your account. Contact the ISP — ask them to enable "IPv6 Prefix Delegation (IA-PD)" for your service. They will assign you a static prefix (e.g. a `/48`). Once enabled it negotiates in seconds.
>
> Without PD, the ISP may assign the WAN address from the *same* `/64` as your LAN prefix, causing an "overlapping" conflict in IOS when you try to configure both interfaces. The workaround is to drop `ipv6 address dhcp` from the WAN, use only `ipv6 enable`, and assign the `/64` statically to the LAN — but LAN clients won't have IPv6 internet routing until PD is sorted.

### LAN Interface (Gi0/0)

```
interface GigabitEthernet0/0
 ip address 192.168.8.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
 ipv6 address ISP_PREFIX ::1/64
 ipv6 enable
 ipv6 nd other-config-flag
 ipv6 dhcp server LAN-V6
 no shutdown
```

- `ipv6 address ISP_PREFIX ::1/64` — assigns the first `/64` from the delegated `/48` to the LAN interface. With `2403:580c:d9d4::/48` delegated, this gives the router `2403:580c:d9d4::1/64` and advertises `2403:580c:d9d4::/64` to LAN clients via SLAAC. The remaining `/64`s (`2403:580c:d9d4:1::/64`, `2403:580c:d9d4:2::/64`, ...) are available for future VLANs or subnets.
- `ipv6 nd other-config-flag` — tells clients to use **SLAAC** for their address but get DNS/domain via DHCPv6 (stateless DHCPv6)
- `ipv6 dhcp server LAN-V6` — serves the DNS info from our pool above

### Static IPv6 Default Route

With only `ipv6 enable` or `autoconfig` on the WAN interface, you need a static default route so the router knows to send IPv6 traffic via the ISP:

```
ipv6 route ::/0 GigabitEthernet0/1 FE80::2A2:FF:FEB2:C2
```

Replace `FE80::2A2:FF:FEB2:C2` with your ISP gateway's link-local address — visible via `show ipv6 neighbors GigabitEthernet0/1` once the interface is up. If `ipv6 address autoconfig default` is configured, IOS will install a default route via RA automatically, so the static route is a belt-and-suspenders fallback.

### Enable SSH Version 2

While we're at it, fix the SSHv1 problem for the future:

```
ip domain name yourdomain.com
crypto key generate rsa modulus 1024
ip ssh version 2
```

## Step 4: Pushing Config via Serial

`screen`'s paste buffer drops characters at serial speeds. The most reliable approach is writing directly to the TTY device:

```bash
#!/bin/sh
TTY=/dev/ttyS0
stty -F $TTY 9600 cs8 -cstopb -parenb clocal -echo raw

send_line() {
    printf '%s\r\n' "$1" > $TTY
    sleep 0.8
}

# Authenticate
printf '\r\n' > $TTY; sleep 1
send_line "hello"           # console password
send_line "enable"
send_line "cisco2"          # enable password
send_line "configure terminal"

# Send config line by line
while IFS= read -r line; do
    printf '%s\r\n' "$line" > $TTY
    sleep 0.6
done < /home/dev/cisco/delta.cfg

send_line "end"
send_line "write memory"
```

Key points:
- Use `stty raw` to disable line buffering on the serial device
- 0.6–0.8 seconds per line is safe at 9600 baud
- Include `exit` commands between sub-modes (e.g. after `interface`, `line`) to avoid the router getting confused about context
- Read the config back with `show running-config` to verify

### Reading Output Back from the Serial Port

To capture what the router is saying, read from the TTY in the background while sending commands:

```bash
sudo cat /dev/ttyS0 > /tmp/router_output.txt &
CAT_PID=$!

# ... send commands ...

kill $CAT_PID
cat /tmp/router_output.txt
```

## Step 5: Verifying the Config

Once applied, check interfaces and IPv6:

```
show ip interface brief
show ipv6 interface brief
show ip nat statistics
show ip ssh
```

Expected output after correct configuration:

```
GigabitEthernet0/0    192.168.8.1     YES manual up   up
GigabitEthernet0/1    192.168.1.x     YES DHCP   up   up
```

And on the LAN client connected to `alpine`'s eth1:

```
eth1: inet6 2403:580c:ffff:35db:xxxx:xxxx:xxxx:xxxx/64
```

SLAAC is working when clients automatically pick up a `2403:580c:ffff:35db::/64` address.

## Step 6: Fixing Dual-NIC Asymmetric Routing on alpine

With `alpine` having two NICs on the same subnet (`192.168.8.0/24`) — one to the main router, one to the backup — Linux routing gets confused. Both are on `192.168.8.0/24`, so the kernel uses whichever it prefers (usually `eth0`) for both.

This means pings sent **out** `eth1` get their **replies routed back via eth0** — the backup router never sees the return traffic.

The fix is **policy routing** — a separate routing table for traffic sourced from `eth1`:

```bash
# Create a routing table for eth1
echo '200 eth1table' >> /etc/iproute2/rt_tables

# Populate it
ip route add 192.168.8.0/24 dev eth1 src 192.168.8.2 table 200
ip route add default via 192.168.8.1 dev eth1 table 200

# Rule: if source is eth1's IP, use table 200
ip rule add from 192.168.8.2 table 200
```

Verify:

```bash
ping -I eth1 -c 3 8.8.8.8
# Should succeed with ~20ms latency
```

To make it persistent, add to `/etc/network/interfaces`:

```
auto eth1
iface eth1 inet dhcp
    post-up ip route add 192.168.8.0/24 dev eth1 src 192.168.8.2 table 200
    post-up ip route add default via 192.168.8.1 dev eth1 table 200
    post-up ip rule add from 192.168.8.2 table 200
    pre-down ip rule del from 192.168.8.2 table 200
```

## Step 7: SSH Shortcut on alpine

Add to `~/.ssh/config` on `alpine` so you can reach the backup router with a simple alias:

```
Host backup-router
    HostName 192.168.8.1
    User admin
    BindAddress 192.168.8.2
    KexAlgorithms +diffie-hellman-group14-sha1
    HostKeyAlgorithms +ssh-rsa
    Ciphers +aes128-cbc
```

Then just:

```bash
ssh backup-router
```

The `BindAddress 192.168.8.2` forces SSH to source from `eth1`, which the policy routing table picks up and routes correctly through the backup router.

## Step 8: Lock Down SSH to LAN Only

Once the router is live on the internet, its public WAN IP (`117.20.x.x`) is immediately visible to automated scanners. By default, SSH is accessible from anywhere. Lock it down to LAN-only with an access-class:

```
ip access-list standard SSH_ACCESS
 permit 192.168.8.0 0.0.0.255
 deny any log
!
line vty 0 4
 access-class SSH_ACCESS in
```

The `deny any log` is important — it logs every blocked attempt, so you can see `show ip access-lists SSH_ACCESS` and watch the hit counter climb. Within minutes of the router going live, you'll see external scanners already hitting it.

The router still listens on TCP port 22 from the WAN side (Cisco doesn't close the port), but drops the connection before authentication for any source IP outside `192.168.8.0/24`. SSH from inside your LAN works normally.

## The Swap

Once verified, the physical swap is straightforward:

1. Unplug the main router's WAN and LAN cables
2. Plug them into the backup router's Gi0/1 (WAN) and Gi0/0 (LAN)
3. The backup router takes over at `192.168.8.1` with identical config + IPv6

LAN clients don't notice — same gateway IP, same DHCP pool, same DNS. They'll additionally start getting IPv6 addresses via SLAAC automatically.

> **Note — give the modem time after the swap.** After plugging the WAN cable into the new router, `Gi0/1` may show `up/up` physically but receive zero DHCP response for several minutes. This is normal — the modem (especially after a reset) needs time to fully come back up and accept a new client. If `show dhcp lease` shows state `Purging` and `DHCP Lease server: 0.0.0.0`, just wait 2–5 minutes before troubleshooting further. You can monitor with:
> ```
> show interface GigabitEthernet0/1 | include input rate|Last input
> ```
> Once you see `input rate` climbing above 0 and `Last input` ticking, the modem is responding and a DHCP lease will follow shortly.

{/* ## Scripts

All scripts used in this process are saved at `/home/dev/cisco/` on `alpine`:

<Table data={{
    headers: ["Script", "Purpose"],
    rows: [
        ["backup_router.cfg", "Full config mirrored from main router + IPv6"],
        ["delta.cfg", "Interface, NAT, line configs (the parts applied separately)"],
        ["send_config.sh", "Push full config via /dev/ttyS0"],
        ["send_delta.sh", "Push delta config via /dev/ttyS0"],
        ["get_config.sh", "Read show running-config from router via serial"],
        ["check_wan.sh", "Check WAN interface + test ping from router"],
        ["check_wan2.sh", "Check interface brief + NAT + ping alpine"],
        ["check_nat.sh", "Check NAT translations and statistics"],
        ["fix_ssh.sh", "Generate RSA keys + enable SSH v2"],
        ["fix_users.sh", "Reset admin/eric passwords via console"],
    ],
}}
/> */}

## Key Takeaways

- **SSHv1 routers**: use an old Ubuntu 14.04 Docker container with OpenSSH 6.6 — it's the cleanest solution
- **Serial config push**: write directly to the TTY device, not through `screen`'s paste buffer — it's more reliable
- **IPv6 on Cisco IOS**: `ipv6 nd other-config-flag` + `ipv6 dhcp server` gives you SLAAC addresses + DHCPv6 DNS info, which is all most LAN clients need
- **ISP IPv6 PD**: if DHCPv6-PD stays in SOLICIT indefinitely, your ISP doesn't have it enabled — contact them and ask for "IPv6 Prefix Delegation (IA-PD)". Once enabled, the ISP assigns a `/48` and LAN clients get full IPv6 internet via SLAAC automatically
- **Dual-NIC same subnet**: always use policy routing — two default routes with different metrics is not enough when both NICs share a subnet
- **Lock down VTY immediately**: apply `access-class` to restrict SSH to LAN-only before the router goes live on the internet — scanners hit public IPs within minutes


1d37c2e2bb3d699a8390560aa6eabec749a70dec63abf926ec97e8ce5c4d57d4

A practical walkthrough of configuring a Cisco 1941/K9 router as a home office gateway backup. Covers console access from Alpine Linux using screen, pulling config from a live router via SSHv1 in Docker, adding IPv6 (SLAAC + DHCPv6), pushing config over serial, and fixing policy routing on a dual-NIC Linux host.

How to set up a Cisco 1941/K9 as a home office gateway — including console access via serial, cloning config from a running router, adding IPv6 with SLAAC and DHCPv6, and fixing dual-NIC asymmetric routing on Alpine Linux.

Configure a Cisco 1941 Router as Your Home Office Gateway with IPv6

{/* banner prompt: Of course! Here are a few creative and engaging prompts for a banner image, designed to capture the essence of the blog post.

### Main Prompt (Cinematic & Evocative)

This prompt focuses on the "resurrection" theme of bringing old hardware back to life with modern, efficient software.

**Prompt:**
> A cinematic, dramatic shot of a dusty, forgotten 90s-era beige PC tower sitting on a workbench in a dimly lit workshop. A brilliant, glowing blue USB cable connects a modern Android smartphone to the PC's front port, acting as a lifeline. Ethereal, glowing data streams, shaped subtly like the Alpine Linux mountain logo, emanate from the PC's vents, illuminating the scene and casting long shadows. The old machine hums with new purpose, transformed into a powerful network gateway. Moody, high-contrast lighting, tech noir aesthetic, hyper-realistic, 8k. --ar 16:9

---

### Alternative Prompts

Here are two other options with different artistic styles.

#### Option 2 (Technical & Stylized)

This prompt emphasizes the technical, blueprint-like nature of the project.

**Prompt:**
> A stylized, futuristic blueprint schematic on a dark grid background, illustrating the post's network topology. At the top, a sleek icon of an Android phone connects via a glowing USB data line to a central, semi-transparent block representing an old computer. The iconic Alpine Linux mountain logo glows brightly from within this central block. From the block, a solid ethernet data stream flows out to a home router icon. The entire image has a clean, technical, and cyberpunk feel, with digital annotations and connection labels. Widescreen, vector illustration style, highly detailed. --ar 16:9

#### Option 3 (Minimalist & Symbolic)

This prompt focuses on the core components in a clean, modern, and symbolic way.

**Prompt:**
> A minimalist composition on a clean, dark grey background. In the center, a simple, iconic representation of an old PC case is given new life. A single, vibrant cyan USB cable plugs into it from one side, originating from an abstract smartphone shape. From the other side, a stable, glowing ethernet cable extends outwards. The Alpine Linux mountain symbol is subtly but clearly etched onto the side of the PC case, glowing with a soft white light. The image conveys elegance, efficiency, and the concept of "powerful simplicity." Professional product shot style, soft studio lighting. --ar 16:9 */}

# Building a 4G/5G Backup Mobile Gateway with Alpine Linux on Old Hardware

*How to repurpose an outdated PC into a reliable backup internet gateway using Android USB tethering*

---

## The Problem

Internet goes down. It always does at the worst possible time. However, even though that only happens a couple of times a year, an always-on 4G/5G backup isn't strictly necessary. It would be nice to have a plug-and-play solution—like using your phone (which probably has plenty of data) as a rescue for those few hours of downtime.

If you have a spare Android phone and an old PC gathering dust, you can build a proper backup gateway that kicks in when your main connection fails — no expensive hardware required.

This guide walks through the entire process, including the real-world gotchas encountered along the way.


## Hardware Requirements

- An old/low-powered PC (sub 1GHz, even under 512MB RAM is workable)
- One or two ethernet NICs (two is ideal, one works fine)
- A USB port for the Android phone
- An Android phone with a mobile data plan
- A USB stick (2GB+ is plenty) to install from

For this project, I used a rugged industrial PC made by **Advantech Australia**. It's equipped with an **Intel Atom D510 CPU** and **2GB of memory**. Its industrial build and dual NIC cards make it a perfect "warhorse" for a dedicated gateway role. 

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/04/repurpose-your-old-pcs-how-to-install-alpine-linux-on-them/2026-04-20-19-42-33.png)

The machine used in this guide had two onboard NICs, an 8GB internal flash module (DOM — Disk on Module), and a larger hard drive set aside for backups. Very modest specs, perfectly adequate for a router role. Please execute me for the dust collected on this machine.


## Why Alpine Linux?

For low-powered hardware doing a single job, Alpine Linux is the right tool:

- Boots in under 5 seconds
- Idles at ~30–50MB RAM
- Tiny base install (~130MB on disk)
- Uses OpenRC (no systemd overhead)
- The `rndis_host` kernel module handles Android USB tethering natively — no extra drivers needed
- Actively maintained, security-focused

Other options like Debian minimal or OpenWrt on x86 are also viable, but Alpine wins on resource footprint for truly constrained hardware.


## Network Topology

With two NICs the setup is clean — no double-NAT:

```
Android Phone (USB tethering)
        |
        | usb0
  [Alpine Linux PC]
   eth0  |  eth1
    |         |
Your Router  (spare / unused)
 (WAN port)
```

With a single NIC, the same approach works — `eth0` connects to your router's WAN port, `usb0` is the phone. Your router handles the LAN side as normal. The Alpine box just NATs between `usb0` and `eth0`.

> **Note on double-NAT:** With a single NIC and a typical home router, you'll have three layers of NAT (phone → Alpine → router). For general internet use this is completely fine. Port forwarding gets complicated, but for a backup scenario it's a non-issue.


## Step 1: Creating the Install USB

Download the Alpine Linux "standard" x86 ISO from [alpinelinux.org](https://alpinelinux.org/downloads/).

Write it to a USB stick using `dd`:

```bash
dd if=alpine-standard-x.x.x-x86.iso of=/dev/sdX bs=4M status=progress
```

Replace `/dev/sdX` with your USB stick's device. **Double-check this** — `dd` will happily wipe whatever you point it at.

> **Important:** Use `dd` directly, or Rufus in **DD mode** if on Windows. Do not let Rufus create a FAT32 partition layout — old BIOSes can choke on it. Alpine ISOs are hybrid MBR images and work correctly when written raw.


## Step 2: BIOS Boot Gotchas (American Megatrends)

Old AMI BIOS boards have a few quirks that can stop USB boot from working even when USB is set as first priority:

- **Plug the USB in before powering on.** USB drives often only appear in the boot device list if they're present at POST time.
- **Check for two separate boot priority menus.** AMI BIOS often has a top-level device-type priority (Removable → HDD → CD) *and* a separate submenu for ordering within each type. The USB stick needs to be first in *both* places.
- **Enable Legacy USB Support** — usually found under Advanced or Integrated Peripherals.
- **Use a rear USB 2.0 port.** Front panel ports and USB 3.0 ports are often ignored by old BIOSes during boot enumeration.
- **Use a USB 2.0 flash drive** if possible. USB 3.0 sticks can fail to enumerate on old BIOSes.
- **Try the one-time boot menu first** — on most AMI boards, hit **F8** or **F11** at POST. This bypasses the boot order entirely and lets you pick the USB stick directly, which is useful for confirming the stick itself is bootable before adjusting BIOS settings.

In my case, I was finally able to have it boot into the Alpine Linux installation interface after plugging the USB drive into the back (rear) of the machine.

---

## Step 3: Identifying Your Disks

Once booted into the Alpine live environment, identify your drives before installing:

```sh
cat /proc/partitions
```

Example output:

```
8    0  312571224  sda     ← ~150GB (Windows drive)
8   16    7880544  sdb     ← ~7.5GB (target install drive)
8   32    3910656  sdc     ← ~3.7GB (Alpine USB stick)
```

To confirm the nature of a drive — particularly whether it's a USB device or an internal flash module:

```sh
dmesg | grep sdb
```

If the output references `sd X:X:X:X` (SCSI enumeration) rather than `usb-storage`, the drive is connected internally — likely a DOM (Disk on Module), common on older appliance-grade hardware. These are perfectly fine for Alpine and handle the low write workload of a router OS well.

> **Tip:** A drive formatted as FAT32 with write cache disabled and a single partition spanning the whole disk is a typical signature of a factory-installed DOM or embedded flash device.

---

## Step 4: Installing Alpine

Run the setup script:

```sh
setup-alpine
```

Work through the prompts:

- **Keyboard layout:** your preference
- **Hostname:** something useful, e.g. `mobile-gateway`
- **Network interface:** configure `eth0` for DHCP during install; skip `eth1` for now
- **Root password:** set something strong
- **Timezone:** your local timezone
- **NTP client:** choose **chrony** (see note below)
- **Mirror:** pick a geographically close one or `r` for auto-detect
- **SSH:** `openssh` is fine
- **Disk mode:** type `sys` — this is a standard permanent install to disk
- **Which disk:** type the device name of your target drive (e.g. `sdb`)

The installer will wipe the selected disk and install Alpine. When finished, remove the USB stick and reboot.

### Why chrony for NTP?

For a machine that's offline frequently or has intermittent connectivity — exactly the backup router use case — chrony is the best choice. It handles time jumps gracefully when connectivity comes and goes, and compensates for clock drift between connections. `busybox-ntpd` is too basic for this, and `openntpd` handles irregular connectivity less gracefully than chrony.

---

## Step 5: Post-Install — Making It Boot

If the machine boots back into another OS after install, the BIOS is still preferring the wrong drive. Go back into BIOS and find the **Hard Disk Drive Priority** submenu — set your Alpine drive (the 8GB/DOM one) as first in that list.

Use the one-time boot menu (F8/F11) to test first — select your Alpine drive directly to confirm it boots before making the BIOS change permanent.

---

## Step 6: Configuring USB Tethering

Plug in the Android phone and enable **USB Tethering** from the phone's settings (usually under Settings → Network → Hotspot & Tethering).

The Alpine kernel will load `rndis_host` automatically and a new interface will appear — typically `usb0`:

```sh
ip link show
```

Bring it up and get an IP via DHCP from the phone:

```sh
udhcpc -i usb0
```

Test connectivity:

```sh
ping -c 3 1.1.1.1 -I usb0
```

---

## Step 7: NAT and IP Forwarding

Install iptables if not already present:

```sh
apk add iptables
```

Enable IP forwarding:

```sh
echo 1 > /proc/sys/net/ipv4/ip_forward
```

Set up NAT (assuming `usb0` = WAN, `eth0` = LAN out to router):

```sh
iptables -t nat -A POSTROUTING -o usb0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o usb0 -j ACCEPT
iptables -A FORWARD -i usb0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
```

Make IP forwarding persistent — add to `/etc/sysctl.conf`:

```
net.ipv4.ip_forward = 1
```

Save and persist iptables rules across reboots:

```sh
rc-update add iptables
/etc/init.d/iptables save
```

---

## Step 8: Hotplug Script (Plug In and It Just Works)

Rather than running commands manually every time, Alpine's `mdev` can detect when the phone is plugged in via USB and automatically bring up the tethering interface and routing.

Create `/etc/mdev/usb-tether.sh`:

```sh
#!/bin/sh
# Called by mdev when a USB network interface appears

IFACE="usb0"

if [ "$ACTION" = "add" ] && ip link show "$IFACE" > /dev/null 2>&1; then
    udhcpc -i "$IFACE" -q
    iptables -t nat -A POSTROUTING -o "$IFACE" -j MASQUERADE
    iptables -A FORWARD -i eth0 -o "$IFACE" -j ACCEPT
    iptables -A FORWARD -i "$IFACE" -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
fi
```

Make it executable:

```sh
chmod +x /etc/mdev/usb-tether.sh
```

This gives you plug-in activation — connect the phone, USB tethering enables on the phone, and the gateway comes up automatically. Unplug to deactivate.

---

## Second NIC — What to Do With It

If your machine has two NICs, you have more flexibility:

- **eth0** — connect to your router's WAN port (LAN out)
- **eth1** — leave unconfigured unless you need a second downstream device connected directly

There's no need to bridge or assign an IP to `eth1` for a basic single-router setup. Leave it unconfigured and use it later if needed.

---

## Summary

{/* | Component | Choice | Reason |
|-----------|--------|--------|
| OS | Alpine Linux | Minimal footprint, fast boot, good hardware support |
| NTP | chrony | Handles intermittent connectivity gracefully |
| Tethering | Android USB (rndis_host) | Native kernel support, no extra drivers |
| Install mode | sys | Standard permanent disk install |
| Activation | mdev hotplug | Plug-in and it just works | */}

The end result is a machine that sits quietly on the shelf, draws minimal power, and becomes a functional mobile gateway the moment you plug a phone into it — no GUI, no fuss, no expensive hardware.

---

*Built and tested on an old low-powered PC with AMI BIOS, dual onboard NICs, and an internal DOM flash drive. Alpine Linux 3.x.*

c8c2ab02361c09d64f2eb621cac10b95098bd376448ee41d0eb5bbc43ad13478

A step-by-step guide to installing Alpine Linux on old, low-powered hardware and configuring it as a mobile backup gateway using Android USB tethering. Covers BIOS quirks, disk identification, NAT/iptables setup, and hotplug automation with mdev.

Turn an old, dusty PC into a reliable 4G/5G backup internet gateway using Alpine Linux and Android USB tethering — no expensive hardware required.

Repurpose Your Old PCs: How To Install Alpine Linux on Them

{/* banner prompt: Of course! Based on the blog post, here are three creative and engaging banner image prompts, each with a different artistic direction. They all capture the core idea of using a phone as a clever, cost-effective backup internet solution.

---

### Option 1: The Digital Lifeline (Conceptual & Dynamic)

**Prompt:**

A dynamic, conceptual illustration of network failover. In the center, a large, futuristic fiber-optic data highway is visibly broken and dark, representing a failed primary internet connection. From the side, a glowing green USB cable connects a modern Android smartphone to a small, sleek mini-PC. This connection creates a vibrant, glowing data bridge that bypasses the broken highway, seamlessly redirecting streams of light (data packets) to a small office network. The scene is set in a dark, tech-themed environment with blue and green neon accents. Style: digital art, high detail, cinematic lighting. **--ar 16:9**

**Why it works:** This prompt visualizes the core concept of a "backup bridge" in a dramatic and futuristic way. The contrast between the broken primary path and the active mobile lifeline makes the solution's purpose instantly clear.

### Option 2: The Smart Switch (Infographic & Clean)

**Prompt:**

A sleek, futuristic infographic-style artwork. Visualize two incoming data streams: a massive, primary blue stream labeled "WAN" that is flickering and disrupted, and a secondary, vibrant green stream originating from an Android phone, labeled "Mobile Backup". Both streams converge on a central, intelligent cube-like gateway device featuring a subtle Alpine Linux penguin logo. The gateway is actively routing the green stream forward, while the blue stream is blocked. The background is a clean, dark gray with a digital grid pattern. The entire image has a clean, high-tech aesthetic with glowing lines and crisp vectors. **--ar 16:9**

**Why it works:** This prompt focuses on the "intelligence" of the solution. It presents the Alpine Linux box as the central brain making a smart decision, which aligns perfectly with the technical, how-to nature of the blog post.

### Option 3: The DIY Hero (Photorealistic & Relatable)

**Prompt:**

A photorealistic shot of a modern tech enthusiast's desk. In the foreground, a compact mini-PC (like an Intel NUC) is the hero, with a subtle Alpine Linux logo visible on a tiny monitor next to it. A braided USB-C cable connects the PC to an Android smartphone, its screen showing the 'USB Tethering Active' notification. Glowing ethernet ports show data flowing out. In the soft-focus background, a primary internet router is clearly visible with a blinking red warning light, indicating an outage. The mood is resourceful and clever, with warm, focused lighting on the foreground setup. **--ar 16:9**

**Why it works:** This prompt grounds the concept in reality, showing the actual components described in the article. It creates a relatable, "I can build that" feeling and highlights the cost-effective, DIY spirit of the solution.

Additional instructions: a flying angel holding a mobile phone */}

## The Problem

For small businesses and home offices, internet downtime is painful. When your primary NBN connection goes down — whether it's a cable fault, an ISP outage, or a modem failure — everything grinds to a halt. Emails stop, video calls drop, cloud tools become unreachable.

The obvious fix is a backup WAN connection, but most solutions involve:
- A dedicated 4G/5G router (~$200–$500)
- A second SIM card with a separate data plan (~$20–$50/month)

That's recurring cost for something you only need occasionally.

## The Better Solution: Use Your Existing Phone

Here's the insight: you almost certainly already have a mobile phone on a plan that includes data. When NBN goes down, that phone in your pocket already *is* a backup internet connection — you just need a way to share it with your whole network instead of just your phone.

Android's **USB tethering** feature turns your phone into a USB network adapter. Plug it into a Linux box and it appears as a regular ethernet interface (`usb0`). From there, you can route your entire network's traffic through it.

**What this costs you:**
- An old PC, mini PC, or single-board computer (even a Raspberry Pi works) — something you may already have spare
- A USB cable
- Your existing phone and mobile plan
- No extra SIM, no extra monthly bill

## Architecture

The setup uses a dedicated Alpine Linux box (we'll call it `mg`) sitting on your LAN alongside your primary NBN router.

```
Normal (NBN up):
  Clients → mg (192.168.8.99) → NBN router (192.168.8.1) → Internet

Failover (NBN down):
  Clients → mg (192.168.8.99) → usb0 (Android phone) → Mobile Internet
```

Clients point their default gateway to `mg`. When NBN is up, `mg` transparently forwards traffic to the NBN router. When NBN fails, you run one command on `mg` and it switches to routing via the phone. Clients don't change anything.

## Requirements

- A PC or mini box with at least one ethernet port, running **Alpine Linux**
- An **Android phone** with USB tethering capability (virtually all modern Android phones)
- A USB cable (USB-A to USB-C or Micro-USB depending on your phone)
- SSH access to the Alpine box

## Step 1: Install iptables

Alpine Linux is minimal by default. Install `iptables` for NAT:

```bash
sudo apk add iptables
```

## Step 2: Enable IP Forwarding

The box needs to forward packets between interfaces like a router:

```bash
echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/99-forwarding.conf
sudo sysctl -w net.ipv4.ip_forward=1
```

## Step 3: Add USB Interface to Network Config

When Android USB tethering is active, it appears as `usb0`. Tell Alpine to obtain a DHCP address from the phone on that interface:

```bash
sudo tee -a /etc/network/interfaces > /dev/null << 'EOF'

# USB tethering (Android backup WAN)
iface usb0 inet dhcp
EOF
```

The kernel modules `rndis_host` and `cdc_ether` handle Android USB tethering. On a standard Alpine install with a reasonably modern kernel, these are already present and loaded automatically when the phone is plugged in with tethering enabled.

## Step 4: iptables Rules at Boot

Create a startup script that sets up NAT and forwarding rules every time the machine boots:

```bash
sudo tee /etc/local.d/wan-setup.start > /dev/null << 'EOF'
#!/bin/sh
# WAN gateway iptables setup

sysctl -w net.ipv4.ip_forward=1

# FORWARD chain — drop by default, allow LAN traffic out via either WAN
iptables -P FORWARD DROP
iptables -F FORWARD
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i eth0 -o eth0 -j ACCEPT
iptables -A FORWARD -i eth0 -o usb0 -j ACCEPT

# NAT — masquerade outbound on both WAN paths
iptables -t nat -F POSTROUTING
iptables -t nat -A POSTROUTING -o eth0 ! -d 192.168.8.0/24 -j MASQUERADE
iptables -t nat -A POSTROUTING -o usb0 -j MASQUERADE
EOF

sudo chmod +x /etc/local.d/wan-setup.start
sudo rc-update add local default
```

Apply the rules immediately without rebooting:

```bash
sudo sh /etc/local.d/wan-setup.start
```

**Why MASQUERADE on `eth0`?** When `mg` forwards a client's packet to the NBN router, without masquerading the source IP is the client's (e.g. `192.168.8.50`). The NBN router sends the reply directly back to that client, bypassing `mg` — breaking the connection tracking. MASQUERADE rewrites the source to `mg`'s own IP (`192.168.8.99`), so all return traffic flows back through `mg` correctly.

## Step 5: The Failover Script

This script handles detecting whether the primary WAN is up and switching routes accordingly. Save it to `/usr/local/bin/wan-failover.sh`:

```bash
sudo tee /usr/local/bin/wan-failover.sh > /dev/null << 'EOF'
#!/bin/sh
# wan-failover.sh — WAN failover between NBN (eth0) and USB tethering (usb0)
# Usage: wan-failover.sh [auto|primary|backup|status|check]

PRIMARY_GW="192.168.8.1"
LAN_IFACE="eth0"
USB_IFACE="usb0"
CHECK_HOST="8.8.8.8"
CHECK_HOST2="1.1.1.1"
STATE_FILE="/var/run/wan-mode"
LOG="/var/log/wan-failover.log"

log() {
    printf '%s %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$1" | tee -a "$LOG"
}

get_mode() {
    cat "$STATE_FILE" 2>/dev/null || echo "primary"
}

check_primary_wan() {
    # Force the check through the primary gateway by adding temporary host routes.
    # This ensures the test uses eth0 even if the current default route is via usb0.
    ip route replace "${CHECK_HOST}/32" via "$PRIMARY_GW" dev "$LAN_IFACE" 2>/dev/null
    ping -c 3 -W 2 "$CHECK_HOST" > /dev/null 2>&1
    result=$?
    ip route del "${CHECK_HOST}/32" 2>/dev/null

    if [ $result -ne 0 ]; then
        ip route replace "${CHECK_HOST2}/32" via "$PRIMARY_GW" dev "$LAN_IFACE" 2>/dev/null
        ping -c 3 -W 2 "$CHECK_HOST2" > /dev/null 2>&1
        result=$?
        ip route del "${CHECK_HOST2}/32" 2>/dev/null
    fi
    return $result
}

bring_up_usb() {
    if ! ip link show "$USB_IFACE" > /dev/null 2>&1; then
        log "ERROR: $USB_IFACE not found — enable USB tethering on the phone first"
        return 1
    fi
    ip link set "$USB_IFACE" up
    udhcpc -i "$USB_IFACE" -q -n -t 5 > /dev/null 2>&1
    ip addr show "$USB_IFACE" | grep -q 'inet ' && return 0
    log "ERROR: Failed to get DHCP lease on $USB_IFACE"
    return 1
}

clear_default_routes() {
    # Remove all current default routes to start clean
    while ip route del default 2>/dev/null; do :; done
}

switch_to_backup() {
    log "Primary WAN DOWN — switching to USB backup..."
    bring_up_usb || return 1

    USB_GW=$(ip route show dev "$USB_IFACE" | awk '/^default/{print $3}' | head -1)
    if [ -z "$USB_GW" ]; then
        log "ERROR: Could not determine USB gateway IP"
        return 1
    fi

    clear_default_routes
    ip route add default via "$USB_GW" dev "$USB_IFACE" metric 10
    echo "backup" > "$STATE_FILE"
    log "Routing via USB ($USB_GW). Mode: BACKUP"
}

switch_to_primary() {
    log "Primary WAN UP — switching back to NBN..."
    clear_default_routes
    ip route add default via "$PRIMARY_GW" dev "$LAN_IFACE" metric 1
    echo "primary" > "$STATE_FILE"
    log "Routing via $PRIMARY_GW. Mode: PRIMARY"
}

status() {
    printf 'Mode:     %s\n' "$(get_mode)"
    printf 'Default:  '; ip route show default
    printf 'eth0:     '; ip addr show "$LAN_IFACE" | awk '/inet /{print $2}'
    if ip link show "$USB_IFACE" > /dev/null 2>&1; then
        printf 'usb0:     '; ip addr show "$USB_IFACE" | awk '/inet /{print $2}'
    else
        printf 'usb0:     not present\n'
    fi
}

case "${1:-auto}" in
    auto)
        mode=$(get_mode)
        if check_primary_wan; then
            if [ "$mode" = "backup" ]; then
                switch_to_primary
            else
                log "Primary WAN OK"
            fi
        else
            if [ "$mode" = "primary" ]; then
                switch_to_backup
            else
                log "Primary WAN still DOWN, staying on backup"
            fi
        fi
        ;;
    primary)  switch_to_primary ;;
    backup)   switch_to_backup ;;
    status)   status ;;
    check)
        if check_primary_wan; then echo "Primary WAN: UP"
        else echo "Primary WAN: DOWN"; fi
        ;;
    *)
        echo "Usage: $0 [auto|primary|backup|status|check]"
        exit 1
        ;;
esac
EOF

sudo chmod +x /usr/local/bin/wan-failover.sh
```

## Step 6: Point Clients to mg

For each machine on your network that you want covered by the failover, change its default gateway from the NBN router's IP to `mg`'s IP (`192.168.8.99`).

- **Windows:** Network adapter settings → IPv4 → Default gateway
- **macOS:** System Settings → Network → Details → TCP/IP → Router
- **Linux:** See below

Alternatively, do it centrally: log into your NBN router's admin page and change the **DHCP gateway option** to `192.168.8.99`. All clients pick this up on their next DHCP renewal with no per-machine changes required.

### Linux: Gateway Switcher Scripts

On Linux, DHCP may leave behind its own default route when you try to change the gateway, resulting in two conflicting defaults. The safest approach is to flush all default routes first, then add the one you want.

Create two convenience scripts in `~/bin/`:

**`~/bin/gw-backup`** — switch to mg (backup WAN):

```bash
#!/bin/bash
while sudo ip route del default 2>/dev/null; do :; done
sudo ip route add default via 192.168.8.99 dev enp2s0 metric 100
echo "Gateway → mg (192.168.8.99) [backup WAN]"
ip route show default
```

**`~/bin/gw-primary`** — switch back to NBN router:

```bash
#!/bin/bash
while sudo ip route del default 2>/dev/null; do :; done
sudo ip route add default via 192.168.8.1 dev enp2s0 metric 100
echo "Gateway → NBN router (192.168.8.1) [primary WAN]"
ip route show default
```

Make them executable:

```bash
chmod +x ~/bin/gw-backup ~/bin/gw-primary
```

Then switching is just:

```bash
gw-backup    # route via mg when NBN is down
gw-primary   # route via NBN router when NBN is back
```

> **Note:** Replace `enp2s0` with your actual interface name (`ip link show` to check). These changes are not persistent across reboots or DHCP renewals — run the script again if needed.

## Using the Failover Script

```bash
wan-failover.sh            # auto: check primary WAN, switch if needed
wan-failover.sh check      # test if primary WAN is up (without switching)
wan-failover.sh status     # show current mode and interface IPs
wan-failover.sh backup     # manually force switch to phone
wan-failover.sh primary    # manually force switch back to NBN
```

## When NBN Goes Down: Step by Step

**1. Enable USB tethering on your Android phone**

On your phone: *Settings → Network & Internet → Hotspot & Tethering → USB Tethering* (on)

Plug the phone into `mg` via USB.

**2. SSH into mg and switch to backup**

```bash
ssh mg
sudo wan-failover.sh backup
```

**3. Verify**

```bash
sudo wan-failover.sh status
ping 8.8.8.8
```

You should see `usb0` with an IP address and pings going through. Clients on the network are now using mobile data transparently — no changes needed on their end.

**4. When NBN comes back**

```bash
sudo wan-failover.sh primary
```

Clients resume using NBN automatically.

## How It Works Under the Hood

When you run `wan-failover.sh backup`, the script:

1. Checks `usb0` exists (phone is plugged in with tethering on)
2. Brings the interface up and runs `udhcpc` to get a DHCP lease from the phone
3. Reads the gateway IP assigned by the phone (typically `192.168.42.129`)
4. Flushes all existing default routes, then adds a single clean one: `ip route add default via 192.168.42.129 dev usb0`

The flush step matters. When `udhcpc` gets a lease it automatically injects its own default route. Without clearing first, you end up with two default routes — the old primary and the new backup — and the kernel picks whichever has the lower metric, which may not be what you want. The `clear_default_routes` loop removes all of them before adding the correct one.

Android's USB tethering uses the RNDIS protocol, which the `rndis_host` kernel module handles. The phone acts as both a DHCP server and a NAT gateway to the mobile network. From the kernel's perspective, `usb0` is just another ethernet interface.

The `check_primary_wan` function tests internet reachability *through* the primary gateway specifically — even if the default route is already pointing at `usb0`. It does this by temporarily injecting a host-specific route (`8.8.8.8/32 via 192.168.8.1`), pinging, then removing it. This prevents the check from falsely succeeding via the backup connection while testing the primary.

## Keeping Costs at Zero

The entire ongoing cost of this solution is **zero**. Your phone's existing mobile plan absorbs the data usage during outages. Most modern plans include more than enough data for light business use — email, web browsing, cloud tools for a small team — for the duration of a typical NBN outage.

Contrast this with a dedicated 4G router plus a SIM plan that costs money every single month regardless of whether your NBN ever goes down.

## Conclusion

A spare computer, Alpine Linux, a USB cable, and a phone you already own is all it takes to give your home or small business network a solid backup WAN. The setup is straightforward, the failover is a single command, and the ongoing cost is nothing.

For the occasional outage — which is exactly the scenario most small businesses and home offices face — this is hard to beat.

<ContactFooter />


5129b9ed386b26385cc2c2e7793f2ca6a76585cfa6373e032a2014f1673cf49f

A step-by-step guide to setting up an Alpine Linux box as a backup WAN gateway for your home or small business network, using Android USB tethering as the mobile data source. Includes a failover script and full network configuration.

Turn an old PC or mini box running Alpine Linux into a backup WAN gateway using your existing Android phone's USB tethering — no extra SIM, no extra plan, no extra cost.

Create a Backup Mobile Data WAN for Your Small Business / Home Network

{/* banner prompt: Of course! Here are a few creative and engaging prompts for a banner image, designed to capture the essence of the blog post.

### Primary Prompt

This prompt uses the metaphor of a "gatekeeper" to represent the brokers restricting access, which is the central theme of the article.

**Prompt:**
> A sleek, futuristic digital art piece. In the foreground, a single, glowing line of code representing a trader's algorithm is stopped just short of a massive, imposing digital gateway. The gateway is labeled "cTrader FIX API". The gate is mostly sealed, with panels glowing red and showing error messages like "RET_ACCOUNT_DISABLED". However, three small, distinct entry ports on the gate are glowing a welcoming green, symbolizing the few brokers that allow access. The overall atmosphere is dark and tech-focused, with dramatic cinematic lighting highlighting the contrast between restriction and opportunity. --ar 16:9

---

### Creative Alternatives

Here are two other options that explore the same theme from different angles.

**Alternative 1 (The Labyrinth Concept):**

This prompt visualizes the search for a compatible broker as navigating a complex maze.

**Prompt:**
> A top-down view of an intricate, glowing circuit board that forms a complex labyrinth. A bright neon data stream snakes through the pathways. Many paths lead to dead ends marked with red, flashing "ACCESS DENIED" nodes. Only a few clear, illuminated pathways lead towards a brightly lit central core labeled "LIVE TRADING". The walls of the labyrinth are subtly embedded with the faded logos of various forex brokers, creating a sense of a difficult, selective journey. Cyberpunk aesthetic, high detail, digital illustration. --ar 16:9

**Alternative 2 (The Data Funnel Concept):**

This prompt uses a data visualization metaphor to show how a large pool of options is filtered down to a select few.

**Prompt:**
> A concept art illustration of a large, translucent holographic funnel suspended in a dark, high-tech environment. Dozens of glowing 3D logos of various forex brokers are being poured into the wide top of the funnel. The funnel narrows sharply, and only a tiny, powerful stream containing three specific, brightly glowing logos emerges from the bottom, flowing directly into a sleek, modern trading terminal displaying financial charts. The rejected logos are stuck and dimming within the funnel, visually representing the filtering process described in the blog. --ar 16:9 */}

So you've built your automated trading system and you're ready to connect it to cTrader via FIX API. Sounds straightforward, right? Well, let me tell you about my recent reality check that'll save you some headaches.

I thought I had it all figured out. Open a cTrader account with any broker offering the platform, hook up my FIX connector, and boom - automated trading paradise. Turns out it's not quite that simple.

If you haven't seen it yet, check my other post about using cTrader: [Trading with cTrader as Your Trading Platform](https://www.tyolab.com/blog/trading-with-ctrader-as-your-trading-platform/).

As I mentioned there, there aren't many big-name brokers that offer cTrader. The brokers I focused on are:
- Pepperstone
- IC Markets
- FP Markets
- Go Markets
- BlackBull Markets

## The BlackBull Markets Surprise

Before I got to BlackBull, I had been trading on FP Markets with a cTrader FIX API connection for quite a while, and everything worked fine until they switched my account from hedged to netted. That setup no longer fit what I needed, so I started looking for another broker that could still offer cTrader trading with FIX API access.

That search led me to open an account with BlackBull Markets. They advertise cTrader as one of their platforms, so naturally, I assumed FIX API access would just work. Wrong.

As soon as my program attempted to connect, the server immediately sent a logout message. Here's what the server spat out:

```
FromAdmin:8=FIX.4.49=11435=534=149=cServer50=QUOTE52=20260211-10:29:17.59356=live.blackbull.106578089=QUOTE58=RET_ACCOUNT_DISABLED10=015
```

The key part? `58=RET_ACCOUNT_DISABLED`. My cTrader account was explicitly restricted from FIX API connections. I could trade through the GUI just fine, but programmatic access was a no-go.

I emailed BlackBull asking for FIX API permissions. Their response? Thanks, but no thanks. Since I'm not an institutional trader with massive holdings, my request was rejected. Fair enough.

## The Great Broker Email Experiment

This got me thinking - which brokers actually support FIX API for regular retail traders? Time for some research. I reached out to the other three main cTrader brokers:

- Pepperstone
- IC Markets  
- Go Markets

Here's the email I sent to each:

```
Dear XXXX Support,

I would like to open a live cTrader account with FIX API enabled. 
Is there any particular requirements for opening such an account?

Kind regards,
Eric
```

Let's break down their responses.

### Go Markets: "Not Available"

Go Markets was refreshingly direct:

> "We regret to inform you that FIX API access is not currently available. However, if you are interested in opening a cTrader live account, you may do so..."

At least they're honest. No FIX API access, period.

### IC Markets: The Green Light

IC Markets gave me hope:

> "You can connect to our cTrader platform via FIX API. More information regarding this can be found at cTrader help centre."

Short and sweet. They seem to support it without any drama.

### Pepperstone: Two FIX Paths

When I emailed Pepperstone, they explained that they offer two different paths, depending on the trader profile.

> "FIX access is generally reserved for institutional level accounts... We can provide a FIX API for clients that are currently trading at least AUD 250,000,000 (2500 lots) in notional volume per month."

For retail traders, they pointed to cTrader FIX API as the alternative.

> "Another option to consider is the API feature offered by Spotware... built into their cTrader platform."

Their follow-up email simply confirmed this: institutional clients can use Pepperstone direct FIX, while retail clients can use FIX through cTrader.

## The Reality Check: Your Actual Options

After this broker survey, here's the brutal truth for retail traders wanting cTrader FIX API access:

**Definitely Available:**
- **IC Markets** - Clear yes, no restrictions mentioned
- **Pepperstone** - Direct FIX for institutional/high-volume clients; retail traders can use FIX through cTrader
- **FP Markets** - They support it without needing to ask for permission

**Definitely Not Available:**
- **Go Markets** - Explicitly stated not available
- **BlackBull Markets** - Requires institutional status

### The Account Type Complication

Here's another wrinkle - account types matter. FP Markets recently changed their account structure from hedged to netted accounts, which didn't work for my trading strategy. This forced me to look elsewhere.

If you need a hedged account specifically, your options narrow down to:
- Pepperstone
- IC Markets
- FP Markets (though you might need to negotiate this as a successful trader)

{/* ## What This Means for Your Trading Setup

If you're building an automated trading system with cTrader FIX API, don't assume every cTrader broker supports programmatic access. Here's what I'd recommend:

### Before Opening an Account

1. **Email the broker directly** about FIX API access
2. **Ask specifically about retail trader restrictions**
3. **Clarify if hedged accounts are available** (if you need them)
4. **Get confirmation in writing** - support responses can vary

### Testing Your Connection

Once you have an account, test your FIX connection immediately. Here's a basic approach to verify access:

```python
# Pseudo-code for testing FIX connection
try:
    fix_session = create_fix_session(broker_config)
    if fix_session.logon():
        print("FIX API access confirmed")
    else:
        print("Connection failed - check account restrictions")
except Exception as e:
    print(f"Connection error: {e}")
``` */}

### Have a Backup Plan

Don't put all your eggs in one broker basket. Given the limited options, consider opening accounts with multiple brokers that support FIX API. This gives you flexibility if one broker changes their policies or has technical issues.

## The Bottom Line

For retail traders wanting cTrader FIX API access, your realistic options are:
1. **Pepperstone** (most reliable option)
2. **IC Markets** (minimum of USD $200 of deposit to activate the cTrader account)
3. **FP Markets** (if you can work with their account types)

That's it. Three brokers out of the major cTrader providers. It's a smaller pool than you'd expect, but at least now you know where to focus your efforts.

The key takeaway? Don't assume FIX API access comes standard with cTrader accounts. Always verify before you start building your trading infrastructure around a specific broker. Trust me, it'll save you the frustration of discovering account restrictions after you've already committed to a platform.


b4c9c58fd16f969b7e36543845c42bdf8f77a33b5c360042e427fe204dad95b1

Discover which forex brokers genuinely support cTrader FIX API access for retail traders - spoiler alert: it's not as many as you'd think.

A reality check on which forex brokers actually allow retail traders to use cTrader's FIX API

cTrader FIX API: Which Brokers Say Yes to Retail Traders?

{/* banner prompt: Of course! Here are a few creative and engaging prompts for a banner image, designed to capture the essence of the blog post. I've provided three distinct options, each with a different creative angle.

### Option 1 (The Human-Centric "Battle")

This prompt focuses on the stressful, hands-on experience of the network administrator, creating a dramatic, cinematic scene.

**Prompt:**
> A cinematic, wide-angle shot of a network administrator in a dimly lit server room, bathed in the glow of a monitor and the ominous amber alert light of a dead router on a rack. On the screen, the Ubiquiti UniFi interface shows a configuration error, while tangled ethernet cables snake across the desk like digital vines. The administrator's face, a mix of intense focus and frustration, is partially visible. The scene should convey a sense of a high-stakes digital battle against time and technology. --ar 16:9 --style photorealistic, dramatic lighting

**Why it works:**
*   **Theme:** Captures the "3-Day Battle" and "Network Disaster" themes.
*   **Keywords:** Incorporates router, server room, Ubiquiti, and the "amber LED."
*   **Emotion:** Evokes the stress and urgency described in the post.
*   **Aspect Ratio:** The wide-angle shot is perfect for 16:9.

---

### Option 2 (The Abstract "VLAN Gotcha")

This prompt visualizes the technical problem itself in a more abstract, high-tech way, focusing on the story's climax.

**Prompt:**
> An abstract, high-tech visualization of a digital network. In the center, a fractured, dark monolith labeled "Meraki" represents the failed router. From one side, glowing blue data packets flow smoothly through a complex, sleek gateway labeled "Ubiquiti." However, these packets are stopped dead by an imposing, semi-transparent red wall labeled "VLAN: DROP UNTAGGED." The entire scene is set against a dark, grid-like background, symbolizing the network infrastructure. The image should be clean, modern, and conceptual. --ar 16:9 --style digital art, cyberpunk, minimalist

**Why it works:**
*   **Theme:** Directly visualizes the core technical conflict, especially the final "VLAN gotcha."
*   **Keywords:** Abstractly represents Meraki, Ubiquiti, and VLANs.
*   **Creativity:** Offers a unique, non-literal interpretation of the problem.
*   **Aspect Ratio:** The horizontal flow of data is ideal for a 16:9 banner.

---

### Option 3 (The Narrative "Cascade of Failure")

This prompt tells the story chronologically within a single image, showing the progression of the problem.

**Prompt:**
> A triptych-style digital illustration within a single 16:9 frame. The left panel shows a sleek Meraki router with a glowing red "CONNECTION FAILED" symbol above it. The center panel shows a complex diagram of PPPoE settings and network switches, with question marks and tangled red lines indicating confusion and struggle. The right panel shows the final solution: a network diagram where ethernet ports are highlighted, with a glowing green shield icon labeled "VLAN 1" finally allowing data to flow to computers and printers. The overall aesthetic is a blend of schematic diagrams and dynamic energy. --ar 16:9 --style infographic, graphic illustration, tech-noir

**Why it works:**
*   **Theme:** Follows the narrative arc of the blog post—failure, struggle, and resolution.
*   **Keywords:** Visually represents the key concepts: Meraki failure, PPPoE struggle, and the VLAN fix.
*   **Clarity:** Clearly communicates the main ideas of the post in a visually engaging way.
*   **Aspect Ratio:** The three-panel layout works perfectly in a 16:9 widescreen format. */}

Nothing strikes fear into a network administrator's heart quite like a dead router during peak business hours. What started as a routine internet outage on a Wednesday afternoon turned into a three-day odyssey through hardware failures, vendor limitations, and configuration nightmares that taught me invaluable lessons about network resilience and the hidden complexities of modern networking equipment.

## The Initial Disaster: When Redundancy Fails

Our office's Meraki router had been rock-solid for years. When the internet went down Wednesday morning, I initially wasn't concerned – these things happen, and our ISP usually resolves fiber issues quickly. But as minutes turned to hours, and our backup USB mobile data modem showed no signs of life in the Meraki dashboard, I knew this wasn't a typical outage.

### The Investigation Begins

Arriving on-site, the symptoms were clear but puzzling:
- Solid amber LED on the Meraki router
- No WAN link lights on the fiber optic connection
- No activity from the USB mobile data modem
- All ethernet ports and USB appeared powerless

The smoking gun came when I connected the WAN cable directly to a switch – instant link lights. The fiber wasn't the problem; the router was dead.

```bash
# Quick fiber connectivity test
# Connect WAN cable directly to switch/laptop
# If link lights appear immediately = router hardware failure
# If no link lights = ISP/fiber issue
```

To confirm my suspicions, I grabbed a consumer-grade home router from our emergency stash. One connection attempt later, we had internet – but with constant WiFi dropouts serving 30+ devices. Not exactly a sustainable business solution.

## The Search for an Emergency Replacement

Here's where the real challenge began. Meraki products come with lifetime warranty, but RMA (Return to Manufacturer Authorization) processes don't respect business emergencies. Even in the best-case scenario, we were looking at several days without proper networking infrastructure.

### Why Finding Business-Grade Alternatives Is Hard

The requirements seemed simple enough:
- Available immediately (no ordering delays)
- Cost-effective for temporary use
- Support 30+ concurrent devices
- Business-grade security and firewall features
- Compatible with our PPPoE internet connection

After researching our options, we settled on:
- **Ubiquiti Cloud Gateway Ultra (UCG-ULTRA)** for routing
- **Ubiquiti U7 Lite WiFi 7** for wireless access
- **24V DC 30W PoE Injector** for powering the access point

## The Ubiquiti Configuration Nightmare

### PPPoE Connection Struggles

What should have been a straightforward setup turned into hours of frustration. The initial PPPoE configuration simply refused to work – the router kept displaying "no internet found" regardless of what I tried.

```yaml
# Initial PPPoE Configuration Attempt
Connection Type: PPPoE
Username: [ISP_USERNAME]
Password: [ISP_PASSWORD]
VLAN ID: [if required by ISP]
MTU: 1500 (or ISP-specific value)
```

Suspecting a faulty unit, I implemented a workaround by converting Port 4 to a WAN port and using the Ubiquiti router as a client of our temporary home router. This bridged connection worked, proving the hardware was functional.

### The Mysterious PPPoE Success

Then something inexplicable happened – after extensive testing and configuration changes, the direct PPPoE connection suddenly started working. No clear explanation, no obvious trigger. Sometimes networking equipment just needs time to "settle in," though this isn't documented anywhere in the manual.

### WiFi Access Point Woes

The networking gods weren't finished with us yet. Our 24V PoE injector turned out to be completely dead – it couldn't power the WiFi access point or even our office VoIP phone. 

**Emergency solution**: We sourced a TP-LINK 8-port PoE Gigabit Switch, which successfully powered the U7 Lite (confirmed by the characteristic blue LED ring).

### UniFi Console Connection Issues

The WiFi access point refused to find the UniFi console during initial setup, consistently failing at exactly 2 minutes and 53 seconds with a "setup was disrupted" error message.

**Resolution**: A factory reset of the access point solved this issue entirely. Sometimes the simplest solutions are the most effective.

### UniFi Settings for the Business Workflow

Beyond the basic setup, we ran into some extra settings we needed for our specific business workflow. With our old Meraki router, we had this sweet IP-based access control for port-forwarding. It let us get to our in-premise NAS drive right from the cloud server.

Now, this Ubiquiti Cloud Gateway Ultra router? It *seems* like it's got that functionality, but honestly, we just couldn't get it to work (I assumed it was my fault entirely). Other than that, the UniFi dashboard itself can feel a bit overwhelming, too. There's info everywhere, but finding exactly what you need? That's another battle entirely.

It has some nice business features though. Take VPNs, for example:
*   You *can* set up WireGuard, OpenVPN, and L2TP servers. I hit a snag trying to create the OpenVPN server though; it kept spitting out an error with a cryptic "default" message.
*   Site-to-site VPN. But didn't test it.

I also had a few headaches setting up fixed IP addresses for certain devices. The error message kept saying the addresses were "already assigned," which was pretty frustrating. But hey, these weren't critical blockers for us, as we're just biding our time for new replacement devices from Cisco anyway. Just a couple more days to wait!

## Meraki RMA: When "Plug and Play" Isn't

Friday arrived with our RMA replacement Meraki router. The promise of Meraki's cloud management is simple: swap the hardware, and everything should work seamlessly. Reality had other plans.

### The PPPoE Déjà Vu

Despite using the correct credentials via the local configuration page (http://192.168.0.1), the new Meraki router refused to establish a PPPoE connection. The status LED remained stubbornly amber.

```bash
# Meraki Local Configuration Access
URL: http://192.168.0.1
Username: [Router Serial Number]
Password: [Leave blank for initial setup]
```

### USB Modem Drama

I still had that USB modem—definitely not the best 4G mobile data modem that Meraki supports, let's just say that. It's temperamental. Sometimes it's a contact issue where it won't even recognize the SIM card. Other times, the SIM registers, but it barely grabs a signal. 

So, here was the situation: 
1) The new router couldn't connect to the internet through the main WAN port.
2) It couldn't connect through the ethernet ports as a client of the Ubiquiti Cloud Gateway.
3) And it definitely wasn't connecting through that USB modem.

Talk about a triple failure, right?

Luckily, after fidgeting with the USB modem on a PC for a while, I finally got an internet connection out of it. My hands were actually shaky as I plugged it back into the router. I was literally praying it would keep working once it was connected. And then—the miracle happened. The router light went from that dreaded amber to a solid, beautiful white. Voilà!

### Multiple WAN Port Strategy

With our temperamental USB mobile data modem providing intermittent connectivity, I configured Port 4 as WAN 2. Using the Ubiquiti router as an intermediate gateway (WAN 2 → Ubiquiti → Internet), we achieved connectivity through DHCP. Another milestone.

The breakthrough came when I switched WAN 2 from DHCP to PPPoE and connected the WAN 2 port with the ISP cable while maintaining the connection through the USB modem. Suddenly, the direct ISP connection worked perfectly.

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/02/router-down-my-3-day-battle-with-meraki-ubiquiti--vlans/2026-02-10-17-22-01.png)

## The VLAN Gotcha That Almost Broke Everything

With internet connectivity restored, I assumed our troubles were over. Wrong. All ethernet-connected devices – computers, printers, phones – couldn't reach the network. WiFi devices worked perfectly.

### The Diagnostic Process

The printer's error message provided the crucial clue: "Cannot connect to DHCP server." This suggested a Layer 2 connectivity issue rather than a DHCP configuration problem.

### The VLAN Configuration Trap

Consulting various resources (including AI assistants), the consensus pointed to VLAN configuration issues. In the Meraki dashboard's "Per-port VLAN Settings" every ethernet port was configured to "Drop Untagged Traffic" instead of the expected "VLAN 1 (Default)."

Go to the VLAN setting page, and find the "Per-port VLAN Settings" section:

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/02/router-down-my-3-day-battle-with-meraki-ubiquiti--vlans/Screenshot_2026-02-10_17-37-04.png)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/02/router-down-my-3-day-battle-with-meraki-ubiquiti--vlans/Screenshot_2026-02-10_17-36-22.png)

```yaml
# Problematic VLAN Configuration
Port 1-8: Drop Untagged Traffic ❌

# Correct VLAN Configuration  
Port 1-8: VLAN 1 (Default) ✅
```

After changing this setting across all ports, ethernet connectivity was immediately restored.

![VLAN Configuration Fix](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/02/router-down-my-3-day-battle-with-meraki-ubiquiti--vlans/2026-02-10-17-21-01.png)

## Lessons Learned: Building Resilient Networks

### 1. Hardware Redundancy Is Non-Negotiable

- Maintain spare networking equipment appropriate for your business size
- Don't rely solely on warranty replacement timelines
- Consider keeping emergency consumer-grade equipment for absolute emergencies

### 2. Understand Your Network's Critical Dependencies

- Document all configuration requirements (PPPoE credentials, VLAN settings, port forwarding rules)
- Test backup connectivity methods regularly
- Ensure mobile data modems are properly maintained and tested

### 3. VLAN Configuration Can Break Everything Silently

Modern networking equipment often ships with conservative security defaults that may not match typical business network expectations. Always verify:
- Default VLAN assignments
- Tagged vs. untagged traffic handling
- Port-specific VLAN policies

### 4. Vendor Lock-In Has Real Costs

While Meraki's cloud management is convenient, the inability to quickly source compatible replacement hardware during emergencies creates significant business risk.

## Configuration Reference for Future Emergencies

### Essential Network Documentation

```yaml
ISP Configuration:
  Connection Type: PPPoE
  Username: [DOCUMENT_THIS]
  Password: [DOCUMENT_THIS]  
  DNS Servers: [PRIMARY], [SECONDARY]
  
VLAN Settings:
  Default VLAN: 1
  Management VLAN: [if different]
  Port Assignments: [document per-port requirements]
  
Critical Port Forwarding:
  Service: [Port Range] → [Internal IP]
  NAS Access: [External Port] → [Internal IP:Port]
```

### Emergency Replacement Checklist

1. **Hardware Inventory**
   - [ ] Spare router compatible with current configuration
   - [ ] PoE switch or injectors
   - [ ] Ethernet cables of various lengths
   - [ ] Console cables for configuration

2. **Configuration Backup**
   - [ ] Current router configuration export
   - [ ] ISP connection parameters
   - [ ] DHCP reservations and ranges
   - [ ] Firewall and port forwarding rules

3. **Testing Procedures**
   - [ ] Verify internet connectivity
   - [ ] Test all VLANs and port assignments
   - [ ] Confirm wireless access point connectivity
   - [ ] Validate critical service access (VPN, port forwarding)

## Conclusion: Preparation Prevents Panic

This three-day networking nightmare reinforced the critical importance of preparation in network administration. While we ultimately resolved every issue and restored full functionality, the business impact of extended downtime could have been significantly reduced with better emergency planning.

The combination of hardware failure, vendor dependency, and subtle configuration issues created a perfect storm that tested every aspect of our network knowledge. The VLAN configuration issue, in particular, demonstrates how modern networking equipment's security-first approach can create unexpected gotchas for administrators familiar with older, more permissive defaults.

Moving forward, we've implemented a more robust emergency response plan, including spare hardware inventory and comprehensive configuration documentation. Because the next networking emergency isn't a matter of if – it's when.

*Have you experienced similar networking emergencies? What lessons did you learn from your network disasters? Share your war stories in the comments below.*

d6ab09a6b47b526abe2e10f3086bc22732aad159e090534a158281bf840d7573

When our office Meraki router died, we scrambled through Ubiquiti replacements, PPPoE struggles, and a sneaky VLAN configuration that taught us valuable lessons.

A real-world network disaster story: router failure, emergency replacement, and the VLAN gotcha that almost broke everything

Router Down! My 3-Day Battle with Meraki, Ubiquiti & VLANs


{/* banner prompt: Of course! Here are three creative and engaging prompts for a banner image, each with a different artistic direction but all reflecting the core theme of the blog post.

---

### Prompt 1: The Beacon in the Void

This prompt uses a dramatic, cinematic approach to visualize the problem and solution as a source of light in darkness.

**Prompt:**
```
A cinematic, photorealistic image of a vast, dark, digital abyss representing a system failure. In the center, a single computer monitor floats, its screen pitch black except for the faintly glowing, ominous text "Please Wait...". From the top right corner, a powerful, clean beam of light, symbolizing a Windows update, descends and strikes the monitor. Where the light hits, the black screen begins to dissolve, revealing a vibrant, fully-loaded Windows desktop underneath. The light is composed of tiny, flowing data particles and the iconic circular update arrows. Moody, dramatic lighting, deep shadows, and a strong contrast between the dark void and the bright, hopeful light. --ar 16:9
```
**Why it works:** This prompt creates a powerful metaphor. The "black screen" is an empty void, and the "Windows Update" is a literal beacon of hope and functionality, cutting through the darkness to restore the system. It's visually striking and tells the story of the blog post in a single image.

---

### Prompt 2: The Digital Locksmith

This prompt takes a more conceptual, cyberpunk-inspired approach, framing the problem as a lock and the solution as the key.

**Prompt:**
```
A sleek, futuristic digital art piece. An IT professional is seen from behind, facing a massive, impenetrable digital wall that glows with a faint grid pattern. The wall has a single line of text embedded in it: "Please Wait". In the professional's hand is a glowing, holographic key forged from the Windows logo and circuitry patterns. They are about to insert this "update key" into a barely visible keyhole on the wall, causing cracks of bright blue light to emanate from the insertion point, ready to shatter the barrier and reveal the bright, working remote desktop behind it. Cyberpunk aesthetic with neon blues, blacks, and whites. --ar 16:9
```
**Why it works:** This prompt personifies the solution. The update isn't just a process; it's the "key" that the IT professional uses to unlock the problem. It highlights the user's frustration (the impenetrable wall) and the simple, elegant power of the solution (the key).

---

### Prompt 3: The System Corruption and Repair

This prompt visualizes the issue from a high-tech, schematic point of view, perfect for a more technical audience.

**Prompt:**
```
A high-tech, holographic schematic of a computer's system architecture, rendered in glowing blue and green lines against a dark background. A critical section labeled "RDP Service" is shown as dark, corrupted, and glitching, with a red "Please Wait" error message flickering nearby. From a main system node labeled "Windows Update Core," a clean, flowing stream of pristine data particles travels along the circuits to the corrupted RDP section. As the data arrives, it purges the corruption, turning the dark, glitching lines back to a healthy, vibrant blue, fixing the error in real-time. Style of an advanced UI display, sharp focus, depth of field. --ar 16:9
```
**Why it works:** This prompt appeals directly to a technical understanding of the problem. It visualizes the "pending update" as a form of system corruption or inconsistency that the update process actively "repairs." It’s a clean, modern, and intelligent way to represent the blog post's main idea. */}

If you've ever tried to connect to a Windows machine via RDP only to be greeted by an endless black screen displaying "Please Wait," you know how frustrating this can be. The good news? It could be an easy fix.

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/02/rdp-please-wait-black-screen-a-simple-fix/please-wait.png)

In my case the target Windows machine has pending system updates that require installation or a restart to complete. Windows Update can interfere with the RDP service's ability to properly initialize the desktop environment, leaving users stranded at the "Please Wait" screen.

## The Simple Solution: Windows Updates

No matter how much we don't want to, but we have to: **update Windows**.

### Step-by-Step Fix

#### Option 1: Physical or Local Access

If you have physical access to the machine:

1. **Log in locally** to the Windows machine experiencing the RDP issue
2. **Check for Windows Updates**:
   - Press `Windows + I` to open Settings
   - Navigate to "Update & Security"
   - Click "Check for updates"
3. **Install all available updates**
4. **Restart the machine** when prompted
5. **Test RDP connection** after the restart completes

#### Option 2: Command Line Approach

If you have limited access but can run commands:

```shell
# Check for updates using PowerShell
Get-WindowsUpdate

# Install updates (requires PSWindowsUpdate module)
Install-WindowsUpdate -AcceptAll -AutoReboot

# Alternative: Use Windows Update command line
UsoClient StartScan
UsoClient StartDownload
UsoClient StartInstall
```

#### Option 3: Group Policy Method

For domain environments:

```shell
# Force Group Policy update
gpupdate /force

# Trigger Windows Update check
wuauclt /detectnow /updatenow
```

### Verification Steps

After applying updates and restarting:

1. **Test RDP connectivity**:
   ```shell
   # Test RDP port connectivity
   telnet target-machine-ip 3389
   ```

2. **Check RDP service status**:
   ```powershell
   # Verify Remote Desktop Services are running
   Get-Service -Name "TermService"
   Get-Service -Name "UmRdpService"
   ```

3. **Verify Windows Update status**:
   ```powershell
   # Check for pending updates
   Get-WindowsUpdate
   
   # Check update history
   Get-WUHistory | Select-Object Date, Title, Result
   ```



9e362af613eefa3f6ac31621c3de02c39258939f9572d02f57ca4c7930aaf05a

Struggling with RDP showing a black 'Please Wait' screen? Learn how pending Windows updates cause this issue and get the simple fix.

Fix RDP black screen with 'Please Wait' message by updating Windows - a simple solution that works.

RDP 'Please Wait' Black Screen? A Simple Fix

{/* banner prompt: Here is a creative and engaging prompt for a banner image, based on the blog post:

**Prompt:**

A cinematic, futuristic banner image in a **16:9 aspect ratio**, depicting the technological duality of China and Japan. The image is a split composition:

*   **On the left:** A vibrant, neon-drenched Shanghai night street. A hand holds a smartphone, scanning a large, glowing QR code that hovers in the air above a food stall. The background is filled with diverse, sleek electric cars and bustling activity, capturing a world of hyper-digital convenience.
*   **On the right:** A clean, high-tech Tokyo scene. A different hand holds an iPhone with a Suica card visible in the wallet, tapping it against a glowing train station turnstile as a Shinkansen bullet train glides by in the background, embodying precision and integrated infrastructure.

The two worlds are divided by a subtle, shimmering digital light, symbolizing the bridge between two distinct technological futures. The style is polished digital art with a touch of cyberpunk aesthetic. */}

As a tech professional traveling through Asia, you'll quickly discover that China and Japan represent two fascinating extremes of digital adoption. While both countries showcase incredible technological advancement, their approaches to cashless payments and electric vehicle integration couldn't be more different. Here's what I learned navigating the digital landscape of both nations.

## The Great Digital Divide: Payment Systems

### China - The QR Code Kingdom

China has achieved something remarkable: a truly cashless society. Walking through any Chinese city, you'll witness the seamless integration of digital payments into every aspect of daily life. From street vendors to luxury retailers, everything revolves around scanning QR codes.

However, there's a catch for foreign visitors. The dominant payment platforms - WeChat Pay and Alipay - traditionally required Chinese bank accounts and phone numbers for full functionality. Even as someone originally from China, I found myself locked out of anonymous WeChat Pay payments, highlighting the system's focus on local users.

#### The Foreign Visitor's Solution

Fortunately, **Alipay International** has emerged as a game-changer for travelers:

```markdown
Alipay International Setup Process:
1. Download Alipay app (any version that is available in your region)
2. Register an account
3. Bind credit card(s) of your choice (Wise or Revolut, Wise might be better for China as you can pre-load Chinese Yuan and avoid the transaction fee)
4. Use your bank's app for transaction authorization
5. Start scanning QR codes everywhere
```

The beauty of this system lies in its simplicity - no complex verification, no Chinese phone number required, just your existing credit cards.

Now enjoy your Chengdu hotpot:

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-09-25.png)

Get some spicy snacks:

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-07-53.png)

See a panda:

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-10-39.png)

Visit a museum (Sanxingdui):
![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-11-40.png)


![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-13-30.png)


![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-14-21.png)

Get some breakfast:
![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-16-10.png)

See a bianlian:
![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-19-07.png)

Visit another museum:

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-20-06.png)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-20-52.png)

Walk on an ancient wall:

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-22-18.png)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-22-45.png)

Enjoy a particular Cantonese cuisine:
![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-18-00-04.png)

This is my favorite:
![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-18-01-22.png)

#### Managing Exchange Rates and Fees

Here's where the technical details matter for your budget:

```markdown
Payment Options Comparison:
├── Local Bank Cards (e.g., CBA)
│   └── ~4% international transaction fee
├── Wise (Multi-currency)
│   ├── Chinese Yuan support ✓
│   ├── Japanese Yen support ✓
│   └── Better exchange rates
└── Revolut
    ├── Chinese Yuan support ✗
    ├── Japanese Yen support ✓
    └── Competitive rates for Japan
```

While 4% seems steep, consider that physical exchange shops typically charge around 15% profit on each transaction. Suddenly, digital payments look quite reasonable.

### Japan - The Cash-Digital Hybrid

Japan presents a more complex payment landscape. While cashless options exist and are growing, cash remains king in many situations. This hybrid approach requires a more nuanced strategy.

#### The iPhone Advantage

For tech professionals carrying iPhones, you have access to a secret weapon: **Suica integration**. Unlike Android devices, iPhones support Japan's Suica transportation card directly in Apple Wallet.

```markdown
Suica Use Cases:
├── Transportation
│   ├── JR trains
│   ├── Subway systems
│   └── Local buses
└── Retail
    ├── Fast food chains (e.g., McDonald's, KFC, etc)
    ├── 7-Eleven convenience stores
    ├── Supermarkets
    ├── Ramen shops
    ├── Department stores
    └── Vending machines
```

The Suica card functions even where traditional credit cards are declined, making it invaluable for minimizing cash dependence.

#### Pre-Trip Preparation for Japan

Unlike China's universal QR code acceptance, Japan requires preparation:

- **Load cash before arrival** - Some tourist area lockers only accept coins
- **Set up Suica on iPhone** - Load it with Wise/Revolut for better rates
- **Carry backup cash** - Not all vendors accept digital payments

#### When We Were in Japan
![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-27-12.png)
(Beautiful Snow View in Hokkaido)

Grab a drink (cash only):
![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-31-19.png)

Alcohol in Japan is just too cheap:

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-56-43.png)

But taxi not so much:

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-55-00.png)

## Transportation Technology: A Tale of Two Approaches

### China's Ride-Sharing Revolution

China's transportation ecosystem showcases deep digital integration, but it comes with foreign visitor challenges. My experience with Didi (China's Uber equivalent) in Dongguan illustrates both the potential and pitfalls:

> **The Tragedy of Dongguan**
>
> On a roadside in Dongguan, I opened Alipay and called a ride through the embedded Didi system, the local version that feels slightly different from what foreigners expect. The app assured me the car was a few minutes away, so I waited by the curb, listening for the hum of the approaching motor.
>
> The system then announced that the ride had arrived, but the street in front of me was empty. I scanned the road in both directions, searching for any sign of the vehicle, and realized I needed to contact the driver. A phone call was not an option without international credits, and even if it were, the cost would have been higher than the ride itself.
>
> I turned to the in-app messaging feature, but the send button would not work. I kept tapping, trying to ask where the driver was, and the frustration grew with every failed attempt.
>
> Later I understood what had happened: the system believed I was underground in a basement while I was standing on the street above. I had to cancel the ride, and then Alipay declined the payment and flagged my account as a risk, locking me out at the worst possible moment.

```markdown
Didi Integration Challenge:
Problem: GPS location mismatch (basement vs. ground level)
Issue: Unable to contact driver
- No international calling credits
- Message function broken ("send" button non-responsive)
- Language barrier in app interface

Result: Cancelled ride → "Account risk" status → Payment blocked
```

This experience highlights the importance of having backup transportation options and understanding the apps' limitations for international users.

Another practical hack is to keep a backup phone with Alipay registered to a different phone number and email, just in case.


### Electric Vehicle Landscape

#### China: EV Paradise

The sheer variety of electric vehicles on Chinese streets is staggering. What struck me most was the accessibility:

```markdown
Chinese EV Market Snapshot:
├── Local Brands
│   ├── BYD
│   ├── NIO
│   ├── XPeng
│   └── Li Auto
├── International Brands
│   ├── Tesla
│   ├── Volkswagen ID series
│   ├── Mercedes EQC
│   └── Audi e-tron
└── Price Points
    └── VW Tiguan-sized EV: ~150,000 yuan (~$30,000 AUD)
```

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-17-23-45.png)
(Bus is charging)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2026/01/a-glance-for-china-and-japan-cashless-future--electric-cars/2026-01-20-18-02-24.png)
(Haven't seen a Nissan electric car before)

The integration into ride-sharing services means you'll likely experience multiple EV models during your stay, often without even realizing it.

#### Japan: Precision Over Proliferation

Japan's approach emphasizes refinement over rapid adoption. While you won't see the EV variety found in China, you'll witness incredible attention to detail in transportation infrastructure. The precision of train platform heights matching door levels perfectly exemplifies Japanese engineering philosophy.

## Final Thoughts

Traveling through China and Japan as a tech professional provides fascinating insights into different approaches to digital transformation. China's aggressive push toward cashless payments creates incredible convenience for locals but challenges for visitors. Japan's more measured approach maintains accessibility while gradually introducing digital options.

The key to navigating both systems successfully lies in preparation, understanding the technical limitations, and having backup plans. Whether you're scanning QR codes in Shanghai or tapping your iPhone's Suica card in Tokyo, these experiences offer valuable perspectives on how different cultures integrate technology into daily life.

For tech professionals, these journeys aren't just about tourism - they're field research into the future of digital payments and transportation technology. The lessons learned from both successes and failures in these advanced digital economies can inform better product development and user experience design for global applications.

*Planning a tech-focused trip to Asia? Consider these payment and transportation technologies as essential tools rather than conveniences - they'll transform your travel experience and provide invaluable insights into the future of digital integration.*


398af4e1a7a8188eca79ba44c2f3a1e65dc2a84699bdca3f9fe2ea4f3420c6c1

Navigate China and Japan's digital payment systems and electric vehicle landscape with practical tips for tech professionals traveling abroad

Technical insights on digital payments and EV adoption across China and Japan for tech travelers

China & Japan at a Glance: Cashless Futures and Electric Cars

4ed299827a75af8f7aefa9e95bbdce268e40ae1125bec5fc9dfc74f5e382cbd2

Learn about the React2Shell vulnerability (CVE-2025-66478) affecting Next.js apps, how attackers exploit it, and immediate steps to secure your applications.

Critical CVE-2025-66478 vulnerability allows remote code execution in Next.js applications worldwide.

React2Shell: Critical RCE Flaw in Next.js & How to Fix It


## The Problem: VirtualBox Was Driving Me Crazy

Picture this: You're running Windows 11 in VirtualBox on Linux. Everything seems fine at first, but then... freeze. Every. Few. Minutes.

Your mouse cursor stops moving. Windows hangs. You force-kill the VM and start again. The cycle repeats.

Sound familiar? That was my reality until yesterday.

## The Symptoms

My VirtualBox Windows 11 VM exhibited all the classic signs of trouble:
- **Constant freezing** after a few minutes of use
- **CPU spinning at 80-100%** on the host
- **Freezes even when actively working** (not just idle)
- **8GB of swap fully utilized** with barely any free RAM
- Getting stuck in **UEFI firmware loops** after tweaking settings

I had tried everything:
- Adjusting RAM allocation (4GB, 6GB, 8GB)
- Tweaking CPU settings
- Disabling 3D acceleration
- Switching graphics controllers (VBoxSVGA, VMSVGA)
- Messing with paravirtualization settings (Default, Hyper-V, None)
- EFI vs BIOS firmware modes

Nothing worked consistently.

## Bingo! The Real Culprit: VirtualBox + Windows 11 = Bad Chemistry

Here's what I eventually discovered: **VirtualBox 7.0.x has known stability issues with Windows 11**, especially with:
1. **EFI firmware** - Windows 11 requires UEFI, but VirtualBox's EFI implementation is buggy
2. **Memory pressure** - The host system was critically low on RAM with swap exhausted
3. **Graphics driver crashes** - VirtualBox's display adapters don't play nice with Windows 11

The root cause wasn't just one thing - it was a perfect storm of VirtualBox limitations, Windows 11's modern requirements, and my system running too many memory-hungry applications simultaneously.

**Why this matters:** Windows 11 is designed for modern virtualization features (TPM 2.0, Secure Boot, UEFI) that VirtualBox implements as an afterthought. KVM/QEMU, on the other hand, was built from the ground up for Linux and integrates these features natively.

## The Solution: Migrate to KVM/QEMU

After diagnosing the issue, the solution became clear: **KVM/QEMU is simply better for Windows 11 on Linux**.

But here's the problem: KVM/QEMU requires mastering an intimidating array of command-line tools. Who remembers all the `virsh`, `virt-install`, `qemu-img` commands and their countless flags?

This is where AI saved the day.

## The AI-Assisted Migration Journey

### Old Way vs AI-Assisted Way

**The Traditional Approach (what I would have done before):**
1. Google "How to install Windows 11 on KVM"
2. Find a blog post from 2019 (outdated)
3. Try commands blindly
4. Hit errors
5. Back to Google "KVM Windows 11 no disk found"
6. Forum posts with conflicting advice
7. Try random solutions
8. More errors
9. Give up and stick with broken VirtualBox
10. Total time: Hours to days, with mounting frustration

**The AI-Assisted Way:**
1. Describe the problem to AI
2. AI diagnoses VirtualBox + Windows 11 compatibility issues
3. AI suggests KVM/QEMU migration
4. AI guides through systematic installation
5. Hit roadblocks? AI troubleshoots in real-time
6. Total time: Less than 2 hours, working solution

The key difference: **AI doesn't just give you commands to copy-paste**. It:
- Explains *why* each step is necessary
- Adjusts when things don't work
- Provides context for troubleshooting
- Creates helper scripts so you never need to remember commands again

## Step-by-Step: How We Did It

### Phase 1: System Preparation

First, we needed to switch from VirtualBox to KVM (they can't run simultaneously):

```bash
# Check if KVM modules are loaded
lsmod | grep kvm

# Unload VirtualBox modules
sudo modprobe -r vboxdrv vboxnetflt vboxnetadp

# Load KVM modules (for Intel CPU)
sudo modprobe kvm
sudo modprobe kvm_intel
```

**Dead End #1:** Initially tried to run both side-by-side. **Error: "VirtualBox can't operate in VMX root mode"**. Turns out, both need exclusive access to CPU virtualization features.

**The Fix:** Created a switcher script (`virt-switch.sh`) to toggle between them:

```bash
#!/bin/bash
# Switch between VirtualBox and KVM

case "$1" in
    vbox)
        sudo modprobe -r kvm_intel kvm
        sudo modprobe vboxdrv
        ;;
    kvm)
        sudo modprobe -r vboxdrv vboxnetflt vboxnetadp
        sudo modprobe kvm kvm_intel
        ;;
esac
```

### Phase 2: Installing KVM Tools

```bash
# Install everything needed for KVM/QEMU
sudo apt-get install -y qemu-kvm libvirt-daemon-system \
    libvirt-clients bridge-utils virt-manager virtinst

# Add user to libvirt group
sudo usermod -aG libvirt,kvm $USER

# Start libvirt service
sudo systemctl start libvirtd
sudo systemctl enable libvirtd
```

### Phase 3: Creating the Windows 11 VM

This is where things got interesting. The initial attempt:

```bash
sudo virt-install \
  --name=Windows11 \
  --ram=8192 \
  --vcpus=4 \
  --disk path=/data/vms/windows11.qcow2,size=80,bus=virtio \
  --cdrom=/path/to/Win11_25H2_English_x64.iso \
  --os-variant=win11 \
  --graphics spice \
  --boot uefi
```

**Dead End #2:** VM created successfully, but stuck in **UEFI firmware loop**. Pressing "Continue" just returned to the same screen.

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/12/escaping-virtualbox-hell-how-i-migrated-windows-11-to-kvm-with-ai/Screenshot_2025-12-04_10-21-03.png)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/12/escaping-virtualbox-hell-how-i-migrated-windows-11-to-kvm-with-ai/Screenshot_2025-12-04_10-21-36.png)

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/12/escaping-virtualbox-hell-how-i-migrated-windows-11-to-kvm-with-ai/Screenshot_2025-12-04_10-21-52.png)

**Why:** The boot order defaulted to HDD first, but there's no OS installed yet. It needs to boot from CDROM first.

**The Fix:** Edit the VM XML configuration:

```bash
sudo virsh dumpxml Windows11 > /tmp/win11.xml
# Edit the <os> section to add:
#   <boot dev='cdrom'/>
#   <boot dev='hd'/>
sudo virsh define /tmp/win11.xml
```

### Phase 4: The Disk Driver Mystery

Windows installation started, but then: **"Where do you want to install Windows?" showed NO DRIVES**.

**Dead End #3:** Windows couldn't see the virtio disk because it doesn't have the drivers built-in.

**Why:** VirtIO is a paravirtualized storage interface - much faster than emulated SATA, but requires special drivers.

**Solution Options:**
1. Add a second CD with VirtIO drivers ISO (complex)
2. **Switch to SATA** (Windows supports it natively, slightly slower but works instantly)

We went with option 2 for simplicity:

```bash
# Recreate VM with SATA disk
sudo virsh undefine Windows11 --nvram
sudo virt-install \
  --name=Windows11 \
  --ram=8192 \
  --vcpus=4 \
  --disk path=/data/vms/windows11.qcow2,bus=sata \
  --cdrom=/var/lib/libvirt/images/Win11_25H2_English_x64.iso \
  --network model=e1000e \
  --os-variant=win11 \
  --boot uefi \
  --tpm backend.type=emulator,backend.version=2.0
```

**Bingo!** Windows installer now saw the 80GB drive.

### Phase 5: Display Resolution Hell

Windows installed successfully, but the display was stuck at **1280x800** with everything looking blurry and oversized. The **Display Settings resolution dropdown was grayed out**.

**Why:** Windows needs the SPICE Guest Tools (includes QXL display drivers) to support dynamic resolutions.

**Dead End #4:** Initially tried increasing VRAM in the VM config while it was running - changes didn't take effect.

**The Fix:**

1. Increase video RAM to 512MB:
```bash
sudo virsh destroy Windows11  # Stop VM
sudo virsh dumpxml Windows11 | \
  sed "s/ram='262144'/ram='524288'/" | \
  sed "s/vram='262144'/vram='524288'/" > /tmp/win11-hires.xml
sudo virsh define /tmp/win11-hires.xml
sudo virsh start Windows11
```

2. Install SPICE Guest Tools in Windows:
   - Download from: `https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe`
   - Install and reboot Windows
   - **Result:** Resolution dropdown now shows 1920x1080!

### Phase 6: Making It User-Friendly

The problem with KVM/QEMU: Too many commands to remember. Who wants to type:

```bash
sudo virsh start Windows11 && virt-viewer --connect qemu:///system Windows11
```

...every time they want to launch Windows?

**Solution:** Created a management script (`~/bin/win11`):

```bash
#!/bin/bash
# Windows 11 KVM Manager

case "$1" in
    start)
        # Switch to KVM if needed
        if ! lsmod | grep -q "^kvm"; then
            sudo /home/dev/virt-switch.sh kvm
        fi
        # Start VM
        sudo virsh start Windows11
        sleep 3
        virt-viewer --connect qemu:///system Windows11 &
        ;;
    stop)
        sudo virsh shutdown Windows11
        ;;
    status)
        sudo virsh dominfo Windows11
        ;;
    # ... more commands ...
esac
```

Now I just type: `win11 start`

Also created a desktop launcher (`~/Desktop/Windows11.desktop`):

```ini
[Desktop Entry]
Name=Windows 11
Exec=/home/dev/bin/win11 start
Icon=computer
Type=Application
```

Double-click to launch. Simple.

## The Complete Solution (TL;DR)

For those who want the quick copy-paste solution:

### 1. Install KVM/QEMU

```bash
sudo apt-get install -y qemu-kvm libvirt-daemon-system \
    libvirt-clients bridge-utils virt-manager virtinst
sudo usermod -aG libvirt,kvm $USER
sudo systemctl start libvirtd && sudo systemctl enable libvirtd
```

### 2. Create Windows 11 VM

```bash
# Switch to KVM
sudo modprobe kvm kvm_intel

# Create VM with proper Windows 11 support
sudo virt-install \
  --name=Windows11 \
  --ram=8192 \
  --vcpus=4 \
  --cpu host-passthrough \
  --disk path=/data/vms/windows11.qcow2,size=80,format=qcow2,bus=sata \
  --cdrom=/path/to/Win11_ISO.iso \
  --os-variant=win11 \
  --network network=default,model=e1000e \
  --graphics spice,listen=0.0.0.0 \
  --video qxl \
  --boot uefi \
  --features smm.state=on \
  --tpm backend.type=emulator,backend.version=2.0,model=tpm-crb
```

### 3. Fix Boot Order

```bash
# Edit VM to boot from CD first
sudo virsh destroy Windows11
sudo virsh dumpxml Windows11 > /tmp/win11.xml
# Add these lines in the <os> section:
#   <boot dev='cdrom'/>
#   <boot dev='hd'/>
sudo virsh define /tmp/win11.xml
sudo virsh start Windows11
```

### 4. Increase Video RAM

```bash
sudo virsh destroy Windows11
sudo virsh dumpxml Windows11 | \
  sed "s/ram='[0-9]*'/ram='524288'/" | \
  sed "s/vram='[0-9]*'/vram='524288'/" > /tmp/win11-hires.xml
sudo virsh define /tmp/win11-hires.xml
sudo virsh start Windows11
```

### 5. Install SPICE Guest Tools (in Windows)

Download and install: https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe

Reboot Windows, then set resolution to 1920x1080 in Display Settings.

### 6. Create Management Script

Save this as `~/bin/win11`:

```bash
#!/bin/bash
VM_NAME="Windows11"

case "${1:-help}" in
    start)
        # Ensure KVM is loaded
        if ! lsmod | grep -q "^kvm"; then
            sudo modprobe kvm kvm_intel
        fi
        sudo virsh start "$VM_NAME"
        sleep 3
        virt-viewer --connect qemu:///system "$VM_NAME" &
        ;;
    stop)
        sudo virsh shutdown "$VM_NAME"
        ;;
    force-stop)
        sudo virsh destroy "$VM_NAME"
        ;;
    status)
        sudo virsh dominfo "$VM_NAME"
        ;;
    *)
        echo "Usage: $0 {start|stop|force-stop|status}"
        ;;
esac
```

Make it executable: `chmod +x ~/bin/win11`

Now just type: `win11 start`

## Key Takeaways

1. **VirtualBox + Windows 11 = Problematic** - Known issues with EFI, stability, and modern features
2. **KVM/QEMU is superior for Windows on Linux** - Native support for TPM 2.0, Secure Boot, better performance
3. **The command-line complexity is real** - But solvable with AI guidance
4. **SATA > VirtIO for beginners** - Slightly slower, but zero driver hassles during installation
5. **SPICE Guest Tools are essential** - For proper display resolution and copy/paste support
6. **Scripts defeat complexity** - Create abstractions so you never memorize commands

## Bonus: Additional Optimizations

### Performance Tuning

```bash
# Give VM more CPUs
sudo virsh setvcpus Windows11 8 --maximum --config
sudo virsh setvcpus Windows11 8 --config

# Increase RAM to 16GB
sudo virsh setmem Windows11 16G --config

# Enable huge pages for better performance
sudo virsh edit Windows11
# Add in <memoryBacking>: <hugepages/>
```

### Snapshots for Safety

```bash
# Create snapshot before risky changes
sudo virsh snapshot-create-as Windows11 "before-updates" \
  "Snapshot before Windows updates"

# List snapshots
sudo virsh snapshot-list Windows11

# Restore snapshot if something breaks
sudo virsh snapshot-revert Windows11 "before-updates"
```

### Auto-start on Boot

```bash
# VM starts automatically when host boots
sudo virsh autostart Windows11

# Disable autostart
sudo virsh autostart --disable Windows11
```

## Conclusion

The migration from VirtualBox to KVM/QEMU solved all my Windows 11 VM stability issues. No more freezing, no more crashes, and significantly better performance.

The real revelation? **AI transforms command-line complexity from a barrier into a non-issue**. You get expert guidance, real-time troubleshooting, and custom scripts - all for less than the cost of a coffee.

Sure, you could eventually figure this all out through Google, StackOverflow, and trial-and-error. But why spend hours or days when AI can guide you through it in under two hours?

The future isn't about memorizing arcane command-line syntax. It's about having an expert assistant who remembers it for you and explains it when needed.

---

**Have you migrated from VirtualBox to KVM? Share your experience in the comments!**



ccbd39e97c7d470a72d6f6e05c53000d03e13239ab019edca5af67c70bd3d60f

A detailed journey from VirtualBox chaos to a stable Windows 11 VM on KVM/QEMU, with AI assistance turning complex command-line wizardry into a manageable migration.

When VirtualBox kept freezing my Windows 11 VM, I turned to KVM/QEMU. The problem? Nobody remembers all those command-line incantations.

Escaping VirtualBox Hell: How I Migrated Windows 11 to KVM with AI

{/* banner prompt: Of course! Based on the blog post, here are a few creative and engaging prompts for a 16:9 banner image. I've provided three distinct options, each with a different artistic focus.

### Option 1 (The Diagnostic Vision)

This prompt focuses on the AI's role as a diagnostic partner, blending the digital and physical worlds.

**Prompt:**
A cinematic, high-detail shot of a lawnmower's underside in a dimly lit garage. A futuristic, semi-transparent AI interface is overlaid on the image, with glowing blue lines and data points scanning the blades. One line of light intensely focuses on and highlights a tiny, unassuming wood chip wedged in the mechanism, identifying it as the "Root Cause: Rotational Imbalance." The background is softly blurred, emphasizing the blend of high-tech analysis and a gritty, real-world mechanical problem. **--ar 16:9 --style raw**

### Option 2 (The Abstract Connection)

This prompt is more conceptual and artistic, visualizing the connection between AI logic and the mechanical fault.

**Prompt:**
A dynamic concept art masterpiece with a split-screen effect. On the left, a glowing, abstract neural network representing AI thought, with pulses of light converging on a single point. These lines of light cross over to the right side of the image, which shows a hyper-detailed, photorealistic view of a lawnmower's blade assembly. The AI's light illuminates the tiny wood chip causing the imbalance, bridging the gap between digital problem-solving and physical reality. The overall tone is one of discovery and clarity. **--ar 16:9**

### Option 3 (The Simple Culprit)

This prompt creates drama by focusing on the tiny object that caused the massive problem, with a subtle hint of the AI's involvement.

**Prompt:**
An epic macro photograph focusing on a small wood chip wedged between the metal blades of a lawnmower. The lighting is dramatic, casting deep shadows and highlighting the texture of the wood and steel. The depth of field is extremely shallow, making the rest of the mower a mysterious, blurred backdrop. A subtle, faint overlay of a blue diagnostic grid is visible over the chip, suggesting it has been identified by an intelligent system. The image captures the theme of a simple fix for a complex problem. **--ar 16:9 --style raw** */}

When my trusty Honda mower (when I got it, it was "Hon", yes, my hon indeed)—now bearing only the battle-worn letters "Ho" from what once proudly displayed "Honda"—suddenly started shaking violently during use.

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/12/ai-found-the-simple-fix-for-my-shaking-mower/ho.png)

I found myself facing a familiar yet frustrating problem. As developers, we're used to debugging complex systems, but mechanical issues can feel like working with undocumented legacy code. This time, I decided to approach the problem systematically by leveraging AI as my diagnostic partner.

## The Problem: When Experience Leads You Astray

My initial hypothesis was rooted in experience. I'd dealt with mower shaking before, and the culprit had always been fuel-related issues. Bad petrol or contaminated fuel had caused similar symptoms in the past, and replacing it with fresh fuel had always resolved the problem.

This time felt different. Even after filling up with fresh fuel, the violent shaking persisted. The symptoms were clear:

- Severe vibration during operation
- Fresh, quality fuel in the tank
- Old engine oil (overdue for change, but seemingly unrelated)

As technical professionals, we often fall into the trap of pattern matching based on previous experiences. Just like assuming a bug is caused by a recent code change, I was convinced this was another fuel issue. But when the obvious solution failed, it was time for a more systematic approach.

## AI as a Diagnostic Partner

Instead of randomly checking components or diving into forums, I decided to treat AI as my debugging partner. Here's how I framed the problem:

```
Problem Statement:
- Honda mower shaking violently during operation
- Fresh fuel recently added
- Engine oil is old but mower ran fine before
- Vibration is severe and consistent
- Previous shaking issues were fuel-related but resolved
```

The AI's response was methodical and eliminated my assumptions immediately. It confirmed that **old engine oil does not cause vibration issues**—a crucial piece of information that redirected my troubleshooting efforts.

Instead, the AI guided me toward a more likely root cause: **imbalance in rotating parts**.

## The Systematic Diagnosis Process

The AI provided a structured troubleshooting approach, listing potential causes in order of likelihood:

### Primary Suspects (Most Common)
- **Bent blade**: Impact damage from hitting rocks or debris
- **Loose blade**: Bolts that have worked their way loose over time
- **Debris accumulation**: Material stuck under the deck or around blades

### Secondary Issues (Less Common)
- **Damaged blade adapter**: The component that connects blade to engine
- **Worn blade mounting hardware**: Washers, spacers, or mounting brackets

### Worst-Case Scenarios
- **Bent crankshaft**: Usually from major impact damage

This methodical approach reminded me of how we debug software—start with the most common causes before diving into edge cases.

## The Discovery: A Tiny Culprit

Following the AI's guidance, I flipped the mower and inspected the underside. The four-blade system appeared intact at first glance, but closer inspection revealed the issue:

**A small wood chip was jammed between one blade and the mower deck.**

This tiny piece of debris was preventing one blade from rotating freely, creating an imbalance in the four-blade system. Like an unbalanced load balancer in a distributed system, one component's irregular behavior was causing the entire system to become unstable.

## The Fix: Simple but Effective

The solution was remarkably straightforward:

```bash
# The repair process (metaphorically speaking)
1. Power down system (turn off mower, disconnect spark plug)
2. Inspect rotating components (check all four blades)
3. Identify foreign object (wood chip wedged in blade assembly)
4. Remove obstruction (carefully extract debris)
5. Test system (restart and verify smooth operation)
```

After removing the wood chip by using a screw driver , the mower returned to its normal, smooth operation. The violent shaking was completely eliminated.

![](https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/12/ai-found-the-simple-fix-for-my-shaking-mower/blades.png)

{/* ## Technical Analysis: Why This Works

From a mechanical perspective, the diagnosis makes perfect sense:

- **Rotational balance**: Four-blade mowers depend on all blades spinning at identical speeds
- **Harmonic vibration**: Even small imbalances create amplified vibration at operating RPMs
- **System feedback**: The engine's mounting system transmits blade imbalances throughout the entire mower

This mirrors many software performance issues where a single bottleneck component can cause system-wide problems.

## Lessons for Technical Problem Solving

This experience highlighted several principles that apply beyond mower repair:

### 1. Don't Let Experience Bias Your Diagnosis
Just as we shouldn't assume every performance issue is database-related because that was the cause last time, mechanical problems require fresh analysis for each incident.

### 2. AI as a Structured Thinking Partner
Modern AI excels at providing systematic troubleshooting approaches. It's like having a senior engineer guide you through a debugging session, offering structured thinking when you're too close to the problem.

### 3. Start with High-Probability, Low-Cost Checks
The AI correctly prioritized checking for debris and loose components before suggesting more complex issues like crankshaft damage. This mirrors good debugging practice—check the simple stuff first.

### 4. Document Your Findings
I now maintain a simple troubleshooting log for mechanical issues:

```yaml
equipment: Honda_mower_4_blade
issue_date: 2025-12-01
symptoms:
  - violent_shaking
  - consistent_during_operation
investigation:
  - fuel: confirmed_fresh
  - oil: old_but_unrelated
  - rotating_parts: debris_found
root_cause: wood_chip_in_blade_assembly
resolution: debris_removal
status: resolved
```

## The Broader Implications

This experience demonstrates AI's potential in domains beyond software development. The same pattern recognition and systematic thinking that helps us debug code can be applied to:

- **Equipment maintenance**: Structured diagnostic approaches
- **Process optimization**: Identifying bottlenecks and inefficiencies  
- **Root cause analysis**: Moving beyond symptoms to underlying issues
 */}
## Quality vs. Cost: A Side Note

It's worth noting that my battle-scarred Honda mower, despite losing most of its brand lettering, continues to be a reliable workhorse. Meanwhile, a more expensive "mid-level premium" mower I purchased has had multiple component failures.

This echoes what we see in technology: sometimes the simple, well-engineered solution outlasts the feature-rich alternative. The Honda's straightforward design makes it easier to diagnose and repair—much like well-architected software systems.

## Conclusion: AI-Powered Problem Solving

This seemingly mundane mower issue illustrates AI's practical value in systematic problem-solving. By providing structured diagnostic approaches and eliminating incorrect assumptions, AI can guide us toward efficient solutions.

The key takeaway isn't that AI magically fixes problems, but rather that it can serve as an excellent thinking partner—offering methodical approaches when we're stuck in our own assumptions or experience patterns.

Whether you're debugging a distributed system or diagnosing a shaking mower, the principles remain the same: approach problems systematically, challenge your assumptions, and don't be afraid to leverage AI as a diagnostic tool in your problem-solving toolkit.

Sometimes the most valuable debugging sessions happen in your garage, not your IDE.

ba9b7baea9330f51a7da1f93513e1b7b346ffd21cdb5313d577475d27ecae9f0

A technical case study showing how AI can effectively diagnose mechanical problems, turning a frustrating mower issue into a quick fix.

How AI helped diagnose a violent mower shake caused by a tiny wood chip, saving time and guesswork.

AI Found the Simple Fix for My Shaking Mower

{/* banner prompt: Of course! Based on the blog post, here are a few creative and engaging prompts for a 16:9 banner image, each with a slightly different creative direction.

### Option 1 (The Human-AI Partnership)

This prompt focuses on the collaboration between the user and the AI, contrasting the hands-on mechanical work with the high-tech digital guidance.

**Prompt:**
> A cinematic, photorealistic shot of a modern, sleek front-loading washing machine on a concrete basement floor. A person kneels beside it, holding a wrench, looking intently at a glowing blue holographic interface projected in the air. The interface displays a 3D wireframe model of the washer, highlighting an unstable leveling foot with a bright red circle and showing a simple instruction: "Raise Left Foot". The scene is lit with dramatic, focused lighting, contrasting the high-tech AI solution with the practical, hands-on task. --ar 16:9

**Why it works:** It directly visualizes the core story: AI providing clear, actionable instructions for a real-world mechanical problem. It's relatable, modern, and technology-themed.

---

### Option 2 (The Diagnostic Vision)

This prompt is more conceptual and visually dynamic, emphasizing the AI's ability to "see" and diagnose the problem.

**Prompt:**
> Futuristic digital art. A sleek, modern front-loading washing machine is the central subject in a clean, minimalist laundry room. A translucent, glowing blue grid of light envelops the machine, scanning it for imbalances. Bright, holographic data points and equalization graphs hover around the machine, pinpointing the source of a vibration (represented by subtle, artistic motion trails on one side). The overall aesthetic is clean, technical, and optimistic, symbolizing AI bringing order to mechanical chaos. --ar 16:9

**Why it works:** This is a highly visual and tech-forward interpretation. It focuses on the "root cause analysis" aspect mentioned in the blog post, making the AI the star of the show.

---

### Option 3 (The Moment of Insight)

This prompt captures the "aha!" moment where the AI provides the key piece of information, turning a frustrating problem into a simple solution.

**Prompt:**
> A dynamic concept art illustration of a vibrating washing machine. A luminous, ethereal AI entity, composed of glowing blue circuits and lines of code, hovers over the machine. The AI is pointing a beam of light directly at one of the machine's feet, projecting a holographic diagram showing how to adjust it with a wrench. The background is a dark, modern basement, making the AI's glow the primary light source. The image captures the moment of revelation, where technology illuminates the solution to a physical problem. --ar 16:9

**Why it works:** This version is more artistic and metaphorical. It portrays the AI as a "guiding light," which reflects the author's positive experience and the broader theme of AI as a helpful problem-solving partner. */}

Black Friday deals can be both a blessing and a curse. After my faithful washing machine of nearly 20 years finally gave up, I found myself with a shiny new Fisher & Paykel WH1060SG1 front-loader and what I assumed would be a simple installation process. How wrong I was.

What started as a straightforward appliance swap turned into a physics lesson, complete with violent shaking, mysterious leftward drifting, and enough vibration to wake the neighbors. But here's where the story gets interesting: instead of calling an expensive repair service or enduring weeks of trial-and-error adjustments, I turned to AI for help.

The result? A systematic, logical approach to troubleshooting that not only solved my problem but taught me valuable lessons about both appliance mechanics and the practical applications of AI in everyday problem-solving.

## The Problem: When Your Washer Becomes a Wanderer

### Initial Symptoms

After connecting all the pipes and firing up my new front-loader for the first test run, I quickly discovered that modern washing machines aren't as plug-and-play as I'd assumed. During the spin cycle, my machine exhibited three concerning behaviors:

1. **Violent shaking** that made me question whether I'd bought a washing machine or a paint mixer
2. **Consistent leftward drift** across my concrete basement floor
3. **Rocking motion** that suggested at least one foot wasn't making proper contact

The most puzzling aspect was the directional movement. Why left? Was it related to the drum's rotation direction? These questions led me to what became a fascinating AI-assisted troubleshooting session.

### The Core Question

The fundamental question that stumped me was surprisingly nuanced:

> *If the washing machine drum spins clockwise or counter-clockwise, which leveling foot should I adjust when the machine drifts to one side?*

This seemed like it should have a straightforward answer, but the relationship between drum rotation and machine movement isn't immediately obvious.

## AI to the Rescue: Systematic Problem Solving

### Breaking Down the Physics

Rather than diving into random adjustments, I decided to consult AI for a methodical approach. The conversation that followed provided insights that would have taken hours of frustrated experimentation to discover.

The first breakthrough came when AI clarified a common misconception:

**The drum's rotation direction is irrelevant to leveling adjustments.**

This was counterintuitive to me. I had assumed that a clockwise-spinning drum would create forces that would affect which side to adjust. But AI explained that the key factor isn't the direction of rotation—it's the direction the machine moves.

### The Golden Rule of Washing Machine Movement

AI provided what became the most valuable piece of information in the entire troubleshooting process:

> **A washing machine always moves toward the side that's lower or less stable.**

This simple rule transforms a confusing problem into a logical diagnostic process:

- Machine drifts **left** → The **left side** is low or unstable → Raise the **left** leveling feet
- Machine drifts **right** → The **right side** is low or unstable → Raise the **right** leveling feet

### Practical Implementation

Once I understood the principle, the implementation became straightforward. Here's the step-by-step process AI guided me through:

```markdown
Leveling Foot Adjustment Process:
1. Identify drift direction during spin cycle
2. Locate leveling feet on the low/drifting side
3. Turn feet counter-clockwise to extend them downward
4. Test with another spin cycle
5. Fine-tune as needed
6. Tighten lock nuts to secure position
```

The key mechanical detail: turning leveling feet **counter-clockwise** extends them downward, effectively raising that corner of the machine.

## Diagnosing the Rocking Problem

### Concrete Floor Complications

My concrete basement floor presented a different challenge than the typical sliding problem. Instead of the machine walking across a smooth surface, it was rocking in place. AI helped me understand that this indicated a classic "three-point contact" scenario—one corner wasn't touching the floor properly.

### The Corner Test Method

AI suggested a simple diagnostic technique:

```markdown
Corner Diagnosis Process:
1. Press down on each corner of the machine
2. Identify which corner feels "floaty" or unstable
3. Check if you can rock the machine by pressing that corner
4. Adjust the leveling foot for that specific corner
5. Repeat until all four corners are solid
```

This methodical approach eliminated guesswork and provided immediate feedback on whether adjustments were working.

### Anti-Vibration Solutions

Beyond leveling, AI also suggested complementary solutions:

- **Anti-vibration pads**: Rubber pads that absorb residual vibrations (Yes, you can get those from hardware stores, appliance stores such as bunnings, amazon, etc.)
- **Lock nuts**: Preventing leveling feet from loosening over time  
- **Load balancing**: Ensuring clothes are distributed evenly in the drum

{/* ## The Technical Perspective: Why This Matters for Developers

### AI as a Diagnostic Tool

As someone who spends most of my time debugging code, the parallels between software troubleshooting and appliance repair were striking. AI provided:

1. **Systematic methodology** rather than random attempts
2. **Root cause analysis** instead of symptom treatment
3. **Verification steps** to confirm fixes worked
4. **Documentation** of the process for future reference

### Structured Problem Solving

The AI approach mirrored good debugging practices:

```python
def diagnose_washing_machine():
    # Step 1: Identify symptoms
    symptoms = observe_machine_behavior()
    
    # Step 2: Form hypothesis
    if symptoms.includes('leftward_drift'):
        hypothesis = 'left_side_unstable'
    
    # Step 3: Test hypothesis
    result = adjust_left_leveling_feet()
    
    # Step 4: Verify fix
    return verify_stable_operation()
```

This structured approach eliminated the frustration of random trial-and-error fixes.

## Lessons Learned: AI in Practical Problem Solving

### Beyond Code: AI's Versatility

This experience highlighted AI's potential beyond traditional software applications. The same logical reasoning that helps with coding problems applies equally well to:

- **Mechanical troubleshooting**
- **Process optimization**
- **Root cause analysis**
- **Step-by-step guidance**

### The Value of Systematic Thinking

What impressed me most was how AI transformed a frustrating hands-on problem into a logical sequence of diagnostic steps. Instead of:

- Random adjustments
- Guessing based on intuition
- Multiple service calls
- Hours of trial and error

I got:

- Clear diagnostic criteria
- Logical reasoning chains
- Immediate verification methods
- Documented solutions for future reference

## Implementation Results and Validation

### Before and After

The transformation was dramatic:

**Before AI assistance:**
- 30+ minutes of violent shaking and drifting
- Frustrated attempts at random adjustments  
- Concern about damaging the new machine
- No clear understanding of the underlying problem

**After implementing AI guidance:**
- Stable operation within 10 minutes of proper adjustment
- Silent spin cycles with minimal vibration
- Confident understanding of leveling principles
- Documented process for future maintenance

### Verification Process

AI also provided a verification checklist to ensure the fix was complete:

```markdown
Stability Verification:
□ No movement during spin cycle
□ No rocking when pressing corners
□ Minimal vibration during operation
□ All leveling feet making solid contact
□ Lock nuts properly tightened
```

## Broader Implications: AI as Your Technical Copilot

### Beyond Appliance Repair

This washing machine experience demonstrated AI's potential as a general-purpose troubleshooting assistant. The same methodology applies to:

- **Network connectivity issues**
- **Performance optimization problems**
- **Hardware installation challenges**
- **System configuration puzzles**

### The New Troubleshooting Paradigm

Rather than immediately reaching for forums, YouTube tutorials, or service manuals, AI provides:

1. **Immediate availability** - no waiting for responses
2. **Customized guidance** - tailored to your specific situation
3. **Interactive problem-solving** - back-and-forth refinement
4. **Educational value** - understanding the why, not just the how */}

## Conclusion: Practical AI in Action

What started as a simple appliance installation became a masterclass in practical AI application. The experience taught me that AI's value extends far beyond code generation and data analysis—it's an incredibly effective partner for real-world problem-solving.

The key insights:

- **AI excels at systematic troubleshooting** when provided with clear symptoms
- **Physics principles remain constant** regardless of specific models or brands
- **Structured approaches beat random attempts** every time
- **Understanding root causes prevents recurring problems**

My Fisher & Paykel now runs reasonably acceptably and stays perfectly in place, provided I avoid attempting the maximum spin cycle, a setting I am still hesitant to try.

For fellow developers facing similar challenges—whether in code or in your laundry room—consider AI as your first troubleshooting partner. You might be surprised at how effectively it translates complex problems into manageable, step-by-step solutions.

*Have you used AI to solve unexpected real-world problems? Share your experiences in the comments below—I'd love to hear about other creative applications of AI in everyday troubleshooting.*

197ea58d1a28113e9ed1eb5ff464c2896f685a03fe506aa9cf25e2bd9f783afb

Learn how AI can assist with practical home problems by walking through a real troubleshooting session that fixed a shaky washing machine.

How AI helped diagnose and fix a vibrating washing machine through step-by-step troubleshooting.

AI Fixed My Shaky Washer: A Troubleshooting Guide

{/* banner prompt: Excellent. Here is a creative and engaging prompt for a banner image, based on the provided blog post.

---

### Banner Image Prompt

**A cinematic digital art illustration of a sleek, modern Linux server in a dark, quiet room, symbolizing a system trying to sleep. A soft, pulsing blue crescent moon icon is faintly visible on the server's chassis, indicating its attempt to enter suspend mode. However, a single Ethernet port on the back glows with a disruptive, jagged, electric orange light. This light sends a 'wake-up' signal, visualized as a sharp, glitchy energy wave, preventing the system from resting.**

**Floating above the scene is a serene, abstract AI construct, represented by a glowing network of interconnected nodes and lines of light. The AI is analyzing the disruptive signal, with a focused beam of clean, white light pointing directly at the problematic Ethernet port, highlighting it as the culprit. In the background, faint, out-of-focus lines of code and system logs hint at the debugging process.**

**The overall mood is a contrast between the calm, deep blue tones of sleep and the sharp, aggressive orange of the wake-up signal, all being systematically resolved by the intelligent AI presence.**

**Style:** Technology-themed, highly detailed, dramatic lighting, cinematic, digital art.
**Aspect Ratio:** 16:9

Additional instructions: A Penguin holding a screw driver and spanner */}

## The Problem: A Restless Linux Box

Have you ever tried to put your Linux machine to sleep, only to watch it wake up immediately like an overtired toddler fighting bedtime? That's exactly what happened on my Ubuntu 24.04 system running kernel 6.14.0-35. The moment I initiated suspend, the system would wake up within 2-3 seconds. Every. Single. Time.

In the old days, this kind of problem would send you down a rabbit hole of forum posts, kernel logs, and trial-and-error fixes that could consume hours or even days. But thanks to AI assistance (specifically Claude Code), what could have been a frustrating ordeal became a systematic, almost enjoyable debugging session.

## Initial Investigation: Checking the Usual Suspects

The first step in any suspend issue is understanding what's actually happening. We started by checking the system's suspend service status and logs:

```bash
systemctl status systemd-suspend.service
journalctl -u systemd-suspend.service -n 50
```

The logs revealed an interesting pattern: the system was suspending successfully, but returning from sleep operation within seconds. The suspend service itself was working, but something was immediately waking the system.

## The Wake Source Hunt

Linux systems have multiple potential wake sources configured in `/proc/acpi/wakeup`. Let's see what was enabled:

```bash
cat /proc/acpi/wakeup
```

Several devices showed as `*enabled`:
- **XHCI** (USB controller) - pci:0000:00:14.0
- **TXHC** (Thunderbolt USB controller) - pci:0000:00:0d.0
- **PEG0** (PCIe Graphics Port)
- **RP08** (PCIe Root Port)
- **AWAC** (ACPI Wake Alarm Clock)

The USB controllers seemed like obvious culprits, right? Let's investigate further.

## Red Herring #1: The Keyboard

My Lenovo keyboard had wakeup enabled:

```bash
cat /sys/bus/usb/devices/3-7.1.1.4/power/wakeup
# Output: enabled
```

Natural assumption: the keyboard must be triggering spurious wake events. Let's disable it:

```bash
sudo bash -c 'echo disabled > /sys/bus/usb/devices/3-7.1.1.4/power/wakeup'
```

Test suspend... still wakes immediately. Not the keyboard.

## Red Herring #2: The USB Controllers

Maybe it's the USB controllers themselves? Let's disable them:

```bash
sudo bash -c 'echo XHCI > /proc/acpi/wakeup'  # Disable main USB controller
sudo bash -c 'echo TXHC > /proc/acpi/wakeup'  # Disable Thunderbolt USB controller
```

Test suspend... still wakes immediately. What?!

## The Wakeup Sources Investigation

A deeper look at `/sys/kernel/debug/wakeup_sources` revealed something interesting:

```bash
sudo cat /sys/kernel/debug/wakeup_sources | head -30
```

The data showed:
- Keyboard (3-7.1.1.4): **0 wakeup events** - completely innocent!
- USB controller (0000:00:14.0): 76 wake events
- Thunderbolt (0000:00:0d.0): 76 wake events
- device:63: 76 events

So the keyboard wasn't causing any wakes at all, despite being enabled. The USB controllers were active, but disabling them didn't help. What was going on?

## The Deep Sleep Detour

At this point, we noticed the system was using **s2idle** (suspend-to-idle) rather than deep sleep:

```bash
cat /sys/power/mem_sleep
# Output: [s2idle] deep
```

The brackets indicate s2idle was active. S2idle is a lighter sleep mode where the system stays in a low-power idle state rather than truly suspending. It's more susceptible to spurious wakeups.

My system supported **deep sleep** (S3), so let's try that:

```bash
sudo bash -c 'echo deep > /sys/power/mem_sleep'
```

Test suspend... Success! The system went into deep sleep properly!

...but then it wouldn't wake up. Not with keyboard, not with mouse, not even with the power button. The system was completely unresponsive. Hard reset required.

**Lesson learned**: My system has broken S3 resume support. This is actually common on modern systems, especially laptops with newer Intel chipsets. Manufacturers often don't properly implement or test S3 suspend/resume, focusing instead on "Modern Standby" (s2idle).

Back to square one with s2idle.

## Finding the Real Culprit: Detective Work

With USB controllers ruled out and deep sleep off the table, we needed to look at the other enabled wake sources. Let's check what PCIe devices were behind those enabled ports:

```bash
lspci -s 00:1c.0  # Check RP08
# Output: 00:1c.0 PCI bridge: Intel Corporation Alder Lake PCH-P PCI Express Root Port

lspci -s 02:00.0  # Check what's behind it
# Output: 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411
```

**Bingo!** A **Realtek Ethernet controller** was sitting behind the RP08 wake source. Realtek NICs are notorious in Linux circles for causing spurious wakeup events, especially in s2idle mode.

The fix:

```bash
sudo bash -c 'echo RP08 > /proc/acpi/wakeup'
```

This toggles RP08 from enabled to disabled. Test suspend...

**It worked!** The system now suspends properly and stays suspended until I wake it with the keyboard.

## Understanding the Root Cause

Why does the Realtek Ethernet controller cause this issue?

1. **s2idle behavior**: In s2idle mode, the system doesn't fully power down. It keeps more devices active in low-power states.

2. **Network polling**: The Realtek driver may continue some level of network activity or polling that generates wake events.

3. **Firmware quirks**: Some Realtek NICs have firmware that doesn't properly handle s2idle states, generating spurious PME (Power Management Event) signals.

4. **PCIe power management**: The interaction between PCIe power management and the network controller's power states can cause issues.

Even though we disabled Wake-on-LAN at the network interface level, the PCIe-level wake source (RP08) was still enabled and responding to these spurious events.

## How AI Revolutionized This Process

Let me be honest: this entire troubleshooting session, from problem statement to solution, took about 30 minutes. In the "old days" (yes, I'm that old), this kind of issue could easily consume an entire evening or weekend:

1. **Searching forums**: Hours spent reading threads where people describe similar-but-not-quite-the-same issues
2. **Trial and error**: Random suggestions to try without understanding why
3. **Dead ends**: Following promising leads that turn out irrelevant to your specific hardware
4. **Documentation diving**: Parsing through kernel documentation to understand power management subsystems
5. **Frustration**: Giving up and just living with the problem

With AI assistance (Claude Code in this case), the process was:

1. **Systematic investigation**: Each step built logically on the previous findings
2. **Parallel exploration**: Checking multiple potential causes simultaneously
3. **Context awareness**: Understanding the relationships between different system components
4. **Hypothesis testing**: Methodically ruling out suspects with clear tests
5. **Knowledge synthesis**: Combining information from logs, sysfs, and system state

The AI didn't just tell me "disable RP08" - it guided me through the investigation, explained what each command revealed, and helped build a mental model of how Linux power management works.

## The Solution: Step-by-Step

Here's the final solution for anyone with similar issues:

### 1. Identify your sleep mode
```bash
cat /sys/power/mem_sleep
```

If you see `[s2idle]`, you're in the light sleep mode susceptible to spurious wakeups.

### 2. Check wake sources
```bash
cat /proc/acpi/wakeup | grep enabled
```

### 3. Investigate enabled PCIe wake sources
```bash
# For each enabled PCIe port, check what device is behind it
lspci -s <address>
```

### 4. Look for Realtek Ethernet controllers
```bash
lspci | grep -i ethernet
```

### 5. If you find a Realtek NIC, check its wake source
Look in `/proc/acpi/wakeup` for the PCIe root port (RPxx) corresponding to the NIC's address.

### 6. Disable the problematic wake source
```bash
sudo bash -c 'echo RP08 > /proc/acpi/wakeup'
```
(Replace RP08 with whatever your PCIe root port is called)

### 7. Make it permanent
The above change won't survive a reboot. To make it permanent, create a systemd service:

```bash
sudo nano /etc/systemd/system/disable-rp08-wake.service
```

```ini
[Unit]
Description=Disable RP08 wake source
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c 'echo RP08 > /proc/acpi/wakeup'

[Install]
WantedBy=multi-user.target
```

Enable it:
```bash
sudo systemctl enable disable-rp08-wake.service
```

## Alternative Solutions

If disabling the PCIe wake source doesn't work for you, or you need the network controller to be able to wake the system, here are alternatives:

### Try updating the NIC driver
```bash
sudo ethtool -s enp2s0 wol d  # Disable Wake-on-LAN explicitly
```

### Check BIOS settings
Some BIOS/UEFI settings affect suspend behavior:
- Look for "Modern Standby" vs "S3" options
- Check "Wake on LAN" settings
- Review "USB wake" settings

### Kernel parameters
Try adding to your kernel command line (in GRUB):
```
mem_sleep_default=s2idle
```

### Last resort: blacklist the driver
If nothing else works and you don't need the onboard Ethernet:
```bash
echo "blacklist r8169" | sudo tee -a /etc/modprobe.d/blacklist.conf
```

## Conclusion

Linux power management can be tricky, with multiple layers (ACPI, PCIe, USB, driver-level) all interacting. What appears as a simple "suspend doesn't work" issue often has a complex root cause requiring systematic investigation.

The key lessons:

1. **Spurious wakeups aren't always from USB devices** - Check PCIe wake sources too
2. **Realtek NICs are common culprits** - Especially in s2idle mode
3. **S3 deep sleep isn't always supported** - Modern systems may have broken S3 resume
4. **Disable at the right level** - Individual USB device settings won't help if the controller or PCIe port is the issue
5. **AI assistance makes debugging systematic** - What used to take hours can now take minutes

And remember: sometimes the culprit isn't the keyboard you suspected all along, but the innocent-looking network card quietly sending spurious wake signals in the background.

Sleep tight, Linux boxes!

---

*Have you encountered suspend issues on Linux? What was your culprit? Share your experiences in the comments below!*

fa825505d34ef4d9ede6f8e4147410aa326d916701be0fab0cfc88300b287a07

Debugging Linux suspend issues can be frustrating. This article chronicles the systematic troubleshooting of immediate wake-from-suspend problems, exploring USB controllers, sleep modes, and ultimately discovering a Realtek Ethernet controller as the culprit.

A deep dive into troubleshooting Linux suspend issues, from USB controllers to sleep modes, and how AI transformed a potentially frustrating debugging session into a systematic investigation.

How to put your baby, sorry, your Linux box to sleep

{/* banner prompt: Of course! Based on the blog post, here are a few creative and engaging prompts for a 16:9 banner image, each with a slightly different focus.

### Option 1: The AI Architect (Focus on Creation)

This prompt visualizes the AI as the central creator, building the solution from abstract code into a tangible product.

**Prompt:**
> A cinematic, technology-themed digital art piece. In the center, a luminous, abstract representation of an AI brain pulsates with soft blue and green light. From this core, intricate streams of glowing code flow outwards, weaving together to form a holographic interface of a Chrome extension featuring a friendly, minimalist koala logo. In the background, out-of-focus stock market charts and data tickers provide a financial context. The overall mood is one of intelligent, autonomous creation and calm efficiency. --ar 16:9

**Why it works:**
*   **Technology-Themed:** Directly mentions AI, code, and holographic interfaces.
*   **Reflects Main Idea:** Puts the AI at the center of the creation process, highlighting that it was "coded entirely by AI."
*   **Incorporates Key Elements:** Mentions the koala logo, Chrome extension, and the CommSec/stock market context.
*   **Engaging:** "Luminous," "pulsates," and "cinematic" create a visually dynamic image.

---

### Option 2: The Collaborative Swarm (Focus on Teamwork)

This prompt emphasizes the collaboration between different AI tools (Gemini, Codex, Claude) mentioned in the post.

**Prompt:**
> A dynamic and futuristic scene depicting three distinct streams of colored light—one brilliant blue (for Gemini/Codex logic), one creative magenta (for Claude's refinement), and one vibrant green (for AI Studio's assets)—swirling and converging in a sleek, dark command center. Where the lights meet, they are actively constructing a 3D, glowing icon of a koala. Lines of JavaScript code are subtly integrated into the light streams. The background is a large, panoramic screen displaying the CommSec interface with a "Session Timed Out" dialog box just beginning to fade away, replaced by a green "Session Active" status. --ar 16:9

**Why it works:**
*   **Reflects Main Idea:** Visually represents the "collaboration of AI tools" which is a key point in the article.
*   **Tells a Story:** Shows the problem (the timeout dialog) being actively solved by the AI team.
*   **Creative and Engaging:** The concept of colored light streams as different AIs is abstract yet easy to understand, making for a compelling visual.
*   **Technology-Themed:** Uses a "command center" setting, code, and glowing UI elements.

---

### Option 3: The Silent Guardian (Focus on the User Benefit)

This prompt focuses on the outcome: a seamless, uninterrupted user experience, with the AI working quietly in the background.

**Prompt:**
> A hyper-detailed, photorealistic image from the perspective of a trader. The main focus is a crisp, clean computer monitor displaying a complex CommSec stock watchlist with glowing green and red charts. To the side, a "Continue session" pop-up is visible but is being dissolved into digital particles by an ethereal, softly glowing koala-shaped energy field that emanates from the browser's toolbar. The rest of the room is dark, emphasizing the trader's focus on the screen, which remains uninterrupted. The feeling is one of powerful, silent automation and focus. --ar 16:9

**Why it works:**
*   **Reflects Main Idea:** It directly visualizes the problem (the pop-up) being solved by the AI-powered extension (the koala).
*   **User-Centric:** It frames the benefit from the user's point of view, highlighting the "uninterrupted market analysis."
*   **Engaging and Creative:** The "dissolving" pop-up and "ethereal energy field" are strong visual metaphors for the extension's function.
*   **Technology-Themed:** The glowing data, digital particles, and overall aesthetic are modern and tech-focused.

Additional instructions: make three cartoon like builders as representation of Google Gemini, OpenAI Codex, Claude Code */}

What happens when you give AI a frustrating problem and ask it to solve it completely autonomously? You get **Koala-Tinue**, a Chrome extension that was coded entirely by artificial intelligence—no human programming required.

This isn't just another AI-assisted development story. This is about AI tools working together to create a functional, polished browser extension from concept to completion, with a human developer serving only as a tester.

## The Problem: CommSec Session Management Headaches

Anyone who's tried to monitor the Australian Stock Exchange (ASX) through CommSec knows the pain. You're deep in analysis, watching market movements, when suddenly—boom—you're logged out due to inactivity. The dreaded "Continue session" prompt appears, breaking your focus and forcing you to babysit your browser tab instead of the market data that actually matters.

For retail traders without access to professional trading platforms, CommSec is often the go-to solution. But the constant session timeouts create a frustrating user experience that pulls attention away from trading decisions.

## Enter Koala-Tinue: The AI-Generated Solution

Meet Koala-Tinue, your friendly CommSec session companion. This Chrome extension automatically handles those annoying session continuation prompts, letting you focus on market analysis instead of tab management.

Here's what makes this project remarkable: **every line of code was written by AI**.

### The AI Development Team

The extension was built using a collaboration of AI tools:

- **Google Gemini & Codex**: Handled all the core JavaScript functionality and Chrome extension architecture
- **Google AI Studio**: Generated the extension icons and visual assets
- **Claude AI**: Managed image editing and icon refinement

The human developer's role? Pure testing. No coding, no debugging logic, no architectural decisions—just running the extension and reporting whether it worked.

Relax — every line of code is double-checked, security risks are mitigated, and your privacy stays safeguarded.

## How Koala-Tinue Works

The extension operates with elegant simplicity:

1. **Install and activate**: Toggle the toolbar icon between Play (paused) and Pause (active)
2. **Open your CommSec Watchlist**: The extension monitors for session continuation prompts
3. **Automatic clicking**: When the "Continue session" dialog appears, Koala-Tinue clicks it for you
4. **Smart scheduling**: The extension aligns with actual ASX trading hours

### Smart Mode: Intelligence Built-In

The most impressive feature is Smart Mode, which demonstrates sophisticated logic implemented by the AI based on detailed prompts:

```javascript
// AI-generated logic for ASX trading hours
const ASX_TRADING_HOURS = {
  start: 10, // 10:00 AM Sydney time
  end: 16,   // 4:00 PM Sydney time
  timezone: 'Australia/Sydney'
};

function isMarketHours() {
  const now = new Date();
  const sydneyTime = new Intl.DateTimeFormat('en-AU', {
    timeZone: 'Australia/Sydney',
    hour: 'numeric',
    weekday: 'short'
  }).formatToParts(now);
  
  // Smart mode logic continues...
}
```

Smart Mode automatically:
- **Activates during market hours**: 10:00 AM to 4:00 PM Sydney time on weekdays
- **Sleeps after hours**: Pauses clicking and resets counters overnight
- **Contextual behavior**: Auto-enables during market hours even if previously paused
- **Resource conservation**: Stops unnecessary processing outside trading hours

## The AI Development Process

### Code Generation Strategy

The AI tools approached the problem systematically:

1. **Architecture Planning**: Gemini analyzed Chrome extension requirements and CommSec's DOM structure
2. **Core Logic**: Gemini initially handled the session detection and automatic clicking functionality, with Codex providing assistance when challenges arose.
3. **Smart Features**: Advanced time zone handling and market hours logic were generated autonomously
4. **User Interface**: Clean, intuitive controls with play/pause toggle functionality

### Visual Asset Creation

#### Icon Generation

Google AI Studio was the creative director chief in charge of the creative work:

>A minimalist, flat design vector logo icon. A cute, stylized soft-grey koala is peeking over the top of a large, central white circle. Inside the white circle is a simple, eucalyptus-green "play" button symbol (▶). The design must have clean lines, no gradients, and be on a transparent background. It should be designed to be clear and recognizable as a small browser extension icon.

For comparison, I also consulted with ChatGPT; below is what ChatGPT generated:

<Icon src="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/11/this-commsec-app-was-coded-entirely-by-ai/chatgpt-icon.png" alt="ChatGPT-generated koala play icon concept" />

But I really prefer Gemini's version:

<Icon src="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/11/this-commsec-app-was-coded-entirely-by-ai/icon-play.png" alt="Google AI Studio koala play icon" />

It is cute and keeps the main focus on the `Play` button.

Okay, this is now my icon for `Koala-Tinue`.


#### Artwork refinement

Unfortunately, we still cannot ship whatever AI produces. The generated icon has a white background, and our current tooling makes it awkward to maintain separate variants for every background color. We therefore prefer icons with a transparent background so they can adapt to any context.

After a quick browser test, the icon also felt a bit invisible because it was mostly gray, so I asked Google AI Studio to add a gold background.

<Icon src="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/11/this-commsec-app-was-coded-entirely-by-ai/icon-unavailabe.png" alt="Koala icon with added gold background attempt" />

If you ask **Google AI Studio** to remove the white background it usually tries, but the result is rarely a truly transparent background—at best you get a convincing white fill.

For example, I asked it to remove the background (which turned out to be fake) and replace it with gold. Below is the result:

<Icon src="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/11/this-commsec-app-was-coded-entirely-by-ai/icon-unavailable-square-bright.png" alt="Gold square koala icon with white background removal attempt" />

Even when it fails to produce transparency, it can still recolor regions independently, which is genuinely useful.

Finally, it was time to ask our senior developer and artwork editor (Claude Code) to finish the job:
1. Remove the original background 
2. Add a gold background tone
3. Create a round-cornered icon
4. Adjust the image size 

The original image created by Google AI Studio:
<Icon src="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/11/this-commsec-app-was-coded-entirely-by-ai/icon-unavailable-white-bg.png" alt="Original koala play icon with white background" />

After Claude Code removed the background and added the gold tone:

<Icon src="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/11/this-commsec-app-was-coded-entirely-by-ai/icon-unavailable-square.png" alt="Gold square koala icon with transparent background" />

We still could not easily target only the circle via command line tooling, so we went back to **Google AI Studio**, and it delivered.

After making the circle white, we asked Claude to round the corners, giving us an icon we can confidently use in the extension.

The final result:

<Icon src="https://storage.googleapis.com/www-tyolab-com-blog-assets/images/blog/2025/11/this-commsec-app-was-coded-entirely-by-ai/icon-unavailable-rounded.png" alt="Rounded gold koala play icon" />

{/* Claude AI refined the visual elements, ensuring the koala mascot was both professional and friendly—perfectly matching the extension's helpful nature. */}

{/* ## Real-World Performance

The testing phase revealed impressive results:

### Reliability Metrics
- **Session detection accuracy**: 100% success rate on CommSec's continuation prompts
- **False positive rate**: Zero unintended clicks on other page elements
- **Resource usage**: Minimal CPU impact, efficient DOM monitoring
- **Market hours alignment**: Perfect synchronization with ASX schedule

### User Experience Improvements
- **Eliminated manual intervention**: No more babysitting browser tabs
- **Maintained focus**: Uninterrupted market analysis during trading hours  
- **Resource efficiency**: Automatic shutdown outside market hours
- **Simple controls**: One-click toggle between active and paused states

## Technical Architecture Deep Dive

### Extension Manifest (AI-Generated)

```json
{
  "manifest_version": 3,
  "name": "Koala-Tinue",
  "version": "1.0",
  "description": "Keep your CommSec session alive automatically",
  "permissions": [
    "activeTab",
    "storage",
    "alarms"
  ],
  "content_scripts": [{
    "matches": ["*://www2.commsec.com.au/*"],
    "js": ["content.js"]
  }],
  "background": {
    "service_worker": "background.js"
  },
  "action": {
    "default_popup": "popup.html",
    "default_icon": "icon16.png"
  }
}
```

### Smart Detection Logic

The AI implemented sophisticated DOM monitoring:

```javascript
// AI-generated session detection
function detectSessionPrompt() {
  const selectors = [
    '[id*="session"]',
    '[class*="continue"]',
    'button[onclick*="session"]',
    '.modal-dialog button'
  ];
  
  return selectors.some(selector => {
    const element = document.querySelector(selector);
    return element && element.textContent.toLowerCase().includes('continue');
  });
}
```

## Lessons from AI-First Development

### What Worked Exceptionally Well

1. **Problem Understanding**: The AI accurately identified the core user pain point
2. **Feature Logic**: Smart Mode demonstrates genuine intelligence in feature design
3. **Edge Case Handling**: Time zone calculations and market hour logic were implemented correctly
4. **User Experience**: The interface is intuitive and the functionality is reliable

### The Human Role in AI Development

The developer's testing role revealed important insights:

- **Quality Assurance**: Human testing validated AI logic against real-world conditions
- **User Experience Validation**: Ensuring the solution actually solves the intended problem
- **Integration Testing**: Verifying compatibility with CommSec's actual interface
- **Performance Monitoring**: Confirming efficient resource usage and reliability

## The Future of AI-Generated Software

Koala-Tinue represents more than just a useful Chrome extension—it's a proof of concept for AI's current capabilities in software development.

### Current AI Strengths
- **Complete feature implementation**: From concept to working code
- **Complex logic handling**: Market hours, time zones, and conditional behavior
- **User interface design**: Intuitive controls and visual feedback
- **Resource optimization**: Efficient algorithms and smart scheduling

### Limitations and Considerations
- **Testing dependency**: Human validation remains crucial for real-world deployment
- **Domain knowledge**: Success depends on clear problem definition and requirements
- **Iterative improvement**: Human feedback loops enhance AI-generated solutions
 */}
## Extension Publishing

After the artwork was finalized and the code thoroughly tested, the next step was to create a distribution package and publish the extension to the Chrome Web Store, making Koala-Tinue available to all CommSec users.


## Installing and Using Koala-Tinue

Ready to try this AI-generated solution? Here's how to get started:

1. **Install the extension** from the [Chrome Web Store](https://chromewebstore.google.com/detail/koala-tinue/bcgbjabijhfedggiaefjbjbaodgflcgi?authuser=0&hl=en)
2. **Navigate to CommSec** and open your Watchlist
3. **Click the Koala icon** in your toolbar to toggle between active/paused
4. **Let Smart Mode handle the rest** during ASX trading hours

The extension works silently in the background, clicking continuation prompts so you can focus on market analysis instead of session management.

## Final Thoughts

Koala-Tinue proves that AI can move beyond code assistance to complete software creation. While human oversight remains valuable for testing and validation, the technical implementation—from architecture to advanced features—can be handled entirely by AI tools working in collaboration. It is also worth noting that a significant portion of the project time was dedicated to the artwork, a crucial step in delivering a polished and decent final product.

For developers, this suggests a future where AI handles implementation while humans focus on problem definition, testing, and user experience validation. For users, it means solutions to everyday frustrations can be developed faster and more efficiently than ever before.

The next time you're watching the ASX through CommSec without interruption, remember: you're experiencing software that was conceived, designed, and coded entirely by artificial intelligence. The future of development isn't just AI-assisted—it's AI-driven.

*Ready to experience AI-generated software firsthand? Try [Koala-Tinue](https://chromewebstore.google.com/detail/koala-tinue/bcgbjabijhfedggiaefjbjbaodgflcgi?authuser=0&hl=en) and see how seamless your CommSec monitoring can become.*


<ContactFooter />


609eba1a7c76416ac8f39482fc020ff4193341b70934095b66ac3b0d0acb68c7

Discover how Google Gemini, Codex, and Claude AI collaborated to build Koala-Tinue, a Chrome extension that keeps CommSec sessions alive automatically.

How AI tools created a Chrome extension that solves CommSec session timeouts with zero human coding.

This Chrome Extension for CommSec Was Coded Entirely by AI

{/* banner prompt: Of course! Based on the blog post's core theme of Docker resolving the chaotic and frustrating .NET build hangs on Linux, here are three creative and engaging prompts for a 16:9 banner image.

---

### Prompt Option 1 (The Sanctuary)

**Prompt:**
A digital art illustration in a cinematic, futuristic style. In the chaotic background, a storm of tangled red wires, glitching code snippets, and faint error messages like `GLIBC_2.38` swirls around a distressed Linux penguin (Tux). In the foreground, a glowing, translucent blue Docker container acts as a perfect shield. Inside the container, the .NET logo is pristine and a set of gears are turning smoothly, representing a successful build. The scene is illuminated by the calm, cool light from the Docker container, pushing back the chaotic red darkness. Aspect ratio 16:9.

**Why it works:** This prompt visually captures the central conflict: the "dependency hell" of Linux (chaos, errors, frustration) versus the clean, isolated, and reliable environment provided by Docker (the sanctuary).

---

### Prompt Option 2 (The Assembly Line)

**Prompt:**
A dynamic vector illustration showing a transformation from left to right. On the left, the .NET logo is stuck in a complex, rusty, and jammed machine made of mismatched parts, with a progress bar frozen at 99%. A frustrated Linux penguin is looking at it with a wrench. On the right, the iconic Docker whale is calmly overseeing a sleek, modern, and efficient assembly line made of Docker containers. This assembly line is smoothly processing the .NET logo, which emerges on the far right as a polished, glowing, and complete application icon. The entire image has a clean, tech-focused aesthetic. Aspect ratio 16:9.

**Why it works:** This prompt visualizes the blog post's narrative of a broken process being fixed. It directly contrasts the "build hang" (jammed machine) with the "reproducible build" (streamlined assembly line), highlighting Docker's role as a workflow optimizer.

---

### Prompt Option 3 (The Breakthrough)

**Prompt:**
Conceptual digital art with dramatic lighting. The .NET logo is depicted as being trapped and frozen inside a complex, dark labyrinth made of glitching, fragmented code representing the Linux environment. A powerful and luminous Docker whale is breaking through the walls of the labyrinth, shattering the digital "ice" and untangling the code with streams of clean, blue data. The breakthrough creates a wave of light that illuminates the path forward. The scene conveys a powerful sense of resolution and freedom from a persistent problem. Aspect ratio 16:9.

**Why it works:** This prompt is more metaphorical and heroic. It portrays the build hang as a formidable obstacle (a labyrinth) and Docker as the powerful solution that provides a "breakthrough," which directly reflects the feeling of solving a long-standing, frustrating technical issue. */}

Microsoft's transformation from a proprietary software giant to an open-source advocate has been remarkable. The release of .NET Core for Linux marked a pivotal moment in cross-platform development. However, despite this progress, developers working with .NET on Linux have encountered persistent challenges that can significantly impact productivity.

## The Promise and Reality of .NET on Linux

When Microsoft announced .NET Core's availability on Linux, it opened doors for developers who had been yearning to use their favorite .NET technologies outside the Windows ecosystem. The framework itself is remarkably solid – applications run smoothly, performance is excellent, and the development experience is largely seamless.

But there's a catch that many developers have discovered the hard way: the build process.

## The Persistent Build Hang Problem

The `dotnet build` command on Linux has a notorious reputation for hanging unexpectedly. This isn't a minor inconvenience – it's a productivity killer that affects development workflows across teams and projects.

### Characteristics of the Problem

The hanging behavior exhibits several frustrating patterns:

- **Unpredictable occurrence**: It works perfectly on one machine but fails consistently on another with seemingly identical configurations
- **Intermittent failures**: The build might succeed several times, then suddenly hang without any code changes
- **Repository dependency**: Sometimes checking out a fresh clone of the same repository resolves the issue temporarily
- **No clear error messages**: The process simply stops responding, providing no actionable feedback

### The Library Dependency Nightmare

The build process is only half the battle. A common scenario unfolds like this: you successfully build your .NET application on your development machine (e.g., running the latest Ubuntu), and everything looks great. You then deploy the compiled artifacts to a production server, which might be running an older but more stable distribution like Debian or CentOS.

When you try to run the application on the target machine, it crashes with errors like these:

```bash
Nov 12 10:47:53 workstation run.sh[290219]: ./[YOUR_APP]: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by ./[YOUR_APP])
Nov 12 10:47:53 workstation run.sh[290219]: ./[YOUR_APP]: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by ./[YOUR_APP])
```

These errors indicate version mismatches between the system's installed libraries and what the .NET runtime expects – a classic Linux dependency hell scenario.

## Why These Issues Persist

A simple Google search for "dotnet build linux hung" reveals the scope of this problem. Results span back at least four years, with countless developers reporting similar experiences across different Linux distributions, .NET versions, and hardware configurations.

The persistence of these issues a couple of underlying factors. The .NET build system performs numerous operations under the hood – package restoration, compilation, linking, and asset generation. Any step in this pipeline can encounter environment-specific issues that cause hanging. Also Linux systems handle process scheduling and resource allocation differently than Windows. The .NET build process might encounter unexpected contention or deadlock scenarios specific to Linux environments. The exact root cause remains elusive. Despite numerous bug reports and community discussions, a definitive, universal solution from the Microsoft .NET team has not materialized. The problem's intermittent and environment-specific nature makes it incredibly difficult to diagnose. Rather than getting stuck trying to solve a mystery that has stumped even the core development team, this guide focuses on a pragmatic and reliable workaround that bypasses the problem entirely.

## Docker as the Solution

While the root causes remain complex, Docker provides a practical workaround that many development teams have adopted successfully.

### Why Docker Works

Docker containers offer several advantages for .NET builds on Linux:

**Consistent Environment**: Every build runs in an identical, controlled environment regardless of the host system's configuration.

**Isolated Dependencies**: Container images include all necessary runtime dependencies at known versions, eliminating library mismatch issues.

**Reproducible Builds**: The same container image produces identical results across different machines and CI/CD environments.

### Setting Up Dockerized .NET Builds

Here's a practical implementation for containerizing your .NET builds:

#### Basic Dockerfile

```dockerfile
# Simple single-stage Dockerfile for building [YOUR_APP] on Ubuntu 22.04
FROM ubuntu:22.04

# Prevent interactive prompts
ENV DEBIAN_FRONTEND=noninteractive

# Install .NET SDK 8.0 for Ubuntu 22.04
RUN apt-get update && apt-get install -y wget ca-certificates \
    && wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb \
    && dpkg -i packages-microsoft-prod.deb \
    && rm packages-microsoft-prod.deb \
    && apt-get update \
    && apt-get install -y dotnet-sdk-8.0 \
    && rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Configure environment
ENV DOTNET_CLI_TELEMETRY_OPTOUT=1 \
    DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1 \
    MSBUILDDISABLENODEREUSE=1

# Copy all project files
COPY . .

# Fix NuGet.Config
RUN printf '%s\n' \
    '<?xml version="1.0" encoding="utf-8"?>' \
    '<configuration>' \
    '  <packageSources>' \
    '    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />' \
    '  </packageSources>' \
    '</configuration>' > NuGet.Config

# Remove global.json to allow any .NET 8 SDK
RUN rm -f global.json

# Restore, build, and publish with runtime identifier
RUN echo "=== RESTORE (with RID) ===" \
    && dotnet restore [YOUR_APP].csproj -r linux-x64 2>&1 | tee restore.log \
    && echo "=== BUILD ===" \
    && dotnet build [YOUR_APP].csproj -c Release --no-restore -r linux-x64 2>&1 | tee build.log \
    && echo "=== PUBLISH ===" \
    && dotnet publish [YOUR_APP].csproj -c Release --no-restore --no-build \
       -o /app/publish \
       -r linux-x64 \
       --self-contained false \
       2>&1 | tee publish.log \
    && echo "=== BUILD COMPLETE ===" \
    && ls -lh /app/publish/

# Set entrypoint
WORKDIR /app/publish
CMD ["/bin/bash"]

```

Remember to replace [YOUR_APP] in the above Dockerfile to your actual project name.


#### Build Script

Create a shell script to streamline the Docker build process:

```bash
#!/bin/bash
#
# Simple Docker build script with better error handling
#

set -e

PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$PROJECT_DIR"

echo "=========================================="
echo "Building [YOUR_APP] in Docker "
echo "Target: Ubuntu 22.04"
echo "=========================================="
echo

# Configuration
IMAGE_NAME="[YOUR_APP]-docker"
CONTAINER_NAME="[YOUR_APP]-extract"
OUTPUT_DIR="./bin/Release/net8.0/publish"

# Clean up
echo "Cleaning up previous containers..."
docker rm -f "$CONTAINER_NAME" 2>/dev/null || true

# Build the Docker image
echo
echo "Building Docker image..."
if ! docker build -f Dockerfile -t "$IMAGE_NAME:latest" . 2>&1 | tee docker-build.log; then
    echo
    echo "=========================================="
    echo "BUILD FAILED!"
    echo "Check docker-build.log for details"
    echo "=========================================="
    exit 1
fi

echo
echo "Creating container to extract build..."
docker create --name "$CONTAINER_NAME" "$IMAGE_NAME:latest"

# Create output directory
mkdir -p "$OUTPUT_DIR"

# Copy published files
echo "Extracting build artifacts..."
if docker cp "$CONTAINER_NAME:/app/publish/." "$OUTPUT_DIR/"; then
    echo "Build artifacts extracted successfully"
else
    echo "Warning: Could not extract all artifacts, checking what\'s available..."
    docker run --rm "$IMAGE_NAME:latest" ls -la /app/publish/ || true
fi

# Clean up container
docker rm "$CONTAINER_NAME"

echo
echo "=========================================="
echo "Build process completed!"
echo "=========================================="
echo
if [ -d "$OUTPUT_DIR" ] && [ "$(ls -A $OUTPUT_DIR)" ]; then
    echo "Output directory: $OUTPUT_DIR"
    echo "Files:"
    ls -lh "$OUTPUT_DIR" | head -20
    echo
    echo "Binary file check:"
    file "$OUTPUT_DIR/[YOUR_APP]" 2>/dev/null || echo "[YOUR_APP] binary not found"
    echo
    echo "To test on target workstation:"
    echo "  scp -r $OUTPUT_DIR/* workstation:/apps/[YOUR_APP]/current/"
else
    echo "ERROR: No files in output directory!"
    echo "Check docker-build.log for errors"
    exit 1
fi

```

Again, remember to replace [YOUR_APP] in the above Dockerfile to your actual app name.

{/* ```bash
#!/bin/bash
# build.sh

set -e

echo "Starting Docker build..."

docker build -t your-app:latest .

echo "Extracting build artifacts..."

# Create output directory
mkdir -p ./dist

# Copy build output from container
docker create --name temp-container your-app:latest
docker cp temp-container:/app/publish ./dist/
docker rm temp-container

echo "Build completed successfully!"
```

#### Docker Compose for Development

For more complex scenarios, use Docker Compose:

```yaml
version: '3.8'

services:
  build:
    build: .
    volumes:
      - ./dist:/app/publish
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    
  runtime:
    image: mcr.microsoft.com/dotnet/aspnet:8.0
    ports:
      - "5000:80"
    volumes:
      - ./dist:/app
    working_dir: /app
    command: dotnet YourProject.dll
    depends_on:
      - build
```

## Implementation Best Practices

### 1. **Multi-Stage Builds**

Use multi-stage Docker builds to optimize image size and build performance:

```dockerfile
# Build stage
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY . .
RUN dotnet publish -c Release -o out

# Runtime stage
FROM mcr.microsoft.com/dotnet/aspnet:8.0
WORKDIR /app
COPY --from=build /src/out .
ENTRYPOINT ["dotnet", "YourProject.dll"]
```

### 2. **Layer Optimization**

Structure your Dockerfile to maximize Docker layer caching:

```dockerfile
# Copy and restore dependencies first (changes less frequently)
COPY *.csproj ./
RUN dotnet restore

# Copy source code (changes more frequently)
COPY . .
RUN dotnet build
```

### 3. **Build Arguments for Flexibility**

Use build arguments to customize builds for different environments:

```dockerfile
ARG BUILD_CONFIGURATION=Release
ARG DOTNET_VERSION=8.0

FROM mcr.microsoft.com/dotnet/sdk:${DOTNET_VERSION} AS build
RUN dotnet build -c ${BUILD_CONFIGURATION}
```

## Measuring the Impact

Teams implementing Docker-based builds typically report:

- **Eliminated hanging**: Complete resolution of build hang issues
- **Improved consistency**: Identical builds across development, staging, and production environments
- **Reduced debugging time**: Less time spent troubleshooting environment-specific issues
- **Better CI/CD reliability**: More predictable automated build processes

## Potential Drawbacks and Considerations

While Docker solves the immediate problems, it introduces some complexity:

### **Increased Resource Usage**

Docker containers consume additional memory and disk space. Ensure your development machines have adequate resources.

### **Learning Curve**

Team members unfamiliar with Docker need time to adapt to containerized workflows.

### **Build Time Overhead**

Initial Docker builds can be slower than native builds, though layer caching mitigates this over time.

### **Debugging Challenges**

Debugging applications running inside containers requires additional setup and tooling.

## Alternative Approaches

Before committing to Docker, consider these alternatives:

### **Using Different .NET Versions**

Sometimes switching between LTS and current .NET versions resolves hanging issues:

```bash
# Try different SDK versions
sudo apt install dotnet-sdk-6.0
sudo apt install dotnet-sdk-7.0
```

### **Environment Variables**

Some developers report success with specific environment settings:

```bash
export DOTNET_CLI_TELEMETRY_OPTOUT=1
export DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
export NUGET_XMLDOC_MODE=skip
```

### **Build Isolation**

Running builds in isolated directories or using different user accounts sometimes helps:

```bash
# Build in temporary directory
mkdir /tmp/build-$(date +%s)
cd /tmp/build-$(date +%s)
git clone your-repo.git
cd your-repo
dotnet build
``` */}

## Looking Forward

The .NET build hanging issue on Linux remains an active concern in the .NET community. Microsoft continues working on improvements, but the complexity of supporting diverse Linux environments means progress is incremental.

Docker provides a pragmatic solution that works today. While it adds complexity to your development workflow, the trade-off between operational overhead and build reliability often favors containerization.

As the .NET ecosystem matures on Linux, we can expect better native support and fewer environment-specific issues. Until then, Docker remains a reliable path forward for teams serious about .NET development on Linux.

## Conclusion

Microsoft's embrace of open source and cross-platform development has created exciting opportunities for .NET developers. However, the reality of cross-platform development includes challenges that require practical solutions.

The persistent build hanging issues on Linux demonstrate that even mature frameworks can struggle with platform-specific complexities. Docker offers a proven workaround that many teams have adopted successfully, trading some additional complexity for significantly improved reliability.

Whether this Docker-based approach will remain necessary long-term remains to be seen. For now, it provides a stable foundation for .NET development on Linux, allowing teams to focus on building great applications rather than fighting build system issues.

If you're experiencing similar challenges with .NET builds on Linux, consider implementing a Docker-based workflow. While it may seem like overkill initially, the improved consistency and reliability often justify the additional setup complexity.

---

Do you have similar experience? Share your comments below.

320e7b744ccc39f7c0ff72a382e896737ceb0b5cd8969c9b0974d2ae0fff33f0

Comprehensive guide to resolving .NET Core build hangs and library dependency errors on Linux using Docker containerization for consistent builds.

Learn how Docker containers solve persistent .NET build hanging issues on Linux distributions.

TYO LabWe Build

Latest

Measurable Beats Powerful: Grounding AI in Tangible Checks

Shipping a native Go binary inside an Android APK: ELF linkers, 16 KB pages, and the memfd trick

Introducing TYO Reach: A Travel Companion for Your Browser

Introducing reach-dl: a Local, Private Download Manager for TYO Reach

The Model That Aced Olympiad Math Can't Write a Tweet

Artificial Intelligence

You Don't Need to Be Bunnings to Build an AI Assistant

The 4 GB Card You Already Own Can Reason Now

The Small Model's Real Job Is Dispatch, Not Work

What a Native Tool Call Actually Is

The 4 GB Experiment: What Actually Makes a Small Model a Good Coding Agent

How a Coding Agent Actually Writes Code

Trading & Finance

cTrader FIX API: Which Brokers Say Yes to Retail Traders?

Building Blocks of Algorithmic Trading: Collecting End-of-Day Data

Finding Your Edge: Build a RAG System with Onyx for Better Trading Strategies

Tech

Serverless Web Proxy on Cloud Run: A Better VPN?

Managing Windows Made Easy with Cygwin and Claude

Router Down! My 3-Day Battle with Meraki, Ubiquiti & VLANs

Linux

Repurpose Your Old PCs: How To Install Alpine Linux on Them

Escaping VirtualBox Hell: How I Migrated Windows 11 to KVM with AI

How to put your baby, sorry, your Linux box to sleep

Networking

Cloud Run Domain Mappings with Cloudflare: Why Your SSL is Stuck

Configure a Cisco 1941 Router as Your Home Office Gateway with IPv6

Create a Backup Mobile Data WAN for Your Small Business / Home Network

Windows

RDP 'Please Wait' Black Screen? A Simple Fix

Fix the 'Could Harm Your Computer' File Preview Error

Why That 'I Ditched Linux for Windows' Article Misses the Point Entirely