{/* banner prompt: A playful, polished 3D clay-render illustration in a soft matte palette of sage green, warm cream and terracotta orange. The scene shows a chunky, rounded, stylized NVIDIA graphics card sculpted from smooth modelling clay, sitting on a flat cream surface. From one end of the card a long ribbon of pale paper unfurls and coils gently into a neat spiral stack, like a scroll of memory. A tiny terracotta pot holding a small green bonsai tree with rounded clay leaves rests on top of the card. Everything has a tactile hand-sculpted matte finish with gently rounded edges, soft diffuse studio lighting, warm subtle shadows, shallow depth of field, and a simple uncluttered pastel background. Wide 16:9 composition, centred, clean and minimal. */}

Last month I put a [27-billion-parameter reasoning model on this same RTX 3060](/blog/2026/08/18-now-it-thinks-a-27b-language-model-on-the-same-rtx-3060/) — Qwen3.8-27B, at 4-bit, about 18GB. It worked, but only just: a bit more than half the model sat on the card and the rest streamed from system RAM, and it typed back at roughly reading speed, about five words a second.

This post is the same model — and a much better story. Someone re-compressed Qwen3.8-27B down to **under 6GB**, small enough that it now fits *entirely* on the 12GB card with room to spare. It runs about **four times faster**. And with the space freed up, it can hold something the 4-bit version never could: a genuinely enormous context — up to **262,000 tokens**, a fat novel's worth of text, read all at once.

## The machine (still the same one)

Same box as every post in this series: an RTX 3060 with **12GB of VRAM**, 64GB of system RAM, Debian. Nothing upgraded. The card is the constant; only what we ask of it changes.

## What changed: ternary instead of 4-bit

Last time, quantization squeezed each of the model's numbers down to 4 bits. This model goes further — it uses **ternary** quantization, where almost every weight is stored as just one of three values: -1, 0, or +1. That is a hair under two bits per weight, and it drops the file from over 50GB in full precision to **5.95GB**.

The payoff is that the model no longer has to be split between the card and RAM. All of it lives on the GPU, where the math is fast, and none of it has to be streamed across from system memory. That single change — everything on-card instead of 57% on-card — is most of where the speed-up comes from.

## The honest numbers

Same model, same card, last month versus this month:

<Table data={
    {
        headers: ['', 'Last month (4-bit)', 'This month (ternary)'],
        rows: [
            ['Size on disk', '~18 GB', '5.95 GB'],
            ['Runs on GPU', '~57% on card, rest from RAM', '100% on card'],
            ['VRAM used', '11 of 12 GB', '~8 GB (at a 32K context)'],
            ['Speed', '~4.7 words/sec', '~20 words/sec (about 27 tokens/sec)'],
            ['Math, coding, reasoning', 'Correct', 'Correct'],
        ]
    }
} />

Twenty words a second is a comfortable reading pace that no longer feels like waiting. For a single thoughtful answer on your own machine, it is genuinely pleasant to use.

## The part you actually asked about: how much can it remember?

"Context" is how much text a model can hold in its head at once — the question you ask, plus every document, transcript, or code file you paste in with it. The bigger the context, the more it can read before it starts forgetting the start.

Once the weights only take ~6GB, the rest of the 12GB card is free for exactly this. So I measured it: how large a context actually fits before the card runs out of memory. The lever is the *KV cache* — the running memory the model keeps of everything it has read so far — and you can store that cache at different precisions, just like the weights.

<Table data={
    {
        headers: ['KV cache precision', 'Largest context that fits', 'VRAM used', 'Speed'],
        rows: [
            ['16-bit (default)', '~64K tokens (about 48,000 words)', '~10.3 GB', '~27 tokens/sec'],
            ['8-bit', '128K tokens (about 96,000 words)', '~11.1 GB', '~26 tokens/sec'],
            ['4-bit', '262K tokens (about 196,000 words)', '~12 GB (tight)', '~26 tokens/sec'],
        ]
    }
} />

So the answer is: on a 12GB card with the GPU to itself, you comfortably get a **64K-token** context out of the box, **128K** by storing the cache at 8-bit, and the model's full **262K** by dropping the cache to 4-bit — right up against the edge of the card's memory. That top figure is roughly 196,000 words: a long novel, a year of meeting notes, or a thick stack of contracts, all in one prompt.

Two things worth noting. First, the speed barely moves as the context grows — the model reads a huge prompt about as fast as a small one, because most of its layers use a lighter form of attention that does not balloon with length. Second, the 4-bit cache is the trade-off you pay for the very largest window: less precise memory of what it read, and only a few hundred megabytes of headroom left. For everyday use, the 8-bit / 128K setting is the sweet spot.

## Does squeezing it that hard break it?

That was my worry — a model stored at under two bits per weight sounds like it should be a lobotomised version of itself. It is not. On the checks I ran it stayed sharp: a multi-step arithmetic word problem came out correct, a "write a function to check balanced brackets" request produced clean, idiomatic code with the right answers, and it explained the trade-offs of its own compression method accurately. Nothing garbled, no repetition, no collapse. The compression is impressive precisely because it is so hard to notice.

## The catch

Last month's post ended with two commands and [Ollama](https://ollama.com). This one cannot, and it is only fair to say so. The ternary format is custom — standard tools like Ollama and the usual `llama.cpp` build refuse to load it. You have to build the model author's own version of `llama.cpp` first:

```sh
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j
```

That compiles in a few minutes on a normal machine. Then download the 5.95GB model file and run it:

```sh
./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 65536 \
  -st -p "Summarise the document I am about to paste."
```

To open it up to the full 262K context, add a 4-bit cache: `-ctk q4_0 -ctv q4_0 -c 262144`.

It is a real trade-off: you buy the smaller size, the extra speed, and the huge context with a bit more setup, and a dependency on one team keeping their build going. For a quick try, last month's Ollama route is still the easier door. For a model you plan to lean on — long documents, private data, all day — this one earns the extra step.

## Where this leaves a small business

Five posts, one unchanged computer. The same RTX 3060 has gone from a single still image, to video, to video with sound, to a reasoning model that spilled over the edge of the card — and now to a 27B model that fits entirely on it, runs at a comfortable reading pace, and can swallow a novel's worth of your own documents in a single prompt, without a byte leaving the building.

That last part is the point. A context this large on a private, paid-for-once card means you can hand the model a whole contract, a full support-ticket history, or a year of notes and ask real questions about it — with none of it going to someone else's server, and no per-page cloud bill. The card did not get bigger. We just got much better at using it.

*It's a five-year-old graphics card. What it can do keeps surprising me.*


100a41bc708fc536b7140f337f846e8d7f5c6f80db39818675d03b244d8cbec2

How ternary quantization shrinks a 27B model to 5.95GB so it runs fully on a consumer RTX 3060, roughly four times faster than the 4-bit version, plus a measured look at exactly how much context fits in 12GB of VRAM: 64K by default, 128K or the full 262K with a quantized KV cache.

The fifth in our RTX 3060 series. The same Qwen3.8-27B we ran last month, re-quantized to under 6GB, now runs entirely on the 12GB card, about four times faster, and holds up to 262,000 tokens of context. A whole novel, on a five-year-old gaming card.

Tag: Quantization

Tag: Quantization

Now It Remembers: A 27B Model With a 262K-Token Context on the Same RTX 3060