Now It Remembers: A 27B Model With a 262K-Token Context on the Same RTX 3060eric

Last month I put a 27-billion-parameter reasoning model on this same RTX 3060 — Qwen3.8-27B, at 4-bit, about 18GB. It worked, but only just: a bit more than half the model sat on the card and the rest streamed from system RAM, and it typed back at roughly reading speed, about five words a second.

This post is the same model — and a much better story. Someone re-compressed Qwen3.8-27B down to under 6GB, small enough that it now fits entirely on the 12GB card with room to spare. It runs about four times faster. And with the space freed up, it can hold something the 4-bit version never could: a genuinely enormous context — up to 262,000 tokens, a fat novel's worth of text, read all at once.

The machine (still the same one)

Same box as every post in this series: an RTX 3060 with 12GB of VRAM, 64GB of system RAM, Debian. Nothing upgraded. The card is the constant; only what we ask of it changes.

What changed: ternary instead of 4-bit

Last time, quantization squeezed each of the model's numbers down to 4 bits. This model goes further — it uses ternary quantization, where almost every weight is stored as just one of three values: -1, 0, or +1. That is a hair under two bits per weight, and it drops the file from over 50GB in full precision to 5.95GB.

The payoff is that the model no longer has to be split between the card and RAM. All of it lives on the GPU, where the math is fast, and none of it has to be streamed across from system memory. That single change — everything on-card instead of 57% on-card — is most of where the speed-up comes from.

The honest numbers

Same model, same card, last month versus this month:

Twenty words a second is a comfortable reading pace that no longer feels like waiting. For a single thoughtful answer on your own machine, it is genuinely pleasant to use.

The part you actually asked about: how much can it remember?

"Context" is how much text a model can hold in its head at once — the question you ask, plus every document, transcript, or code file you paste in with it. The bigger the context, the more it can read before it starts forgetting the start.

Once the weights only take ~6GB, the rest of the 12GB card is free for exactly this. So I measured it: how large a context actually fits before the card runs out of memory. The lever is the KV cache — the running memory the model keeps of everything it has read so far — and you can store that cache at different precisions, just like the weights.

So the answer is: on a 12GB card with the GPU to itself, you comfortably get a 64K-token context out of the box, 128K by storing the cache at 8-bit, and the model's full 262K by dropping the cache to 4-bit — right up against the edge of the card's memory. That top figure is roughly 196,000 words: a long novel, a year of meeting notes, or a thick stack of contracts, all in one prompt.

Two things worth noting. First, the speed barely moves as the context grows — the model reads a huge prompt about as fast as a small one, because most of its layers use a lighter form of attention that does not balloon with length. Second, the 4-bit cache is the trade-off you pay for the very largest window: less precise memory of what it read, and only a few hundred megabytes of headroom left. For everyday use, the 8-bit / 128K setting is the sweet spot.

Does squeezing it that hard break it?

That was my worry — a model stored at under two bits per weight sounds like it should be a lobotomised version of itself. It is not. On the checks I ran it stayed sharp: a multi-step arithmetic word problem came out correct, a "write a function to check balanced brackets" request produced clean, idiomatic code with the right answers, and it explained the trade-offs of its own compression method accurately. Nothing garbled, no repetition, no collapse. The compression is impressive precisely because it is so hard to notice.

The catch

Last month's post ended with two commands and Ollama. This one cannot, and it is only fair to say so. The ternary format is custom — standard tools like Ollama and the usual llama.cpp build refuse to load it. You have to build the model author's own version of llama.cpp first:

Copy git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

That compiles in a few minutes on a normal machine. Then download the 5.95GB model file and run it:

Copy ./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 65536 \
  -st -p "Summarise the document I am about to paste."

To open it up to the full 262K context, add a 4-bit cache: -ctk q4_0 -ctv q4_0 -c 262144.

It is a real trade-off: you buy the smaller size, the extra speed, and the huge context with a bit more setup, and a dependency on one team keeping their build going. For a quick try, last month's Ollama route is still the easier door. For a model you plan to lean on — long documents, private data, all day — this one earns the extra step.

Where this leaves a small business

Five posts, one unchanged computer. The same RTX 3060 has gone from a single still image, to video, to video with sound, to a reasoning model that spilled over the edge of the card — and now to a 27B model that fits entirely on it, runs at a comfortable reading pace, and can swallow a novel's worth of your own documents in a single prompt, without a byte leaving the building.

That last part is the point. A context this large on a private, paid-for-once card means you can hand the model a whole contract, a full support-ticket history, or a year of notes and ask real questions about it — with none of it going to someone else's server, and no per-page cloud bill. The card did not get bigger. We just got much better at using it.

It's a five-year-old graphics card. What it can do keeps surprising me.