Now It Thinks: A 27B Language Model on the Same RTX 3060eric

The first three posts in this series were about making things — the same RTX 3060 learning to generate images, then video, then video with sound. This one is different. This time the card isn't drawing or filming anything — it's thinking. I put a 27-billion-parameter language model on the same 12GB card, and it works.

The headline: Qwen3.8-27B, a reasoning model released only a few days ago, runs locally on a consumer RTX 3060 that shipped back in 2021. It answers questions, writes code, reasons step by step — and it can even look at a picture and tell you what's in it. The kind of model people assume needs a data centre, or at least a $2,000 graphics card, runs on hardware a small business already owns.

The machine (still the same one)

Same box as every post in this series: an RTX 3060 with 12GB of VRAM, 64GB of system RAM, Debian. Nothing upgraded. The card is the constant.

Why this is surprising

A 27B model is big. The rule of thumb people quote is that you need a 24GB card (an RTX 4090 or better) to run one at all — and its full-precision form is over 50GB, firmly data-centre territory. So "27B on 12GB" sounds like it shouldn't be possible.

It is, because of two things working together: quantization and offloading.

Fitting an 18GB model into 12GB

Quantization compresses the model's numbers to 4 bits each, shrinking it from 50GB+ down to about 18GB with very little quality lost. That still doesn't fit in 12GB — but it doesn't have to. The card holds a bit more than half the model (about 57%), and the rest lives in the 64GB of system RAM, streamed onto the GPU as it's needed. The GPU does the heavy math; the spare layers wait in RAM.

The remarkable part is how easy it was. The video-with-sound model in the last post took patches and careful memory tuning. This time it was genuinely two commands — download, and run. The tool (Ollama) works out the GPU/RAM split automatically.

The honest numbers

Four-ish words a second isn't fast — you watch the answer type itself out, roughly at reading speed. This is not the tool for churning through a thousand requests an hour. It's the tool for asking one good question and getting a thoughtful answer, privately, on your own machine, for the cost of the electricity.

It reasons — out loud

Qwen3.8 is a reasoning model: before it answers, it thinks. Ask it a coding question and you can watch it work the problem — considering approaches, catching its own mistakes, checking its logic — then give a clean final answer. Ask it to find the longest palindrome in a string, and it doesn't just spit out code; it reasons about edge cases and complexity first. That visible thinking is a big part of why these newer models are so much more useful than the ones from a year ago.

And it sees

Here's the part that closes the loop on this whole series. The model is multimodal, so I showed it one of the AI-generated animal photos from our image experiments and asked: is this a real photograph, or AI-generated?

It nailed it. It described the scene correctly — a brown bear in a rushing river, autumn forest blurred behind it — and then, unprompted, listed exactly why it looked artificial: fur that was "suspiciously smooth and painted," the "characteristic milky AI bokeh" in the background, "melted" rocks, a glassy eye lacking real reflections. A local model, on a 12GB card, giving a genuinely sharp critique of a locally-generated image. The tools in this series can now check each other's work.

Try it yourself

If you have an RTX 3060 (or any 12GB card) and Ollama installed, this is the whole thing:

Copy ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:Q4_K_M

The first run downloads the model; after that it's instant to start. Ask it anything. To feed it an image, just put a file path in your message.

Where this leaves a small business

Four posts, one unchanged computer. The same RTX 3060 went from a single 512px still, to short video, to video with a soundtrack — and now to a reasoning model that can answer questions, help write code, and read an image, all without a single byte leaving the building.

That last point matters more than the speed. Running the model locally means your questions, your customer data, your documents never touch someone else's server. No per-message cloud bill, no data leaving your office, no dependency on a subscription that can change its terms next quarter. For a lot of small businesses, "good enough, private, and paid-for-once" beats "fastest" every time.

It isn't a data centre. It doesn't need to be.

It's a five-year-old graphics card. What it can do keeps surprising me.