Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp Guide)
This video shows how a 22.36 GB four-bit Qwen3.6 35B A3B build runs at 17 tokens a second on a 6 GB GTX 1060 by using llama.cpp flags (--n-cpu-moe, --no-mmap, --n-gpu-layers, --mlock, and a larger micro-batch) to park idle mixture-of-experts weights in system RAM while attention, embeddings, the shared expert, and the KV cache stay on the GPU. It then names the three places the trick breaks: memory-bandwidth ceilings, TurboQuant KV compression that never merged upstream, and speculative decoding that backfires on sparse models.
Cloud Codes12 minTranscript found
Quick learning frame
Read this before watching.
AI-native interfaces are control surfaces for intent, artifacts, context, preview, inspection, and iteration.
New playlist item from Cloud Codes; queued for transcript-backed review, topic mapping, and a practical learning artifact.
Skill you build: The ability to decide which weights of a sparse mixture-of-experts model deserve scarce VRAM and to tune llama.cpp offload flags and batch sizes accordingly, instead of treating VRAM capacity as a hard wall.
Watch for the shift from claim to mechanism. The learning value is the point where the transcript reveals a repeatable action, tool boundary, context move, review habit, or artifact.
Concept diagram
Where this video fits.
01Intent
02Canvas
03Artifact
04Preview
05Feedback
06Iteration
Deep lesson
Turn this video into working knowledge.
2,028 cleaned transcript words reviewed across 596 timed caption segments.
Thesis
Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp Guide) teaches a practical interfaces + open design move: This video shows how a 22.36 GB four-bit Qwen3.6 35B A3B build runs at 17 tokens a second on a 6 GB GTX 1060 by using llama.cpp flags (--n-cpu-moe, --no-mmap, --n-gpu-layers, --mlock, and a larger micro-batch) to park idle mixture-of-experts weights in system RAM while attention, embeddings, the shared expert, and the KV cache stay on the GPU. It then names the three places the trick breaks: memory-bandwidth ceilings, TurboQuant KV compression that never merged upstream, and speculative decoding that backfires on sparse models.
The goal is not to remember the video. The goal is to extract the operating principle, tie it to timestamped evidence, test how far the claim transfers, and make something reusable.
1:23
Sparsity beats capacity
“single token, so attention goes on the GPU. Embeddings, the shared expert, the key value cache, all GPU. The routed experts are idle most of the time. Idle weights can live in system RAM. Until the middle of...”
Qwen3.6 35B A3B has 256 experts but the router wakes only eight plus one shared expert per token, so it moves under 1 GB of weights per token where a dense 35B at four bits would stream roughly 20 GB; that makes the real question not whether the model fits but which parts must be fast, so attention, embeddings, the shared expert, and the KV cache go to GPU while the 248 idle experts live in system RAM. Write out the per-token memory traffic for a dense 35B versus this 35B-A3B model at four bits, then list which tensors you would pin to VRAM and why.
6:54
Prompt versus decode
“pointing a coding agent at a repository, you will use every one of them. Here's what that costs on this specific model. 20,480 bytes of key value cache per token per sequence, 20 KB. So, the full 256,000...”
The CUDA backend compares every expert matmul against op_offload_min_batch_size, default 32: decoding one token at a time falls under it so the CPU does the math at about 51 GB/s (your 17 tok/s), while a pasted prompt of hundreds of tokens exceeds it, so llama.cpp ships expert weights over PCIe to the GPU; raising the 512-token micro-batch matters, and the maintainer's benchmark went from 22 to 345 tokens a second moving 128 up to 2048. Run one short chat turn and one long pasted-file prompt against the same local model and record decode versus prompt-processing throughput, then re-run with the micro-batch raised to 2048.
9:45
Where it breaks
“across the memory bus. That paper measured verification running two to three times slower and speculation making the whole system up to one and a half times slower than not speculating at all. The workaround is multi-token prediction,...”
TurboQuant's 3-bit rotated KV cache cuts KV memory at least six times and takes a 6 GB card from 64k to 256k context, but it is not in llama.cpp: all 75-odd related pull requests are closed unmerged (two tagged an AI policy violation), it needs a fork 300 commits ahead, and after the MoE attention kernel rewrite Turbo 2 decodes at 45 percent of F16. Speculative decoding fails too, because guessing several tokens ahead wakes most of the experts instead of eight, measured as up to 1.5 times slower than not speculating. List the three failure modes named here (bandwidth ceiling, unmerged TurboQuant, expert-waking speculation) and, for each, write the one measurement that would tell you it is hurting your setup.
01
Intent
Start with this video's job: This video shows how a 22.36 GB four-bit Qwen3.6 35B A3B build runs at 17 tokens a second on a 6 GB GTX 1060 by using llama.cpp flags (--n-cpu-moe, --no-mmap, --n-gpu-layers, --mlock, and a larger micro-batch) to park idle mixture-of-experts weights in system RAM while attention, embeddings, the shared expert, and the KV cache stay on the GPU. It then names the three places the trick breaks: memory-bandwidth ceilings, TurboQuant KV compression that never merged upstream, and speculative decoding that backfires on sparse models. Treat "Intent" as the outcome you are trying to make visible, not a topic label. Anchor it to 1:23, where the video says: “single token, so attention goes on the GPU. Embeddings, the shared expert, the key value cache, all GPU. The routed experts are idle most of the time. Idle weights can live in system RAM. Until the middle of...”
02
Canvas
Use "Canvas" to locate the part of the interfaces + open design workflow the video is demonstrating. Ask what changes in your real setup if this claim is true. Anchor it to 6:54, where the video says: “pointing a coding agent at a repository, you will use every one of them. Here's what that costs on this specific model. 20,480 bytes of key value cache per token per sequence, 20 KB. So, the full 256,000...”
03
Artifact
Turn "Artifact" into the reusable artifact for this lesson: A UI critique sheet for judging whether an AI interface improves control. This is where watching becomes something you can inspect and reuse.
04
Preview
Use "Preview" as the application surface. Decide whether the idea touches a browser flow, a local file, a model choice, a source document, a UI, or a review step.
05
Feedback
Use "Feedback" to prove the lesson. The evidence should connect back to the video title, transcript anchors, and a concrete output, not a generic best-practice claim.
06
Iteration
Use "Iteration" to carry the idea forward: save the prompt, checklist, diagram, or operating rule that would make the next agent run better.
Example
Source-backed work packet
Convert the video into a scoped task that includes the transcript claim, target workflow, acceptance criteria, and proof. The output should be a ui critique sheet for judging whether an ai interface improves control..
Example
Claim vs. demo brief
Separate what the speaker claims, what the demo actually proves, and what still needs outside verification before you adopt the workflow.
Example
Teach-back module
Transform the lesson into a definition, a mechanism diagram, one misconception, one practice exercise, and a check-for-understanding question.
Do not learn it wrong
Treating the title as the lesson without checking what the transcript actually says.
Letting the prompt drift into generic advice that could apply to any video in the playlist.
Copying the tool setup without identifying the operating principle that transfers to your own stack.
Skipping the artifact, which means the learning never becomes operational or inspectable.
Do not count this as learned until these are true.
01
State the transcript-backed claim in your own words: This video shows how a 22.36 GB four-bit Qwen3.6 35B A3B build runs at 17 tokens a second on a 6 GB GTX 1060 by using llama.cpp flags (--n-cpu-moe, --no-mmap, --n-gpu-layers, --mlock, and a larger micro-batch) to park idle mixture-of-experts weights in system RAM while attention, embeddings, the shared expert, and the KV cache stay on the GPU. It then names the three places the trick breaks: memory-bandwidth ceilings, TurboQuant KV compression that never merged upstream, and speculative decoding that backfires on sparse models.
02
Explain the practical stakes without hype: New playlist item from Cloud Codes; queued for transcript-backed review, topic mapping, and a practical learning artifact.
03
Map the idea onto the Intent -> Canvas -> Artifact -> Preview -> Feedback -> Iteration sequence and name the weakest link.
04
Produce the artifact and include the evidence that proves it: A UI critique sheet for judging whether an AI interface improves control.
Put it into practice
Give this grounded prompt to Codex or Claude after watching.
You are helping me turn one specific YouTube video into real, durable learning.
Source video:
- Title: Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp Guide)
- URL: https://www.youtube.com/watch?v=7AExwNFlXU4
- Topic: Interfaces + Open Design
- My current learning frame: Load a four-bit mixture-of-experts model on your own GPU, walk --n-cpu-moe down from every layer until the loader runs out of VRAM, then push --n-gpu-layers back up and record tokens per second alongside your RAM speed.
- Why this matters: New playlist item from Cloud Codes; queued for transcript-backed review, topic mapping, and a practical learning artifact.
Transcript anchors from this exact video:
- 1:23 / Evidence 1: "single token, so attention goes on the GPU. Embeddings, the shared expert, the key value cache, all GPU. The routed experts are idle most of the time. Idle weights can live in system RAM. Until the middle of..."
- 3:54 / Evidence 2: "of system memory, run faster with memory mapping left on because the page cache is doing real work. Flag three is arithmetic. Once the experts move out, most of your 6 gigs is empty again. So, push layers..."
- 6:54 / Evidence 3: "pointing a coding agent at a repository, you will use every one of them. Here's what that costs on this specific model. 20,480 bytes of key value cache per token per sequence, 20 KB. So, the full 256,000..."
- 9:45 / Evidence 4: "across the memory bus. That paper measured verification running two to three times slower and speculation making the whole system up to one and a half times slower than not speculating at all. The workaround is multi-token prediction,..."
- 11:18 / Evidence 5: "You have stopped buying capacity and started deciding which weights deserve the fast memory. 6 GB, a card from 2016, 35 billion parameters answering in real time. Go run it, then tell me your tokens per second and..."
Your task:
1. Use the transcript anchors above as the primary source packet. If you add outside context, label it clearly as outside context and keep it secondary.
2. Create a source-check table with columns: timestamp, claim, what the demo proves, confidence, and what still needs verification.
3. Extract the actual teachable claims from the video. Do not invent claims that are not supported by the title, lesson frame, or transcript anchors.
4. Build a reusable learning artifact: A UI critique sheet for judging whether an AI interface improves control.
5. Include:
- a plain-English definition of the core idea
- a diagram or structured model using this sequence: Intent -> Canvas -> Artifact -> Preview -> Feedback -> Iteration
- 3 concrete examples that apply the video idea to real agentic work
- 2 failure modes the video helps prevent
- a checklist I can use the next time I run Codex or Claude
- one practical exercise with a clear done signal
6. Add a "learning transfer" section: what changes in my workflow tomorrow if I actually learned this?
7. Add a "source check" section that cites which transcript anchor supports each major takeaway.
Quality bar:
- Make this specific to "Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp Guide)", not a generic Interfaces + Open Design essay.
- Prefer operational examples, failure modes, and reusable artifacts over broad definitions.
- Call out uncertainty instead of smoothing over weak evidence.
- If evidence is weak, say what transcript segment or timestamp needs review instead of guessing.
- Finish with a concise artifact I could paste into my learning app.
Misconceptions
What to stop believing.
A beautiful page is automatically a good learning tool.
Learning requires sequence, active recall, feedback, and application.
Generated UI should be accepted as-is.
Generated UI needs critique, revision, and browser verification.
Practice studio
Learning only counts when you make something.
01
Transcript evidence map
Separate what the video actually says from what you already believe about the topic.
3 source-backed takeaways with timestamps, confidence, and a transfer note.02
One useful artifact
Apply the video to a real workflow and produce a ui critique sheet for judging whether an ai interface improves control..
A reusable artifact with a done signal and one verification step.03
Teach-back card
Explain the lesson to someone who has not watched the video yet.
A 90-second explanation, one diagram, one example, and one misconception to avoid.
Recall check
Answer first, then reveal — without rewatching.
Why can a 22 GB model run usefully on a 6 GB card when a dense model of the same parameter count cannot?
What decides whether an expert matrix multiply runs on the CPU or gets copied to the GPU?
Why does the video say TurboQuant KV compression is a memory win rather than a speed win, and why is it hard to use?
Source shelf
Use the video as a doorway, then verify with primary sources.
