I Tested Apple’s Secret macOS AI… The Results Made No Sense
Alex Ziskind benchmarks FM, the LLM CLI quietly built into the macOS 27 Golden Gate beta, discovers off-the-shelf benchmarkers and even Apple's own meters misreport it, and proves through a contention experiment that FM's decode runs on the Neural Engine while prompt processing runs on the GPU, which is why a Mac Mini matches a far pricier M3 Ultra Mac Studio.
Alex Ziskind11 minTranscript found
Quick learning frame
Read this before watching.
AI-native interfaces are control surfaces for intent, artifacts, context, preview, inspection, and iteration.
New playlist item from Alex Ziskind; queued for transcript-backed review, topic mapping, and a practical learning artifact.
Skill you build: The ability to design a controlled contention experiment to identify which hardware engine a workload actually uses when monitoring tools give false readings.
Watch for the shift from claim to mechanism. The learning value is the point where the transcript reveals a repeatable action, tool boundary, context move, review habit, or artifact.
Concept diagram
Where this video fits.
01Intent
02Canvas
03Artifact
04Preview
05Feedback
06Iteration
Deep lesson
Turn this video into working knowledge.
2,210 cleaned transcript words reviewed across 608 timed caption segments.
Thesis
I Tested Apple’s Secret macOS AI… The Results Made No Sense teaches a practical interfaces + open design move: Alex Ziskind benchmarks FM, the LLM CLI quietly built into the macOS 27 Golden Gate beta, discovers off-the-shelf benchmarkers and even Apple's own meters misreport it, and proves through a contention experiment that FM's decode runs on the Neural Engine while prompt processing runs on the GPU, which is why a Mac Mini matches a far pricier M3 Ultra Mac Studio.
The goal is not to remember the video. The goal is to extract the operating principle, tie it to timestamped evidence, test how far the claim transfers, and make something reusable.
0:36
A free LLM in macOS
“context. A lot of helpful AI these days still depends on you noticing the issue first, then writing the prompt, then managing and following up. Even when the tools are good, they still assume the right person catches...”
The macOS 27 Golden Gate beta ships FM, a built-in CLI that runs a large language model with no need to install Ollama, LM Studio, or MLX: a free, private, scriptable AI endpoint sitting in the terminal. It has two modes, a local on-device model and Apple's Private Cloud Compute, and in testing the cloud path ran about three times faster than local. If you are on the macOS 27 beta, open a terminal and invoke FM in both local and cloud modes with the same prompt, noting the speed difference yourself.
3:00
Benchmarks that lie
“little built-in model was doing 600,000 tokens a second. That's for prompt processing and decode was 22,000. Come on, really? That's definitely not a benchmark. That's a typo. Here's the thing. It's not really the tool's fault. Apple...”
Pointing the standard LLM bencher at FM returned an absurd 600,000 tokens per second for prompt processing because Apple's server is not fully OpenAI-compatible and does not hand back the timing numbers benchmarks expect, so the author built his own tool, Apple FM Bench. Real on-device numbers on the M4 Pro Mac Mini were around 1,142 for prompt processing and 56 tokens per second for decode. Write down the two inference stages (prefill/prompt processing and decode/generation) and which hardware trait each depends on, since every number in this video hangs on that split.
7:41
The contention experiment
“something bugged me. The numbers just didn't feel right. So instead of testing FM directly, I tested the tools. I built a little workload that has to use the neural engine. If you use CoreML model and you...”
Every meter (asitop, mactop, and Apple's own powermetrics underneath them) showed the Neural Engine at 0% even while a CoreML workload provably ran over 2x faster with the ANE enabled, so the author hammered each engine while FM ran: hammering the ANE halved decode only, hammering the GPU halved prompt processing only. That proves decode runs on the Neural Engine, which is essentially the same 16-core block from M2 through M5, explaining why the $10k Mac Studio could not beat the Mac Mini. Sketch the contention test as a reusable recipe: run the mystery workload, saturate one candidate engine at a time, and read which stage slows down instead of trusting utilization meters.
01
Intent
Start with this video's job: Alex Ziskind benchmarks FM, the LLM CLI quietly built into the macOS 27 Golden Gate beta, discovers off-the-shelf benchmarkers and even Apple's own meters misreport it, and proves through a contention experiment that FM's decode runs on the Neural Engine while prompt processing runs on the GPU, which is why a Mac Mini matches a far pricier M3 Ultra Mac Studio. Treat "Intent" as the outcome you are trying to make visible, not a topic label. Anchor it to 0:36, where the video says: “context. A lot of helpful AI these days still depends on you noticing the issue first, then writing the prompt, then managing and following up. Even when the tools are good, they still assume the right person catches...”
02
Canvas
Use "Canvas" to locate the part of the interfaces + open design workflow the video is demonstrating. Ask what changes in your real setup if this claim is true. Anchor it to 3:00, where the video says: “little built-in model was doing 600,000 tokens a second. That's for prompt processing and decode was 22,000. Come on, really? That's definitely not a benchmark. That's a typo. Here's the thing. It's not really the tool's fault. Apple...”
03
Artifact
Turn "Artifact" into the reusable artifact for this lesson: A UI critique sheet for judging whether an AI interface improves control. This is where watching becomes something you can inspect and reuse.
04
Preview
Use "Preview" as the application surface. Decide whether the idea touches a browser flow, a local file, a model choice, a source document, a UI, or a review step.
05
Feedback
Use "Feedback" to prove the lesson. The evidence should connect back to the video title, transcript anchors, and a concrete output, not a generic best-practice claim.
06
Iteration
Use "Iteration" to carry the idea forward: save the prompt, checklist, diagram, or operating rule that would make the next agent run better.
Example
Source-backed work packet
Convert the video into a scoped task that includes the transcript claim, target workflow, acceptance criteria, and proof. The output should be a ui critique sheet for judging whether an ai interface improves control..
Example
Claim vs. demo brief
Separate what the speaker claims, what the demo actually proves, and what still needs outside verification before you adopt the workflow.
Example
Teach-back module
Transform the lesson into a definition, a mechanism diagram, one misconception, one practice exercise, and a check-for-understanding question.
Do not learn it wrong
Treating the title as the lesson without checking what the transcript actually says.
Letting the prompt drift into generic advice that could apply to any video in the playlist.
Copying the tool setup without identifying the operating principle that transfers to your own stack.
Skipping the artifact, which means the learning never becomes operational or inspectable.
Do not count this as learned until these are true.
01
State the transcript-backed claim in your own words: Alex Ziskind benchmarks FM, the LLM CLI quietly built into the macOS 27 Golden Gate beta, discovers off-the-shelf benchmarkers and even Apple's own meters misreport it, and proves through a contention experiment that FM's decode runs on the Neural Engine while prompt processing runs on the GPU, which is why a Mac Mini matches a far pricier M3 Ultra Mac Studio.
02
Explain the practical stakes without hype: New playlist item from Alex Ziskind; queued for transcript-backed review, topic mapping, and a practical learning artifact.
03
Map the idea onto the Intent -> Canvas -> Artifact -> Preview -> Feedback -> Iteration sequence and name the weakest link.
04
Produce the artifact and include the evidence that proves it: A UI critique sheet for judging whether an AI interface improves control.
Put it into practice
Give this grounded prompt to Codex or Claude after watching.
You are helping me turn one specific YouTube video into real, durable learning.
Source video:
- Title: I Tested Apple’s Secret macOS AI… The Results Made No Sense
- URL: https://www.youtube.com/watch?v=8vDuIVlfeV0
- Topic: Interfaces + Open Design
- My current learning frame: Reproduce the investigation on your own Apple Silicon Mac: run Apple FM Bench (or the FM CLI directly), then apply the engine-contention technique to confirm decode sits on the Neural Engine and prompt processing on the GPU, and decide whether tools like MLX that do scale with GPU are a better fit for your hardware.
- Why this matters: New playlist item from Alex Ziskind; queued for transcript-backed review, topic mapping, and a practical learning artifact.
Transcript anchors from this exact video:
- 0:36 / Evidence 1: "context. A lot of helpful AI these days still depends on you noticing the issue first, then writing the prompt, then managing and following up. Even when the tools are good, they still assume the right person catches..."
- 3:00 / Evidence 2: "little built-in model was doing 600,000 tokens a second. That's for prompt processing and decode was 22,000. Come on, really? That's definitely not a benchmark. That's a typo. Here's the thing. It's not really the tool's fault. Apple..."
- 5:00 / Evidence 3: "decode stage. It also has a pretty nice little U GPU inside for processing. That's the prompt processing stage. By the way, if you don't know what I'm talking about, I explain this in a bunch of other..."
- 7:41 / Evidence 4: "something bugged me. The numbers just didn't feel right. So instead of testing FM directly, I tested the tools. I built a little workload that has to use the neural engine. If you use CoreML model and you..."
- 9:25 / Evidence 5: "the neural engine and its prompt processing runs on the GPU. So that's why the studio couldn't beat the Mini. Decode is on the neural engine. And that's basically the same exact chip across the M3 and the..."
- 11:03 / Evidence 6: "bigger and chunkier models running on Apple Silicon, watch this video here. Thanks for watching and I'll see you next time."
Your task:
1. Use the transcript anchors above as the primary source packet. If you add outside context, label it clearly as outside context and keep it secondary.
2. Create a source-check table with columns: timestamp, claim, what the demo proves, confidence, and what still needs verification.
3. Extract the actual teachable claims from the video. Do not invent claims that are not supported by the title, lesson frame, or transcript anchors.
4. Build a reusable learning artifact: A UI critique sheet for judging whether an AI interface improves control.
5. Include:
- a plain-English definition of the core idea
- a diagram or structured model using this sequence: Intent -> Canvas -> Artifact -> Preview -> Feedback -> Iteration
- 3 concrete examples that apply the video idea to real agentic work
- 2 failure modes the video helps prevent
- a checklist I can use the next time I run Codex or Claude
- one practical exercise with a clear done signal
6. Add a "learning transfer" section: what changes in my workflow tomorrow if I actually learned this?
7. Add a "source check" section that cites which transcript anchor supports each major takeaway.
Quality bar:
- Make this specific to "I Tested Apple’s Secret macOS AI… The Results Made No Sense", not a generic Interfaces + Open Design essay.
- Prefer operational examples, failure modes, and reusable artifacts over broad definitions.
- Call out uncertainty instead of smoothing over weak evidence.
- If evidence is weak, say what transcript segment or timestamp needs review instead of guessing.
- Finish with a concise artifact I could paste into my learning app.
Misconceptions
What to stop believing.
A beautiful page is automatically a good learning tool.
Learning requires sequence, active recall, feedback, and application.
Generated UI should be accepted as-is.
Generated UI needs critique, revision, and browser verification.
Practice studio
Learning only counts when you make something.
01
Transcript evidence map
Separate what the video actually says from what you already believe about the topic.
3 source-backed takeaways with timestamps, confidence, and a transfer note.02
One useful artifact
Apply the video to a real workflow and produce a ui critique sheet for judging whether an ai interface improves control..
A reusable artifact with a done signal and one verification step.03
Teach-back card
Explain the lesson to someone who has not watched the video yet.
A 90-second explanation, one diagram, one example, and one misconception to avoid.
Recall check
Answer first, then reveal — without rewatching.
What is FM in the macOS 27 Golden Gate beta, and what are its two ways of running?
Why did the off-the-shelf benchmark report FM doing 600,000 tokens per second?
How did the contention test prove which engines FM uses, without trusting any meters?
Source shelf
Use the video as a doorway, then verify with primary sources.
