← All posts

Build Log

How My Agent System Works (And Where It's Different)

A plain-language explanation of the five-agent system I built for a recent project: how it compares to other AI agent tools, where it is genuinely better, where it is not, and why it matters for product design work.

Jay TrainerJuly 15, 202612 min read
Agent ArchitectureMulti-Agent SystemsProduct DesignModel Mixing

The Elevator Pitch

Most AI agent systems are like hiring a team of interns who are fast, tireless, and forget everything the next day. My system is like having a team of five specialists who keep journals, write memos to each other, argue when they disagree, and have a night janitor who keeps the office clean.

The one-liner: Other AI agent systems give you speed. Mine gives me judgment — a team with memory, independent perspectives, and the ability to argue with each other before I commit to a decision.

How Other Agent Systems Work

Hermes Agent

Nous Research's open-source autonomous agent. It lives on your server, keeps bounded, curated memory in MEMORY.md and USER.md files that persist across sessions, runs scheduled tasks from natural language or cron expressions, and writes its own reusable skills from experience. It can delegate work to child subagents with isolated context and restricted toolsets. Its strength is a persistent, self-improving assistant: the longer it runs, the more capable it gets.

OpenClaw

Peter Steinberger's open-source personal assistant — the most widely adopted self-hosted agent framework going. It's a local-first gateway that connects any model to your real channels (WhatsApp, Telegram, Slack, Discord, iMessage, and roughly twenty more), stores its personality and memory as markdown files in a workspace, runs cron jobs and webhooks, and messages you proactively instead of waiting to be asked. Its strength is reach: one assistant, deeply wired into your actual life.

Agent Zero

An autonomous agent framework running in a Docker sandbox. Give it a goal and it decides which tools to use — code execution, browsing, even a full Linux desktop — then loops until the goal is met. Every agent can spawn subordinate agents that keep their own focused contexts and report back up the chain, and its hybrid memory sorts what it learns into facts, solutions, and behavioral adjustments. Its strength is autonomy: give it a goal and walk away.

What they all have in common

  • Speed and autonomy — they can work while you're away
  • Action-first — they execute code, browse, and touch the real world
  • Memory as a profile — persistent markdown or vector memory about you and your preferences, not an evidence base with quality standards
  • One cognitive signature — subagents, workspaces, and subordinates are copies of the same configured brain
  • No opinions — they don't disagree with themselves or evaluate evidence quality
  • Delegation, not debate — a parent hands tasks down and gets reports back; structured peer disagreement isn't a concept

How My System Works

I have five named agents — a researcher (Maren), a strategist (Reid), a designer (Ava), a project manager (Kai), and an engineer (Syd). Each one has:

  • A persistent memory file that carries across every conversation
  • An inbox where other agents leave structured messages for each other
  • A definition file that gives them a specific lens, expertise, and responsibilities
  • Access to a shared evidence base — 108 research signals, 44 design decisions, 120+ cross-linked files in an Obsidian vault

What happens when I ask for something

Single-agent commands (research, design, strategy work): One AI instance loads that agent's definition, state file, and inbox. It thinks through that specific lens — "what would a researcher notice here?" — and produces domain-specific output. This is one brain with a strong persona, not an independent thinker. But it has deep context: the researcher "remembers" every interview, every signal, every open research question from previous sessions.

Multi-agent commands (status updates, scope checks, risk assessments, sprint planning): The system spawns each agent as a separate AI instance running on a different model. They literally cannot see what the others said. Each gets only their own agent definition, state file, and inbox — nothing from the other agents. A more powerful AI model (Opus) then merges their independent reports and explicitly tells me where they disagree.

This is not one mind wearing hats. It's genuinely separate instances with cognitive diversity — the reporting model (Sonnet) notices different things, weights evidence differently, and phrases concerns differently than the synthesis model (Opus). When Opus reconciles their outputs, tensions surface that a single brain would smooth over.

High-stakes decisions: When a design decision moves from "proposed" to "accepted," the system automatically spawns two AI agents: an Advocate (argues FOR the decision) and a Challenger (finds weaknesses). Both run on the most powerful model (Opus). The Challenger gets extra context — emerging signals, the disagreement log, unanswered research questions — to level the playing field against confirmation bias. I see both arguments before I make the call.

Side-by-Side Architecture Comparison

My SystemHermes AgentOpenClawAgent Zero
Agents5 named specialists with distinct expertiseOne assistant plus child subagents with isolated context and restricted toolsOne assistant per workspace, routed across chat channelsSuperior agent spawning subordinate agents per task
MemoryPersistent markdown files in git-tracked vaultMEMORY.md + USER.md across sessions; pluggable backends (Mem0, Honcho)Markdown workspace files (MEMORY.md, SOUL.md, skills) with active memory recallHybrid vector memory of facts, solutions, and behaviors; project-scoped
CommunicationStructured inbox messages between agentsParent-to-child task delegationChannel routing to agents; no agent-to-agent messagingTasks down, reports up the hierarchy
IndependenceSeparate model instances for multi-agent workSubagents run under the same configured brainOne brain per workspaceSubordinates are copies of the same brain
DisagreementFirst-class feature — logged, surfaced, resolved by humanNot a conceptNot a conceptTreated as inconsistency to resolve
Evidence gatesDecisions must cite confirmed signals of sufficient strengthNoneNoneNone
AutonomyHuman-in-the-loop (commands), plus overnight maintenanceScheduled tasks via natural language or cron; background jobsAlways-on daemon with cron, webhooks, and proactive messagesFully autonomous loops with human override
OutputKnowledge artifacts (notes, decisions, signals)Text, voice, and images; terminal, files, and self-written skillsReal-world actions across chat apps, browser automation, code executionCode execution, browsing, full Linux desktop control
PersistenceEverything survives in git foreverLocal files on your server, across sessionsLocal-first workspace plus daemon stateDocker volumes and workspace snapshots
Model portabilityIntelligence is in files, not the modelProvider-agnostic routing with fallbacks, local models includedModel-agnostic gateway with multi-model failoverMulti-provider, integrations expanding

Where It's Genuinely Better

1. Cognitive diversity from model mixing

This is the thing no other system does. My parallel agents run on Sonnet. The synthesizer runs on Opus. The adversarial reviewers run on Opus. These aren't just different instances of the same model — they're different models with different reasoning profiles. Sonnet notices different things than Opus. It weights evidence differently. It phrases concerns differently.

When Opus merges four Sonnet perspectives, the result is richer than any single model would produce. Tensions and disagreements emerge naturally because the models genuinely think differently — not because I programmed them to disagree.

Hermes subagents, OpenClaw workspaces, and Agent Zero subordinates all run copies of the same configured brain. Five copies of the same model will converge on the same answer. My system gets genuinely independent perspectives from models with different cognitive signatures.

2. The vault is the brain, not the model

Every signal, decision, disagreement, and cross-reference is a markdown file tracked in git. The intelligence accumulated over weeks of work — 108 signals from real provider interviews, 44 design decisions with evidence trails, a traceability matrix linking every wireflow to every requirement — lives in the files.

If I swapped Claude for GPT or Gemini tomorrow, the vault would still work. The new model would read the same agent definitions, the same state files, the same signals, and pick up where Claude left off.

To be fair, provider-agnostic plumbing is table stakes now — Hermes and OpenClaw both run on any model, and both keep markdown memory. The difference is what's in the files. Theirs carry preferences and personality. Mine carries the accumulated evidence base of a project: signals, decisions, disagreements, and the links between them. The models are the workforce; the vault is the brain.

3. Evidence-gated decisions

A design decision can't be "accepted" in my system unless it's backed by a confirmed signal with moderate-or-better evidence strength. If I try to accept a decision backed by weak or unvalidated evidence, the system:

  1. Flags it in the Signal Gate Check table
  2. Spawns an Advocate and Challenger for adversarial review
  3. Presents both arguments before I confirm

This means my product decisions have an epistemological standard. I can trace any decision back to specific provider quotes from specific testing sessions. "Why did we choose a single list homepage?" → SIG-099, from the March 31st prototype test, where three providers validated it independently.

No other agent system has evidence quality gates. They'll happily let you make decisions based on vibes.

4. Agents that actually disagree

When my strategist says "this feature is in scope" but my researcher says "we have zero evidence supporting that," the system surfaces the conflict to me instead of quietly picking one. Disagreements are logged in a dedicated file with both positions clearly stated. They're reviewed during retrospectives for calibration.

This is architecturally impossible in systems where "agents" are clones of the same model. Identical clones converge. My system's combination of different personas, different state files, different model instances, and different context bundles produces genuine disagreement.

5. It remembers everything — permanently

My vault has been accumulating knowledge for weeks. Every interview, every meeting, every design decision, every signal — cross-linked and searchable. When I ask "what do providers think about ambient recording?" the system can cite P-01's exact reaction from the March 26th prototype test, P-06's concern about consent, and the three signals that emerged from those conversations.

The newer systems remember too — Hermes curates a MEMORY.md, OpenClaw accumulates a workspace, Agent Zero files solutions into a vector store. But those memories are profiles: who you are, what you like, what worked last time. Mine is an evidence base: a project's entire history with a quality grade on every claim and a link trail behind every decision.

6. It takes care of itself overnight

At 11pm, a maintenance agent (Haiku — cheapest model) prunes bloated files, checks for stale evidence, cleans agent inboxes, and flags overdue items. At 6:57am, a reporting agent (Sonnet) writes my morning briefing with overnight changes, today's due items, and review queue status.

Scheduled tasks aren't unique anymore — Hermes runs cron jobs and OpenClaw's daemon never sleeps. What's different is what my night shift does: it maintains knowledge quality. Pruning, staleness checks, inbox hygiene — housekeeping for the evidence base, not just scheduled prompts. I open my laptop and the system already knows what I need to focus on.

7. Structured async communication

Agents leave structured messages for each other: type (signal, decision, request, alert, question), priority, content, action needed, related files. When Maren discovers a new signal, she writes to Reid's and Ava's inboxes with exactly what they need to know and what they should do about it.

This is like a well-run team's Slack — except the messages are structured, prioritized, and automatically escalated if they sit unread for 3 days. Hermes and Agent Zero have delegation — a parent hands a task down and gets a report back. That's a hierarchy, not a conversation. Peer-to-peer structured messages between specialists with different jobs is a different thing, and I haven't seen it anywhere else.

Where It's Not As Great (Honest Limitations)

1. It doesn't do creative work autonomously

OpenClaw can handle my messages while I get coffee. Agent Zero can research a topic, write code, test it, and iterate — all without me. My agents only think when I type a command. The overnight maintenance helps, but it's cleanup work, not creative work. For anything that requires judgment — design decisions, scope calls, research synthesis — I'm the bottleneck.

This is partly by design. Product decisions require human judgment, and an AI that autonomously makes scope calls or design decisions is dangerous, not helpful. But it means I can't say "go design the Patient 360 page" and walk away.

2. Single-agent commands are still one brain with a persona

When I run /debrief or /synthesize or /critique, it's one AI instance reading one agent's definition and thinking through that lens. It's a well-informed persona with deep context — not an independent thinker. The genuine multi-model independence only kicks in for commands that explicitly spawn parallel instances.

About 60% of my daily commands are single-agent. The multi-agent magic is real but it's reserved for the commands where multiple perspectives matter most.

3. It can't take action in the real world

Agent Zero can browse the web, execute code, and drive a full Linux desktop. OpenClaw can act across every chat app I use. Hermes can run jobs on a schedule and ship the results. My agents produce knowledge artifacts — notes, decisions, analysis, signals — not working software. They inform decisions; they don't ship features.

(They do integrate with tools that take action — Figma for design, Vercel for deployment, the prototype codebase — but the agents themselves are thinkers, not builders.)

4. It's a cockpit, not a package

This isn't something you install with npm install. It's a custom-built knowledge architecture: 10+ protocol documents, 5 agent definitions with distinct personas, 15 templates, 2 scheduled tasks, a frontmatter schema enforced across 120+ files, a signal lifecycle with maturity gates, an inbox system for async agent communication, and cross-linking conventions that make the Obsidian graph view actually useful.

Other agent systems are "install package, run command." Mine took weeks to build and refine. It's powerful once it's built — but it's not turnkey, and it requires ongoing maintenance (which is why the night shift exists).

5. The quality depends on the operator

The system is only as good as what I put into it. If I skip /debrief after an interview, the signals don't get captured. If I don't run /challenge on a shaky decision, the adversarial review doesn't happen. If I ignore the morning briefing's stale signal warnings, evidence quality degrades silently.

Autonomous systems don't have this problem — they just run. Mine requires discipline. The scheduled tasks help (night shift catches what I miss), but the system fundamentally assumes an engaged human operator.

The Model Stack

Not every task needs the most expensive AI model. The system routes work to the right model for the job:

TaskModelWhy
Individual agent reportingSonnetFast, cheaper, good enough for "read your state and report"
Synthesizing multiple agent outputsOpusDetecting contradictions and merging perspectives requires deep reasoning
Adversarial cross-validationOpusBoth Advocate and Challenger need the strongest reasoning available
Overnight maintenanceHaikuChecking file sizes and dates is mechanical — cheapest model works
Morning briefing generationSonnetFormatting a summary from existing data is moderate complexity
Single-agent commandsOpusJay-facing work where reasoning quality directly affects output quality

The principle: Haiku does housekeeping. Sonnet does reporting. Opus does thinking.

The result: I only pay for the most powerful model when genuine reasoning is needed. The cost per multi-agent command dropped ~8x compared to running everything on Opus. And the model mixing itself is a feature — different models reasoning differently produces richer, more diverse outputs than any single model would.

What Makes It Genuinely Novel

Most "multi-agent" AI systems solve the problem of parallel execution — how do I get multiple LLMs working on different parts of a coding task simultaneously?

My system solves a completely different problem: how do I get AI to think from multiple disciplinary perspectives, maintain intellectual honesty between them, and hold decisions to evidence standards?

The things that make it unique, taken together:

  1. Named agents with persistent memory — not clones, not sessions, not threads
  2. Genuinely separate model instances with cognitive diversity from model mixing
  3. Structured async communication between agents via inboxes
  4. Evidence-gated decision making with signal maturity requirements
  5. Adversarial cross-validation that stress-tests decisions before they're accepted
  6. Overnight self-maintenance that prevents knowledge degradation
  7. Model portability — the intelligence is in the vault, not the model
  8. Disagreement as a feature — tensions are surfaced, not smoothed over

No individual piece is unprecedented. But the combination — applied to product design work instead of coding — is something I haven't seen anywhere else.

Mission Control Team Board showing five named agents, in-progress work, decisions awaiting a human tiebreaker, and completed overnight maintenance.
Mission Control: the Team Board view of the five-agent system, with agent statuses, inbox items awaiting a human decision, work in progress, and the overnight maintenance trail.