Designing Autonomous AI Agents: How We Built a Multi-Agent Swarm for UI Development 

Autonomous AI agents are having a moment, and unlike most hype cycles, this one is actually earned. These autonomous AI systems now plan, execute, and iterate with minimal human intervention and increasingly, they do it as a coordinated swarm of specialized agents rather than a single model working in isolation.

Anthropic and Cursor gave us examples of such autonomous agents at full tilt: a team of agents running autonomously for days built a C compiler, a DAW music production program, and a browser. But these are one-off exercises on blank canvases and rarely how they’re used.

In most real-world setups, the swarm doesn’t decide the path. Humans still design the execution path: what to build, in what order, which agent handles what. Agents execute fragments and at every stage — spec creation, planning, task breakdown, validation — developers get pulled back into coordination work instead of actual development. 

Since most real-world work is repetitive by nature: new pages, new modules, new APIs, same patterns, this overhead compounds constantly. Spec-driven development was supposed to help, but in practice it just moves the bottleneck, now someone has to review AI-generated specs and plans before anything gets built. It's exhausting. Most developers don't enjoy it, and honestly, most don't do it consistently.

We wanted the swarm to run autonomously for hours, with developers stepping in only to write and review code, not read AI-generated task lists or validate specifications. Production-quality modules generated autonomously, and developer work returned to developers.

Getting there took six weeks of building and breaking an agentic AI workflow from the ground up which isn't in any of the published playbooks. Here's how we built it:

TL;DR: What we learned from our 6 week setup

• Getting agents to run is easy. Getting them to work reliably is where things break. They skip steps, overlap responsibilities, and invent solutions you never asked for, all with full confidence. You can tighten prompts, add guardrails, and increase context but it doesn’t solve the core issue, which is control and clarity across the system.
• Current models can't see the visual differences that actually matter in UI: a 4px font deviation, a gray that's one shade off. They'll mark it as done. We had to build dedicated visual QA agents and explicitly teach them what to look for, how to compare, and what "wrong" means in pixel terms. The model won't catch it on its own.
• Around 95% of a feature can be generated without human involvement. The final 5% still requires a developer. Unless the feature is trivial or the quality bar is low, full autonomy isn’t realistic yet. This only works if the handoff is clean. A developer should be able to pick up the generated code and finish it, not rewrite it from scratch. And “finish” here means actual engineering work: Validating logic, handling edge cases, tightening the implementation, and polishing for production. Not the boring man-in-the-middle work. The developer is back to being a developer.

Scoping the autonomous AI agent experiment with real codebase, real constraints 

We used our production AI portal as a testing ground: thousands of active users, real codebase, real constraints. We scoped the experiment to UI, with a minor footprint of BFF logic. Mostly React— a bounded domain, still hard enough for 6 weeks of full-time work with a 3-person R&D team.

The input mirrored what we would give a real developer. Figma assets and a minimal description of functionality. No polished specifications created just for this experiment. Before we started, we made three decisions: 

• No humans in the loop during execution. The swarm runs end-to-end and produces output without intervention.
• Results must be reproducible. This isn’t a one-off demo that only works under ideal conditions.
• No platform building. We are not building an agentic framework from scratch. Tools like GitHub Copilot, Cursor, and Claude Code already handle orchestration, MCP integration, context management, and sub-agent workflows. We picked one and built on top of it. 

Success criteria: Functionally correct, pixel-perfect to the Figma design, production-quality code. 

That was the target. Where we actually landed – and why – is what this post is about.

Stage 1: Agent orchestration AI— Getting long-running autonomous coding agents to run reliably

We started by generating a basic agent team using an LLM: an orchestrator, a coder, and a tester. The setup was minimal. A simple loop — plan, code, test — with the orchestrator re-running steps as needed.

The first run looked promising. Within an hour, we had a functional feature. This is what modern agentic platforms give you out of the box: a working multi-agent system with zero infrastructure effort.

But that’s where the illusion of progress ends.

The UI looked similar to the design, but not production quality. Wrong fonts, incorrect icons, misplaced components, boxes and lines that weren't on the design. Good for one hour of work – but that's point zero. We started looking at why.

Output review: Figma Design vs Agent-generated component

Agents improvised and behaved in ways we never intended:

• The orchestrator wrote code instead of delegating, roughly once in three runs. Its prompt said "coordinate the implementation," and coding was the fastest path to that goal.
• The orchestrator skipped a subagent entirely and marked the task complete. The subagent needed terminal access to run the application. The orchestrator didn't have terminal access itself so it decided the subagent didn't either. It projected its own constraints onto agents with different capabilities.
• A tester that couldn't reach the running app quietly pivoted to reading source code and declared tests passed "as a proxy." The prompt said "verify the output." Reading code is technically a form of verification.
• Tool selection was inconsistent. An agent with access to an implement-design skill will sometimes use it, sometimes improvise its own approach. You can't rely on an LLM to choose the right skill at the right moment, explicit skill references in agent prompts are required.

Challenges encountered in running the autonomous AI agent

The deeper problem was consistency. Because agents fill gaps differently each time, same spec and same Figma produced a different result on every run. One run nailed icons and missed typography.

The next got typography right and invented wrapper elements. Stability was well below 95%. Better prompts help but they don't fix this. If you're designing autonomous AI agents for production runs, you're designing for variance.

The fix: Improving agent design, not prompt engineering.

The agents weren't broken. They find ambiguities you didn't know existed and resolve them with confident, plausible, wrong decisions. Adding prompt constraints fixes each specific failure, but the next run invents a different creative interpretation.

It doesn't scale. This is why agent design is its own discipline, and why it's foundational to any LLM agent workflow built for reliability. The single most impactful decision: design each agent around a single responsibility. An agent with one job does that job. An agent with two jobs picks the one it prefers.

Results after two weeks: Stable multi-agent AI system, unresolved visual fidelity

The two weeks we spent here were not glamorous, but they're what turned unpredictable agents into mostly-predictable ones. Our "predictable" agent swarm runs 1-2 hours. Agents follow instructions, stay in their roles, produce consistent-ish output 90% of the time.

What we didn't solve at this stage: visual fidelity was low, visual accuracy unreliable, reproducibility still shaky.

Stage 2: Pixel-perfect design matching — The hardest problem in autonomous AI agents for UI development

After Stage 1 we had agents that work reliably, but the output was around 80% of what we needed. The code worked, the pages rendered, but not close enough to the target design. So the next question was simple: how do we close that gap?

Our main bottleneck was visual testing. Even the most advanced models couldn't reliably spot the differences that matter: a font size of 4 pixels, an icon in dark gray instead of black, padding that's close but not right. A human sees it immediately.

The model doesn't. It sees "a card with a header and an icon" and calls it a match. If the system can't see the gap, it can't close it. From our runs: a back-arrow icon rendered in blue instead of black. We ran multiple automated fix cycles. The agents couldn't detect the difference.

What finally made pixel-perfect UI generation work in our agentic AI workflow

Getting to 80% took a day. Getting to 95% took weeks. The remaining 5% may not be reachable with the current approach at all. Plenty of tools improve how agents generate code but we couldn't find anything that would reliably tell "the padding is 2px but should be 4."

Getting to 95% required a specific combination:

• Native Figma MCP with a DEV seat: It provides design elements as React components with Tailwind classes — implementation-ready code specs, not raw CSS measurements for the agent to interpret, not just screenshots.
• Design decomposer agent: Breaks the Figma design into component-level implementation tasks. Instead of "build this page," the coder gets "build this specific card component with these exact styles."
• Implement-design skill from Figma: A specialized skill guiding agents through pixel-perfect implementation from MCP data. Without it, agents approximate values and ignore edge cases.
• Dedicated visual tester agent: Visual comparison by screenshots alone doesn't work. A full adversarial agent that extracts actual CSS values and compares them against Figma specs numerically.

Adversarial framing is critical — you have to explicitly tell it "find at least five specific issues" or it'll rubber-stamp anything that roughly resembles the target. This is one of the agents where we had to specify both the WHAT and the HOW:

Remove any one of these, and the output quality drops noticeably. Without adversarial framing, testers default to optimism — in early runs, our tester reported "all good" while eight visual mismatches were visible on screen.

Results: Near-production visual fidelity, but code quality still blocking shipment

Visual quality of generated pages reached near-production level — 95% fidelity to the Figma design. Average run: 2–3 hours. What we didn't solve: code quality. Every effort so far went into visual fidelity — and it showed. The output looked right but no developer could reasonably own it. Working, passing tests, still unable to ship. The code underneath was no one's to maintain.

Stage 3: Code quality and engineering standards — Making AI-generated code shippable

We had the design — 95% matching what we wanted. Not all of the remaining 5% is critical; some you accept and ship. But the remaining issues that do matter need a developer to fix. And if the code is unmaintainable, the developer can’t fix anything — they’re stuck.

Code quality isn't the goal. It's the prerequisite for the 95/5 model to work at all. Without it, that 95% of visual fidelity becomes throw-away work that never delivers real value.

Building a code constitution for LLM agent workflows

Agents don't infer your project's conventions, they default to whatever patterns dominate their training data. We already had a working constitution from our spec-driven development workflow, but autonomous execution demanded stricter controls.

We significantly improved it: tighter rules, more explicit forbidden patterns, agent-specific constraints, and guidelines for which shared components and styling conventions to use. Generic React knowledge isn't enough when code must fit an existing codebase.

The constitution benefited both human and agent workflows equally: same architecture decisions, same component patterns, same file organisation, regardless of who's doing the work.

Why autonomous AI agents need adversarial code review, not self-evaluation

Self-evaluation doesn't work — same principle as Stage 2's visual tester. A separate code review agent checks every output against the constitution and project rules. If it fails, the coder fixes and resubmits. The key distinction: you're not reviewing code style; you're reviewing compliance to your standards and architecture.

⚠️ One wrong line breaks everything

Our auto-generated agents.md included a statement "use Material UI" which was not used in our solution. The agent kept trying to reuse Material UI icons on every run until someone caught the line and removed it. 

One wrong sentence propagates through every agent that references the file, on every run, silently. Your agents.md must be manually reviewed and accurate. Configuration is code, and it needs the same rigor.

The quality tax: What production-grade agentic AI workflows actually cost in runtime

Constitution, code reviewer, and test case generation improved quality significantly — but runs that took 2–3 hours in Stage 2 now take up to 5. Actual code implementation accounts for less than a third of that. The majority is review, testing, visual comparison, and fix cycles. The bottleneck isn't writing code; it's verifying it's correct. That's still cheap compared to developer time. Our answer: run overnight, review in the morning.

What developers actually receive vs agents we ended up with

After an overnight run, a developer reviews and refines — mostly code review, then visual polish and minor adjustments. A feature that would take two days of developer effort gets delivered in under a day of total human work. The remaining 5% is real, identifiable, and fixable by someone who can read the code and understand the intent.

We started with three agents: an orchestrator, a coder, and a tester. Single responsibility grew the team: the tester split into a functional tester and a visual tester. We added a code reviewer, a design decomposer, and a test case creator.

Each new agent emerged because an existing one was doing too much. 3 became 7. All project-agnostic — what makes the swarm project-specific is the constitution, the rules, and the codebase structure.

6-week results: What a production-ready agentic AI workflow actually looks like

Average run: up to 5 hours overnight. Production-quality code that reads like a human wrote it, 95% visual fidelity, a reproducible pipeline of seven agents across three phases. A developer picks up the output and finishes it.

Not solved: the last 5% is still manual, stability is not 100%, and every new project needs tuning.

Scaling the agentic AI workflow: where we are and what comes next

The orchestrator is a pure state machine driver — owns the task file, never touches code, decides what runs next based on subagent outputs. If interrupted, it re-reads the state file and resumes. The agents rely on 7 skills — implement-design, ui-component-guidelines, responsiveness-guidelines, download-figma-assets, run-lint, ui-icons-checker, component-generation.

Some existed at the start as we built them for other workflows. Others were built because an agent failed without it — a missing skill showed up as a recurring defect that no amount of prompting could fix.

We proved this works — not as a demo, but on real features. Two features tested end-to-end, both delivered through the full pipeline. We're scaling to more on the same project. A human is still required for the last 5%. That's a deliberate choice — we care about code quality and want to retain control over what ships. The more we adopt this, the closer we get to trusting AI with code quality end-to-end.

But for now, the developer-in-the-last-mile model is the right one for us. We follow a "human owns the code" rule. The shift isn't that humans disappear. It's that developers stop being asked to do things they're skipping anyway, reading AI-generated specs, plans, task breakdowns — and go back to engineering.

Can other teams use this? Yes – but it's not plug-and-play. The 7 agents and skills are project-agnostic. What's project-specific: the constitution, the rules in agents.md, and the codebase structure. Think of it as a scaffolded solution – you get the pipeline, you customize it for your conventions, tooling, and project context. 

The 6 weeks we spent were mostly on the approach itself and the learning walls we hit along the way. Adoption should take significantly less.

Read more: Where Are You on the AI Adoption Curve? The Three Stages Every Enterprise Must Pass Through

Why we're not calling our multi-agent AI system a dark factory (yet)

We're not calling this a dark factory yet, and not out of false modesty. Two things actually disqualify it.

1. The 5% part: A dark factory means no human touches the output. We're not there — a developer reviews and finishes every feature. We chose this deliberately, but it's still a human in the loop, just at the end instead of the middle.
2. We focused only on UI: A real feature isn't just a page, it's UI plus API logic plus backend-for-frontend plus database plus product management decisions. Each of those is a separate domain. Our swarm handles one of them.

The natural question is: can we just add more agents to cover more domains? The answer is both no and yes:

• No to throwing more agents into the pile. We saw what happens with seven agents in a single swarm – orchestration is already fragile. More agents in one swarm doesn't scale. It causes chaos. This one is straightforward.
• Yes to a more structured approach. Our hypothesis: specialized swarms as black boxes. A UI swarm, an API swarm, a persistence swarm — each one focused, each one tuned to its domain. The agents within each swarm stay small and single-responsibility. Then you orchestrate the swarms, not the agents. Define the black boxes, let each one do its job, integrate at the boundaries. Will it work? We don't know yet. We have one box working. The second is within reach. Composing them — autonomous execution across an entire system, not just within a bounded domain — is the next design challenge.

The industry's frontier is probing this space — every major vendor is likely running similar experiments. What gets shared publicly tends to be more marketing than scaffolding, though still impressive. What's in their labs, we don't know.

We're not there yet either, let's admit it. A human in the loop, whether in the middle or at the end, means nobody has fully solved this. But many teams are experimenting and so are we. And what's already clear is that you can build solutions that meaningfully accelerate software engineering today.

Whether you call it an agent swarm, a dark factory in progress, or just a better CI pipeline with opinions, the acceleration is real. The ceiling is just not where the demos make it look.