Qwen vs Gemma: How Local LLM Coding Agents Perform on a 15k-LOC App

Most “local LLM” tests use greenfield toy apps in Python or JavaScript, scored on whether a water simulation or Flappy Bird clone looks right. Maintaining a live application is a much harder test, especially when the model must understand an existing architecture before changing it.

For this Qwen agent review and Gemma review, we put both model families to work on a mutable 15k-LOC Flutter application. They had to navigate the existing codebase, operate the running app and complete tasks ranging from basic controls to settings changes and a complex swipe-to-seek interaction.

The result is a practical Flutter and Dart AI coding agent test, not another isolated code-generation exercise. Read on for a local vs cloud LLM coding comparison that reveals which agent delivered, where each model failed and whether the best local LLM for coding in 2026 can genuinely maintain production-style software.

Testing local LLM agents beyond greenfield toy apps

This is not a standardized coding task for AI models and local LLMs. It is a qualitative field test designed to answer a more practical question: which model behaves like the best LLM coding agent when the codebase is already live and mistakes have consequences?

• We wanted to test on a real, mutable codebase, not greenfield. A live ~15k-LOC app the models had to maintain and read existing structure, fit conventions, and not break it (several did edit the wrong files).

• Hard-for-AI language. Dart/Flutter, far less represented in training data than Python/JS. 

• End-to-end agentic, not code-gen-in-a-vacuum. The model must drive the running macOS build, exercise it, and validate behavior at runtime, and not just emit a diff.

• Every task demands a screenshot; scores come from my own manual verification, not the model's self-report which is how the "declared success, screenshot disproves it" failures got caught.

• Escalating difficulty (smoke-drive → settings + validate → timed swipe-gesture repro) gives a capability gradient that cleanly separates cloud SOTA from local.

• Purpose-built app-driving tools and skills, so a model's failure reflects the model, not a missing scaffold.

Most BAs get everything they need from stages 1 and 2. Stage 3 is optional and worth knowing about, but don't let it intimidate you out of starting.

Test environment

• Hardware: RTX 4090 (24GB), Core i5-13600K, 64GB RAM

• Local runtime: LM Studio 0.4.16 (b2) · llama.cpp CUDA 12 (Windows) v2.21.0

• Hosted: Google AI Studio (Gemma) · Azure (GPT)

• Agent harness: pi coding agent (except gpt-5.3-codex--medium, run via GitHub Copilot)

• Target: Live Flutter app, ~15k LOC Dart drive the real macOS build, validate at runtime, produce screenshot evidence, keep tests green

• Method: single run per cell (qualitative field test, not a benchmark); manual verification of every result; human nudges allowed and noted

The agentic test suite: Smoke-drives, state logic, and swipe gestures

Model results: Local LLMs vs cloud SOTA benchmarks 

Scores were evaluated as: 0–3 (FAIL / BAD / AVG / GOOD) · (s) skill-prompted · 2→3(s) plain then skill · — not run · DNF infra failure · ✅ clean pass

Tiers: cloud SOTA (1–2) → viable local (3, Qwen) → hosted Gemma, glitchy but usable (4–5) → local Gemma, hard-crashes (6–7).

Token economics (pi session counters) & hidden costs: Local vs cloud LLM consumption

Legend: ↑ input · ↓ output · R reasoning · CH cache-hit · Peak high-water context fill (window: 262k local/Google, 400k GPT) · ·s skill-prompted. ↑ is cumulative across every turn in the session; no compaction was ever used.

Cost ($) reported only for GPT: $0.117 (T1), $0.302 (T2). Local models are free; Google-hosted Gemma ran on free tier — so dollars aren't comparable, and the honest cost axis is tokens + wall-clock.

Local LLM agent performance trends

• Input dwarfs output, 30–300×. The work product (↓) is tiny and stable (1.9k–53k) regardless of success; nearly all token volume is context replayed each agentic coding turn. Qwen's looping T3 run hit ↑9.4M for just 32k out (≈294×) — the single biggest session in the test.
• Reasoning blow-up tracks failure. R explodes on the loop/FAIL runs: gemma-26b-it R7.6M (T2 FAIL) and R5.0M (T1 BAD), both at CH0.0%; gemma-31b climbed R2.3M→3.1M→3.9M across its three failed T3 retries. These disproportionately difficult agentic tasks appear to push smaller models into reasoning loops instead of moving the task forward.
• Cache-hit is the hidden cost lever — and erratic on hosted Gemma. GPT held 85.9–99.5%; gemma-31b swung 66.7–98.2%; gemma-26b sat at 0.0% twice. With no caching, every turn in these agent workflows was fully reprocessed, compounding the reasoning blow-up.
• Local counters omit R and CH entirely. LM Studio sessions report neither, so Qwen's "thinking" is invisible here and cross-stack reasoning numbers aren't apples-to-apples. This is an important limitation when comparing hosted AI models with open weight models running through different inference stacks.
• Local burns 10–30× the raw input of GPT for equal-or-worse outcomes — but at $0. GPT got GOOD on T1 with ↑86k; Qwen got AVG with ↑913k (≈11×). The local penalty is latency/wall-clock, not money. It also removes the need to transmit code through an external service or manage an API key.
• Peak context stayed moderate (14–59%) even as cumulative ↑ hit millions — no compaction was ever triggered; each turn just resends the full running transcript, so the millions are plain back-and-forth turnover across many turns, not one giant prompt.
• Token use also reveals that successful code generation is only a small part of the job. Most consumption occurs while models inspect the repository, interpret tool output, make function calls, operate the application and retry failed steps. That distinction matters when evaluating how coding agents work inside a real development workflow, rather than on isolated coding tasks.

Final Verdict: Is Qwen 3.6 or Gemma 4 THE local LLM coding agent for your needs? 

• Qwen3.6-35B-A3B is genuinely usable for hands-on coding on a 24GB consumer GPU, a real shift from "local = toy." Among the tested Qwen models, however, they still needed nudging and were not ready for long-horizon autonomy.
• The result also shows how far local LLMs and open weights have progressed. Models distributed through ecosystems such as Hugging Face can now participate in real application maintenance, although availability does not guarantee dependable agentic performance.
• Runtime stack matters as much as the weights: local llama.cpp Gemma hard-crashed; the same family on Google's API only glitched (recoverable). Neither was crash-free. A model being fine tuned for instruction following or coding does not eliminate failures introduced by the runtime, tool layer or application-driving stack.
• SLM failure mode: disproportionately hard requests trigger 10k+ token reasoning loops (reproduced across chess/arithmetic benches).
• Self-reports are unreliable. Models declared success while their own screenshots disproved it in T3. Runtime evidence must therefore remain part of the evaluation, especially when building agents expected to modify and validate live software.
• Small language models are capable but jagged: lots of human-in-the-loop. Useful for hands-on work, not autonomous swarms.