An agentic playbook for production-scale performance fixes: How we cut p95 latency by 30x in 24 hours without new infra

Software development is shifting from AI-assisted coding to agentic engineering. Instead of prompting AI for each step, we now set goals and let agents execute autonomously. 

The idea is for agents to handle execution with minimal intervention. 

Like everyone else on the spectrum, we were skeptical of the hype too, until a production crisis made the argument for us. When one of our projects started melting down in production and debugging kept hitting walls, we stopped asking the agent what to do next and handed it the wheel entirely. What happened over the next 24 hours changed how we think about engineering work.

The incident: Production scale broke what staging couldn’t see 

There's a particular kind of dread that sets in when you realize your system isn't broken, it just can't handle the people actually using it.

In January 2026, our enterprise AI platform crossed a usage threshold where performance degraded sharply, even though nothing had changed in deployment topology. The staging and production reality were different: 

When performance became a revenue problem 

This technical delta spiraled into a business-critical emergency. Users reported the app "freezing" for 15–20 seconds during project switches, leading to immediate fallout:

• First-run experience (FRE) failure: The onboarding flow ground to a halt. Activation rates fell to 35% as new users bailed during project setup before ever seeing a single AI output. We were losing people at the front door.
• Accelerated active user churn: Reliability issues drove an 18% MoM churn spike. We weren't just losing new signups; our high-frequency power users (5+ sessions/week) stopped using complex workflows due to frequent timeouts. Losing these established accounts created a massive deficit, as the cost to replace a single churned customer surged to 25x the cost of maintaining one.
• Feature erosion: Users started tiptoeing around slow areas. Agent marketplace visits dropped 20% as people learned which parts of the product to avoid. The product was effectively getting smaller while we watched.
• Hyper-extended time-to-value (TTV): The lag buried our core value proposition and handed competitors a window we couldn't afford to leave open.
• Negative word-of-mouth & NPS decline: User frustration translated into poor recommendations, weakening organic growth. We were risking losing a viral growth channel.
• Unsustainable LTV: Every failed activation was a direct hit to the bottom line. With onboarding completion dropping to 35%, we were burning $10k–$50k in LTV per enterprise signup.

System architecture: High-density plugins, complex dependencies

The project was an enterprise AI agent orchestration platform used to build, deploy, and operate multi-step AI workflows at scale. It uses a high-density plugin architecture which allows for rapid feature deployment but creates complex internal dependencies.

The system is distributed across three primary Docker containers, but the internal communication is where the performance "tax" is paid:

• core_service: A single Python/Flask process running 20+ plugins simultaneously. They share the same memory heap and PostgreSQL connection pool. Communication happens via direct function calls or a local event bus.
• membership_service: A separate "gatekeeper" container for security isolation. Every request to core_service triggers a cross-container RPC call to validate JWTs and permissions.
• agent_runtime: The engine that handles heavy lifting like data indexing and background task execution, communicating back to the main API via a Redis-backed event bus.

At production scale (16,000+ projects), the pattern created three specific bottlenecks that were invisible in smaller staging environments:

• RPC amplification: A single user request doesn't just call the API; it triggers a chain of RPC calls (Main ↔ Auth ↔ Worker). At 25,000 users, the latency of these "small" hops compounds exponentially.
• Shared resource contention: Since 20+ plugins live in the same process, they compete for a limited pool of database connections and Redis threads. One slow plugin (e.g., an unoptimized social plugin) can starve the entire "core" API.
• Multi-tenant isolation overhead: To keep data separate, every query includes complex Row-Level Security (RLS) checks. At scale, these checks transformed simple indexed lookups into massive sequential table scans.

Before turning to AI, we applied the standard engineering first aid. Each hit a visibility ceiling because traditional tools lack cross-plugin context:

The usual fixes didn’t move the needle in production– brute-force debugging wasn’t just enough   

We applied the usual techniques first. None solved the problem. Here is how the traditional fixes panned out:

The moment it clicked: Scale wasn't the stress test, it was the product

After days of dead ends, we finally had to admit something uncomfortable: the architecture wasn't broken. It just had never been tested by this many people doing this many things at once.

At staging scale (≈200 projects), the /api/v1/agents endpoint behaved acceptably. Each plugin interaction, query, and serialization step looked harmless in isolation. But at production scale (≈16,000 projects), those same patterns compounded into multi-second latency:

• Permission checks began scanning large membership tables instead of small ones.
• Tag lookups triggered repeated database queries for every agent in the response.
• Plugins called each other inside loops where a single batch would have sufficed.
• Database connections became scarce as concurrent requests increased.
• Response building slowed as object graphs grew larger.
• Large object graphs made serialization expensive.

The frustrating part? This wasn't classic microservices overhead. Most plugins were running inside the same process: core_service. There was barely any network latency to blame. The culprit was instead how data access patterns and coordination logic wasn’t simply written with this kind of density in mind.

Moving beyond dashboards to improve visibility

Once we understood the real problem, the plugin architecture that had caused the headache also handed us a way out. Shared memory and cross-plugin caching meant we could fix things systemically rather than playing whack-a-mole with individual symptoms.

But we couldn't get there with humans squinting at Grafana panels. Now, we needed a system capable of autonomous production-scale analysis:

1. Test at production scale: Simulate 16,000 projects and real user journeys using k6, not synthetic staging data.
2. Analyze traces narratively: Use OpenTelemetry and Jaeger to scan thousands of request traces at once and surface architectural patterns, not just flag the one slow query everyone already noticed.
3. Automate root cause analysis: Map slow spans directly to the exact lines causing them, in seconds rather than hours of archaeological digging.
4. Autonomous verification: A closed-loop system that re-tests every fix against the P95 target before a human ever sees a Pull Request.

The 24-hour timeline: Building an agent orchestrated workflow

We built an agent-orchestrated workflow with minimal human intervention. The architecture used: agent loops, state tracking, goal-oriented prompts, and specialized sub-agents.

Hour 0–4: Instantiate the environment 

The agent's first task was to build a production-faithful sandbox. Performance fixes are useless if they aren't tested against real-world data density.

1. Data restoration: Restored an anonymized dump of 16,000 projects and 25,000 users.

2. Workload simulation: Built k6 scripts based on real user journeys (Project switching, agent browsing, marketplace filtering).

3. Instrumentation: Wired OpenTelemetry and Jaeger into the request path to capture every span.

Hour 4–5: Baseline (Agent sets the starting line)

The agent runs the first baseline load test and extracts the slowest endpoint: /api/v1/agents: avg ≈ 2,847ms, P95 ≈ 3,210ms (slowest). 

From here on, the agent goes into autonomous loop mode.

Hour 5–16: Active optimization via autonomous agents  

Although production serves ~25,000 monthly users, peak concurrency rarely exceeds 50–200. We optimized for 50 concurrent users to reflect real peak pressure rather than synthetic stress.

The system used multiple sub-agents by design:

• Investigation and implementation were strictly separated.
• A read-only RCA agent analyzed traces and code paths.
• An Autofixer agent implemented changes only after evidence-backed handoff.
• This separation enforced safety, prevented speculative fixes, and made the architecture reusable across performance tuning, bug fixing, and feature work.

Here is how the agent architecture looked like:

The agent loop includes: 

Step 1: Parse baseline results

Agent: read_k6_results("baseline-results.json")

Result: /api/v1/agents avg=2,847ms, p95=3,210ms (slowest)
Decision: focus on /api/v1/agents

Step 2: Query Jaeger to fetch slow traces

Agent: query_jaeger_traces

Result: 18 traces > 2s

Sample trace IDs:
- 7f3c8a9b2d1e4f5a
- 9e2d1c4b5a6f7g8h
- 1a2b3c4d5e6f7g8h

Decision: analyze first 5 traces for dominant pattern

Step 3: Delegate to RCA investigator sub-agent

Main agent delegates to autonomous-rca-investigator.

RCA Investigator: fetch_trace_details("7f3c8a9b2d1e4f5a")

Pattern detected: N+1 query
- 50 agents × 50 tag queries
- ~1,400ms wasted on query overhead

RCA Investigator: search_codebase("def list_agents")
File: agents.py:145

RCA conclusion:

• Root cause: N+1 query (51 queries instead of 1)
• Fix: Use SQLAlchemy joinedload() for eager loading
• Expected impact: 2,847ms → ~1,400ms (2x faster)
• Evidence: Trace 7f3c8a9b2d1e4f5a, agents.py:145

Step 4: Delegate fix implementation (Autofixer sub-agent)

The main agent delegates implementation to the rca-autofixer sub-agent. Autofixer actions:

• Adds joinedload(Agent.tags) to the list_agents() query
• Creates a pull request with:
  • rationale
  • reference to the RCA
  • expected performance improvement metrics

Step 5: Verification (main agent)

The main agent merges the PR into a test branch and re-runs the same k6 load test. After fix:

• /api/v1/agents: avg 2,847ms → 701ms
• P95 latency: 3,210ms → 889ms
• Error rate: remains 0.00%

Trace verification confirms:

• Query count reduced from 51 → 1
• Database query time reduced from ~1,400ms → ~185ms

The agent then runs unit and integration tests. No regressions are detected. A human reviewer inspects the PR and approves the merge.

Result: The agent proceeds immediately to the next bottleneck.

Step 6: Human review and approval

Using the same loop and delegation, the agent fixes remaining high-impact issues, such as:

• permission checks scanning bloated membership tables
• missing indexes on frequently-filtered keys (e.g., agent_id)
• loop-driven cross-plugin calls that should have been batched
• over-serialization (100+ fields × 50 agents)
• redundant permission checks repeated across plugins

Each fix is handled as: baseline → traces → pattern classification → code location → fix proposal → approval → PR → verification.

This matters because you rarely have “one bottleneck.” You have a chain, and the agent keeps walking it.

Hour 17–24: Regression and stabilization

Once the endpoint is within target range (or close enough to make remaining work incremental), the agent shifts to stabilization:

• reruns the full load test suite
• validates error rates and throughput
• reruns regression tests
• spot-checks traces to ensure no hidden query fan-out returned
• documents “before vs after” metrics per fix

The patch notes: 5 systemic architectural fixes that drive real gains

The agent identified and resolved five systemic issues that were invisible in staging but catastrophic at production scale.

1. Consolidation of RPC hops

Our "clean" multi-tenant isolation created an RPC Amplification effect. At 16,000 projects, a single user request triggered a chain of 5–10 synchronous service-to-service calls. Jaeger traces showed 3,200ms requests dominated by RPC hops to Auth and Permission services.

Example trace:

The Fix: Consolidated core operations into single service calls and implemented request-scoped caching for authentication.

Result: Reduced cross-service calls by 60%. Latency: 3,200ms → 450ms.

2. Eliminating the N+1 query loop

A classic loop pattern was fetching related tags individually for every agent, causing 51 round-trips to the database for a single list. RCA Agent identified 50 identical SELECT tags spans within a single request trace.

The Fix: Switched from lazy loading to joinedload. This forced the database to handle the relationship via a single LEFT OUTER JOIN instead of 50 separate round-trips.

Result: 15x Speedup (2,800ms → 185ms).

3. Missing database indexes

Large table scans were occurring on foreign key columns and common WHERE clauses, specifically in project-switching flows. The agent ran EXPLAIN ANALYZE on production-scale data and identified Seq Scans on critical columns like agent_id and project_id.

The Fix: Automated creation of targeted indexes on agent_id, user_id, and project_id.

Result: 4.4x faster project switching (1,410ms → 320ms).

4. Decoupling activity logging 

A "read-only" browse operation was triggering 3,000 synchronous INSERT operations. The database was choking on its own audit trail. Metrics showed an 80% write-to-read ratio during simple navigation tests.

The Fix: Moved logging to an async worker that batches in-memory logs and flushes every 5 seconds.

Result: 80% reduction in database write volume.

5. Over-serialization and permission spam 

API responses were bloated with 100+ fields, while the same permission check was being repeated 40+ times in a single request. Span analysis showed serialization taking 300ms+ and redundant auth spans cluttering the trace.

The Fix: Implemented selective Pydantic model_dump for payloads and added a 5-minute Redis cache for permission results.

A simplified example of the change applied to response serialization:

The agent-driven workflow delivered measurable, production-scale results:

After the optimization, we recovered 25% of returning users, and the average session duration increased from 18 minutes to 35 minutes.

The agents also eliminate all 5 major bottlenecks: N+1 queries, missing indexes, over-serialization, redundant permission checks, cross-plugin loops. Zero regressions introduced, validated via load tests, traces, and test suites.

Total time investment:

• 1. Agent autonomous work: 16 hours

• 2. Human review and approval: 2 hours

• 3. Regression testing: 6 hours

• Total: 24 hours

Human effort required:

• 1. Review 5 pull requests: ~30 minutes

• 2. Approve changes: ~15 minutes

• 3. Monitor regression tests: ~45 minutes

• Total human time: ~90 minutes

Why the approach works when everything else failed

Traditional performance optimization is a very human process, and that's exactly the problem. You run a test, dig through traces, form a hypothesis, write a fix, deploy it, and start over.

At production scale, this fails because humans tire quickly, miss large-scale patterns, lose context across tools, and spend hours per iteration.

And there's a very human tendency to declare victory after the first win and move on, even when four more bottlenecks are quietly waiting.

With an autonomous performance agent, the workflow shifts fundamentally with:

• Automated load testing: Simulates production volumes (e.g., 16,000 projects) to trigger scale mutations that staging environments miss.
• Pattern detection: Analyzes thousands of traces in seconds to identify aggregate architectural failures (like redundant permission checks) rather than just "slow queries."
• Code correlation: Maps request spans directly to source code and query plans.
• Evidence-based fixes: Proposes batching, eager loading, or indexing backed by trace IDs and quantified ROI.
• Autonomous iteration: Immediately attacks the next bottleneck once the first is cleared until the target (e.g., P95 < 500ms) is met.
• Zero context switching: The agent maintains a continuous, unified context between load test results, trace data, source code, and query plans without the cognitive load of switching tools.

Human expertise: The mandatory foundation of agentic engineering

Before you get excited about agentic engineering, understand that:

Agentic engineering amplifies expertise. It does not replace it.

When the agent analyzed those 50 tag queries and suggested joinedload(), it wasn't AI being brilliant. It was using knowledge that a senior engineer had already added to its knowledge base.

What the agent actually knew

Architect-level understanding:

• N+1 query patterns and SQLAlchemy eager loading
• Our 3-container plugin architecture
• Database schema (agents, tags, relationships)
• Multi-tenant permission flow

Performance engineering patterns:

• What N+1 looks like in traces (50 identical queries)
• When to use joinedload() vs caching vs indexes
• How to read EXPLAIN ANALYZE output

Framework expertise:

• SQLAlchemy relationship loading strategies
• Flask request lifecycle
• Pydantic serialization optimization

The agent didn’t come up with these solutions on its own. It applied patterns that engineers already know work.

What happens without human expertise?

Same problem. But one agent guesses, the other recognizes the pattern and applies the right solution because the knowledge was already encoded by experienced engineers.

The mandatory prerequisites for agentic engineering to work

To replicate our 24-hour success, you need:

Senior engineer who:

• Understands distributed systems performance
• Knows framework optimization patterns
• Can validate fixes are correct
• Encodes knowledge into agent prompts/tools

Documented architecture:

• How services communicate
• Database schema and relationships
• Framework patterns used
• Scale characteristics (25K users, 16K projects)

Expert-designed tools:

• Not generic "find slow queries"—trace correlation + code mapping
• Not generic "suggest fixes"—pattern recognition + proven solutions

Without these: Agent tries random "performance improvements" for 24 hours.

With these: Agents systematically apply expertise at scale like we did.

Agentic engineering is powerful. But expertise is the foundation.

Technical stack references

• Load Testing: k6 - Open-source load testing tool

• Instrumentation: OpenTelemetry - Vendor-neutral observability framework

• Distributed Tracing: Jaeger - Open-source tracing platform

AI Analysis: Claude | OpenAI GPT-4