Traditional Testing Is Failing GenAI Applications: Introducing the Testing Pyramid 2.0  

If you ask an engineering team what makes GenAI applications hard to ship, they rarely point to prompt design or the growing list of models to choose from. The real friction tends to surface later, in production, when releases slow down, behavior starts to drift, and teams lose confidence in what they are about to ship. Many organizations only recognize this pattern after weeks of regression fixes even with healthy test coverage and mature QA practices.

We ran into this problem ourselves and learned quickly that building a building demo is easy, while making a GenAI system production-ready is not. At one point, we spent nearly two weeks unable to release a new version of our conversational agentic app for EPAM’s AI productivity portal. On paper, everything seemed solid. We had years of experience, established practices, and a well-structured testing pyramid.

In practice, each fix triggered a new set of regressions, forcing the team to spend more time chasing side effects than improving behavior. Bugs escaped into production, response quality drifted across releases, and the system reacted inconsistently under real user traffic. Even with unit tests covering nearly 80% of the codebase, the test suite failed to protect the parts of the system that actually determined user outcomes. 

That pattern made the root cause impossible to ignore: the issue was not a single defect or a flawed release, but a structural mismatch between traditional testing assumptions and the way GenAI systems operate in production.

Understanding that gap is the first step toward a more reliable approach. In the sections that follow, we outline what broke, why the testing pyramid no longer holds, and what a Testing Pyramid 2.0 looks like in practice.

Why GenAI applications cannot be tested like deterministic software

Traditional applications have predictable behavior. Given the same input, a function produces the same output, and teams can validate correctness by testing the code paths they control. Most testing practices, including the classic testing pyramid, assume this model.

GenAI applications break this model completely. When you integrate with GPT, Claude, or similar models, you no longer execute your core logic entirely inside your codebase. Instead, execution is outsourced to an external system that you can influence but not fully control. In practice, the model becomes your business rules engine. It decides intent classification, routing, and even final outcomes.

Take a simple prompt: "Extract user intent from the query 'what GenAI tools can I use for code reviews.' User intent could be one of the following..." 

A small change in wording can shift the model’s interpretation from a concrete task to a broad explanation, or from a specific tool recommendation to a general learning overview. Even without any prompt changes, responses can vary between runs. 

This non-determinism is the core challenge that traditional testing cannot address. It forced us to revisit our testing strategy from the ground up. That reality pushed us to rework our testing approach from its base: When business logic runs through an LLM, the premises of the testing pyramid no longer hold.

The traditional testing pyramid was not built for GenAI systems 

The traditional testing pyramid is built on a simple idea. Most confidence should come from fast, deterministic unit tests at the bottom to catch logic errors early. A few integration tests dominate the middle layer to verify that components work together, while the narrow peak contains a minimal set of end-to-end (UI) tests to simulate real user journeys. This structure has guided how software teams design test suites for years, and it formed the foundation of our own testing strategy.

When we started building GenAI features, we followed the same model. We wrote extensive unit tests and mocked all LLM calls. The system initially appeared stable, and we assumed the pyramid was doing its job.

Production proved otherwise. By mocking LLM interactions, we were not testing the most critical part of the system: the model integration where the real business logic executed.

This is like testing an e-commerce application while mocking every database call. You would miss data integrity issues, security vulnerabilities, and performance bottlenecks. That is what happened with our LLM tests. We validated fixed, deterministic responses with a specific confidence level, while production depended on probabilistic outputs that varied across inputs and runs.

When a model is responsible for interpretation and judgment, tests that bypass it stop providing meaningful assurance. The testing pyramid no longer reflects how the system actually fails.

The new testing pyramid 2.0 for GenAI apps: Integration tests at the core

We restructured the testing pyramid to match how GenAI systems actually behave:

We moved integration tests to the foundation. Every LLM interaction is tested against the real model, not a mock. Tests exercise actual API calls to validate prompts, outputs, and behavior under real conditions. These tests are written and maintained by developers because they encode application intent and failure modes.

Unit tests were also reduced significantly. They remain useful for pure functions and isolated logic, but they no longer dominate the test suite. The bulk of confidence now comes from integration coverage, not local determinism. These tests are also owned by developers.

End-to-end tests operate as black-box validations, focused on user-visible behavior: a question goes in, an answer comes out, and the result is checked against expected outcomes or ground truth. These tests are owned by QA and are not concerned with internal reasoning paths.

Some teams experiment with gray-box end-to-end testing, such as validating intermediate RAG steps or computing metrics like context recall from retrieved embeddings. We found this approach extremely time-consuming to maintain and brittle: tracking inputs, outputs, and intermediate state multiplies maintenance effort and makes tests sensitive to data drift

Integration tests are slower, more expensive to run, and require more infrastructure. But they consistently surfaced the failures users actually experienced in production.

Defining metrics for quality in GenAI systems

Traditional testing relies on binary outcomes. A test either passes or fails. GenAI systems do not behave this way. Their outputs fall on a quality spectrum, ranging from clearly wrong to acceptable, good, or excellent.

That difference forces a change in how tests are automated. 

In limited cases deterministic checks still work. When outputs are constrained to predefined values, such as intent names or enums, simple string equality is sufficient. You verify that the model returned the expected label and not a variation or hallucination.

For most stages, however, testing relies on metrics and thresholds. Instead of asserting actual_result == expected_result, we calculate metrics that quantify quality, then set thresholds for passing. Our app has multiple stages within its execution pipeline: 

• Intent classification used for properly identifying target data sources and user action: As mentioned previously, simple string equality still works in such a case when outputs are constrained to predefined enums.
• Query rephrasing used for adapting user queries for vector database searches: BERT score measures semantic similarity between the user's original query and the document search query 
• Context retrieval: context recall, precision, and F1 scores validate that we're retrieving the right information 

End-to-end testing requires a different approach. We use F1 score with a precision and recall method developed internally by Daniel, a PhD scientist on our team. We generate evaluative statements from ground truth using an LLM, then classify those statements against the model’s response as true positives, false positives, or false negatives. 

For example, when testing a chatbot’s ability to summarize news from a URL, the model generates specific assessment questions from the source article, such as: 

• Does the company have over 500 million monthly active users?
• Do all companies expect increased capex in the coming years?

The generated summary is evaluated against each question. A perfect match scores 1.0. Partial agreement lowers the score proportionally. This produces a repeatable, quantifiable measure of quality.

Yes, this means using AI to evaluate AI. It is not perfect, but it is practical and scalable.

Key takeaways 

• Metrics definition is not an afterthought. It is the foundation of GenAI testing. Different stages of the pipeline require different metrics, and trying to force a single measure across the system leads to misleading results.
• Thresholds should be introduced conservatively. Start with values the system can realistically meet, then tighten them over time as prompts, retrieval, and reasoning improve. This keeps the test suite useful rather than punitive.
• Finally, metrics alone are not enough. Subject matter experts are essential. They define what questions matter, what acceptable answers look like, and where quality truly lies in a given domain. Without their input, even the most sophisticated metrics will fail to capture real-world usefulness.

Building the testing infrastructure behind our GenAI pipeline 

Integration testing is built on DeepEval and pytest. DeepEval provides a mature set of evaluation metrics that map well to individual stages of our GenAI pipeline, while pytest gives us a familiar, reliable execution framework. Together, they let us test real model interactions at the component level without inventing new infrastructure.

End-to-end testing required a different setup and introduced more complexity. When we designed our testing strategy, existing tools did not support the workflow we needed. Rather than force-fit a solution, we built a lightweight internal tool tailored to our requirements.

The tool has two simple responsibilities: 

• Test execution: It sends user questions to the bot, captures responses, and records them for evaluation. This layer is intentionally minimal, relying on straightforward API calls and structured logging.
• Metrics calculation and evaluation: Here we compute F1 scores, compare results against defined thresholds, and generate reports for manual review. This layer is implemented with Python scripts, using DeepEval for metric computation and Plotly for visualization. The entire tool took only a few days to build and refine.

As the test suite grew, an operational issue became impossible to ignore: LLM usage costs scale quickly when running comprehensive tests. To manage this, we integrated Langfuse for granular cost tracking. Over time, it became just as valuable for end-to-end tracing and debugging, allowing us to connect failures and regressions back to specific prompts, model calls, and execution paths.

How to make tests stable in CI/CD pipelines? 

LLMs are probabilistic by design. The same prompt can produce different responses across runs. This creates a direct tension with CI/CD pipelines, which assume tests fail only when something is broken. If left unaddressed, randomness alone can destabilize builds. Stability requires explicit engineering choices. Over time, we converged on a small set of techniques that reduce noise without masking real regressions. Our stability toolkit includes: 

• Retry logic with flaky library: For most tests, we require 2 out of 3 test runs to pass. This absorbs random variation without masking real regressions. Legitimate inconsistencies no longer fail builds, while persistent failures still surface.
• Low LLM temperature: Lower temperature settings reduce randomness in LLM responses, making tests more consistent. This might not work for every scenario, but in our case, having the lowest temperature was acceptable. 
• Structured outputs: Wherever possible, LLM responses are constrained to JSON with predefined schemas. This sharply reduces ambiguity and parsing failures. While now considered standard practice, it was a major stability improvement for us.
• Smoke test suites: A small, critical set of tests runs on every pull request, while full suites run nightly. Fewer tests mean lower cost and higher reliability. Running 10 tests with occasional variance is far more stable than running 100 with the same failure rate.
• Handcrafted test cases: Auto-generated tests are useful, but they require manual curation. Carefully designed test cases produce more predictable behavior and reduce noise. 
• Soft assertions for end-to-end tests: Individual failures do not immediately fail the suite. Each test contributes to an aggregated score, and the build fails only when the aggregate drops below a threshold. It allows you to identify problematic areas without letting occasional fluctuations derail your CI/CD pipeline. 

The human element in GenAI testing 

Fully automating GenAI testing is far harder than most teams expect. Traditional testing already requires judgment in edge cases. With LLMs, non-determinism makes that judgment unavoidable.

We automate what we can: F1 scores are computed automatically, builds pass or fail based on defined thresholds. But automation alone is not enough. Someone still needs to review results, especially failures and borderline cases.

A failing test can mean very different things:

• The ground truth needs adjustment. The response may be acceptable, but the test case is too rigid and does not reflect valid variation.
• The response is genuinely wrong. This is a real defect that needs to be fixed in prompts, retrieval, or logic.
• The metric is misaligned. The chosen metric may not capture what actually matters, which is common early in metric design.
• The threshold is too strict. The system meets user expectations, but the pass threshold does not reflect a realistic definition of “good enough.”

To resolve this, QA engineers, product owners, and domain experts review results together. They examine low-scoring responses, discuss expected behavior, and refine both tests and implementation. The process is slow, but it is necessary.

Subject matter experts are non-negotiable. Without domain experts who understand the nuances of what constitutes a good response in your specific context, you simply cannot achieve the quality your users deserve. Their expertise guides everything from writing test cases to interpreting results to setting appropriate thresholds. 

The tooling ecosystem is still evolving, so human judgment remains a core part of GenAI testing.

The bottom line: Building a new testing model for AI apps

Testing GenAI systems forces you to unlearn long-held assumptions about quality assurance. The traditional testing pyramid collapses once core business logic stops running inside your codebase and starts living inside a model you can only steer through prompts, not control outright.

Our approach accepts that reality. We center testing on real integrations, anchor evaluation in metrics, automate with intent, and keep human review in the loop. This demands more upfront engineering effort and ongoing expert involvement, but it aligns testing with how GenAI systems actually fail.

The payoff shows up in production: Our GenAI app quality improved drastically. Most issues never reach our QA team, let alone our users, and we catch problems before they impact production. 

GenAI is changing the rules for how we build software. It's time we adapt the rules for how we test it.