How to Build an AI Maturity Assessment Framework from Scratch: Lessons from Our 16-Month AI Journey

About two years ago, we faced a fundamental challenge: how do we understand our organization's AI maturity, and how do we track our progress?

Our only metric at the time was the percentage of teams and people who "use AI." We eventually realized this wasn't enough, because "using AI" can mean very different things. One person might hold a license and occasionally log in to an AI app. Another might use GitHub Copilot only for basic code completion. A third might use it consistently, with accelerators and advanced workflows. "Using AI" was too broad to be meaningful.

Without a way to tell an occasional license holder from a team running a near-dark-factory workflow, you can't see where your organization stands or track progress. A maturity model fixes this by defining clear levels of AI use and measuring every team against them.

We built ours at EPAM and refined it over more than a year. Read on to learn how to build an AI maturity assessment program from scratch, from your first model to a validated process that runs continuously at scale.

How We Built Our AI Maturity Model

About a year and a half ago, we recognized that we needed a maturity model that categorizes how teams use AI across defined levels. We went through multiple iterations before finalizing our current model in Q3 2025, and we have followed it since. That same quarter, we launched the assessment process across teams. We began validating self-reported results in Q4.

The 4 Levels of AI Maturity

At a high level, the model defines three levels beyond zero:

• Level 0: Teams use no AI.
• Level 1: Licenses exist, but teams use AI inconsistently.
• Level 2: Teams use AI consistently, with commonly adopted patterns and tools. We review and update the definitions of "consistent" and "commodity tools" each quarter.
• Level 3: Teams use novel AI features: approaches the market positions as the most advanced, where the greatest benefits are expected but few have accumulated real lessons learned. Level 3 offers the highest potential acceleration and benefit, but it requires a team mature and skilled enough to navigate challenges others haven't faced yet.

How to Measure AI Maturity at Scale?

Once the model existed, measurement became the next problem. EPAM has thousands of engineering teams. Even assessing a single team thoroughly takes significant effort: we had to align on rules, cover edge cases, and account for every engagement model and technology, including niche ones well beyond Java and JavaScript. At this scale, manual assessment was impossible.

1. Use a Self-Assessment Questionnaire

We concluded that we needed a survey. Because we couldn't staff a team large enough to assess every project individually, we built a self-assessment questionnaire: a set of very specific questions that teams answer to determine their preliminary maturity level. Based on the responses, our automated system assigns a preliminary score.

Examples of questions:

This approach was an important first step. It let us run assessments at scale and see where we stood as an organization, at least based on self-declared data. The numbers weren't always high, but they were valuable: they showed us where to look.

2. Capture AI Adoption hurdles

Beyond the maturity level itself, we also asked teams for their reasons. For example, if a team wasn't using AI, we asked why. Some engagements weren't ready to permit AI usage, and that turned out to be the dominant reason for low coverage. We also asked which technologies and approaches teams used.

Together, these answers gave us a rich knowledge base: which approaches were most common, and what was working and what wasn't. These insights reflect the experience of our teams

What We Learned Quarter by Quarter

Q4 2025: Knowing Where We Stood

Our biggest achievement was learning our average maturity level and the main reasons teams weren't using AI. That alone was a significant milestone.

Q1 2026: Identifying What Drives Level 3 Outcomes

On top of knowing where teams stood, we identified the approaches that actually drove Level 3 outcomes. Demand for these answers was enormous. EPAM had made Level 3 highly visible and attractive through major campaigns, investments in training, and strong internal motivation. Teams wanted to reach the highest level.

But our goal wasn't just to point people to external training. We needed to understand what actually works on real projects.

This distinction matters. There is a wide gap between what the market claims and what teams can realistically apply in enterprise environments. In practice, they face technical restrictions, security limitations, approval processes, organizational resistance, and quality commitments.

Tools and approaches that look impressive in demos, which often target small, greenfield applications, behave very differently on large, complex enterprise systems with real quality expectations and commitments. Our survey data gave us a ground-truth view of what actually worked.

How to Validate Self-Reported AI Maturity?

By Q4 2025, and more formally in Q1 2026, we recognized that self-reporting alone wasn't sufficient. We didn't distrust people. The problem is that AI terminology is genuinely ambiguous. The word "agent" meant different things to different people. "AI-native," "novel features," and "accelerators" carry different expectations depending on who uses them.

We introduced a validation process. Once a project self-declared a high maturity level, a group of experts from our R&D team conducted a one-hour session to examine the details: what exactly the team considered novel, how they understood and applied specific approaches, and so on.

One hour turned out to be a very short window for the depth of knowledge we wanted to extract. Our goal wasn't only to assign a score. We also wanted to collect insights:

• Which approaches worked and which didn't
• What limitations the team discovered and how they overcame them
• What the team would do differently
• What they would recommend to others

We also explored how teams think about the value and cost of AI. Whether the team felt it got enough value from AI, or tried to optimize AI-related costs in the first place. Teams can't answer these questions well in an unprepared session.

Collecting Artifacts Before the Review

To fix this, we improved the process. Before each call, we asked teams to submit artifacts in advance: quantitative details such as the percentage of work covered by AI (broken down by novel vs. commodity AI), per-role tool usage and ratings, workflow stages, and specifics about the primitives they use (rules, skills, MCPs, and so on). Each submission covers the following areas:

• AI coverage across SDLC phases: Each phase's share of effort, and how much commodity and novel AI cover it. These figures roll up into total and novel AI coverage.
• Daily active usage: Evidence that teams use AI tools every day, not just license them.
• Performance tracking: Confirmation that the team measures flow time, throughput, and quality.
• AI champions: Who drives adoption on the team, and in what role.
• Reusability: Whether reusable artifacts (rules, skills, agents, MCPs) exist, and whether others can find and use them.
• AI cost management: Practices for cost visibility, review, and experimentation.
• Tools by role: Primary tool, the AI tool used with it, satisfaction score (1–10), likes, and pain points.
• AI cost structure: Cost model per tool (per-seat vs. usage-based), usage intensity, and how paid capacity compares to actual consumption.
• GenAI value: A defensible ROI or cost-benefit analysis.

Reviewers could then analyze the artifacts offline before the session. We even built an agent that validates submitted artifacts and tells the submitter whether they are complete, so both sides arrive prepared.

Building an Internal AI Harness to Automate Validation

As validation requests grew to tens of projects, running multiple reviews in parallel became cognitively unmanageable. The volume exceeded what the available R&D experts could cover at the required quality. Reviewers needed to share findings, remember details across sessions, and maintain objectivity.

We realized we had to practice what we preached. If we asked teams how they accelerate their software delivery, we needed to apply the same thinking to our own process.

So we built an internal AI harness: a set of agents, skills, and tooling that accelerates and automates our validation workflow. We didn't conceive it as a product with a roadmap. We built it pragmatically and evolved it iteratively, shaped as much by urgency as by design, to solve our own team's immediate needs.

Here is how it worked:

• Once a team submitted its materials, a GitLab pipeline kicked off automatically.
• Agents analyzed the submission and produced a preliminary score along with a detailed validation report.
• For each assessment criterion, the report indicated whether the team met the bar and why, or flagged gaps with explanations. It also generated a list of targeted questions for the live call, focused on those gaps.

This dramatically accelerated our team's work and ensured structured, consistent knowledge capture across all submitted projects.

Scaling AI Maturity Assessments with Meridian

By the middle of Q2 2026, the harness was proving its value, but success created new pressure. Projects arrived in increasing volume. The community of validated teams grew, we identified champions, we developed training and boot camps, and more teams wanted in.

R&D expert capacity could no longer cover all reviews at acceptable quality. The pipeline-based setup was also showing its limits: as the number of projects grew, so did the operational overhead, and the system became unwieldy.

Two things became clear:

1. The validation process needed to scale. We needed to onboard additional reviewers, coached and governed by the R&D team, who could conduct assessments independently under a shared framework.
2. The harness needed to scale. The GitLab pipeline could not support our volume and complexity.

This is how Meridian was born.

Meridian replaced our ad hoc GitLab harness with a purpose-built application. Because a team deeply experienced in AI built it, and because we applied the same advanced practices we validate in others, development moved fast. The bottleneck wasn't implementation; it was alignment. Integrating with multiple internal enterprise systems and aligning senior stakeholders on requirements and desired outputs took far more time than writing the code.

Building a Knowledge Harness For Continuous Improvement

Beyond maturity assessment itself, we have built a growing knowledge base drawn from practices, success and failures– drawn from years of experiences, documentation and projects.

Generating automated case studies: We use our continuously updated knowledge base to draft case studies of approaches that work. This process already runs today. We plan to expand it so the harness can generate case studies and validate them against the original EPAM project evidence.

A Claude Code plugin for senior staff: We also built a Claude Code plugin for senior EPAM staff. It lets practitioners extract insights grounded in real life experiences and hard-won knowledge: what approaches work, in which contexts, and how teams have navigated common obstacles. This makes hard-won knowledge searchable and immediately applicable.

What's Next: From Counting Licenses to Measuring AI Maturity

Our journey took us from counting license holders, a metric that makes an occasional user look identical to someone running a near-dark-factory workflow, to a mature, validated AI maturity model that runs at scale.

We have applied the model repeatedly over more than a year of operation and refined it continuously. Quarterly rule updates ensure that "novel" always means something genuinely current. The validation process has generated a unique, real-time knowledge base about what actually works in enterprise AI adoption. We also bring these learnings into future engagements to keep evolving how we meet clients where they are.

The process is not a one-time exercise. It runs continuously: we assess teams multiple times per quarter, capture lessons as they go, and continuously identify and disseminate the most valuable practices. The more we learn, the more we can offer. That, ultimately, is the point.