Big Data and Agile: The Perfect Marriage

by Gino Marckx, Head of Agile Competency Center, EPAM Canada

Big Data is 'in' these days. It’s one of those emerging technologies that many organizations feel the need to climb on board for. But just like with anything else, it’s important to understand how Big Data can help your business innovate and grow. Talking about Big Data without mentioning data science is really not possible. It’s the data science that allows organizations to make sense of the data that we are able to collect these days: what information is hidden in the terabytes of data representing click streams, social media interactions, sensory data points, all kinds of structured and unstructured data.

Using the scientific method, hypotheses around the information that is hidden in the data are proved or rejected and based on the outcome of validating the hypothesis, new hypotheses arise or ways of using that information emerge. Because of the uncertain nature of big data projects, it is crucial to be able to rely on a delivery method that allows the project to consume changing requirements or quickly change directions altogether. 

Agile provides such an adaptive approach to delivery. The principles and practices that are collected under the Agile umbrella all focus on validating assumptions as early as possible in the delivery lifecycle, significantly reducing the risk exposure as the project continues. Every software engineering project piles up the assumptions one after the other. The entire planning process is based on the assumption that everything will go as expected, even the padding that is added during the planning is based on assumptions that the amount of padding applied is sufficient. The functionality that is implemented is based on assumptions that the functionality will indeed provide the expected business value.

The design and architecture is based on a whole bunch of assumptions and ultimately every line of code that is written is based on the assumption that the line of code does not contain any bugs. By delivering the work in small increments of working - even production ready - software, those assumptions are all validated early on. All code, design, architecture and requirements and validated every time a new increment is delivered, even the plan is validated as teams get real and accurate data around the progress of the project. But the early validation is not the only benefit that Agile brings, it also allows projects to learn from the feedback, take in new or changing requirements and quickly change direction when necessary, without changing the process at all.

Both the fast feedback and the ability to change direction are attributes that support the scientific method and help big data projects succeed. On the other hand, big data analysis can help agile projects to validate the assumptions about requirements even earlier. So the benefits really flow in both directions.

Watch our webinar to learn more why Big Data and Agile is a happy marriage of science and methodology when done right.