If you ask yourself what the key to being successful in data initiatives is, some answers will be like this:

• you need to know how to clean your data,
• you need to know how to create a model,
• you need to know how to tell a story,
• you need to know how to…

Ultimately, all of that will result in an excellent outcome but will it bring real value to the clients??

Before joining dyvenia, I had already gained some knowledge of data fundamentals. That was useful in understanding how dyvenia’s teams worked. We delivered great data models and insights to our clients. However, over time, I started to wonder how we could improve our processes even further.

After all, we live in a world where the only constant is change. What was fashionable a month ago is being replaced by newer trends and requirements. What our client needed three months ago may not be useful now. As a result, ignoring a client’s evolving needs leads to waste of time, resources and energy.

For this reason, we began looking for a way of working that meets our clients’ expectations, and provide them with what they need at any moment. The agile spirit of our company led us to choose Scrum as the starting point.

Scrum is a lightweight, agile framework that helps teams deal with complex problems.  In a development cycle, I see the following as core enablers of agility:

Experiment – Inspect – Adapt (and repeat)

What does that mean? When dealing with a complex challenge, we usually don’t know the correct solution immediately. When trying to think it through, we need to consider all outcomes and risks, and define the whole scope. The result will either be a poor solution or a massive waste of time (think about all the redefining scope, redoing work every time we learn new things).

Using Scrum, the team works on solving such problems in a lean, limited waste manner.

The Scrum team would usually contain the following roles:

• Data Engineers
• Data Analysts
• Product Owner – a representative of the client’s goals
• Scrum Master – a “coach” facilitating agility.

Scrum is not the definition of agility, but it does a great job of identifying where existing tools, processes, and ways of working block our teams from focusing on the core values that allow them to work together in an efficient and flexible way.

This brings us to the concept of Scrum sprints — short, repeatable phases of varying lengths. Teams usually determine the sprint length when starting to work using Scrum.

Scrum Teams use sprints to reach a sprint goal within a short, time-boxed period. Each sprint should result in a draft, prototype, or workable version of the final deliverable.

Depending on how much uncertainty and risk are involved in a challenge, the sprints can last from one to four weeks, but when we learn the basics, two weeks become almost the “default choice.”

During this interval, the team works on a chunk of the client’s product like:

• Cleaning, verifying, and preparing data for later steps,
• Collecting insights, new knowledge from data experimentation, 
• Processing new data for accessible insights in an automatic way,
• Visualizing both historical and current events on dashboards, and at the end of it, the client (usually represented by the Product Owner, PO) inspects the work product to review if it meets their expectations. After collecting feedback, we adapt by creating a plan for the next interval (sprint).

Whether Scrum is used for software development or for working with data, it prescribes four meetings:

• Sprint Planning, where the team plans what they aim to deliver during the sprint. The team defines the Sprint Goal, answering the question of why the sprint is valuable for stakeholders. 
• During the sprint itself, the team syncs at Daily Scrum meetings. This meeting also facilitates a quick inspection — adaptation approach regarding its Sprint Goal. 
• At the end of the sprint, there are two meetings: Sprint Review and Sprint Retrospective. During the former meeting, the team presents their work and collects feedback. The team then uses the latter meeting to reflect on their ways of working and collaboration in order to improve their facilitation and become more effective.

In our interpretation of Scrum, the core development unit is a Scrum Team. This is a cross-functional, self-managing team, possessing all the skills required to deliver a complete product. As already mentioned, it consists of a Product Owner, a Scrum Master, and a Development Team (in our case, mostly Data Engineers and Data Analysts). Each role has different responsibilities and accountabilities, but all focus on the Product Goal.

The Product Owner owns the vision of the Product — Product Goal. They know WHAT needs to be achieved, learned, or enabled and how important it is for stakeholders. They are a value maximizer. They need to understand the client’s values and point of view, but they don’t necessarily have to come from the client side.

They act as a bridge between the end user and the development team. They translate the vision and requirements into actual work done by the developers. They are using the Product Backlog as a wzay to structure ideas into actual, actionable items by writing User Stories. A User Story is an informal, general explanation of a product feature written from the perspective of the end user. Its purpose is to articulate how a product feature will provide value to the customer.

Developers are committed to creating a usable piece of product each sprint. They own HOW it can be done, and they work with the Product Owner. A Scrum Master is a servant leader who helps the team with effectiveness and collaboration. They are also removing roadblocks during the organization’s agile journey. They are the team’s coach and mentor.

A Product Backlog is used to communicate one’s vision to their team and stakeholders over a longer period of time. It’s a “living” list of work and features missing in the product.

What does “living” mean? It means that every time during the inspection-adaptation cycle, the Product Owner, accountable for maximizing business value, can reprioritize items in the Product Backlog based on feedback from the stakeholders/end-users. Product Backlog also ensures transparency with stakeholders.

You can find out more about Scrum in the Scrum Guide. It’s only a few pages long and in multiple languages which makes him as light as the framework itself.

If we think about data, statistics, modeling, databases, and numbers come to mind.

For the average person, however, it is storytelling that creates true value. The above skills are not the only that Data Analysts must have, but also they have to be able to tell the story behind their data to ensure proper insight delivery. When they do it right, clients know the value of their data. On this basis, they can build their next requirements and ask new and better questions. A Data Engineer must contribute toward better data quality, accessibility, and understanding, so the story is not only “nice” but also valuable.

Those data stories should not be confused with Scrum User Story.

Customer-centric User Stories are a key reason for why a team needs to be cross-functional. They need to be interdisciplinary so they can deliver such functions or knowledge autonomously, without synchronous help from other teams, departments, and people.

There are multiple techniques for creating User Stories. One way is answering the WHO? WHAT? WHY? questions.

In dyvenia, we use INVEST to squeeze from User Stories as much as possible.

What do those letters stand for?

• Independent – It can be done without a big bunch of dependencies.  
• Negotiable – The team can negotiate the actual realization of a story to best fit the other traits.  
• Valuable – It brings value even if the team gets reassigned after the sprint.  
• Estimable – The team has enough knowledge to predict effort and complexity.  
• Testable – The team can agree on how they can test that the User Story is completed.

Good User Stories benefit from cooperation with the business, stakeholders, and the Product Owner, who convey what they expect and how we can transform User Stories into valuable data storytelling.

The quality of User Stories makes the difference between having an agile process and a chaotic waterfall. Good ones convey end-to-end value, communicate a clear vision, and allow the team to execute effectively. Then there are those that are bad, just items that do not add value on their own — often resulting from trying to make them “too small.”

Some good examples:

• “As Sweden Market Head of Sales, I want to qualify current prospects based on their living area and shape of their house coming from available data sources.”  
• “As (previous), I want the qualification to be automated so that a single model can qualify past and new prospects.”  
• “As (previous), I want to improve the accuracy of predictions by removing wrong data from our training datasets.”  
• “As (previous), I want to have a live dashboard where new prospects are automatically qualified when coming in.”

As one can see, the order is counterintuitive compared to what we would call “a project.” The User (representing a Sales organization benefiting from team expertise in our case ) is getting something valuable from the first User Story. Each User Story is also a learning opportunity and a “cancellation point”. Later ones can be abandoned if new ideas seem to bring more value.

If you look at the Agile Manifesto, you will find what we value in dyvenia.

“Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation  

Responding to change over following a plan

That is, while there is value in the items on

the right, we value the items on the left more.”

We started by implementing Scrum, and when we understood how it fits our team and client projects, we did what we believe the key message of Scrum is — we inspected and adapted. And we continuously do it to provide more value, less unnecessary work, and more tailored data usage for our clients.

The only way to succeed is to find methodologies that provide consistency and leverage the most value for clients. To us, Scrum has proven to be the framework that has introduced the right level of empowerment, velocity and quality.

This is dyvenia’s first article in a series of articles where we share our lessons learned when implementing Scrum for data initiatives. We are planning on sharing more content like this in the following weeks, so subscribe to our newsletter to receive it as soon as it is available.