Building a Foundation for Data-Driven Decision-Making at Oatly

Introduction

Data Edge has helped the leading oat product company Oatly to implement a new and modern data platform, that enables them to take data-driven decisions and systematically create business value.

Oatly had reached a point where their data platform could no longer keep up with the needs of an increasingly more data-driven organization. Their platform setup made them dependent on consultants and they struggled to deliver new use cases in an iterative way. Their platform also made it difficult for them to fulfill external regulations and to start working more with advanced analytics.

Identifying Key Capabilities

Together we pinpointed a couple of key capabilities from modern data platforms to solve this:

• Capability to handle batch and streaming data
• Possibility to share data within the organization
• Capability to handle different use-cases, from reporting to advanced analytics and machine learning
• Full traceability within the platform, who did what and when
• Allowing for easy data discovery
• Securing high data quality and highlighting data quality issues
• Limiting access and enhancing security without affecting developers daily work

Implementing a Data Lakehouse Solution

The answer to these needs became a data lakehouse setup built upon a Delta Lake in Databricks. By choosing the Delta format the solution gets the flexibility of a traditional data lake but still incorporates key data warehouse features such as ACID-transactions and schema enforcement.

By keeping the data in the delta lake throughout the whole solution, if granted, all data can easily be shared and used between different use-cases and users in Databricks. For example by a data scientist working in the data science environment or a business intelligence developer working in the more BI-oriented SQL-environment.

From Source to Insight

To be able to handle both streaming and batch data, we developed an ingestion framework based upon the spark engine in Databricks. By utilizing the autoloader concept in Databricks the framework can instantly create the components needed to allow streaming of new files from storage accounts into a bronze ingestion-layer.

For data transformations we choose dbt. This not only makes the setup more flexible but also, due to its code based approach, allows developers to work according to best practices within software development. All new SQL logic must go through both code linting and complete functionality testing in automated CI/CD pipelines, before being permitted to be released between the three existing environments, dev/test/prod.

With the use of highly automated CI/CD flows all manual interaction with both the test- and production environment can be avoided. This gives full traceability of everything residing in these environments. Every small change can be traced all the way back to its origin, which is a key feature to fulfill regulatory requirements.

The data transformations are done in different layers, according to medallion architecture. From the bronze layer with raw data, through the silver layer where data is cleaned and filtered, up to the gold layer where the data gets the right business level granularity.

Finally we developed an access framework that made it easy to share lakehouse data with different business users. This made it possible to share market specific data with analysts working all around the world. To further enhance the sharing of data within the organization we also implemented data discovery tools such as Purview and dbt docs.

Fostering Independence and Ongoing Development

During the whole project we put a lot of effort into sharing knowledge with Oatlys new internal data team. This has given them the skills needed to continue the work of developing the platform without extensive external help. Both costs for consultants and for the platform could be lowered and at the same time capabilities to iteratively create business value added.

With this in place Oatly now has the right technical capabilities to move forward on their journey to become a truly data-driven company.

Let's Make Data Work for You

At Data Edge, we believe in the power of data to transform businesses. Our work with Oatly is a testament to how the right strategies and technologies can elevate a company's data capabilities, driving significant business improvements.

Thinking about your data strategy? We're here to help. With a focus on practical solutions and sustainable results, we guide our clients through the complexities of data-driven development, ensuring they're equipped for the challenges of tomorrow.