About our client
Our client is a European manufacturing company generating over $1B in sales. Their products are used throughout the world in automotive plants, machine tools, and many more industrial applications. The company has over 40 subsidiaries and more than half of these use the SAP ecosystem independently from one another.
For that reason, a single-point-of-access to data for the company was an elusive challenge; updates were done manually and shared sporadically, and there was no way for the organization to acquire a consolidated view of their data or the stories the data wanted to tell.
The client’s challenges
The client’s Finance team approached us with the following objectives in mind:
-
Consolidate data from SAP-using entities;
-
Combine that data with data coming from other, non-SAP sources in a single, easily-accessible location (a data lake) ;
-
All the while ensuring that (interim) results are delivered at speed.
Our solution
Developing a custom data connector that would allow us to connect with SAP and extract data from it.
Load extracted data into a data lake which would act as a single point of access and work foundation for data analysts.
Provide a reporting tool with dashboards and detailed consolidated data views.
OUR APPROACH
In our work approach we follow the agile methodology. This allows us to not only progress fast and test extensively, but also nimbly respond to new requirements while ensuring transparency in actions we take.
Following said approach, we always portion a project into a Minimum Viable Product (MVP) and ensuing releases that build off the learnings acquired in the MVP phase.
In the same fashion, we committed to deliver an MVP to our client in 3 months. The MVP included several data extracts, a data lake implementation, data modeling, and visualization.
The foundation for all these deliverables, however, was developing a data connector – a process that makes extraction of data from SAP (or any other data source) and writing it to its destination (such as a data lake) on a schedule possible.
Why do I need a data connector?
Data connectors enable you to combine various sources of data into one integrated space. The segmented approach to data management will no longer cause disruption in the analysis, but a holistic overview that will enable to disrupt data silos, poor communication, and quality of insights.
5 benefits of data connectors:
-
Improved decision-making process
-
Increased productivity
-
Streamlined operations
-
Improved customer experience
-
Ability to predict a future
Why should I build my own data connector instead of buying one?
The main reasons why you should build your own data infrastructure lie in the ability to retain control over it, increased flexibility and having full ownership and independence.
How did we go about writing the connector?
Why connecting to SAP can be so challenging?
-
SAP does not follow existing industry standards. Instead, they have developed their capabilities, which are not well adapted to third-party systems and are difficult to integrate.
-
Complexity and cost increase with inconsistent configuration. SAP’s most significant benefit is its ability to be highly customized to meet a business’s specific needs, and many companies take advantage of this flexibility. Most companies use SAP’s flexibility to meet their particular needs, which is its most significant benefit. However, this level of customization increases the complexity of integration, increasing the time and cost of connecting to new systems. Multinational corporations, for example, likely have different SAP instances in North America, Europe, and Asia, each individually customized. This customization suits each region’s business operations, posing significant integration challenges. As companies acquire competitors with their SAP instances (or another ERP entirely), integration challenges mount.
-
Slow delivery, which is a result of specialized skill set requirements. Developing and delivering software can be very complicated with SAP’s comprehensive solutions and multiple integration methods. Therefore, to manage this complexity, it is necessary to utilize large teams of developers. It can take months to complete SAP projects because of their complexity and business-critical nature. In the meantime, companies must back-up line-of-business requests awaiting prioritization and resourcing, frustrating end-users who need to innovate in real-time.
What are the principles we follow when developing a data connector?
-
Utmost flexibility so that the deliverable is scalable and easy to adapt. For that reason, we avoid integrating UI-based tools. They are hard to automate, extend and maintain long-term.
-
Efficiency. For that reason, we avoid solutions that require purchasing additional tools. On top of that, vendor-selection processes tend to be lengthy and involve several stakeholders.
-
Complete transparency. Not only do we keep our client in the loop at every step of our process, but we also share the complete work product with them. We don’t believe in vendor lock-in and find it important that our clients are able to use our solutions without us if needed. For that reason, our clients receive full source code which they can adapt and expand in any way they deem necessary.
Given the lack of clear industry standards and extensive customization of SAP on the client’s end, we determined that the fastest and most efficient next step would be developing a proprietary data connector.
Developing the connector was done in two phases: the prototype stage and iteration stage.
The prototype stage was completed in just 2 hours. We arrived at a straightforward product that was able to connect to SAP, extract and load data to a data lake on a daily basis. It built on top of our existing data product repository and followed our nimble development recipes.
We spent the iteration phase adding requested features, data veracity checks and ensuring the reliability of the product. This all happened concurrently with performing data analytics.
The final MVP was delivered in 3 months. It was a robust and feature-rich analytics solution that included infrastructure, code, data models, dashboards and documentation.