Data Lake Implementation
Eficaz ETL & Data Lake
Big Data implementation and development
Eficaz DW Suite
Client is an organization in Africa spread across multiple countries. The client has currently 130+ branches across the African Great Lakes region.
The specific issues that the company faced in achieving this goal were:
- Challenges in extracting, transformation and reporting/analyzing data across all the branches
- Delays in report generation and distribution, as the report hits database for each slicer selection
- Inefficient reporting of high level performance metrics
- Complicated data structure
- Moderate implementation resources are available
The overall deployment was designed to meet the following objectives:
- Achieve significant improvements in accessing, transforming, analyzing , reporting and visualizing the data. This will allow customer’s employees to focus more on the analysis activities.
Benefits of Proposed system
- Efficient and automated data collection, distillation and visualization platform.
- Will provide advanced analytical capabilities to improve analysis and decision making (thresholds, trending, graphical analysis, etc.).
- Dynamic and largely scalable user environment that allows numerous users to simultaneously work with the system.
- Provides an effective and efficient method for collaboration and reporting of the required information to the appropriate stakeholders and management. In addition, it will streamline and automate the decision making.
- Rapidly adapted and scaled to support additional users, new requirements and changing needs.
In a short span of 24 weeks, Lera Tech created a highly scalable Data Lake which had the following:
- Distributed ingestion pipelines from source to Staging environment in Data Lake.
- Bots based pre processing and data consolidation from other data sources.
- Automation of key reporting and analysis processes.
- Seamless, incremental data extraction and transformation layer for extracting data from core source system.
- Advanced analytical environment, enabling key personnel within the organization to perform What If? analysis on several business scenarios.
- Reduction of time required to generate ad hoc reports used across the business to several minutes this has improved productivity of the employees, utilizing the information provided.
- The integrated and accurate reporting translates into better decisions. These factors contribute to a high return on investment from the comprehensive reporting solution.
There were several elements in the decision making process as well as the solution and implementation partner i.e., Lera Tech that contributed heavily to the success of the project
- The Data Access Layer (DAL) includes the core source system database or any other long term data storage including Active Directory/Authorization Store, file systems, externally managed services.
- The Business Services Layer consists of components that abstract the knowledge of how to connect to the Data Layer. It also consists of services that allow for abstract of distributed processing and communications.
- The ERP BI consists of Graphical User Interfaces that use the objects contained in the Business Layer.