Leap Analysis

BUSINESS REQUIREMENT

The extraction and transformation of various types of data from different sources to a single format and database is a time consuming, expensive and redundant task in Data Analytics. Dexlock was keen to take up the challenge as an R&D project to result in performing data analytics tasks with more speed, security and efficiency.

CHALLENGES

• The process of identifying the queries and converting the user defined scripts to the corresponding database queries for processing data.
• The efficiency while converting these scripts into queries and running the queries in the respective databases.

SOLUTIONS

• A semantic web analyzer was built to map data from different databases using a graph structure. The graph structure acts as a commonplace for queries to be performed.
• An extension to an existing framework was custom designed to read data from different types of databases.
• Custom graphs optimization tools were created to improve and monitor the overall performance of the platform.
• The need of writing separate queries for each type of database was replaced with user-defined scripts. A custom solution designed to translate the user-defined scripts into the appropriate query type and format was implemented to achieve the same.

IMPACT

• The platform overall is a major innovation in the field of Data Analytics.
• The complex and time consuming process of ETL is simplified by this solution.
• The redundant operations such as transformation of data into another format, separate query operations and the need for a separate database is no longer needed.
• Data Integrity is maintained with the absence of moving data from the original location.

KEY TECHNOLOGIES

• JAVA
• ReactJS
• D3JS
• Apache TinkerPop
• RDF
• Spring Boot
• Gremlin