Legacy data migration with a custom data factory

Snapshot

In partnership with Databricks, Blueprint designed and built a centralized storage architecture that decreased hardware costs while enabling advanced Data Analytics, 360° Customer views, and improved regulatory compliance.

Our work

  • Modern Data Estate
  • Data Migration
  • Data Science and Analytics
  • Azure Databricks

The Problem

Our client, a global leader in the Entertainment/Restaurant (Eatertainment) industry, had a vital need to analyze a variety of very large data sets, including marketing, operations, and privacy law compliance data due to explosive growth. These objectives were blocked by out-of-date hardware and architecture, the extreme volume of the data itself, and the fact that their four main information types (Game Transactions, In-store Location Data, Point-of-Sale (POS) Data, and Mobile Data) were of varying formats stored in different locations.

The Blueprint Way

Once it was established that all four data sources were vital to achieving the business goals, the focus shifted to planning how to aggregate, store, and make that data available for analysis. Blueprint utilized a data-acquisition pattern called a Metadata Driven Azure Data Factory to move 15 terabytes of data from SQL servers to an Azure Data Lake and then implemented Databricks Delta Lake to form a “Lake House” that married customer-level data from Gaming, POS, and Mobile sources. This data is now queryable for the first time with Databricks. In addition, Blueprint engineers created Azure Event Hubs to stream data that contains Personally Identifiable Information (PII) and split that data, sending PII data to a database with PII-masking triggers, and sending non-PII data into the Data Lake, ensuring that compliance with global privacy regulations is achievable.

Impact

  • Aggregated all data in a centralized, manageable location. Data is now available for analytics and marketing purposes, allowing for ROI recognition, opportunity exploration, and predictive analysis.
  • Greatly increased ability to comply with global privacy regulations, minimizing risk of organizational disruption
  • Enabled 360° Customer view, allowing for optimized marketing and product offerings
  • Significantly reduced hardware costs by eliminating the need for on-prem SQL servers.

Let's build your future.

Share with your network

Share on twitter
Share on facebook
Share on linkedin
Share on email

You may also enjoy

Case Study

Legacy data migration with a custom data factory

Building a data factory to enable analysis of terabytes of legacy data

Case Study

Supply chain monitoring and predictive analytics

Connecting disparate systems for end-to-end supply chain management