Streamlining data infrastructure and analytics for a leading electrical cooperative

A large organization consisting of multiple electrical cooperatives in the energy industry sought to modernize their data infrastructure by migrating extensive datasets to Azure Databricks. Blueprint collaborated with their engineering and data teams to acquire and organize this data for analytics, leveraging the Lakehouse Optimizer to ensure cost transparency and continuous improvement within their Databricks platform. This transformation facilitated their ability to forecast energy prices and anticipate weather-related outages, ultimately projecting an annual savings of over $170,000. Blueprint’s efforts not only streamlined their data processes but also laid the foundation for future analytics projects and for improved operations, enabling them to better serve millions of customers with reliable and affordable electricity.

Client Snapshot

Who:

A large organization of electrical cooperatives

Industry:

Energy

Stakeholders:

Director of Data Science and Director of Analytics

Work Summary

What we did:

  • Migrated large amounts of data to Azure Databricks
  • Organized the data for analytics
  • Implemented the Lakehouse Optimizer powered by Blueprint to enable cost transparency and offer insights into what they can improve within their Databricks platform

Client background

Our customer is one of the largest generation and transmission electric cooperatives in the nation, providing reliable, affordable electricity to millions of customers. Additionally, each non-profit within the cooperative goes a step further to offer innovative energy solutions that enhance the quality of life for millions within their state. They own power generation assets, purchase electricity through contracts, and coordinate transmission resources for their members. Their data is a valuable resource that aids in forecasting energy prices and understanding weather patterns to better plan for outages.

The Blueprint way

This customer aimed to migrate their data to Azure Databricks, emphasizing data quality and understanding data origins. Blueprint assisted in eliminating disparate data sources, enabling a smooth migration. Post-migration, Blueprint’s Lakehouse Optimizer provided cost transparency and long-term monitoring of Azure Databricks.

"

I finally wrapped my head around this UDF and have to say, this is very impressive.
Software Developer

The challenge

As the leading organization overseeing many other cooperatives, their biggest challenge was the siloed data, which was not easily accessible. Blueprint assisted in unifying the orchestration and processing of disparate data sources into a cohesive system. Despite their ambitious plans for utilizing their data, they were hindered by limited time and resources to complete the migration efficiently and at scale. Additionally, being new to Databricks, they needed someone with high expertise to ensure their data was accessible and unified.

The solution

The Blueprint team successfully migrated 3 large data products that include many pipelines within each. Our client created a metadata framework (MDF) driven approach using Microsoft SQL Server and Synapse Pipeline, which our team altered and enhanced. The primary effort was focused on migrating outdated ETL processes that utilized a variety of tools, ranging from Visual Basic to shell scripting, to their MDF process and Databricks. The majority of the pipelines that were transitioned ingested data from FTP/SFTP endpoints.  The Blueprint team saved countless hours of labor and costs by eliminating irrelevant reports and outdated legacy data that had not been used for years.

"

This (Blueprint’s Lakehouse Optimizer) is a very clean and powerful UI. Very intuitive!
Director of Software Services

Impact

Discover how you can become our next success story

Get started with us today.

Share with your network

You may also enjoy

With a focus on migrating from Cloudera to Databricks within a tight timeframe, the Blueprint team executed a seamless transition while addressing critical challenges surrounding data silos, and heavy storage costs.