Scaling for MHHS: how Octopus Energy achieved a 50x cost reduction in margin data engineering
Octopus Energy has significantly reduced costs and improved efficiency in its data engineering processes by re-architecting its margin data pipelines. Facing a 48x increase in data volume due to new UK regulations (MHHS), the company's existing architecture was projected to incur substantial additional costs. By implementing a new system on Databricks, they achieved a 50x cost reduction per settlement date and processed 98.8% fewer rows, enhancing data freshness from weekly to daily. AI
IMPACT Demonstrates how data engineering infrastructure can be optimized to handle massive data growth and reduce operational costs.