Challenge
Databricks was at the heart of a financial services provider’s modernization journey, fuelling fraud detection, real-time forecasting, and data-driven decisions across the business.
The Risk & Compliance team had started strong: flawless pipelines, zero job failures, and costs steady at ~$25 K/month. But six months later, with new workloads across departments, spend nearly doubled to ~$48 K/month.
Leadership saw the value of Databricks. What they needed was clarity.
How much were hidden inefficiencies costing?
Which jobs were truly critical?
Who owned what?
Solution: Unified observability on Databricks
The Impetus team partnered with the financial service provider to complement Databricks operations with intelligent observability, performance optimization, and governance insights without altering how Databricks functions.
How Impetus Prism complemented Databricks
- API and metadata integration: Seamlessly connected to Databricks APIs, pulling usage, billing, and cluster events for analysis
- Smart Virtual Tagging: Automatically mapped spend by workspace, department, job, and notebook—no manual tagging needed
- Inteligent recommendations: Used GenAI to translate metrics into plain-English actions, e.g., “Right-size Cluster X from 32 → 16 cores to save $5.2 K/month”
- Governance engine: Flagged redundant jobs, idle clusters, and scheduling inefficiencies with automated right-sizing and termination guidance
- Unified dashboard: Correlated logs, metrics, and traces to uncover bottlenecks, performance drifts, and anomalies, all in one view
Impact: Clarity, control, and confidence
Within just three months, the results were measurable and meaningful:
- 30% lower Databricks spend, from ~$48 K to ~$34 K/month
- 40% higher cluster utilization via right-sizing and consolidation
- 50% faster issue resolution through unified observability
- 35% better SLA adherence for critical workloads
- 90% tagging accuracy, unlocking true chargeback and governance
- 25% faster analytics delivery for fraud detection and reporting
Different teams saw value instantly:
- Finance gained precise forecasting and chargeback dashboards
- IT got end-to-end cluster and job visibility
- Risk & Audit accessed compliance-ready usage and governance reports
- Leadership finally had confidence to scale Databricks adoption, knowing costs were controlled and transparent
Bigger story: Enabling proactive observability for Databricks
Before Impetus Prism, Databricks already powered data innovation across the enterprise. With Impetus Prism layered on top, Databricks became more transparent, predictable, and efficient, operating at its full potential.
- Performance: Bottlenecks detected and resolved before SLAs broke
- Efficiency: Idle or redundant workloads automatically flagged
- Governance: Clear ownership and cost accountability across teams
- Reliability: Predictable uptime and performance
- Agility: Teams scaling without chaos or overspend
Databricks drives innovation. Impetus Prism ensures it scales efficiently and responsibly. Together, they transform data modernization from a cost challenge into a strategic advantage, unifying visibility, governance, and optimization in one intelligent ecosystem.
