The client is a North American-based financial institution modernizing the data platform its business depends on. Its on-premises SQL Server, Alteryx, and Tableau stack was built before cloud-native platforms were within reach at its scale - capable for individual reports, but not for the data volumes, governance, or AI ambitions it now has.

Leadership wanted one platform where scalability, governance, data quality, and observability were structural from day one, and where AI and machine learning were first-class consumers rather than later add-ons. As a federally regulated institution, it also had to satisfy examiners on demand: who accessed which data and why, how any figure was derived, and what changed, when, and by whom — questions the legacy stack could answer only through manual effort.

Built for Reports, Not for Scale or Change

The stack ran as a single environment with no separation between development, QA, and production where changes went straight to production with no automated gate. Deployments were manual, with no CI/CD or infrastructure-as-code, and fixed on-premises hardware left little room to grow.

Governance and Quality That Couldn’t Face an Examiner

Governance lived in hand-maintained documents with no catalog, lineage, access taxonomy, or PII classification, and no programmatic record of who accessed what. Data quality was enforced ad hoc inside individual workflows, with no shared rules, quarantine, or alerting.

Disconnected Sources, No Single View

Core banking, digital banking, loan origination, and payments were integrated ad hoc, with no shared customer key to tie them together. A trustworthy cross-system view meant manual reconciliation, and clean, integrated data for AI was out of reach.

‍

Compass specified the platform end to end, choosing the right Databricks capability for each stage of the data journey. Source data now lands through Lakeflow Connect managed connectors, Auto Loader, and Lakeflow Jobs which cover batch, change-data-capture, and streaming jobs - and is preserved in open Delta Lake tables. From there it flows one way through a four-layer Medallion model: Bronze preserves the raw record, Silver conforms it into system-specific star schemas and resolves customer identity across sources, Gold serves a unified cross-system model, and a governed Semantic Layer publishes business logic and KPIs as Metric Views. Transformations are expressed declaratively with Lakeflow Declarative Pipelines (DLT), and the whole platform runs on elastic serverless compute.

Engineering discipline is driven through Databricks Asset Bundles with CI/CD with infrastructure-as-code, giving true dev/QA/prod separation, peer review, and automated testing. Unity Catalog is enabled on every environment as a single governance layer providing access control with a defined role taxonomy, automated lineage, audit logging, PII classification, row-level security, column masking, and immutable retention tiers. This means a PII tag applied at Bronze propagates forward automatically. Data quality is enforced in-pipeline with DLT expectations that quarantine failing records against a versioned rule registry, and Unity Catalog system tables keep pipeline health, quality trends, access, and cost continuously observable.

Consumption runs on the same trusted foundation through three governed paths: BI, regulatory, and AI/ML. Analysts self-serve through Databricks SQL Warehouses, Databricks Dashboards, and Genie’s natural-language querying. And because the platform is AI-native from day one, the same governed data feeds MLflow, the Feature Store, AutoML, and Model Serving for classic ML. Mosaic AI, the Agent Framework, Databricks AI Search, and the Unity AI Gateway does the same for governed GenAI and agents, making the organization ready for priority use cases such as fraud and AML monitoring, credit-risk modeling, and customer analytics.

‍

The design replaces a collection of disconnected on-premises components with one integrated platform from which scale, governance, and AI-readiness emerge by design rather than being retrofitted. Three pillars define it.

A Unified Medallion Lakehouse (Databricks + Azure)

One cloud-native Lakehouse on open Delta Lake, organized Bronze → Silver → Gold → Semantic and served through Databricks SQL, replaces the legacy ETL tool, the standalone warehouse, and the disconnected reporting pipelines that shared neither a metadata layer nor a security model.

Benefit: One elastic platform that scales from today’s workload to a much larger data estate.

Governance, Quality & Observability by Design (Unity Catalog)

Access control, automated lineage, audit logging, PII classification, row-level security, and column masking are structural properties of the platform, not retrofits; DLT expectations quarantine failing records; and Unity Catalog system tables expose pipeline health, access, and cost.

Benefit: Regulator-ready by default. The platform answers who accessed what, where a number came from, and what changed and when without manual effort, while bad data is stopped before it reaches a trusted report or an AI model.

An AI-Native Foundation (MLflow + Mosaic AI)

AI and ML are first-class consumers from day one: governed Gold and feature designs feed MLflow, the Feature Store, AutoML, and Model Serving, and Mosaic AI’s Agent Framework, AI Search, and AI Gateway govern generative AI the same way as any table.

Benefit: The governed data built now becomes the lineage-tracked, reproducible input for priority use cases such as fraud and AML monitoring, credit-risk modeling, and customer analytics so AI can be built quickly and defended under examination.

A Sound Investment: Outcome and What Comes Next

The engagement delivered a clear, regulator-ready architecture blueprint and a phased roadmap that de-risks a high-stakes migration off the legacy stack. The organization can retire legacy licensing along the way while establishing a scalable, governed, AI-ready foundation. The Foundation phase stands up the Databricks environment, Unity Catalog, networking, identity, and CI/CD and live-syncs the existing warehouse to keep reporting running, followed by progressive onboarding of the source systems and then AI activation. Compass and Databricks enables teams of all sizes to start with a small workload and grow towards a much larger AI-enabled estate.