Platform architecture

Databricks community and industry best practices

synthesiscuratedofficial / community / practitioner evidenceazure13 min readreview 14 days old

This page combines official Databricks recommendations with practitioner guidance. It focuses on Azure, but much of the guidance applies to other clouds. Use the source links to verify facts.

Topics: liquid clustering, ZORDER, predictive optimization, Photon, serverless, classic compute, cluster policies, FinOps, DABs, Private Link, Lakeflow, and table constraints.

Compiled: 2026-06-24. The page identifies GA and Public Preview features. Confirm each (verify) item in the current documentation before you use it.

Refreshed: 2026-08-08 — catalog design, managed-table retention, ABAC status, compute sizing, serverless limits, predictive optimization, and secure cluster connectivity checked against current docs. The rest of the file still dates from the compile.

Well-Architected Lakehouse Framework#

Five pillars match the Azure Well-Architected Framework. Two pillars cover data and AI governance, interoperability, and usability.

The Well-Architected Lakehouse page contains details from the seven Microsoft Learn pillar pages. This page adds wider vendor and practitioner guidance.

Unity Catalog governance#

UC is the governance layer and is on by default for workspaces created after 2023-11-09. Object model is a three-level namespace: catalog.schema.object.

Catalog and schema design. Databricks recommends domain-based catalogs, such as sales or finance. Environment-based, project-based, and combined patterns also work when those boundaries match the required isolation. Use catalogs as the primary isolation unit. Do not use a fixed catalog count. Create a catalog for a real domain, environment, team, or project boundary. See Databricks platform lessons for an environment and medallion catalog pattern.

Medallion. Current governance guidance puts bronze, silver, and gold schemas inside a domain catalog. Use a different layout only when the lifecycle layers need different isolation.

Prefer managed assets. Databricks recommends managed tables and volumes for new tables and most use cases. The platform supplies compaction, optimization, metadata caching, file sizing, and feature upgrades. Managed tables use Delta or Iceberg. You can use UNDROP for approximately seven days after you drop a managed table. You can configure this period at the catalog, schema, or object level. External engines can read managed tables through Unity REST or Iceberg REST.

Use external assets when you cannot move Hive metastore data. Also use them for unsupported formats, special recovery requirements, or required external writers. Limit external access to read operations when possible. Route write operations through Databricks. Use one external location for each schema at the highest common path.

Tags and ABAC. ABAC policies, governed tags, and data classification became GA in April 2026. Read PII and ABAC governance for feature dates and session-user changes. Governed tags define allowed values and assignment permissions at the account level. They apply to all workspaces and metastores in the account. Use them for classification, discovery, cost attribution, and access control. Tags inherit from a catalog to a schema and then to a table. Apply column tags directly. Do not put sensitive data in tag names or values. Compute resources use separate compute tags for billing.

Lineage. Unity Catalog captures table and column lineage automatically. It has no data before 2024-09-01. An object rename removes its prior lineage. Path references and UDFs can prevent column lineage capture. Submit jobs capture object lineage but do not capture the run link.

Delta Lake and table optimization#

Liquid clustering is GA for Delta tables on Databricks Runtime 15.4 LTS and later. Iceberg support is in Public Preview on 16.4 LTS and later. Use it instead of partitioning and ZORDER for new tables. You can change clustering keys without a data rewrite. Select no more than four keys from frequent query filters. Each key must have statistics. Do not combine CLUSTER BY with PARTITIONED BY.

Z-order is legacy. Migration is low-friction: stop running OPTIMIZE ... ZORDER BY, reuse the same columns as clustering keys, and run plain OPTIMIZE. On DBR 18.1+ there's an in-place ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY [(...) | AUTO] that converts a partitioned table with minimal read/write downtime. CLUSTER BY AUTO seeds from the existing partition columns and lets predictive optimization take over (UC managed tables only). After enabling clustering or changing keys, run OPTIMIZE FULL once to recluster existing data; subsequent OPTIMIZE runs are incremental.

Automatic liquid clustering uses query history to select and change keys. It changes keys only when the predicted savings exceed the clustering cost. It can omit keys for small, well-clustered, or infrequently queried tables.

Predictive Optimization runs OPTIMIZE and VACUUM for Unity Catalog managed tables. Disable scheduled optimization jobs before you enable it. Its VACUUM process reads the Delta log instead of a full directory list.

OPTIMIZE / VACUUM / file sizing. Predictive Optimization runs OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables. Check its status before you create scheduled jobs. Without it, schedule OPTIMIZE for the workload frequency. VACUUM removes unreferenced files. Keep the default seven-day retention period unless a requirement permits a different value. Let the platform manage file sizes on managed tables.

Deletion vectors are active by default with liquid clustering. They mark deleted rows without a file rewrite. They also enable row-level concurrency. Keep them active unless an old reader does not support them. Liquid clustering requires Delta writer version 7 and reader version 3.

Performance, compute sizing, and serverless#

Photon is the native vectorized C++ engine, Spark-API compatible, no code changes. On by default on all SQL warehouses and on serverless. Most benefit on SQL and DataFrame workloads with wide transformations — joins, aggregations, large scans. Minimal benefit on simple sub-2-second ETL. Evaluate recurring jobs for whether Photon is both faster and cheaper.

Serverless and classic compute. Databricks recommends serverless compute for new workloads. It starts quickly, scales automatically, and includes Photon. It has no public IP addresses. Use classic compute for specific instance types, init scripts, custom configurations, or reserved capacity.

Classic compute sizing. Size from executor cores, executor memory, and local storage, not only worker count. Start with general-purpose workers, then read the workload evidence. For a shuffle-heavy job, try fewer, larger workers to reduce network transfer. Increase worker memory when the Spark UI shows spill or out-of-memory errors. Add workers when the job needs more parallel tasks. Measure the next run before you keep the change.

Serverless compatibility check. Serverless notebooks and jobs use Spark Connect. They do not support R, RDD APIs, the Spark UI, Spark logs, compute policies, init scripts, instance pools, or most Spark settings. A serverless job can run for no more than seven days. For Structured Streaming, use Trigger.AvailableNow() or a Lakeflow pipeline. Serverless jobs do not support processing-time or continuous Spark triggers. Use classic compute when the workload needs one of these features, or change the workload first.

Cluster policies control compute configuration. Enforce size standards, restrict expensive instance types, require autoscaling, and limit worker counts. Override the 4320-minute default auto-termination value. Add libraries through the policy instead of init scripts.

Autoscaling. Set the minimum and maximum worker counts within the workspace quota. Use workload evidence to select the values. Always use auto-termination with autoscaling.

Spot instances and pools. Use spot instances for fault-tolerant workloads. Spark reschedules tasks after the cloud provider revokes an instance. Pools keep virtual machines ready and reduce compute start time. Idle pool instances have an infrastructure cost. Size each pool for measured concurrency.

Compute configuration. Avoid init scripts and compute-scoped libraries when serverless is a possible target. Do not hardcode Spark settings without workload evidence. Use Unity Catalog volumes instead of compute-local paths or DBFS mounts. Turn on adaptive query execution (AQE).

Cost and FinOps#

Use system tables. system.billing.usage contains resource, identity, and tag data for each usage record. Join it to system.billing.list_prices for prices. Join it to system.access.workspaces_latest for workspace data. Read the system tables documentation for the available schemas and access requirements.

Add cost tags. Add business unit and project tags to workspaces, clusters, warehouses, and pools. Billing records contain these values in custom_tags. Use governed tags to keep the taxonomy consistent.

Budget policies add cost tags to serverless resources. Give each user at least one budget policy. Account and workspace budgets track spend and send alerts. Confirm the current alert behavior before you set the thresholds.

SQL warehouse sizing. Two independent levers: cluster size (scale up) for query complexity and disk spills — if the query profile shows spill, size up; cluster count (scale out, max-clusters) for concurrency. Start Small/Medium with autoscaling and separate warehouses by workload (dev small, prod larger; don't share one warehouse across mismatched patterns). Prefer serverless warehouses: they start and scale in seconds, scale down sooner than classic, and Intelligent Workload Management right-sizes resources per query — so you get instant availability and aggressive idle termination at once. Set short auto-stop (1 minute is fine for dev).

Serverless costs. Serverless compute has no idle infrastructure cost. Usage charges still apply to long queries and active warehouses. Use budgets and policies to control this cost. Read Databricks platform lessons before you approve an expensive SKU.

CI/CD and DevOps#

Declarative Automation Bundles (DABs; formerly Databricks Asset Bundles) are the standard. dbx is deprecated and no longer maintained. Bundles define jobs, pipelines, and related resources as source files. Read Terraform and DABs ownership boundaries before you assign resource ownership.

  • Repository structure. Use one repository when code and bundle configuration have the same release cycle. Use separate repositories when teams have independent release cycles.
  • Versioned artifacts. Upload with Git commit hashes for traceability and rollback. Parameterize environment-specific values (cluster size, secrets) instead of hardcoding.
  • Environment isolation. Separate dev/staging/prod workspaces; bundle deployment modes (dev vs prod) handle the differences.
  • Testing. Develop notebooks/tests locally or in-workspace; databricks bundle validate for config; pytest (Python wheels) / JUnit (JARs) for units; chispa for Spark DataFrame assertions; integration tests for full pipelines. Lint with Pylint plus the Databricks Labs pylint plugin.
  • Git folders. Use Git folders to control notebook versions when the team does not use bundle deployment pipelines.
  • Known gaps (2025 list — verify against current docs): lifecycle hooks (pre/post-deploy scripts), dependency management beyond wheels, and failure output vs Terraform. The product moved since that note; re-check before you design around a gap.

Security baseline (Azure)#

This section gives the Azure security baseline. See Databricks platform lessons for subnet sizing and Private Link cost guidance.

  • No public IPs — secure cluster connectivity (NPIP). Enable it on every classic workspace. It is the default for new workspaces and will become mandatory. A new workspace can use the Databricks-managed VNet or VNet injection. An existing workspace must use VNet injection before you add SCC. Serverless has no public IPs but does not use SCC.
  • Private Link, three independent legs: inbound/front-end (users → workspace), back-end/classic (classic compute → control plane), and outbound/serverless (serverless → your Azure resources via NCC private endpoints). Enforce private connectivity to make the workspace reject public connections. The "Isolated environment" reference architecture adds VPN/inbound-Private-Link-only access plus a required egress firewall — that's the pattern for HIPAA/PCI/FedRAMP-style workloads.
  • Egress note (Azure-specific). After 2026-03-31, new Azure VNets default to no outbound internet, so new workspaces need an explicit egress method (NAT gateway). Existing workspaces are unaffected.
  • TLS pinning. Do not run TLS inspection (decrypt/re-encrypt) on cluster ↔ control-plane traffic — certificate pinning will fail the cluster. Use service-endpoint policies or a private package repo (Artifactory/Nexus) to cut firewall rules and artifact-download egress cost.
  • Secrets. Use Databricks secret scopes for values that a job reads. Use an external store for deployment secrets. Read Databricks platform lessons.
  • Unity Catalog. Do not use workspace table ACLs, DBFS mounts, or the Hive metastore for new governance. Use Unity Catalog volumes. Grant access to groups, not users. Read Unity Catalog grants.

Data quality and reliability#

Lakeflow expectations are the primary data-quality mechanism in declarative pipelines (Lakeflow Spark Declarative Pipelines, formerly DLT). An expectation is a SQL boolean per row with one of three violation policies:

  • warn (default) — invalid rows are written and flagged in metrics.
  • drop (ON VIOLATION DROP ROW) — invalid rows dropped before write, count logged.
  • fail (ON VIOLATION FAIL UPDATE) — stops the flow on the first bad record; needs manual intervention.

Metrics emit to the pipeline event log regardless of policy, so track quality trends and alert on regressions by querying event_log(). For records you don't want to silently drop, use the quarantine pattern: two flows, one writing clean rows to the target and one routing failures to a separate table for investigation and reprocessing.

Table constraints (outside pipelines too): NOT NULL and CHECK are enforced on Delta tables (CHECK can do regex via REGEXP/RLIKE and range checks). Primary-key and foreign-key constraints are informational only — not enforced, used for documentation and query optimization. In UC-enabled pipelines you can declare PK/FK on streaming tables and materialized views; to validate PK uniqueness for real, use an EXPECT (num_entries = 1) expectation over a GROUP BY.

Pipeline defaults worth adopting: serverless for new pipelines (enhanced autoscaling, UC and lineage on by default); CLUSTER BY over PARTITIONED BY; Auto Loader for incremental file ingestion; predictive optimization runs OPTIMIZE/VACUUM on pipeline tables automatically. Use the event log plus event hooks (Python functions on pipeline events) to push failures and quality breaches to Slack/PagerDuty.

Sources#

Referenced by