# Databricks community and industry best practices

This page combines official Databricks recommendations with practitioner guidance. It focuses on
Azure, but much of the guidance applies to other clouds. Use the source links to verify facts.

**Topics:** liquid clustering, ZORDER, predictive optimization, Photon, serverless, classic
compute, cluster policies, FinOps, DABs, Private Link, Lakeflow, and table constraints.

Compiled: 2026-06-24. The page identifies GA and Public Preview features. Confirm each `(verify)`
item in the current documentation before you use it.

Refreshed: 2026-08-08 — catalog design, managed-table retention, ABAC status,
compute sizing, serverless limits, predictive optimization, and secure cluster
connectivity checked against current docs. The rest of the file still dates from
the compile.

## Well-Architected Lakehouse Framework

Five pillars match the Azure Well-Architected Framework. Two pillars cover data and AI
governance, interoperability, and usability.

The [Well-Architected Lakehouse](well-architected.md) page contains details from the seven
Microsoft Learn pillar pages. This page adds wider vendor and practitioner guidance.

## Unity Catalog governance

UC is the governance layer and is on by default for workspaces created after 2023-11-09. Object model is a three-level namespace: `catalog.schema.object`.

**Catalog and schema design.** Databricks recommends domain-based catalogs, such as `sales` or
`finance`. Environment-based, project-based, and combined patterns also work when those
boundaries match the required isolation. Use catalogs as the primary isolation unit. Do not use
a fixed catalog count. Create a catalog for a real domain, environment, team, or project boundary.
See [Databricks platform lessons](../lessons-learned/platform-lessons.md) for an environment and
medallion catalog pattern.

**Medallion.** Current governance guidance puts bronze, silver, and gold schemas inside a domain
catalog. Use a different layout only when the lifecycle layers need different isolation.

**Prefer managed assets.** Databricks recommends managed tables and volumes for new tables and
most use cases. The platform supplies compaction, optimization, metadata caching, file sizing,
and feature upgrades. Managed tables use Delta or Iceberg. You can use `UNDROP` for approximately
seven days after you drop a managed table. You can configure this period at the catalog, schema,
or object level. External engines can read managed tables through Unity REST or Iceberg REST.

Use external assets when you cannot move Hive metastore data. Also use them for unsupported
formats, special recovery requirements, or required external writers. Limit external access to
read operations when possible. Route write operations through Databricks. Use one external
location for each schema at the highest common path.

**Tags and ABAC.** ABAC policies, governed tags, and data classification became GA in April 2026.
Read [PII and ABAC governance](governance-pii-abac.md) for feature dates and session-user changes.
Governed tags define allowed values and assignment permissions at the account level. They apply
to all workspaces and metastores in the account. Use them for classification, discovery, cost
attribution, and access control. Tags inherit from a catalog to a schema and then to a table.
Apply column tags directly. Do not put sensitive data in tag names or values. Compute resources
use separate compute tags for billing.

**Lineage.** Unity Catalog captures table and column lineage automatically. It has no data before
2024-09-01. An object rename removes its prior lineage. Path references and UDFs can prevent
column lineage capture. Submit jobs capture object lineage but do not capture the run link.

## Delta Lake and table optimization

**Liquid clustering** is GA for Delta tables on Databricks Runtime 15.4 LTS and later. Iceberg
support is in Public Preview on 16.4 LTS and later. Use it instead of partitioning and ZORDER for
new tables. You can change clustering keys without a data rewrite. Select no more than four keys
from frequent query filters. Each key must have statistics. Do not combine `CLUSTER BY` with
`PARTITIONED BY`.

**Z-order is legacy.** Migration is low-friction: stop running `OPTIMIZE ... ZORDER BY`, reuse the same columns as clustering keys, and run plain `OPTIMIZE`. On DBR 18.1+ there's an in-place `ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY [(...) | AUTO]` that converts a partitioned table with minimal read/write downtime. `CLUSTER BY AUTO` seeds from the existing partition columns and lets predictive optimization take over (UC managed tables only). After enabling clustering or changing keys, run `OPTIMIZE FULL` once to recluster existing data; subsequent `OPTIMIZE` runs are incremental.

**Automatic liquid clustering** uses query history to select and change keys. It changes keys
only when the predicted savings exceed the clustering cost. It can omit keys for small,
well-clustered, or infrequently queried tables.

**Predictive Optimization** runs `OPTIMIZE` and `VACUUM` for Unity Catalog managed tables.
Disable scheduled optimization jobs before you enable it. Its `VACUUM` process reads the Delta
log instead of a full directory list.

**OPTIMIZE / VACUUM / file sizing.** Predictive Optimization runs `OPTIMIZE`, `VACUUM`, and `ANALYZE` on Unity Catalog managed tables. Check its status before you create scheduled jobs. Without it, schedule `OPTIMIZE` for the workload frequency. `VACUUM` removes unreferenced files. Keep the default seven-day retention period unless a requirement permits a different value. Let the platform manage file sizes on managed tables.

**Deletion vectors** are active by default with liquid clustering. They mark deleted rows without
a file rewrite. They also enable row-level concurrency. Keep them active unless an old reader
does not support them. Liquid clustering requires Delta writer version 7 and reader version 3.

## Performance, compute sizing, and serverless

**Photon** is the native vectorized C++ engine, Spark-API compatible, no code changes. On by default on all SQL warehouses and on serverless. Most benefit on SQL and DataFrame workloads with wide transformations — joins, aggregations, large scans. Minimal benefit on simple sub-2-second ETL. Evaluate recurring jobs for whether Photon is both faster and cheaper.

**Serverless and classic compute.** Databricks recommends serverless compute for new workloads.
It starts quickly, scales automatically, and includes Photon. It has no public IP addresses.
Use classic compute for specific instance types, init scripts, custom configurations, or
reserved capacity.

**Classic compute sizing.** Size from executor cores, executor memory, and local
storage, not only worker count. Start with general-purpose workers, then read the
workload evidence. For a shuffle-heavy job, try fewer, larger workers to reduce
network transfer. Increase worker memory when the Spark UI shows spill or
out-of-memory errors. Add workers when the job needs more parallel tasks. Measure
the next run before you keep the change.

**Serverless compatibility check.** Serverless notebooks and jobs use Spark
Connect. They do not support R, RDD APIs, the Spark UI, Spark logs, compute
policies, init scripts, instance pools, or most Spark settings. A serverless job
can run for no more than seven days. For Structured Streaming, use
`Trigger.AvailableNow()` or a Lakeflow pipeline. Serverless jobs do not support
processing-time or continuous Spark triggers. Use classic compute when the
workload needs one of these features, or change the workload first.

**Cluster policies** control compute configuration. Enforce size standards, restrict expensive instance types, require autoscaling, and limit worker counts. Override the 4320-minute default auto-termination value. Add libraries through the policy instead of init scripts.

**Autoscaling.** Set the minimum and maximum worker counts within the workspace quota. Use
workload evidence to select the values. Always use auto-termination with autoscaling.

**Spot instances and pools.** Use spot instances for fault-tolerant workloads. Spark reschedules
tasks after the cloud provider revokes an instance. Pools keep virtual machines ready and reduce
compute start time. Idle pool instances have an infrastructure cost. Size each pool for measured
concurrency.

**Compute configuration.** Avoid init scripts and compute-scoped libraries when serverless is a
possible target. Do not hardcode Spark settings without workload evidence. Use Unity Catalog
volumes instead of compute-local paths or DBFS mounts. Turn on adaptive query execution (AQE).

## Cost and FinOps

**Use system tables.** `system.billing.usage` contains resource, identity, and tag data for each
usage record. Join it to `system.billing.list_prices` for prices. Join it to
`system.access.workspaces_latest` for workspace data. Read the
[system tables documentation](https://learn.microsoft.com/azure/databricks/admin/system-tables/)
for the available schemas and access requirements.

**Add cost tags.** Add business unit and project tags to workspaces, clusters, warehouses, and
pools. Billing records contain these values in `custom_tags`. Use governed tags to keep the
taxonomy consistent.

**Budget policies** add cost tags to serverless resources. Give each user at least one budget
policy. Account and workspace budgets track spend and send alerts. Confirm the current alert
behavior before you set the thresholds.

**SQL warehouse sizing.** Two independent levers: **cluster size** (scale up) for query complexity and disk spills — if the query profile shows spill, size up; **cluster count** (scale out, max-clusters) for concurrency. Start Small/Medium with autoscaling and separate warehouses by workload (dev small, prod larger; don't share one warehouse across mismatched patterns). Prefer serverless warehouses: they start and scale in seconds, scale down sooner than classic, and Intelligent Workload Management right-sizes resources per query — so you get instant availability and aggressive idle termination at once. Set short auto-stop (1 minute is fine for dev).

**Serverless costs.** Serverless compute has no idle infrastructure cost. Usage charges still
apply to long queries and active warehouses. Use budgets and policies to control this cost. Read
[Databricks platform lessons](../lessons-learned/platform-lessons.md) before you approve an
expensive SKU.

## CI/CD and DevOps

**Declarative Automation Bundles (DABs; formerly Databricks Asset Bundles) are the standard.
dbx is deprecated** and no longer maintained. Bundles define jobs, pipelines, and related
resources as source files. Read [Terraform and DABs ownership boundaries](terraform-vs-dabs.md)
before you assign resource ownership.

- **Repository structure.** Use one repository when code and bundle configuration have the same
  release cycle. Use separate repositories when teams have independent release cycles.
- **Versioned artifacts.** Upload with Git commit hashes for traceability and rollback. Parameterize environment-specific values (cluster size, secrets) instead of hardcoding.
- **Environment isolation.** Separate dev/staging/prod workspaces; bundle deployment modes (dev vs prod) handle the differences.
- **Testing.** Develop notebooks/tests locally or in-workspace; `databricks bundle validate` for config; pytest (Python wheels) / JUnit (JARs) for units; chispa for Spark DataFrame assertions; integration tests for full pipelines. Lint with Pylint plus the Databricks Labs pylint plugin.
- **Git folders.** Use Git folders to control notebook versions when the team does not use bundle
  deployment pipelines.
- **Known gaps (2025 list — verify against current docs):** lifecycle hooks (pre/post-deploy scripts), dependency management beyond wheels, and failure output vs Terraform. The product moved since that note; re-check before you design around a gap.

## Security baseline (Azure)

This section gives the Azure security baseline. See
[Databricks platform lessons](../lessons-learned/platform-lessons.md) for subnet sizing and
Private Link cost guidance.

- **No public IPs — secure cluster connectivity (NPIP).** Enable it on every classic workspace. It is the default for new workspaces and will become mandatory. A new workspace can use the Databricks-managed VNet or VNet injection. An existing workspace must use VNet injection before you add SCC. Serverless has no public IPs but does not use SCC.
- **Private Link, three independent legs:** inbound/front-end (users → workspace), back-end/classic (classic compute → control plane), and outbound/serverless (serverless → your Azure resources via NCC private endpoints). Enforce private connectivity to make the workspace reject public connections. The "Isolated environment" reference architecture adds VPN/inbound-Private-Link-only access plus a required egress firewall — that's the pattern for HIPAA/PCI/FedRAMP-style workloads.
- **Egress note (Azure-specific).** After 2026-03-31, new Azure VNets default to no outbound internet, so new workspaces need an explicit egress method (NAT gateway). Existing workspaces are unaffected.
- **TLS pinning.** Do not run TLS inspection (decrypt/re-encrypt) on cluster ↔ control-plane traffic — certificate pinning will fail the cluster. Use service-endpoint policies or a private package repo (Artifactory/Nexus) to cut firewall rules and artifact-download egress cost.
- **Secrets.** Use Databricks secret scopes for values that a job reads. Use an external store for
  deployment secrets. Read [Databricks platform lessons](../lessons-learned/platform-lessons.md).
- **Unity Catalog.** Do not use workspace table ACLs, DBFS mounts, or the Hive metastore for new
  governance. Use Unity Catalog volumes. Grant access to groups, not users. Read
  [Unity Catalog grants](unity-catalog-grants.md).

## Data quality and reliability

**Lakeflow expectations** are the primary data-quality mechanism in declarative pipelines (Lakeflow Spark Declarative Pipelines, formerly DLT). An expectation is a SQL boolean per row with one of three violation policies:

- **warn** (default) — invalid rows are written and flagged in metrics.
- **drop** (`ON VIOLATION DROP ROW`) — invalid rows dropped before write, count logged.
- **fail** (`ON VIOLATION FAIL UPDATE`) — stops the flow on the first bad record; needs manual intervention.

Metrics emit to the pipeline event log regardless of policy, so track quality trends and alert on regressions by querying `event_log()`. For records you don't want to silently drop, use the **quarantine pattern**: two flows, one writing clean rows to the target and one routing failures to a separate table for investigation and reprocessing.

**Table constraints** (outside pipelines too): `NOT NULL` and `CHECK` are **enforced** on Delta tables (`CHECK` can do regex via `REGEXP`/`RLIKE` and range checks). Primary-key and foreign-key constraints are **informational only** — not enforced, used for documentation and query optimization. In UC-enabled pipelines you can declare PK/FK on streaming tables and materialized views; to validate PK uniqueness for real, use an `EXPECT (num_entries = 1)` expectation over a `GROUP BY`.

**Pipeline defaults worth adopting:** serverless for new pipelines (enhanced autoscaling, UC and lineage on by default); `CLUSTER BY` over `PARTITIONED BY`; Auto Loader for incremental file ingestion; predictive optimization runs OPTIMIZE/VACUUM on pipeline tables automatically. Use the event log plus event hooks (Python functions on pipeline events) to push failures and quality breaches to Slack/PagerDuty.

## Sources

- Well-Architected Lakehouse: https://learn.microsoft.com/azure/databricks/lakehouse-architecture/well-architected
- Well-architected intro / deployment guide: https://learn.microsoft.com/azure/databricks/lakehouse-architecture/
- Operational excellence best practices: https://learn.microsoft.com/azure/databricks/lakehouse-architecture/operational-excellence/best-practices
- Reliability best practices: https://learn.microsoft.com/azure/databricks/lakehouse-architecture/reliability/best-practices
- Performance efficiency best practices: https://learn.microsoft.com/azure/databricks/lakehouse-architecture/performance-efficiency/best-practices
- Cost optimization best practices: https://learn.microsoft.com/azure/databricks/lakehouse-architecture/cost-optimization/best-practices
- Azure WAF service guide for Databricks: https://learn.microsoft.com/azure/well-architected/service-guides/azure-databricks
- Introducing the Well-Architected Lakehouse (blog): https://www.databricks.com/blog/introducing-well-architected-data-lakehouse-databricks
- Unity Catalog best practices: https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/best-practices
- Data and AI governance best practices: https://learn.microsoft.com/azure/databricks/lakehouse-architecture/data-governance/best-practices
- Managed vs external assets: https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/managed-versus-external
- UC table types: https://learn.microsoft.com/azure/databricks/tables/types
- Design Unity Catalog architecture (deployment guide): https://learn.microsoft.com/azure/databricks/lakehouse-architecture/deployment-guide/unity-catalog
- Governed tags: https://learn.microsoft.com/azure/databricks/admin/governed-tags/
- Apply tags to UC objects: https://learn.microsoft.com/azure/databricks/database-objects/tags
- ABAC core concepts: https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/core-concepts
- ABAC, governed tags, and data classification GA (blog): https://www.databricks.com/blog/abac-row-filtering-and-column-masking-policies-governed-tags-and-data-classification-are-now
- UC managed tables, retention and RETAIN DROPPED: https://learn.microsoft.com/azure/databricks/tables/managed
- Predictive optimization: https://learn.microsoft.com/azure/databricks/optimizations/predictive-optimization
- Data lineage in UC: https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/data-lineage
- Liquid clustering: https://learn.microsoft.com/azure/databricks/tables/clustering
- Announcing Automatic Liquid Clustering (blog): https://www.databricks.com/blog/announcing-automatic-liquid-clustering
- Predictive optimization at scale (blog): https://www.databricks.com/blog/predictive-optimization-scale-year-innovation-and-whats-next
- Classic compute configuration best practices: https://learn.microsoft.com/azure/databricks/compute/cluster-config-best-practices
- Serverless compute limitations: https://learn.microsoft.com/azure/databricks/compute/serverless/limitations
- Photon: https://learn.microsoft.com/azure/databricks/compute/photon
- SQL warehouse sizing, scaling, queuing: https://learn.microsoft.com/azure/databricks/compute/sql-warehouse/warehouse-behavior
- Billable usage system table: https://docs.databricks.com/aws/en/admin/system-tables/billing
- Monitor costs using system tables: https://docs.databricks.com/aws/en/admin/usage/system-tables
- From Chaos to Control: cost maturity (blog): https://www.databricks.com/blog/chaos-control-cost-maturity-journey-databricks
- Declarative Automation Bundles: https://learn.microsoft.com/azure/databricks/dev-tools/bundles/
- CI/CD guidance: https://learn.microsoft.com/azure/databricks/dev-tools/ci-cd/
- Secure cluster connectivity: https://learn.microsoft.com/azure/databricks/security/network/classic/secure-cluster-connectivity
- Azure Private Link concepts: https://learn.microsoft.com/azure/databricks/security/network/concepts/private-link
- Isolated environment architecture: https://learn.microsoft.com/azure/databricks/security/network/deployment-architecture/isolated-environment
- Hardened connectivity architecture: https://learn.microsoft.com/azure/databricks/security/network/deployment-architecture/hardened-connectivity
- Manage data quality with expectations: https://learn.microsoft.com/azure/databricks/ldp/expectations
- Lakeflow pipelines best practices: https://learn.microsoft.com/azure/databricks/ldp/best-practices
- Clean and validate data / constraints: https://learn.microsoft.com/azure/databricks/transform/validate
- Constraints on Azure Databricks: https://learn.microsoft.com/azure/databricks/tables/constraints