Databricks platform lessons · databricks.wiki

Practical guidance for an Azure Databricks platform that teams manage as code. The guidance comes from production work and applies across organizations.

Plan before anyone builds#

Decide dev/test/prod and a workspace + catalog strategy up front. A workspace per environment lets you test infra changes (storage, compute policies, init scripts) in a lower environment before they hit prod.
env + medallion catalogs work well: each workspace gets bronze/silver/gold. Bronze is schema-per-source; silver and gold are schema-per-data-product.
Do not start with notebooks in an ungoverned workspace. Create the catalogs, groups, policies, and service principals before teams build workloads.

Two tools, one boundary: Terraform vs DABs#

Terraform answers "what does the workspace look like?" — networking, storage, workspace, Unity Catalog structure, grants, identity, cluster policies, account groups, service principals.
DABs answer "what runs inside the workspace?" — jobs, pipelines, notebooks, the compute and secrets tied to a workload. The asset's author (data engineering) owns the bundle.
Do not put workload resources in Terraform. Two repositories can cause duplicate work and Terraform state conflicts. See Terraform vs DABs.

Control Unity Catalog grant ownership#

databricks_grants (plural) is authoritative — it owns every grant on the object and removes anything it does not declare. DABs sets its own grants on catalogs/schemas when it deploys, so plural Terraform grants wipe them out on every apply. Use databricks_grant (singular) for objects that DABs also changes. See Unity Catalog grants.
Grant to groups, not users. Bind Entra ID (or Okta) groups to Databricks account groups and let Terraform grant on the group.

Set suitable compute policy defaults#

The default cluster auto-termination of 4320 minutes is not sensible. Override it in a cluster policy.
Size SQL warehouses for their workloads and give them a short timeout. They start and stop in seconds. Do not allow unrestricted compute.

Identity: machines run jobs, not people#

Run production jobs as service principals, one per environment, scoped to only the catalogs they need.
Manage access groups in Entra ID. Use Automatic Identity Management to create and sync the Databricks account groups. Use SCIM only when AIM does not apply.
For CI/CD, use workload identity federation (OIDC) instead of long-lived PAT or service-principal secrets.

Separate secrets by use#

Databricks secret scopes for anything read inside a job's context.
An external store (Key Vault, GitHub Secrets, Doppler) for secrets used by CI/CD. For GitHub-based CI, GitHub Secrets is usually enough.

Plan network capacity before deployment#

In Azure, you cannot resize a subnet after you assign it to a Databricks workspace. You must replace the workspace to change the subnet. A /21 VNet has provided sufficient capacity in production environments, but teams must calculate their own requirements.
Clusters do not need public IP addresses when they have an egress route through a NAT gateway or firewall.
Azure has two tiers, Standard and Premium. There is no Enterprise tier — Enterprise is an AWS tier. An earlier Reddit source incorrectly applied the AWS tiers to Azure. Premium is the prerequisite for Private Link, not the expensive alternative to it: every Private Link leg on Azure requires the Premium plan.
Standard tier is going away. New workspaces have been Premium-only since 2026-04-01, and any Standard workspace left on 2026-10-01 is upgraded to Premium automatically. Check the workspace tier in the account console before this date.
The cost argument underneath the Reddit comment still holds, just without the tier framing: every non-serverless cluster pulls a ~15 GB image from the control plane at start, and that NAT data-processing cost adds up. Private Link avoids it. Price Private Link endpoints against your NAT egress bill.

Sources for the tier correction, checked 2026-08-08: Manage your subscription, End of life for Standard tier workspaces, Private Link concepts.

Cost and observability are platform work#

Many system tables are not enabled by default. Turn them on to emit compute usage to UC tables and build real cost dashboards beyond the built-in reports. See the system tables documentation.
Watch the SKU. PREMIUM_SERVERLESS_REAL_TIME_INFERENCE is one of the most expensive DBU classes — confirm any workload on it is intentional and budgeted.

Cost spike triage (first 15 minutes)#

Use system tables to investigate a cost increase. Check these items in order.

Spend by SKU (last 30 days). Query system.billing.usage grouped by sku_name. One expensive SKU (model serving, serverless inference, oversized SQL) usually explains the jump.
Idle / always-on classic compute. Query system.compute.clusters for auto_termination_minutes of 0 or NULL on live clusters. That is the classic leak. Override the 4320-minute default in cluster policy before the next incident.
Jobs on all-purpose clusters. For high-spend job IDs, check existing_cluster_id on tasks. Job compute or serverless is usually cheaper and isolates failures.
Tags and budget policies. Untagged serverless has no chargeback path. Budget policies exist so serverless notebooks and jobs still land tags in billing — give every user at least one. Detail in Community best practices.

If system tables are disabled, enable system.billing and system.compute first; the CLI fallback is slower and incomplete.

Onboarding a brownfield platform to Terraform#

Import in phases — networking, then storage, then workspace, then grants, then cleanup — so you never interrupt running workloads.
Manual changes to prod will happen (a grant here, a cluster policy there). They have to be reverse-engineered back into Terraform. Codify them and move to PR-based change control so the drift stops.

Operational discipline#

PR-driven Terraform with rendered plan comments, environment CODEOWNERS, and concurrency-locked applies.
Pin provider versions; SHA-pin GitHub Actions; retain plan artifacts (90 days) for audit.

Limit the number of services#

Do not add ADF when DABs already orchestrates the Databricks jobs. Do not add Purview when Unity Catalog already supplies the required catalog. Add a service only for a defined gap.

User interface#

Databricks has a dark theme in the user settings.