# Databricks platform lessons

Practical guidance for an Azure Databricks platform that teams manage as code. The guidance
comes from production work and applies across organizations.

## Plan before anyone builds

- Decide dev/test/prod and a workspace + catalog strategy up front. A workspace
  per environment lets you test infra changes (storage, compute policies, init
  scripts) in a lower environment before they hit prod.
- `env + medallion` catalogs work well: each workspace gets bronze/silver/gold.
  Bronze is schema-per-source; silver and gold are schema-per-data-product.
- Do not start with notebooks in an ungoverned workspace. Create the catalogs, groups, policies,
  and service principals before teams build workloads.

## Two tools, one boundary: Terraform vs DABs

- **Terraform** answers "what does the workspace look like?" — networking,
  storage, workspace, Unity Catalog structure, grants, identity, cluster
  policies, account groups, service principals.
- **DABs** answer "what runs inside the workspace?" — jobs, pipelines,
  notebooks, the compute and secrets tied to a workload. The asset's author
  (data engineering) owns the bundle.
- Do not put workload resources in Terraform. Two repositories can cause duplicate work and
  Terraform state conflicts. See [Terraform vs DABs](../best-practices/terraform-vs-dabs.md).

## Control Unity Catalog grant ownership

- `databricks_grants` (plural) is **authoritative** — it owns every grant on the
  object and removes anything it does not declare. DABs sets its own grants on
  catalogs/schemas when it deploys, so plural Terraform grants wipe them out on
  every apply. Use `databricks_grant` (singular) for objects that DABs also changes.
  See [Unity Catalog grants](../best-practices/unity-catalog-grants.md).
- Grant to **groups, not users**. Bind Entra ID (or Okta) groups to Databricks
  account groups and let Terraform grant on the group.

## Set suitable compute policy defaults

- The default cluster auto-termination of **4320 minutes** is not sensible.
  Override it in a cluster policy.
- Size SQL warehouses for their workloads and give them a short timeout. They start and stop
  in seconds. Do not allow unrestricted compute.

## Identity: machines run jobs, not people

- Run production jobs as **service principals**, one per environment, scoped to
  only the catalogs they need.
- Manage access groups in Entra ID. Use Automatic Identity Management to create
  and sync the Databricks account groups. Use SCIM only when AIM does not apply.
- For CI/CD, use workload identity federation (OIDC) instead of long-lived PAT
  or service-principal secrets.

## Separate secrets by use

- Databricks secret scopes for anything read inside a job's context.
- An external store (Key Vault, GitHub Secrets, Doppler) for secrets used by
  CI/CD. For GitHub-based CI, GitHub Secrets is usually enough.

## Plan network capacity before deployment

- In Azure, you **cannot resize a subnet after you assign it** to a Databricks workspace.
  You must replace the workspace to change the subnet. A `/21` VNet has provided sufficient
  capacity in production environments, but teams must calculate their own requirements.
- Clusters do not need public IP addresses when they have an egress route through a NAT
  gateway or firewall.
- **Azure has two tiers, Standard and Premium. There is no Enterprise tier** —
  Enterprise is an AWS tier. An earlier Reddit source incorrectly applied the AWS tiers to Azure.
  Premium is the **prerequisite** for Private Link, not the expensive
  alternative to it: every Private Link leg on Azure requires the Premium plan.
- Standard tier is going away. New workspaces have been Premium-only since
  **2026-04-01**, and any Standard workspace left on **2026-10-01** is upgraded
  to Premium automatically. Check the workspace tier in the account console before this date.
- The cost argument underneath the Reddit comment still holds, just without the
  tier framing: every non-serverless cluster pulls a ~15 GB image from the
  control plane at start, and that NAT data-processing cost adds up. Private
  Link avoids it. Price Private Link endpoints against your NAT egress bill.

Sources for the tier correction, checked 2026-08-08:
[Manage your subscription](https://learn.microsoft.com/azure/databricks/admin/account-settings/account),
[End of life for Standard tier workspaces](https://learn.microsoft.com/azure/databricks/admin/account-settings/standard-tier),
[Private Link concepts](https://learn.microsoft.com/azure/databricks/security/network/concepts/privatelink-concepts).

## Cost and observability are platform work

- Many **system tables are not enabled by default**. Turn them on to emit
  compute usage to UC tables and build real cost dashboards beyond the built-in
  reports. See the
  [system tables documentation](https://learn.microsoft.com/azure/databricks/admin/system-tables/).
- Watch the SKU. `PREMIUM_SERVERLESS_REAL_TIME_INFERENCE` is one of the most
  expensive DBU classes — confirm any workload on it is intentional and budgeted.

## Cost spike triage (first 15 minutes)

Use system tables to investigate a cost increase. Check these items in order.

1. **Spend by SKU (last 30 days).** Query `system.billing.usage` grouped by
   `sku_name`. One expensive SKU (model serving, serverless inference, oversized
   SQL) usually explains the jump.
2. **Idle / always-on classic compute.** Query `system.compute.clusters` for
   `auto_termination_minutes` of `0` or `NULL` on live clusters. That is the
   classic leak. Override the 4320-minute default in cluster policy before the
   next incident.
3. **Jobs on all-purpose clusters.** For high-spend job IDs, check
   `existing_cluster_id` on tasks. Job compute or serverless is usually cheaper
   and isolates failures.
4. **Tags and budget policies.** Untagged serverless has no chargeback path.
   Budget policies exist so serverless notebooks and jobs still land tags in
   billing — give every user at least one. Detail in
   [Community best practices](../best-practices/community-best-practices.md).

If system tables are disabled, enable `system.billing` and `system.compute`
first; the CLI fallback is slower and incomplete.

## Onboarding a brownfield platform to Terraform

- Import in phases — networking, then storage, then workspace, then grants, then
  cleanup — so you never interrupt running workloads.
- Manual changes to prod **will** happen (a grant here, a cluster policy there).
  They have to be reverse-engineered back into Terraform. Codify them and move to
  PR-based change control so the drift stops.

## Operational discipline

- PR-driven Terraform with rendered plan comments, environment CODEOWNERS, and
  concurrency-locked applies.
- Pin provider versions; SHA-pin GitHub Actions; retain plan artifacts (90 days)
  for audit.

## Limit the number of services

- Do not add ADF when DABs already orchestrates the Databricks jobs. Do not add Purview when
  Unity Catalog already supplies the required catalog. Add a service only for a defined gap.

## User interface

- Databricks has a dark theme in the user settings.