Audit a Databricks environment

Use this guide for Azure, AWS, or GCP Databricks. The commands read metadata and system tables. They do not change the environment. Authentication is the only non-read action.

---

Audit report template#

Complete this report after the audit. Replace each italic placeholder with an audit result.

Environment: tenant / workspaces audited Date: YYYY-MM-DD Auditor: name Coverage: areas reviewed · environments reviewed · exclusions and reasons

Bottom line: 2–3 sentences — overall posture and the single most important thing to fix.

Scorecard (RAG by Well-Architected pillar)#

🟢 no material issues · 🟡 issues to address · 🔴 urgent / business risk

Top 5 findings (severity-ranked)#

Immediate and long-term actions#

• Quick wins (≤1 day): e.g. enable auto-termination; add failure webhooks; stop abandoned apps
• Strategic (needs a project): e.g. migrate hive_metastore → UC; adopt DABs for all resources

Coverage gaps and limits#

• Products not present: …
• Checks blocked by missing permissions or disabled system tables: …

---

Scope#

Use the audit to assess inventory, cost, jobs, pipelines, deployments, serverless compute, Model Serving, Vector Search, Apps, and Lakebase. Record each finding with evidence and a resource ID.

Well-Architected Lakehouse framework#

Every check below maps to a pillar of the official Databricks Well-Architected Lakehouse Framework (learn.microsoft.com/azure/databricks/lakehouse-architecture/well-architected), so findings are grounded in Databricks' own guidance rather than opinion. The seven pillars:

Prerequisites#

• [ ] Databricks CLI ≥ v0.292.0 (databricks --version)
• [ ] An authenticated profile (databricks auth profiles shows Valid: YES)
• [ ] Ideally account-admin read + workspace-admin read. Partial access still works — note gaps.
• [ ] Unity Catalog system tables enabled (system.billing, system.compute, system.lakeflow). If disabled, the SQL steps fall back to slower per-resource CLI loops (noted inline).

Command conventions#

• Replace $P with the profile under audit; $ACCT with the account-console profile.
• Audit each environment and the Databricks account.
• Names are literal. Never normalize hyphens to underscores. Backtick-quote any name part with a hyphen in SQL: `my-catalog`.schema.table.
• ⚠️ Unity Catalog commands take POSITIONAL args, not flags. schemas list <CATALOG> — schemas list --catalog-name X does not exist and will fail.
• Capture resource IDs with every finding so it stays actionable later.

---

Required permissions#

Databricks access spans four independent permission planes: account roles, workspace object ACLs, Unity Catalog privileges, and system-table grants. This audit reads only metadata and system tables — never business data — so SELECT on user tables is never required.

System table requirements:

• An account admin must have enabled the relevant schemas (system.billing, system.compute, system.lakeflow, and system.access for audit logs). These schemas are not all on by default.
• The metastore must be on Unity Catalog Privilege Model v1.0, and you must query from a UC-enabled workspace.
• System tables only contain data for workspaces in the same cloud region. To audit a workspace in another region, run the queries from a workspace in that region.

Recommended "auditor" grant bundle (read-only, least privilege):

1. Workspace admin (read) in each environment workspace — the practical way to CAN_VIEW every job, pipeline, cluster, and notebook at once.
2. USE CATALOG on system + USE SCHEMA + SELECT on system.billing, system.compute, system.lakeflow, and system.access.
3. CAN_USE on one SQL warehouse (to run the queries).
4. BROWSE on all catalogs — or metastore admin if you also need external locations / storage credentials (A3).
5. Account admin — required only for the account inventory. If access is not available, record the gap and audit each workspace.

Grant to a group, not your user. Per UC best practice (and consistent with this audit's own findings), assign the bundle to an auditors group. Whoever grants it must themselves be both an account admin and a metastore admin.

---

Audit procedure#

Authenticate#

databricks auth login --host <WORKSPACE_URL> --profile $P
databricks current-user me --profile $P

Expected: OAuth flow completes; current-user me returns your identity JSON. If it fails: configuration does not support OAuth tokens → re-run the login to convert a PAT/azure-cli profile. On Azure with auth_type = azure-cli, run az login --tenant <TENANT_ID> first — the Databricks profile borrows that identity.

---

Inventory#

Account resources#

databricks account workspaces list --profile $ACCT
databricks account metastores list --profile $ACCT
databricks account groups list --profile $ACCT
databricks account users list --profile $ACCT

Look for: more metastores than regions; users granted directly instead of via groups; orphaned or empty workspaces; duplicate workspace configs. If it fails: subcommand names vary by CLI build — confirm with databricks account --help. No account-admin access → skip the account inventory and record the coverage gap.

Workspace resources#

databricks clusters list --profile $P
databricks warehouses list --profile $P
databricks instance-pools list --profile $P
databricks cluster-policies list --profile $P
databricks jobs list --profile $P
databricks pipelines list --profile $P
databricks serving-endpoints list --profile $P

Expected: one block per resource type; capture counts + IDs. If it fails: databricks sql-warehouses list does not exist — use warehouses list.

Unity Catalog resources#

databricks catalogs list --profile $P
databricks schemas list <CATALOG> --profile $P
databricks tables list <CATALOG> <SCHEMA> --profile $P
databricks external-locations list --profile $P
databricks storage-credentials list --profile $P

Look for: a hive_metastore catalog still in active use (un-migrated legacy data — major governance gap); external locations with broad access; storage credentials shared too widely.

---

Cost and capacity#

Prefer system tables — one query beats iterating the CLI over every resource. Get the warehouse first:

databricks experimental aitools tools get-default-warehouse --profile $P

DBU usage by SKU#

databricks experimental aitools tools query "SELECT sku_name, usage_unit, ROUND(SUM(usage_quantity),1) AS dbus FROM system.billing.usage WHERE usage_date >= DATE_SUB(CURRENT_DATE,30) GROUP BY 1,2 ORDER BY dbus DESC LIMIT 20" --profile $P

This query shows the primary sources of DBU usage. If it returns TABLE_OR_VIEW_NOT_FOUND, record the missing billing schema and use the cluster configuration fallback.

Idle and always-on clusters#

databricks experimental aitools tools query "SELECT cluster_name, auto_termination_minutes, worker_node_type, num_workers FROM system.compute.clusters WHERE delete_time IS NULL ORDER BY auto_termination_minutes DESC" --profile $P

Red flag: auto_termination_minutes = 0 or NULL → burning DBUs 24/7. Top remediation target.

Cluster configuration fallback#

databricks clusters list --profile $P --output json

Scan for autotermination_minutes (0/missing), spark_version (outdated runtime = security + perf debt), oversized node_type_id.

---

Active incident triage (first 15 minutes)#

Use this path before you restart a failed job or pipeline. Preserve evidence first.

1. Record the UTC time, workspace, workload name, resource ID, and the person who reported it.
2. Record the last successful run, the failed run ID, and the exact failure time.
3. Capture the current job or pipeline configuration and note changes since the last success.
4. Capture the exact error before you retry: - Jobs: databricks jobs list-runs --job-id <ID> --limit 10 --profile $P --output json. - Pipelines: databricks pipelines list-pipeline-events <PIPELINE_ID> --max-results 50 --profile $P --output json.
5. Check whether the failure affects one workload or a shared dependency, then continue with the job or pipeline sections. Record every command and result in the findings table.

Do not restart until you capture the failed run ID, error, and current configuration. A retry can replace the useful failure context.

---

Job reliability#

The job timeline contains hourly slices. The result_state value exists only on the final slice. Count final slices as runs, and combine all slices before you calculate a run duration.

Source: https://learn.microsoft.com/azure/databricks/admin/system-tables/jobs

Job failure rates#

databricks experimental aitools tools query "SELECT job_id, COUNT(*) AS runs, COUNT_IF(result_state <> 'SUCCEEDED') AS failures, ROUND(100.0 * COUNT_IF(result_state <> 'SUCCEEDED') / COUNT(*), 1) AS fail_pct FROM system.lakeflow.job_run_timeline WHERE period_start_time >= DATE_SUB(CURRENT_DATE, 30) AND result_state IS NOT NULL GROUP BY 1 HAVING failures > 0 ORDER BY fail_pct DESC, failures DESC LIMIT 30" --profile $P

Worst offenders by failure rate. Column names vary by region — if one errors, run databricks experimental aitools tools discover-schema system.lakeflow.job_run_timeline --profile $P.

Job duration outliers#

databricks experimental aitools tools query "WITH runs AS (SELECT job_id, run_id, (UNIX_TIMESTAMP(MAX(period_end_time)) - UNIX_TIMESTAMP(MIN(period_start_time))) / 60.0 AS duration_min FROM system.lakeflow.job_run_timeline WHERE period_start_time >= DATE_SUB(CURRENT_DATE, 30) AND period_end_time IS NOT NULL GROUP BY job_id, run_id) SELECT job_id, ROUND(AVG(duration_min), 1) AS avg_min, ROUND(MAX(duration_min), 1) AS max_min, COUNT(*) AS runs FROM runs GROUP BY job_id ORDER BY max_min DESC LIMIT 20" --profile $P

Review condition: max_min ≫ avg_min can identify runs without a suitable timeout.

Job configuration review#

For each high-risk job from the previous queries:

databricks jobs get <JOB_ID> --profile $P --output json

Review each job against the following configuration checks:

Quick jq to surface the two most common offenders:

databricks jobs get <JOB_ID> --profile $P --output json | jq '{name, tasks: [.settings.tasks[] | {task_key, uses_all_purpose: (.existing_cluster_id != null), retries: .max_retries}], on_failure: .settings.email_notifications.on_failure}'

Recent run fallback#

databricks jobs list --profile $P
databricks jobs list-runs --job-id <ID> --limit 25 --profile $P

---

Pipeline reliability#

Pipeline inventory and health#

databricks pipelines list --profile $P
databricks pipelines get <PIPELINE_ID> --profile $P --output json

Score each pipeline against the table below:

Pipeline failures#

databricks pipelines list-pipeline-events <PIPELINE_ID> --max-results 50 --profile $P

⚠️ Read the real cause from error.exceptions[0].message, NOT the top-level .message (which only says "Update X is FAILED"):

databricks pipelines list-pipeline-events <PIPELINE_ID> --max-results 50 --profile $P --output json | jq -r '.events[] | select(.level=="ERROR") | .error.exceptions[0].message' | sort | uniq -c | sort -rn

Note: a pipeline stuck INITIALIZING on serverless is a normal cold start (a few minutes), not a failure. Do not report it as an error.

Pipeline source review#

Pull the pipeline's source notebooks/files and scan. Legacy DLT syntax is migration debt and a reliability/perf risk — flag every hit:

rg -n "import dlt|@dlt\.|dlt\.apply_changes|LIVE\.|CREATE LIVE TABLE|CREATE STREAMING LIVE TABLE|CREATE OR REPLACE (STREAMING TABLE|MATERIALIZED VIEW)" <pipeline_src_dir>

---

Shared reliability controls#

• Alerting coverage: cross-reference C3 (jobs) + D1 (pipelines) — what fraction of production workloads have any failure notification? A low number is usually the single biggest finding.
• Full-refresh exposure: note any pipeline/job that triggers a pipeline full refresh on a schedule — it reprocesses streaming sources from scratch and can cause data loss.
• Runtime/version drift: outdated spark_version (jobs) and channel: PREVIEW (pipelines) → security + reproducibility risk.
• Tagging: are clusters/jobs/warehouses tagged (team/cost-center)? Untagged = no cost attribution.

---

Deployment and IaC controls#

Run from each bundle root (the directory containing databricks.yml). The audit question: is this environment managed as code, or hand-clicked?

databricks bundle validate --strict --profile $P
databricks bundle validate --strict -t <TARGET> --profile $P
databricks bundle summary -t <TARGET> --profile $P

Compare the bundle resources with the live inventory to find manually created resources:

databricks bundle summary -t <TARGET> --profile $P   # IaC-managed resources
databricks jobs list --profile $P                    # everything that exists
# anything in `jobs list` but absent from `bundle summary` was created by hand

Command notes: There is no bundle list-targets command. Read targets from the databricks.yml file. Confirm available commands with databricks <group> --help.

---

Serverless migration opportunities#

"Use serverless architectures" is the first principle of the Performance Efficiency pillar. This section identifies classic compute that is a serverless candidate.

databricks experimental aitools tools query "SELECT sku_name, usage_type, ROUND(SUM(usage_quantity),1) AS dbus FROM system.billing.usage WHERE usage_date >= DATE_SUB(CURRENT_DATE,90) GROUP BY 1,2 ORDER BY dbus DESC" --profile $P

Workloads whose usage_type is not SERVERLESS_COMPUTE are the candidate spend. Then per job/pipeline, the migration signal is in the spec JSON:

databricks jobs get <JOB_ID> --profile $P --output json   # job_clusters/new_cluster = classic; environments/environment_key = serverless
databricks pipelines get <PIPELINE_ID> --profile $P --output json

Candidates: all-purpose clusters backing notebook/Python work; jobs on classic job_clusters; spark_jar_task on classic; outdated DBR (13.x/14.x); no scale-to-zero. Compatibility limits: RDD/SparkContext APIs, %scala/%r cells, custom Spark data-source JARs, unsupported spark.conf.set(...), DBFS mounts / Hive Metastore, continuous-trigger streaming, ML libs not pre-installed on serverless.

Caveat: Confirm the system.compute.clusters columns in the target environment before you use them.

---

Model Serving#

databricks serving-endpoints list --profile $P -o json
databricks serving-endpoints get <NAME> --profile $P -o json
databricks serving-endpoints get <NAME> --profile $P -o json | jq '.config.served_entities[] | {entity_name, entity_version, scale_to_zero_enabled, workload_size, min_provisioned_throughput, max_provisioned_throughput}'
databricks serving-endpoints get-permissions <ENDPOINT_ID> --profile $P   # takes ID, not name
databricks serving-endpoints export-metrics <NAME> --profile $P           # Prometheus metrics

Cost: system.billing.usage WHERE billing_origin_product = 'MODEL_SERVING'. Caveat: The Databricks CLI has no command to list Unity Catalog model aliases. The endpoint response supplies entity_name and entity_version for the served model.

---

Vector Search#

databricks vector-search-endpoints list-endpoints --profile $P
databricks vector-search-endpoints get-endpoint <ENDPOINT_NAME> --profile $P
databricks vector-search-indexes list-indexes <ENDPOINT_NAME> --profile $P
databricks vector-search-indexes get-index <CATALOG>.<SCHEMA>.<INDEX> --profile $P

Health states to watch on get-endpoint: YELLOW_STATE, RED_STATE, OFFLINE, stuck PROVISIONING. Caveat: The public documentation does not identify the VECTOR_SEARCH billing SKU or its system table mappings. Confirm the value before you report cost numbers.

---

Databricks Apps#

databricks apps list --profile $P -o json          # verify with --help
databricks apps get <APP_NAME> --profile $P -o json # app_status.state, url, compute, resources[], service_principal
databricks apps logs <APP_NAME> --profile $P        # OAuth auth only — silently fails on PAT

Limits: 100 apps/workspace; 10-min start; 120s proxy timeout (504 not in app logs); ephemeral FS — only stdout/stderr survive. UX (from databricks-app-design): missing loading/empty/error states; Genie/AI surfaces that hide the generated SQL, execution identity, or "verify" disclaimer.

---

Lakebase#

CLI group is postgres (not lakebase); needs CLI ≥ v0.294.0. Discover first — the surface is narrow: databricks postgres -h.

databricks postgres list-branches projects/<PROJECT_ID> --profile $P
databricks postgres list-endpoints projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile $P
databricks postgres list-databases projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile $P
databricks postgres get-endpoint projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID> --profile $P
databricks postgres get-synced-table "synced_tables/<CATALOG>.<SCHEMA>.<TABLE>" --profile $P

Command notes: There is no documented list-projects, get-project, or list-synced-tables command — enumerate via the Python SDK (w.postgres), REST /database/, or Catalog Explorer UI. Provisioned-era instances use the separate databricks database group. Never run generate-database-credential during an audit — it mints a live DB token. Cost: CU-hours (~2 GB RAM/CU) plus the synchronized table pipeline cost. Both appear in system.billing.usage. Confirm the exact SKU in the target environment.

---

Review the results#

• [ ] One completed inventory block per environment + the account
• [ ] The spend and job failure queries returned data, or the fallback checks ran
• [ ] Each high-risk job and pipeline has a configuration review
• [ ] Product-specific checks include only products that exist in the environment
• [ ] Each finding tagged with its Well-Architected pillar (so the report rolls up by pillar)
• [ ] Findings captured with resource IDs

Findings table#

Troubleshooting#

Referenced by

• Overview
• Community best practices
• New features to watch
• Well-Architected Lakehouse
• Platform lessons
• Databricks migration guide