# Databricks PII governance: governed tags, ABAC, and Terraform

Use Unity Catalog to classify sensitive data and apply a small governed tag taxonomy. Use ABAC
for masks and row filters. Use Terraform for stable governance controls.

Read the [identity guide](identity-entra-scim.md) and the
[Terraform and DABs guide](terraform-vs-dabs.md) before you apply this governance model.

## The governance layers

### 1. Classification, tagging, and masking

- Use **Databricks Data Classification** (GA 2026-04-20) to detect sensitive data
  in Unity Catalog.
- Use **governed tags** to standardize labels such as `classification=restricted`,
  `pii=email`, `pii=phone`, `pii=ssn`, `row_security=region`.
- Require **data-steward review** before broad auto-tagging in production.
- Use **ABAC policies** to apply masks or row filters based on governed tags.
- Use DABs or SQL migrations to apply workload-specific tags when pipelines create
  new tables.
- Use **Terraform** for the stable governance plane: tag policies, ABAC policies,
  Entra groups, grants, and stable tag assignments.

Use Unity Catalog, Entra-backed groups, and group grants as the foundation. Add governed tags,
Data Classification, steward review, and ABAC policies for PII governance.

### 2. ABAC vs governed tags vs tag policies

Each object has a specific role:

- **Governed tags** describe the data.
- **Tag policies** define allowed tag keys and values.
- **ABAC policies** enforce row filters or column masks based on those tags.
- **Entra-backed groups** decide who is exempt or allowed to see unmasked data.

```text
Governed tag:
  pii=email on prod_catalog.gold.customer.email_address

ABAC policy:
  Any column tagged pii=* is masked for account users
  except approved PII-reader groups

Entra ID:
  Membership in a "PII Unmasked Readers" group
  determines whether the UDF returns raw or masked values
```

Rows are not the governance target. Do not tag rows. Tables or columns are
tagged, and row-filter UDFs evaluate row values such as `region_code`,
`tenant_id`, or `business_unit`.

### 3. Row filters and column masks

- **RBAC** gets a user to the catalog/schema/table.
- **Column masks** decide whether sensitive column values are raw or masked.
- **Row filters** decide which rows are returned.
- **ABAC** applies those controls centrally when matching governed tags are present.

```text
Table:   prod_catalog.gold.customer
Column:  region_code
Tags:    table row_security=region
         column security_key=region_code
Policy:  Pass region_code to governance.allow_region(region_code)
UDF:     Return true only if current user is in the matching Entra-backed group
```

Do not create one policy for each table. Define a small governed tag
taxonomy, apply tags at the highest useful Unity Catalog object level, tag
sensitive columns specifically, and use catalog-level ABAC policies so the same
mask/filter logic follows the tags.

### 4. Terraform's role at scale

Use Terraform for stable, repeatable governance objects:

- Entra security groups and Databricks account groups.
- Unity Catalog grants and service principal permissions.
- Governed tag definitions and allowed values.
- Catalog/schema/table/column tag assignments for stable, high-value assets.
- ABAC policy resources (`databricks_policy_info`).
- External locations, storage credentials, and managed-identity access paths.

Do not use Terraform for frequently changing tables or generated columns. These resources create
noisy plans and stale governance metadata.

```text
Terraform:            platform controls and stable governance
DABs / pipeline SQL:  workload-owned table creation and workload-specific tags
Data Classification:  sensitive-data discovery and optional auto-tagging
Data stewards:        review false positives, approve tag activation, exceptions
```

### 5. DABs in the governance model

DABs are not a replacement for Terraform-owned ABAC.

- Terraform owns the platform and control plane.
- DABs own jobs, pipelines, notebooks, SQL tasks, dashboards, and deployment targets.
- DABs run production jobs as service principals that Terraform created and granted.
- Pipelines can apply tags to the tables they produce, but central ABAC policies
  stay in Terraform or a restricted governance repository.

Terraform defines the guardrails; DABs deploy the workloads inside them. A bundle
can write a gold customer table, but Unity Catalog grants, governed tags, ABAC
policies. A central governance team must control the service principal's access.

### 6. Periodic access reviews

Make access reviews identity-led, not workspace-led:

- Entra ID is the source of truth for group membership.
- Databricks account groups are synced or mapped from Entra.
- Terraform grants permissions to groups, not individual users.
- A quarterly review exports group membership, UC grants, service principals, PAT
  usage, and job ownership.
- Business data owners approve or remove access.
- Remove access in Entra first. Then, let the change propagate to Databricks.

Review Entra group membership and service principal ownership first, then validate
Unity Catalog grants and workspace permissions against those approved groups.

### 7. PAT tokens, service principals, and secrets

The modern direction is workload identity federation and service principals
instead of long-lived secrets:

- Prefer OAuth / service principal auth for automation.
- Avoid human PATs for production automation.
- Restrict PAT creation and lifetime where PATs are still allowed.
- Store secrets in Azure Key Vault-backed secret scopes or an external secret system.
- Use managed identities for Azure resource access where possible.
- Rotate secrets and tokens, audit usage, and alert on stale credentials.

## Feature status (GA dates)

The Unity Catalog governance stack went GA in April 2026 (verified against
Microsoft Learn release notes):

- **Governed tags GA: 2026-04-02**
- **Databricks Data Classification GA: 2026-04-20**
- **ABAC GA: 2026-04-28**
- **SQL DDL for governed tags** (`CREATE/ALTER/DROP/DESCRIBE GOVERNED TAG`,
  `SHOW GOVERNED TAGS`) documented 2026-04-13

The ABAC GA release introduced a **breaking change**: row filters and column masks
on views/functions now evaluate as the **session user**, not the view/function
owner. Existing customers got a 3-month grace period; new customers get the new
behavior by default.

- ABAC compute requirements: **serverless**, **standard compute on DBR 16.4+**, or
  **dedicated compute on DBR 16.4+ with fine-grained access control enabled**.
  Older runtimes cannot read ABAC-protected tables.
- Governed tags SQL DDL (`CREATE GOVERNED TAG`, `ALTER GOVERNED TAG`) requires
  **DBR 18.1+** or Databricks SQL.
- Use non-sensitive metadata for governed tags and ABAC. Do not put actual PII
  values in tag keys or values.

## Tag inheritance

- Tags applied to a **catalog** inherit to schemas and tables underneath.
- Tags applied to a **schema** inherit to tables underneath.
- Tags applied to a **table** do NOT inherit to its columns.
- **Column tags must always be applied directly.**

Classify schemas with `data_classification`. Then, tag individual PII columns. Schema tags do not
propagate to columns.

## SQL DDL reference

Built-in functions for ABAC `WHEN` and `MATCH COLUMNS` clauses are **snake_case**:
`has_tag('pii')`, `has_tag_value('pii', 'ssn')`. Not `hasTag()`.

```sql
-- 1. Create the governed tag taxonomy (account-level, one time)
CREATE GOVERNED TAG pii
  DESCRIPTION 'Type of PII contained in the column'
  VALUES ('ssn', 'email', 'phone', 'dob', 'address');

CREATE GOVERNED TAG data_classification
  VALUES ('public', 'internal', 'confidential', 'restricted');

-- 2. Define the masking UDF (one per masking strategy)
CREATE FUNCTION governance.mask_ssn(val STRING)
  RETURNS STRING
  RETURN CASE
    WHEN is_account_group_member('pii_readers') THEN val
    ELSE '***-**-' || right(val, 4)
  END;

-- 3. Define the ABAC policy at the highest useful scope (catalog)
CREATE POLICY mask_ssn_columns
ON CATALOG prod_catalog
COLUMN MASK governance.mask_ssn
TO `account users`
FOR TABLES
MATCH COLUMNS has_tag_value('pii', 'ssn') AS ssn_col
ON COLUMN ssn_col;

-- 4. Row filter pattern (uses USING COLUMNS, not ON COLUMN)
CREATE FUNCTION governance.allow_region(region STRING, allowed STRING)
  RETURNS BOOLEAN
  RETURN region = allowed;

CREATE POLICY regional_access_emea
ON CATALOG sales
ROW FILTER governance.allow_region
TO `emea team`
FOR TABLES
MATCH COLUMNS has_tag('region') AS rgn
USING COLUMNS (rgn, 'EMEA');
```

## Terraform resources — exact names

Current Databricks Terraform provider resource names. There is one unified tag
assignment resource. The earlier `databricks_schema_tag` and `databricks_column_tag` names are
incorrect.

| Concern | Resource |
|---|---|
| Define a governed tag key + allowed values | `databricks_tag_policy` |
| Look up an existing tag policy | `databricks_tag_policy` (data source) |
| List all tag policies | `databricks_tag_policies` (data source) |
| Assign a tag to a UC entity (catalog, schema, table, column, volume, model) | `databricks_entity_tag_assignment` |
| Assign a tag to a workspace entity (dashboard, notebook, app, Genie space) | `databricks_workspace_entity_tag_assignment` |
| Create an ABAC row-filter or column-mask policy | `databricks_policy_info` |
| List policies on a securable | `databricks_policy_infos` (data source) |
| Manage who can assign or manage tag policies (`roles/tagPolicy.manager`, `roles/tagPolicy.assigner`) | `databricks_access_control_rule_set` |

`databricks_entity_tag_assignment` takes `entity_type` (`catalogs`, `schemas`,
`tables`, `columns`, `volumes`) and `entity_name` (fully qualified, e.g.
`prod_catalog.customers.users.email_address`).

## Quotas and limits

| Limit | Value |
|---|---|
| Governed tags per account | 1,000 |
| Allowed values per governed tag | 500 |
| Policies per metastore | 10,000 |
| Policies per catalog or schema | 100 |
| Policies per table | 50 |
| Principals per policy (TO + EXCEPT combined) | 20 |
| Column conditions per MATCH COLUMNS clause | 3 |
| Tag key/value max length | 256 chars, UTF-8, case-sensitive |

The design uses a small set of policies with many tag assignments. A metastore that approaches
the policy limit needs a design review.

## Separation of duties — who does what

Databricks publishes a four-role split:

1. **Tag taxonomy owner** (security or governance team) — `CREATE` on tags at
   account level. Defines `pii`, `data_classification`, etc.
2. **Data stewards / classifiers** — `ASSIGN` on the tag + `APPLY TAG` on the
   object. They stamp tags on data.
3. **Governance admins** — `MANAGE` on the catalog/schema + `EXECUTE` on UDFs.
   They write the ABAC policies.
4. **Data consumers** — query through the enforced policies.

Tagging is a security boundary. If a user can change tags, they can change which
policies apply. Audit tag changes.

## ABAC limitations

Review these limitations before you enable ABAC:

1. **No information schema view for ABAC policies.** `information_schema.row_filters`
   and `information_schema.column_masks` show only table-level filters/masks, not
   ABAC. To list ABAC policies use the Unity Catalog REST API; audit trail comes
   from `system.access.audit`. Build "periodic access review" on UC system tables +
   REST API, not info_schema for policy enumeration.
2. **Materialized views and streaming tables require the pipeline owner in the
   `EXCEPT` clause.** If the pipeline run-as identity is subject to the ABAC policy,
   the refresh fails. Every policy covering tables a pipeline reads must exempt the
   pipeline's run-as service principal.
3. **Time travel and clones fail on ABAC-protected tables** unless the principal is
   in `EXCEPT`. ETL service principals that `RESTORE` or `CREATE TABLE ... CLONE`
   need explicit exemption.
4. **Vector search indexes do not enforce ABAC.** The index syncs all rows from the
   source and serves them without applying filters/masks. For PII columns, use the
   "columns to sync" setting to exclude masked columns — otherwise PII leaks out the
   vector search path.
5. **Delta Sharing requires the share owner in `EXCEPT`.** Tables with ABAC policies
   (or views over them) can only be shared if the share owner is exempt. The policy
   does not govern the recipient. Recipients enforce their own ABAC.
6. **Multiple conflicting policies on the same column for the same user → query
   blocked.** Only one row filter resolves per (table, user) and one column mask per
   (column, user). If two distinct policies resolve, Databricks blocks access. Same
   UDF + same args is fine; different masks for the same column for the same user is
   not.
7. **Standard or dedicated compute on DBR versions before 16.4 cannot read ABAC tables.** The
   escape hatch is to scope the policy to a specific group via `TO` and put the
   legacy workload's principal in `EXCEPT`. Useful migration pattern.

Pipeline refreshes and Delta Sharing need the operating identity in the `EXCEPT` clause. Vector
Search indexes do not inherit ABAC. Exclude masked columns when you synchronize an index.

## Operating principles

- Do not tag at the row level. Row filters use column values. Tag schemas with
  classification, then columns with PII type only where it matters. Policies are
  written against tags, so a new PII table does not change policy code. A few
  PII-type policies cover the whole platform, and a CI check against
  `information_schema.column_tags` identifies columns that do not have a required tag.
- Confirm all production compute is on DBR 16.4+ before adopting ABAC, otherwise
  ABAC-protected tables are unreadable. If a team has dynamic views built around
  owner identity, revisit them for the session-user GA change.

## Microsoft Learn sources

Concept + setup:
- ABAC overview — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/
- ABAC core concepts — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/core-concepts
- Governed tags overview — https://learn.microsoft.com/azure/databricks/admin/governed-tags/
- Create and manage governed tags — https://learn.microsoft.com/azure/databricks/admin/governed-tags/manage-governed-tags
- Apply tags to UC securable objects — https://learn.microsoft.com/azure/databricks/database-objects/tags

Hands-on tutorials:
- Tutorial: Configure ABAC (UI) — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/tutorial
- Tutorial: Configure ABAC with SQL — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/tutorial-sql

SQL DDL reference:
- CREATE GOVERNED TAG — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-syntax-ddl-create-governed-tag
- ALTER GOVERNED TAG — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-syntax-ddl-alter-governed-tag
- Row filter syntax — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-syntax-ddl-row-filter
- Column mask syntax — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-syntax-ddl-column-mask

Best practices, perf, limits:
- Create and manage ABAC policies — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/policies
- Best practices — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/best-practices
- Common patterns (VARIANT UDFs, struct redaction) — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/common-patterns
- Performance — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/performance
- Requirements, quotas, limitations — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/requirements
- Resource limits (governed tag, tag-assignment, and policy quotas) — https://learn.microsoft.com/azure/databricks/resources/limits
- ABAC vs table-level filters/masks — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/abac/abac-vs-rls-cm
- Access control overview — https://learn.microsoft.com/azure/databricks/data-governance/unity-catalog/access-control/

Audit + observability:
- System tables reference — https://learn.microsoft.com/azure/databricks/admin/system-tables/
- Audit log system table (`system.access.audit`) — https://learn.microsoft.com/azure/databricks/admin/system-tables/audit-logs
- Information schema overview — https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-information-schema
- `INFORMATION_SCHEMA.TABLE_TAGS` — https://learn.microsoft.com/azure/databricks/sql/language-manual/information-schema/table_tags
- `INFORMATION_SCHEMA.SCHEMA_TAGS` — https://learn.microsoft.com/azure/databricks/sql/language-manual/information-schema/schema_tags
- `INFORMATION_SCHEMA.ROW_FILTERS` — https://learn.microsoft.com/azure/databricks/sql/language-manual/information-schema/row_filters

Terraform provider:
- `databricks_policy_info` — https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/policy_info
- `databricks_tag_policy` — https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/tag_policy
- `databricks_entity_tag_assignment` — https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/entity_tag_assignment
- `databricks_access_control_rule_set` — https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/access_control_rule_set