Databricks PII governance: governed tags, ABAC, and Terraform

Use Unity Catalog to classify sensitive data and apply a small governed tag taxonomy. Use ABAC for masks and row filters. Use Terraform for stable governance controls.

Read the identity guide and the Terraform and DABs guide before you apply this governance model.

The governance layers#

1. Classification, tagging, and masking#

Use Databricks Data Classification (GA 2026-04-20) to detect sensitive data in Unity Catalog.
Use governed tags to standardize labels such as classification=restricted, pii=email, pii=phone, pii=ssn, row_security=region.
Require data-steward review before broad auto-tagging in production.
Use ABAC policies to apply masks or row filters based on governed tags.
Use DABs or SQL migrations to apply workload-specific tags when pipelines create new tables.
Use Terraform for the stable governance plane: tag policies, ABAC policies, Entra groups, grants, and stable tag assignments.

Use Unity Catalog, Entra-backed groups, and group grants as the foundation. Add governed tags, Data Classification, steward review, and ABAC policies for PII governance.

2. ABAC vs governed tags vs tag policies#

Each object has a specific role:

Governed tags describe the data.
Tag policies define allowed tag keys and values.
ABAC policies enforce row filters or column masks based on those tags.
Entra-backed groups decide who is exempt or allowed to see unmasked data.

Governed tag:
  pii=email on prod_catalog.gold.customer.email_address

ABAC policy:
  Any column tagged pii=* is masked for account users
  except approved PII-reader groups

Entra ID:
  Membership in a "PII Unmasked Readers" group
  determines whether the UDF returns raw or masked values

Rows are not the governance target. Do not tag rows. Tables or columns are tagged, and row-filter UDFs evaluate row values such as region_code, tenant_id, or business_unit.

3. Row filters and column masks#

RBAC gets a user to the catalog/schema/table.
Column masks decide whether sensitive column values are raw or masked.
Row filters decide which rows are returned.
ABAC applies those controls centrally when matching governed tags are present.

Table:   prod_catalog.gold.customer
Column:  region_code
Tags:    table row_security=region
         column security_key=region_code
Policy:  Pass region_code to governance.allow_region(region_code)
UDF:     Return true only if current user is in the matching Entra-backed group

Do not create one policy for each table. Define a small governed tag taxonomy, apply tags at the highest useful Unity Catalog object level, tag sensitive columns specifically, and use catalog-level ABAC policies so the same mask/filter logic follows the tags.

4. Terraform's role at scale#

Use Terraform for stable, repeatable governance objects:

Entra security groups and Databricks account groups.
Unity Catalog grants and service principal permissions.
Governed tag definitions and allowed values.
Catalog/schema/table/column tag assignments for stable, high-value assets.
ABAC policy resources (databricks_policy_info).
External locations, storage credentials, and managed-identity access paths.

Do not use Terraform for frequently changing tables or generated columns. These resources create noisy plans and stale governance metadata.

Terraform:            platform controls and stable governance
DABs / pipeline SQL:  workload-owned table creation and workload-specific tags
Data Classification:  sensitive-data discovery and optional auto-tagging
Data stewards:        review false positives, approve tag activation, exceptions

5. DABs in the governance model#

DABs are not a replacement for Terraform-owned ABAC.

Terraform owns the platform and control plane.
DABs own jobs, pipelines, notebooks, SQL tasks, dashboards, and deployment targets.
DABs run production jobs as service principals that Terraform created and granted.
Pipelines can apply tags to the tables they produce, but central ABAC policies stay in Terraform or a restricted governance repository.

Terraform defines the guardrails; DABs deploy the workloads inside them. A bundle can write a gold customer table, but Unity Catalog grants, governed tags, ABAC policies. A central governance team must control the service principal's access.

6. Periodic access reviews#

Make access reviews identity-led, not workspace-led:

Entra ID is the source of truth for group membership.
Databricks account groups are synced or mapped from Entra.
Terraform grants permissions to groups, not individual users.
A quarterly review exports group membership, UC grants, service principals, PAT usage, and job ownership.
Business data owners approve or remove access.
Remove access in Entra first. Then, let the change propagate to Databricks.

Review Entra group membership and service principal ownership first, then validate Unity Catalog grants and workspace permissions against those approved groups.

7. PAT tokens, service principals, and secrets#

The modern direction is workload identity federation and service principals instead of long-lived secrets:

Prefer OAuth / service principal auth for automation.
Avoid human PATs for production automation.
Restrict PAT creation and lifetime where PATs are still allowed.
Store secrets in Azure Key Vault-backed secret scopes or an external secret system.
Use managed identities for Azure resource access where possible.
Rotate secrets and tokens, audit usage, and alert on stale credentials.

Feature status (GA dates)#

The Unity Catalog governance stack went GA in April 2026 (verified against Microsoft Learn release notes):

Governed tags GA: 2026-04-02
Databricks Data Classification GA: 2026-04-20
ABAC GA: 2026-04-28
SQL DDL for governed tags (CREATE/ALTER/DROP/DESCRIBE GOVERNED TAG, SHOW GOVERNED TAGS) documented 2026-04-13

The ABAC GA release introduced a breaking change: row filters and column masks on views/functions now evaluate as the session user, not the view/function owner. Existing customers got a 3-month grace period; new customers get the new behavior by default.

ABAC compute requirements: serverless, standard compute on DBR 16.4+, or dedicated compute on DBR 16.4+ with fine-grained access control enabled. Older runtimes cannot read ABAC-protected tables.
Governed tags SQL DDL (CREATE GOVERNED TAG, ALTER GOVERNED TAG) requires DBR 18.1+ or Databricks SQL.
Use non-sensitive metadata for governed tags and ABAC. Do not put actual PII values in tag keys or values.

Tag inheritance#

Tags applied to a catalog inherit to schemas and tables underneath.
Tags applied to a schema inherit to tables underneath.
Tags applied to a table do NOT inherit to its columns.
Column tags must always be applied directly.

Classify schemas with data_classification. Then, tag individual PII columns. Schema tags do not propagate to columns.

SQL DDL reference#

Built-in functions for ABAC WHEN and MATCH COLUMNS clauses are snake_case: has_tag('pii'), has_tag_value('pii', 'ssn'). Not hasTag().

-- 1. Create the governed tag taxonomy (account-level, one time)
CREATE GOVERNED TAG pii
  DESCRIPTION 'Type of PII contained in the column'
  VALUES ('ssn', 'email', 'phone', 'dob', 'address');

CREATE GOVERNED TAG data_classification
  VALUES ('public', 'internal', 'confidential', 'restricted');

-- 2. Define the masking UDF (one per masking strategy)
CREATE FUNCTION governance.mask_ssn(val STRING)
  RETURNS STRING
  RETURN CASE
    WHEN is_account_group_member('pii_readers') THEN val
    ELSE '***-**-' || right(val, 4)
  END;

-- 3. Define the ABAC policy at the highest useful scope (catalog)
CREATE POLICY mask_ssn_columns
ON CATALOG prod_catalog
COLUMN MASK governance.mask_ssn
TO `account users`
FOR TABLES
MATCH COLUMNS has_tag_value('pii', 'ssn') AS ssn_col
ON COLUMN ssn_col;

-- 4. Row filter pattern (uses USING COLUMNS, not ON COLUMN)
CREATE FUNCTION governance.allow_region(region STRING, allowed STRING)
  RETURNS BOOLEAN
  RETURN region = allowed;

CREATE POLICY regional_access_emea
ON CATALOG sales
ROW FILTER governance.allow_region
TO `emea team`
FOR TABLES
MATCH COLUMNS has_tag('region') AS rgn
USING COLUMNS (rgn, 'EMEA');

Terraform resources — exact names#

Current Databricks Terraform provider resource names. There is one unified tag assignment resource. The earlier databricks_schema_tag and databricks_column_tag names are incorrect.

Concern	Resource
Define a governed tag key + allowed values	`databricks_tag_policy`
Look up an existing tag policy	`databricks_tag_policy` (data source)
List all tag policies	`databricks_tag_policies` (data source)
Assign a tag to a UC entity (catalog, schema, table, column, volume, model)	`databricks_entity_tag_assignment`
Assign a tag to a workspace entity (dashboard, notebook, app, Genie space)	`databricks_workspace_entity_tag_assignment`
Create an ABAC row-filter or column-mask policy	`databricks_policy_info`
List policies on a securable	`databricks_policy_infos` (data source)
Manage who can assign or manage tag policies (`roles/tagPolicy.manager`, `roles/tagPolicy.assigner`)	`databricks_access_control_rule_set`

databricks_entity_tag_assignment takes entity_type (catalogs, schemas, tables, columns, volumes) and entity_name (fully qualified, e.g. prod_catalog.customers.users.email_address).

Quotas and limits#

Limit	Value
Governed tags per account	1,000
Allowed values per governed tag	500
Policies per metastore	10,000
Policies per catalog or schema	100
Policies per table	50
Principals per policy (TO + EXCEPT combined)	20
Column conditions per MATCH COLUMNS clause	3
Tag key/value max length	256 chars, UTF-8, case-sensitive

The design uses a small set of policies with many tag assignments. A metastore that approaches the policy limit needs a design review.

Separation of duties — who does what#

Databricks publishes a four-role split:

Tag taxonomy owner (security or governance team) — CREATE on tags at account level. Defines pii, data_classification, etc.
Data stewards / classifiers — ASSIGN on the tag + APPLY TAG on the object. They stamp tags on data.
Governance admins — MANAGE on the catalog/schema + EXECUTE on UDFs. They write the ABAC policies.
Data consumers — query through the enforced policies.

Tagging is a security boundary. If a user can change tags, they can change which policies apply. Audit tag changes.

ABAC limitations#

Review these limitations before you enable ABAC:

No information schema view for ABAC policies. information_schema.row_filters and information_schema.column_masks show only table-level filters/masks, not ABAC. To list ABAC policies use the Unity Catalog REST API; audit trail comes from system.access.audit. Build "periodic access review" on UC system tables + REST API, not info_schema for policy enumeration.
Materialized views and streaming tables require the pipeline owner in the EXCEPT clause. If the pipeline run-as identity is subject to the ABAC policy, the refresh fails. Every policy covering tables a pipeline reads must exempt the pipeline's run-as service principal.
Time travel and clones fail on ABAC-protected tables unless the principal is in EXCEPT. ETL service principals that RESTORE or CREATE TABLE ... CLONE need explicit exemption.
Vector search indexes do not enforce ABAC. The index syncs all rows from the source and serves them without applying filters/masks. For PII columns, use the "columns to sync" setting to exclude masked columns — otherwise PII leaks out the vector search path.
Delta Sharing requires the share owner in EXCEPT. Tables with ABAC policies (or views over them) can only be shared if the share owner is exempt. The policy does not govern the recipient. Recipients enforce their own ABAC.
Multiple conflicting policies on the same column for the same user → query blocked. Only one row filter resolves per (table, user) and one column mask per (column, user). If two distinct policies resolve, Databricks blocks access. Same UDF + same args is fine; different masks for the same column for the same user is not.
Standard or dedicated compute on DBR versions before 16.4 cannot read ABAC tables. The escape hatch is to scope the policy to a specific group via TO and put the legacy workload's principal in EXCEPT. Useful migration pattern.

Pipeline refreshes and Delta Sharing need the operating identity in the EXCEPT clause. Vector Search indexes do not inherit ABAC. Exclude masked columns when you synchronize an index.

Operating principles#

Do not tag at the row level. Row filters use column values. Tag schemas with classification, then columns with PII type only where it matters. Policies are written against tags, so a new PII table does not change policy code. A few PII-type policies cover the whole platform, and a CI check against information_schema.column_tags identifies columns that do not have a required tag.
Confirm all production compute is on DBR 16.4+ before adopting ABAC, otherwise ABAC-protected tables are unreadable. If a team has dynamic views built around owner identity, revisit them for the session-user GA change.