# Provision Databricks identities from Microsoft Entra ID

## Background

Use **Automatic Identity Management (AIM)** to make Entra ID the identity source
for Databricks. AIM creates an account group when an administrator adds the Entra
group to Databricks. Databricks sets the account group's `externalId` from the
Entra `ObjectId` and refreshes it each day.

Do not create a second Databricks group for the same Entra group. Do not build a
workflow that depends on `externalId`, because Databricks can update it. Manage
the group and its membership in Entra ID. Manage the workspace assignment and
Unity Catalog grants in Databricks, with Terraform where useful.

A typical naming convention for the paired groups is
`App Access - Databricks <Role> <Env>` so the Entra group and the Databricks
group are obviously the same thing.

Two provisioning paths exist:

- **AIM** is the Databricks recommendation for supported identity providers.
  It does not need an enterprise application or a SCIM token.
- **SCIM** remains the fallback for an unsupported identity provider, a
  non-federated workspace, or a cross-tenant Entra design.

## Configure the target state

- **AIM**: enabled on the account.
- **Identity federation**: enabled for each workspace that uses AIM.
- **JIT provisioning**: enabled by AIM. Users become active on first login.
- **Entra ID groups**: managed in Entra ID, with Terraform if the team uses it.
- **Databricks account groups**: created and synced by AIM.
- **Workspace assignments**: managed by an administrator or Terraform.
- **Grants**: in Terraform (`databricks_grants` / `databricks_grant`) at catalog,
  schema, and external-location levels.
- **Verification**: audit logs confirm AIM provisioning users and syncing group
  membership on login (queries below).

### How the pieces fit together

```
Entra ID                          Terraform                           Databricks
─────────                         ─────────                           ──────────
azuread_group ──→ Entra group ──AIM──→ Account group ──→ Workspace access
                         │                    │
                         │                    └──databricks_grants──→ UC privileges
                         │
                         └──authentication──→ User or service principal sync
```

1. **Terraform** can create the Entra ID groups.
2. **AIM** creates the Databricks account groups and syncs their membership.
3. **Terraform** can read the existing account group by name.
4. **Terraform** can assign the group to workspaces and grant catalog privileges.
5. **AIM** activates users and service principals when they authenticate.

### Distinguish app registrations from provisioning connectors

An app registration named like an Entra connector (for example
`sp_<something>_entraid`) is **not necessarily** a SCIM provisioning connector. A
plain app registration with `User.Read.All` Graph permissions that is registered
in Databricks as an account admin is just a service principal, not an identity
sync path. AIM does the synchronization. Do not assume that a name proves a provisioning path.

## How Automatic Identity Management works

### Overview

AIM syncs users, service principals, and groups (including nested groups) from
Entra ID to Databricks without an enterprise app, SCIM token, or extra Terraform.
Databricks uses Entra ID as the source of record.

### Sync triggers

AIM does **not** run on a fixed schedule. It syncs lazily based on user activity:

| Activity | Sync cooldown |
|---|---|
| Browser login | 5 minutes |
| Token authentication, job runs | 40 minutes |

When a sync triggers, Databricks fetches transitive (nested) group memberships
from Entra ID for groups that have been added to Databricks.

### What gets synced

- Users (on first login via JIT)
- Group memberships (on each login/activity, resolving nested groups)
- Service principals (on first use — token auth or job run)

### What stays inactive or incomplete until use

- Users and service principals that have not authenticated
- Parent group hierarchy that no added Databricks group needs
- Group name renames (only update when an admin opens the group detail page in
  the account console)

### User and group statuses

| Status | Meaning |
|---|---|
| Inactive: No usage | The user has not logged in, or the group has not been added to a workspace |
| Active | Identity is active in Databricks |
| Active: Removed From EntraID | Deleted from Entra ID, will be auto-deactivated on next sync |
| Deactivated | Deactivated in Entra ID or auto-deactivated by Databricks |

### Sharing and permissions

- **Account-level assets** (Unity Catalog objects, dashboards, Genie spaces,
  workspace assignment): Entra ID groups are available immediately.
- **Workspace-level assets** (notebooks, jobs, SQL warehouses, alerts): a
  workspace admin must first add the group to the workspace.

## Terraform architecture

AIM owns the Databricks account group and its membership. Do not declare an
AIM-managed group as a `databricks_group` resource. Do not declare its Entra
membership as `databricks_group_member` resources.

Terraform can manage the boundaries around the AIM-owned group:

### Identity resources (`groups.tf`)

```
modules/entra_id_group  → Creates Entra ID security group
data.databricks_group   → Reads the AIM-created account group by display_name
permission_assignment   → Assigns the existing group to a workspace
```

The account group must exist before Terraform plans the data-source lookup. Add
the Entra group to Databricks first, then apply the workspace assignment and
grants.

### Grant resources (`grants.tf`)

- `databricks_grants.catalog` — catalog-level grants per role group
- `databricks_grants.schema` — schema-level grants per role group + per-schema group
- `databricks_grants.external_location` — external location grants

### Example role-group model

A workable six-tier RBAC model, with privileges tuned per role:

| Group | Catalog | Schema | External Location |
|---|---|---|---|
| Admins | USE_CATALOG, CREATE_SCHEMA, MANAGE, CREATE_VOLUME | USE_SCHEMA, MANAGE | ALL_PRIVILEGES, MANAGE |
| Data Architects | USE_CATALOG | MODIFY, CREATE_TABLE, CREATE_VOLUME, USE_SCHEMA, SELECT | READ_FILES |
| Data Engineers | USE_CATALOG | MODIFY, CREATE_TABLE, CREATE_FUNCTION, USE_SCHEMA, SELECT | READ_FILES |
| Data Analysts | USE_CATALOG | USE_SCHEMA, SELECT | — |
| QA | USE_CATALOG | USE_SCHEMA, MANAGE | — |
| Service principals | USE_CATALOG | scoped per workload | — |
| Per-schema group | — | USE_SCHEMA, SELECT, EXECUTE | — |

## Verify AIM

### Audit logs (SQL)

```sql
-- Users provisioned by AIM
SELECT event_time, action_name, request_params.targetUserName
FROM system.access.audit
WHERE action_name = 'add'
  AND request_params.endpoint = 'autoUserCreation'
  AND event_time > current_timestamp() - INTERVAL 7 DAYS
ORDER BY event_time DESC;

-- Group membership synced from Entra ID
SELECT event_time, request_params.targetUserName, request_params.targetGroupName
FROM system.access.audit
WHERE action_name IN ('addPrincipalToGroup', 'removePrincipalFromGroup')
  AND request_params.groupMembershipType = 'IdentityProvider'
  AND event_time > current_timestamp() - INTERVAL 7 DAYS
ORDER BY event_time DESC;

-- User logins
SELECT DISTINCT user_identity.email
FROM system.access.audit
WHERE action_name = 'aadBrowserLogin'
  AND event_time > current_timestamp() - INTERVAL 7 DAYS;
```

### API verification

Use a Databricks CLI profile to check whether a user is active in the workspace:

```bash
databricks users list \
  --filter 'userName eq "<USER_NAME>"' \
  --profile "$P" \
  --output json
```

Use the audit queries above to confirm that AIM created the user and refreshed the
group membership.

## Known behaviors and limitations

1. **Users become active on first login** — an unused identity stays inactive.
2. **Nested groups work for permissions but not API/Terraform** — nested groups
   inherit permissions automatically, but are not retrievable through the API or Terraform
   unless explicitly provisioned to the account.
3. **Service principals provision on first use** — a group can contain service principals.
   Databricks provisions them only after they authenticate or run a job.
4. **Group name changes are not proactive** — the name updates when an administrator opens the
   group detail page in the account console.
5. **Cross-tenant Entra ID is not supported** — use SCIM with Entra B2B
   collaboration for cross-tenant scenarios.
6. **Do not mix AIM and SCIM** — adding the same identity through both causes
   duplicate entries and permission conflicts. Use AIM as the single source of
   truth.
7. **Do not depend on `externalId`** — Databricks can update it from the Entra
   `ObjectId`.

## Required identifiers

Keep these values for each environment in the approved secret store:

| Resource | Where to find it |
|---|---|
| Databricks Account ID | Account console → top-right user menu |
| Entra ID Tenant ID | Entra admin center → Overview |
| Workspace URL/ID | Workspace URL (`adb-<id>.<n>.azuredatabricks.net`) |
| Catalog name | Unity Catalog → Catalog Explorer |
| Group object IDs | Entra group → Overview (Object ID) |

## References

- [Sync users and groups automatically from Microsoft Entra ID (AIM)](https://learn.microsoft.com/azure/databricks/admin/users-groups/automatic-identity-management)
- [Establish your first account admin](https://learn.microsoft.com/azure/databricks/admin/admin-concepts#establish-first-account-admin)
- [Identity best practices](https://learn.microsoft.com/azure/databricks/admin/users-groups/best-practices)
- [Identity federation](https://learn.microsoft.com/azure/databricks/admin/users-groups/#enable-identity-federation)
- [databricks_group data source](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/group)
- [databricks_group Terraform resource](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/group)
- [databricks_mws_permission_assignment Terraform resource](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_permission_assignment)