New and recent Databricks features to watch

azure6 min read

These Azure Databricks changes merit a platform review. Each entry identifies a capability and a decision to test before adoption. This selection is not a complete release history. Use the Azure release notes for the wider announcement stream.

Availability can differ by region, account, compute, and preview enrollment. A general availability announcement does not prove that a feature meets your workload's requirements.

Data engineering and orchestration#

Arrow Flight for Zerobus Ingest is generally available. It provides an Arrow path into Zerobus Ingest. Before adoption, test producer authentication, schema compatibility, retries, and reconciliation with a representative input stream. See Arrow Flight ingestion and the September release notes.

Bundles support a direct engine without a Terraform dependency. Existing bundles can require migration. Review the deployed CLI version, state conversion, and resulting plan before adoption. The direct engine's wider resource support does not change your ownership boundary: keep shared infrastructure separate from workload deployment. See the direct engine guide and Terraform versus bundles.

Storage and table formats#

Automatic change data feed is generally available. It computes row changes during reads. The documented requirements include Databricks Runtime 19 or later and supported Unity Catalog tables. Delta tables need row tracking; managed Iceberg tables need the v3 format. External Iceberg readers cannot query this feed. Test downstream compatibility and replay requirements before you replace an existing feed. See change data feed and the September announcement.

For retention and maintenance decisions, use the Delta storage guide.

Governance and Unity Catalog#

Fine-grained write privileges are in Beta. INSERT, UPDATE, and DELETE can separate data changes from the wider authority of MODIFY. They require supported compute and Databricks Runtime 18.1 or later; dedicated access mode does not support them. Test every required write operation and its inherited grants before you replace MODIFY. See fine-grained DML privileges and the grants guide.

Metastore-level ABAC policies are in Beta. A policy can cover matching objects across current and future catalogs. SQL administration requires Runtime 19 or later, and policy changes require a metastore admin. Managed disaster recovery does not replicate these policies. Include policy restoration in the recovery test before adoption. See metastore policies.

Identity, networking, and security#

Identity attributes are in Beta. Account attributes from the identity provider can support ABAC column mask conditions. Test missing attributes and delayed identity updates, not only the expected user record. Keep attribute administration separate from data access. See identity attributes and PII governance.

The private network gateway is in Beta. It connects serverless compute to a VNet and connected networks. It complements Private Link rather than replacing it. The current preview requires the account REST API; it has no UI or Terraform configuration support. Review destination scope, DNS, routing precedence, and the deployment owner before a pilot. See private network gateway.

Resilience, cost, and operations#

Check predictive optimization at the table boundary. Its automatic maintenance applies to supported Unity Catalog managed tables. Do not assume an account rollout means every table has the intended settings. Check inherited configuration, operation history, retention, and serverless cost before you remove existing maintenance jobs. See predictive optimization.

The query history system table is generally available. Use system.query.history to investigate query incidents. Before an incident, confirm that it contains the query records and time period you need. Check that the incident responders can query those records. See query history and the September announcement.

Keep a separate review for product deprecations. Use the official upcoming changes to identify affected workloads, owners, and migration dates. A planned retirement is not a feature to adopt, and an expired forecast is not evidence that a rollout completed.

Compute and runtime#

Environment version 6 is available. Review package changes and the documented dbutils behavior before you change a workload's environment. Test notebooks and libraries that use internal entry points, as well as the normal job path. See environment version 6.

Serverless JAR tasks are generally available. Java and Scala workloads can use this job task option. Test dependency packaging, network access, and runtime behavior before you retire classic compute. See serverless JAR tasks and the August announcement.

SQL and analytics#

Metric view sharing through OpenSharing is in Beta. It supports sharing metric views with Databricks recipients in other metastores or accounts. Test recipient access and metric results before you use it as a cross-team contract. Do not assume every external BI client has the same support. See metric views in shares and the September announcement.

AI/BI, Genie, and the Microsoft surface#

The Genie One MCP service is generally available. Migrate clients from /api/2.0/mcp/genie before its October 31, 2026 retirement. The replacement endpoint is /ai-gateway/mcp-services/system.ai.genie_one_mcp. On-behalf-of OAuth clients use the ai-gateway scope instead of genie. Test authentication and required tools before the switch. See Genie MCP migration and the September announcement.

Genie Code scheduled tasks are generally available. Runs use the creator's permissions and automatically approve tool actions, subject to the product's checks. Review the prompt's scope and the creator's privileges before you schedule unattended work. Start with a bounded read-only task and inspect its run history. See scheduled task behavior and the September announcement.

Mosaic AI and model serving#

The unified trace table is in Beta. It collects Unity Gateway requests and responses in a Unity Catalog table. Treat those payloads as potentially sensitive data. Define access and retention before you enable collection. Trace delivery is best-effort, so retain audit logs as the compliance record. See unified tracing.

OLTP and apps#

Lakebase Postgres has a separate application lifecycle. Its current documentation covers branches, autoscaling, restore, and lakehouse synchronization. Lakebase Change Data Feed into Delta tables remains Public Preview. Select the specific capability and region for your evaluation; do not apply one availability label to the entire product. Test recovery and synchronization delay against the application's requirements. See Lakebase Postgres.

Sources#

Generated 2026-09-06 by an agent-assisted source review of the official Azure documentation. Selected entries updated 2026-09-25 against the linked product documentation. The evaluation questions are wiki guidance, not vendor guarantees.