Databricks Community discussions

Platform-focused threads from the Databricks Community discussion boards, filtered and accumulated. This page stores thread titles and links only. A thread can contain incomplete or incorrect advice; verify technical claims against official documentation.

Sources:

• Administration & Architecture — https://community.databricks.com/tmcxu86974/rss/board?board.id=administration-and-architecture
• Data Engineering — https://community.databricks.com/tmcxu86974/rss/board?board.id=data-engineering
• Data Governance — https://community.databricks.com/tmcxu86974/rss/board?board.id=data-governance
• Generative AI — https://community.databricks.com/tmcxu86974/rss/board?board.id=GenAI-Insight-Hub
• Machine Learning — https://community.databricks.com/tmcxu86974/rss/board?board.id=machine-learning
• Warehousing & Analytics — https://community.databricks.com/tmcxu86974/rss/board?board.id=warehousing-and-analytics

Added 2026-08-21#

• Power BI dashboards using Direct Lake — Data Engineering · Aug 21, 2026
• Questions about "Store OpenTelemetry traces in Unity Catalog" — Generative AI · Aug 20, 2026
• Using autoloader with multiple object types in load path — Data Engineering · Aug 20, 2026
• Streaming Doesn't Mean Your Compute Needs to Run Forever — Data Engineering · Aug 20, 2026
• Bedrock and Genie — Generative AI · Aug 19, 2026
• Microsoft Fabric One and Databricks Unity Catalog — Bidirectional Access Without Data Movement — Administration & Architecture · Aug 19, 2026
• Databricks delta tables and Iceberg — Data Engineering · Aug 19, 2026
• how to enable column level lineage in databricks — Data Engineering · Aug 18, 2026
• Account Console environment for demonstrating workspace management capabilities to clients ? — Administration & Architecture · Aug 18, 2026
• Azure Databricks Serverless Compute Unable to Connect to Azure SQL MI Using Failover Group FQDN via — Administration & Architecture · Aug 17, 2026
• Databricks streamlit app with write back capability and audit trail display — Warehousing & Analytics · Aug 12, 2026
• Unity Catalog Metric Views to be accessible to Custom Apps outside of DBX environment — Administration & Architecture · Aug 12, 2026
• Unity Catalog in Microsoft Azure Government — Administration & Architecture · Aug 07, 2026
• Genie LLM audit — Data Governance · Aug 04, 2026
• Genie task running for over 20 hours. — Generative AI · Jul 30, 2026
• Billing structure and LLM usage costs for no-code agents — Generative AI · Jul 30, 2026
• Can I connect Fabric Data Agent With Databricks Genie One as external connection — Data Governance · Jul 27, 2026
• Enterprise Unity Catalog RBAC model in Databricks — Data Governance · Jul 27, 2026
• Does user_identity in system.access.audit reflect the actual logged-in user for Service-Principal-mo — Data Governance · Jul 27, 2026
• 'Schedules' feature in Genie One — Generative AI · Jul 24, 2026
• UI sends empty managed_identity_id, breaks storage credential creation with system-assigned Access — Data Governance · Jul 22, 2026
• Genie One – Unable to Extract Usage Details — Warehousing & Analytics · Jul 14, 2026
• Can't select Serverless GPU on notebooks — Machine Learning · Jul 09, 2026
• Dashboard Creation Requiring Workspace Access Entitlement? — Warehousing & Analytics · Jul 08, 2026
• Serverless ML — Machine Learning · Jun 24, 2026
• Can Databricks Jobs Run on Kubernetes Clusters? — Machine Learning · Jun 23, 2026
• Network time-out — Warehousing & Analytics · Jun 22, 2026

Added 2026-08-27#

• Title: Oracle CDC pipeline (Lakeflow Connect) never terminates — Data Engineering · Aug 26, 2026
• How do ISVs prevent customers modifying a wheel deployed into their own Databricks workspace? — Administration & Architecture · Aug 25, 2026
• Distinguishing Runs Related to an All-Purpose Cluster — Data Engineering · Aug 24, 2026
• Lakehouse Federation (Snowflake) — large query results fail to download from internal stage — Data Engineering · Aug 24, 2026
• Got error using Opensharing Get Table API to foreign iceberg table — Warehousing & Analytics · Aug 24, 2026
• Databricks-hosted Foundation Model APIs blocked by workspace rate limit of 0 — Generative AI · Aug 24, 2026
• An AI agent that validates and PRs your Unity Catalog access requests, and knows when to say no — Data Engineering · Aug 24, 2026

Added 2026-08-30#

• Unity Catalog - Can i delete system tags? — Data Governance · Aug 28, 2026
• Model Serving An internal error occurred during feature store lookup all deploys failing — Machine Learning · Aug 27, 2026

Added 2026-08-31#

• How Can a Databricks App Render the Same Detailed Response as the Genie UI? — Generative AI · Aug 30, 2026

Added 2026-09-01#

• Best practices for data quality in lakeflow — Data Engineering · Sep 01, 2026
• Lakeflow connect — Data Engineering · Sep 01, 2026
• Unable to create Compute — Data Engineering · Aug 31, 2026

Added 2026-09-03#

• Is it possible to embed a Genie Space inside a Power BI report? — Warehousing & Analytics · Sep 03, 2026
• Serverless Scala JAR: Scala UDFs that read Row input fail — Data Engineering · Sep 03, 2026
• Streaming Considerations for Daily Batch Ingestion — Data Engineering · Sep 03, 2026
• Difference between Workspace and Unity Catalog experiments when using MLflow autologging? — Machine Learning · Sep 02, 2026
• Lakebase Postgre updating Delta Table. — Data Engineering · Sep 02, 2026
• Partitioning vs Liquid Clustering (per-table): — Data Engineering · Sep 01, 2026

Added 2026-09-05#

• Does enabling Photon improve performance while lowering compute costs? — Administration & Architecture · Sep 04, 2026
• Lakeflow connect Ingestion pipeline notification for gateway pipeline — Data Engineering · Sep 04, 2026
• Lakaflow connect Data Ingestion gatway pipeline fail often and restarts automatically — Data Engineering · Sep 03, 2026
• Bajar costos de lectura en databricks — Data Engineering · Sep 03, 2026

Added 2026-09-07#

• Lakeflow SDP Append Flow — Data Engineering · Sep 06, 2026

Added 2026-09-07#

• Genie Agent content search on a managed volume fails with "The request was invalid" — Generative AI · Sep 04, 2026

Added 2026-09-09#

• How are you separating dev, staging and prod in Unity Catalog without duplicating everything? — Data Engineering · Sep 09, 2026
• Harmonizing Informatica Governance Policies Across Databricks and Power BI Import Mode — Data Governance · Sep 08, 2026
• Could Databricks app support the integration of Genei Room and Power BI? — Warehousing & Analytics · Sep 08, 2026
• How are you attributing serverless costs back to individual jobs and teams? — Data Engineering · Sep 08, 2026
• Now that Unity Catalog manages Iceberg natively, is anyone actually switching? — Data Engineering · Sep 08, 2026
• Databrick Oauth federation — Administration & Architecture · Sep 08, 2026
• Using Databricks Asset Bundles and Lakeflow Jobs in a Real Project — Data Engineering · Sep 08, 2026
• Genie Agent response Export to PDF via API — Generative AI · Sep 08, 2026
• Using Delta Sharing to hold ISV entitlement data outside the customer's metastore — reasonable fit? — Administration & Architecture · Sep 08, 2026

Added 2026-09-11#

• Changing Root Metastore Location — Data Governance · Sep 11, 2026
• Free Trial – “Daily limit for workspace creation” error — Administration & Architecture · Sep 11, 2026
• Auto Loader stream fails on RocksDB checkpoint after enabling managed file events — Data Engineering · Sep 11, 2026
• Migrating Large-Scale Z-Ordered Tables to Liquid Clustering: Strategies for Production Pipelines — Data Engineering · Sep 11, 2026
• OAuth M2M (client-credentials) - getting error with github run — Data Engineering · Sep 11, 2026
• Allow Public Network Access - Disabled, I can still access the workspace from public internet — Administration & Architecture · Sep 10, 2026
• Databricks Asset Bundles: How to manage dependencies between volumes/files and cluster creation? — Administration & Architecture · Sep 10, 2026
• databricks-langchain PyPI installation stuck for 30+ minutes on multiple Azure Databricks clusters — Administration & Architecture · Sep 10, 2026
• shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for log — Data Engineering · Sep 10, 2026
• Auto Loader Strategy: Balancing Cloud Notification Costs vs. Directory Listing in Massive Migrations — Data Engineering · Sep 10, 2026

Added 2026-09-13#

• Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs — Data Engineering · Sep 12, 2026
• igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences? — Data Engineering · Sep 12, 2026
• Best Practice for Handling Schema Evolution with Auto Loader in Production? — Data Engineering · Sep 11, 2026

Added 2026-09-15#

• Lakeflow connect CT pipeline keeps running multiple connections in source — Data Engineering · Sep 15, 2026
• Feature enablement for Foundation Model Unity Catalog permissions — Data Governance · Sep 15, 2026
• Are attached-but-idle serverless notebooks billed? Idle timeout? 32 min job, 86 min billed — Data Engineering · Sep 15, 2026
• Non-deterministic ROW_NUMBER() results across Unity Catalog environments — Data Engineering · Sep 13, 2026

Added 2026-09-17#

• [FGAC_ON_DEDICATED_COMPUTE_FAILED] Fine-grained access control (FGAC) on dedicated compute failed — Administration & Architecture · Sep 17, 2026
• using remote_query with SQL Serverless Warehouse — Data Engineering · Sep 17, 2026
• Error with Databricks ODBC Driver OAuth 2.0 — Administration & Architecture · Sep 16, 2026
• Photon enabled but a large share of the plan is falling back, cost up and runtime flat — Data Engineering · Sep 16, 2026
• Notebook commands hang after restarting a cluster with custom library installed — Data Engineering · Sep 15, 2026
• Databricks architecture drawn step by step: control plane, compute plane, Delta, Unity Catalog — Data Engineering · Sep 15, 2026

Added 2026-09-19#

• Predictive Optimization for Streaming tables in Lakeflow pipelines — Data Engineering · Sep 19, 2026
• Data ingestion: Setting up a connector for Postgres with CDC enabled. — Data Engineering · Sep 18, 2026
• Liquid Clustering vs Z-Order vs Partitioning — Data Engineering · Sep 18, 2026
• Genie Agent refuses to add scalar SQL function from Unity Catalog — Generative AI · Sep 18, 2026
• Personal Compute Policies — Data Governance · Sep 18, 2026

Added 2026-09-21#

• How I explain classic vs serverless compute, and the limitation list I check before migrating a job — Administration & Architecture · Sep 21, 2026
• Unauthorized network access to workspace — Administration & Architecture · Sep 21, 2026
• Serverless Capabilities Not Available In My Workspace — Data Engineering · Sep 21, 2026
• How I explain account vs workspace vs Unity Catalog metastore to new platform engineers — Data Engineering · Sep 20, 2026
• Power of Genie — Generative AI · Sep 19, 2026
• Managing "Secret" Injection in DABs across Dev/Stage/Prod — Data Engineering · Sep 19, 2026
• Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Consumption — Data Engineering · Sep 19, 2026

Added 2026-09-23#

• Genie Higher Environment deployment using DAB — Generative AI · Sep 23, 2026
• Question about MySQL Integrated‑CDC pipeline on Classic compute workspace — Data Engineering · Sep 23, 2026
• Unity Catalog v.2.0? — Data Engineering · Sep 22, 2026
• Unity Catalog Volume as spark checkpoint location (in 2026) — Data Engineering · Sep 22, 2026

Added 2026-09-25#

• DeepSeek V4.1 Flash returns workspace ITPM rate-limit error on fresh/idle workspace — Generative AI · Sep 25, 2026
• Genie One Foundations: What Data Teams Should Get Right Before Rolling It Out — Data Engineering · Sep 25, 2026
• org.apache.iceberg.connect.IcebergSinkConnector to sink data from kafka to databricks - large volume — Data Engineering · Sep 25, 2026
• Operationalizing Lakeflow Connect:Handling Upstream Schema Evolution & Historical Backfill — Data Engineering · Sep 24, 2026
• Access to Lakehouse Real-Time (RT) in the preview tab — Administration & Architecture · Sep 24, 2026
• Genie agent hyperlink creation — Data Engineering · Sep 24, 2026
• Is Advanced edition required for Serverless pipeline? — Data Engineering · Sep 24, 2026
• Genie Agents citation (bug) — Generative AI · Sep 24, 2026

Added 2026-09-29#

• Connecting to Snowflake via Workload Identity Federation — Data Engineering · Sep 28, 2026
• Liquid clustering vs. partitioning for a 5 TB Silver table with frequent MERGEs — Data Engineering · Sep 28, 2026

Added 2026-10-01#

• Lakeflow connect SQL server ingestion can be made elastic? — Data Engineering · Sep 30, 2026
• How to build FinOps chargeback by cost centre — allocating the whole cloud bill, not just your DBUs — Warehousing & Analytics · Sep 30, 2026
• Photon enabled but a large share of the plan is falling back, cost up and runtime flat — Data Engineering · Sep 30, 2026
• Built my first Databricks App with Lakebase — Data Engineering · Sep 30, 2026
• Implementing a "Zero-Bus" Architecture with Unity Catalog — Data Engineering · Sep 29, 2026
• Implementing a "Zero-Bus" Architecture in a Lakehouse: Best Practices for Shared Dimensions? — Data Engineering · Sep 29, 2026

Added 2026-10-03#

• Lakeflow connect — Data Engineering · Oct 03, 2026
• Serverless compute is not available in my account/workspaces — Administration & Architecture · Oct 03, 2026