Practitioner notes from r/dataengineering and r/databricks
This archive preserves the last monthly snapshot from r/dataengineering and r/databricks. Reddit retired public RSS on 2026-11-13.
Verify technical claims against official documentation.
Databricks and lakehouse#
- Data Engineering Lab to Practice on your Laptop — r/dataengineering · 2026-10-19 Hi, I created a DE stack (fully integrated services) and learning material for those interested in self learning. I kept Databricks, fabric in mind while building this stack. Feel free to use it and let know if there are any improvements/suggestion I can make - Work independently in your own data lakehouse. - Invite
- A full month-end finance app built on Databricks Apps — r/databricks · 2026-10-24 Below are the things covered in the video: 1. Overview: drag the month slider and every figure on the page moves with it, including a cash flow diagram from sales through to profit 2. P&L: the statement drawn as a ladder from revenue down to profit, plus a bridge showing why we beat or missed budget 3. Cafes: ranking,
- Fabric vs Snowflake/Databricks — r/dataengineering · 2026-11-05 Not a troll post but an honest question, and maybe more about Microsoft vs Snowflake/Databricks. Why is Fabric so far behind its competitors. Not so much from the perspective of specific UI/UX or functionality, but more of a question of why MS engineers are seemingly incapable of just copying and improving on their
- I enjoy managing access control — r/dataengineering · 2026-10-23 I admit it. I just spent weeks tidying up our Databricks access layer. I created entra groups for sensitive schemas, terraform to deploy them and set the entitlement and grants for admins, engineers, and default users. My technical colleagues would ask what I was up to, and when I told them they would sigh with
- Had a call with a Databricks PM this week and found out about all the cool things! — r/databricks · 2026-10-22 Spent 30 minutes on a product discovery call with u/saad-the-engineer from Databricks on Tuesday. Supposed to be about Unity Catalog triggers for jobs. We wandered. At one point I was explaining how I run bronze to silver on most client builds. It’s table-driven, with three Delta tables holding the config: one for
- Databricks Genie Ontology Explained | Demo with Databricks Product Manager — r/databricks · 2026-10-19 Hey everyone! Elise Georis (PM at Databricks) partnered with me to do a 35-min deep-dive (chat + demo) on Genie Ontology. We covered a lot of ground and hope you enjoy learning more about it. Some of the topics covered: -Modeled ontology: governed metric views, Pages, and domains. -Learned ontology: snippets extracted
- Should I move from a snowflake environment to a fabric environment to get back to a DE role? — r/dataengineering · 2026-10-17 I have been a DE for over 10 years working on SSIS and later building pipelines on databricks and dbt. Earlier this year I took up a new job as a manager of an analytics team. I hate my role and the lack of ownership and hands on work. I am primarily tasked with leading a team on reporting and data delivery and
- Which Databricks feature do you think is the most underrated right now? — r/databricks · 2026-11-12 For me, it’s system tables. I ignored them for a long time, but now I use them to see which jobs cost the most, who’s running what, and which clusters sit idle. It helped us find waste we didn’t even know about. It made me wonder what other useful features people are missing. -What feature do you think doesn’t get
- Databricks Metric Views are coming back to Power BI! [Private Preview] — r/databricks · 2026-11-12 First I'd like to give a bit of background because if you feel like you've heard about this feature before, you're right. Previous attempt to bring Metrics Views to PBI was "blocked" when MS removed the required opiton from the connector. In February 2026 - the Power BI Databricks connector introduced Metric View BI
- Data Engineer Niche tech Stack? — r/dataengineering · 2026-11-04 Hey DE super heroes, Let's help each other. What are the niche tech stacks/skills you are coming across in the Job Description and what we can learn? According to my knowledge - Python, Pyspark, sql = Mandatory Snowflake, dbt, ADF, Databricks= I am seeing in most of the JD. Now, it's your turn. 8+ YOE in the DE world 🌎
- Databricks Genie agent. — r/databricks · 2026-10-24 My company wants to roll out a Databricks Genie agent to around 5,000 users, with these requirements: Accuracy: Answers should consistently meet 90%+ accuracy. Reliability as data grows: The agent should not hallucinate or degrade when more tables are added. Scalability: The solution should extend from one business
- Databricks World. I think they forgot the Pirates of Renaming — r/databricks · 2026-10-20 I really liked the Databricks World map they showed at DAIS, so I decided to make my own version. But I felt like something was missing... The Pirates of Renaming Not long ago, I was trying to figure out the difference between DLT and SDP. But it seems like the last name is Lakeflow Pipelines. What should I add to the
- Job completion triggers just hit Beta, and they replace a pattern I've built too many times!! — r/databricks · 2026-11-09 The usual way to chain jobs across teams has always been a bit ugly. Either a parent job with Run Job tasks that has to own everything downstream, or a schedule set 30 minutes after the upstream job usually finishes, which works until the day it doesn't. Databricks just added a job completion trigger in Beta. A job
- Databricks compute management is so bad — r/databricks · 2026-10-20 Why serverless pipelines have to wait 10 minutes to get compute? Why don’t you track the compute usuage across region or pool and already keep certain number pre warmed computes available?? I don’t want to pay more money for performance optimized serverless compute.
- Cut Your MCP Token Bill by 97%: Code Mode on Databricks — r/databricks · 2026-10-26 submitted by /u/Limp-Park7849 to r/databricks [link] [comments]
- DataFrame caching is now in Public Preview on Databricks serverless compute: df.cache() and df.persist() finally work there — r/databricks · 2026-10-26 But it is not the cache you know from classic clusters, and the differences will bite if you port code without reading them. Until now, an iterative workload on serverless recomputed its plan on every action. A notebook that hits one expensive join ten times, or a feature loop over the same base table, was a common
- Starting out on Databricks — r/databricks · 2026-10-30 Hi everyone! I’m a software engineer who moved into data engineering around a year ago and have been learning and working with Databricks. So far, I’ve worked on loading data from ADLS and external locations, basic PySpark transformations, implementing the medallion architecture using notebooks and SQL, and scheduling
- Databricks Apps: Are you using them in production yet or keeping it to internal prototypes? — r/databricks · 2026-11-08 Hey everyone, I’ve been diving into Databricks Apps recently and wanted to see how others are adopting it in their organizations. Historically, whenever we wanted to spin up a Streamlit app or a custom Python dashboard connected to our data in Databricks, it meant dealing with a headache of external
- Audio transcription in SQL — r/databricks · 2026-10-22 ai_transcribe is another databricks AI function. It converts audio into timestamped text with speaker labels, directly from SQL. The Beta currently supports English and Spanish. more news
- This Halloween, I'm building a horror game where the only way to survive is to migrate to Databricks — r/databricks · 2026-10-31 What could be scarier than a haunted house? A legacy data warehouse—the kind some companies spend years trying to migrate away from. This Halloween, I’d like to share a small project of mine: a game called SAWhouse: Migration to Databricks. It’s an educational horror game inspired by the atmosphere of the SAW film
- Modern Family TV Show: Analytics Dashboard Built with Databricks Apps — r/databricks · 2026-11-01 A favourite show + excuse to build something beyond the usual dashboards. Behind the scenes -> six tables bring everything together + Python. The data comes from Wikipedia and IMDb's non-commercial datasets.
- Genie App Builder — r/databricks · 2026-10-30 Genie App Builder creates a Databricks app from a short description, and we can refine it in chat. Great for building enterprise apps, and I don't need Power Automate anymore! more news
- Databricks Genie Ontology is now GA, and new Genie plugins bring it into ChatGPT and Claude. 🧞 — r/databricks · 2026-11-12 Read more on LinkedIn:
- JDBC connections in Unity Catalog — r/databricks · 2026-11-02 JDBC connections in Unity Catalog are now generally available on serverless compute and DBR 19 or newer. We can store the driver, endpoint, and credentials once, then use the connection by name in notebooks. More news
- Is Databricks the right solution for a team of 3 data scientists? — r/databricks · 2026-10-29 As a small 3 person data science team we really need a way of easily deploying and managing our models, as well as connecting to our siloed data sources. We currently use Vertex AI (as I still call it) where we spend way too long productionising and we are performing several different roles to deliver the models. It
- Databricks 3D — r/databricks · 2026-11-10 Just found a cool Databricks 3D Model, will try to print it today!
Other practitioner signal#
- Job completion triggers (now available in Beta!) — r/databricks · 2026-11-03 You can now trigger a job on completion of one or more upstream jobs. You could always trigger downstream jobs using the run job task type but that required the parent having knowledge of all downstream jobs. Now you can simply select an upstream job and watch for its completion to trigger the downstream job. Bonus:
- Shifting from DE to AI Ops Engineer — r/dataengineering · 2026-11-01 I have been searching DE job since January 2026 and although I had few initial interviews but I was not selected for the next round. From 2021 to may 2024, I've held Data Engineer job title with 4 contract roles. Prior to that I was doing MSSQL Development, SSIS. In my free time I did DevOps/DE projects on AWS Cloud
- What does the day to day life of a solutions architect look like? — r/databricks · 2026-11-10 I am a SWE and will be having a call to discuss a solutions architect role. Trying to learn more about the day to day life and see if it's a role I'm interested in. main concern is that I have only been in the industry for a few years and being a solutions architect might keep me from gaining technical depth and
- What's new in AIBI Dashboard - October 2026 — r/databricks · 2026-11-04 I couldn't add the screenshots easily you can find them over here This release brings extensive additions to AI/BI Dashboards, expanding layout controls, visualization formatting, BI migration capabilities via Genie Code, and interactive filtering options. AIBI, Genie and BI Migration Bulk import from Power BI and