Data engineering and Lakeflow

Lakeflow data engineering

guideazure2 min read4 days old

Use this page to choose the ingestion, transformation, and orchestration boundary for an Azure Databricks data product. Check current product documentation for connector capabilities and feature status.

The three responsibilities#

Responsibility Databricks surface Own explicitly
Ingest Lakeflow Connect or custom ingestion Source cursor, schema drift, retries, quarantine, and reconciliation
Transform Lakeflow Spark Declarative Pipelines or jobs Data contracts, expectations, state, backfills, and table ownership
Orchestrate Lakeflow Jobs Dependencies, parameters, identities, notifications, and recovery

Do not collapse all three into one notebook. A source extraction failure, a data-quality failure, and a publish failure need different recovery paths and different evidence.

Default flow#

  1. Land source-faithful data with ingestion metadata and a repeatable cursor.
  2. Quarantine malformed or contract-breaking records instead of silently dropping them.
  3. Standardize types, identifiers, deduplication, and late-arriving behavior in a durable layer.
  4. Publish business tables only after reconciliation and data-quality checks pass.
  5. Make every stage rerunnable for a bounded time range without duplicating results.
  6. Record source counts, target counts, rejected rows, freshness, and the code version for each run.

Managed connector or custom code?#

Prefer a managed connector when it supports the required source objects, authentication, incremental semantics, network path, and recovery behavior. Use custom ingestion when one of those is missing or when the source contract requires transformations before landing. The decision is operational. Choose the path that the team can reconcile and restore during an incident.

Production checklist#

  • Service principals or workload identities own production runs; personal identities do not.
  • Schemas and data-quality expectations are versioned with the pipeline.
  • Backfill, replay, and schema-change procedures are tested before cutover.
  • Notifications identify the failed stage and link to useful evidence.
  • Compute policies, tags or budget policies, and retention are defined.
  • Deployment uses Declarative Automation Bundles or another reviewed promotion path.

Official sources#

Referenced by