Delta and storage

Delta table and storage design

guideazure1 min read4 days old

Table design is an operating decision: layout, retention, optimization, and ownership affect cost and reliability long after the first write succeeds. Prefer Unity Catalog managed tables unless an explicit interoperability or lifecycle requirement justifies external ownership.

Design rules#

  1. Choose ownership first. Record whether Databricks or another platform owns the files, metadata, retention, and deletion lifecycle.
  2. Partition sparingly. Do not copy a source partition scheme by habit. Use query patterns, data volume, and file size to justify physical layout.
  3. Evaluate liquid clustering for new large tables. Choose clustering keys from selective, frequent filters and joins, then verify benefit from query evidence rather than assumptions.
  4. Control small files at the writer. Tune ingestion cadence and write behavior before adding an endless repair schedule.
  5. Treat VACUUM as destructive. Retention must cover the real rollback, streaming, clone, and concurrent-reader windows. Never shorten it merely to save storage.
  6. Use one optimization owner. If predictive optimization owns OPTIMIZE and VACUUM, remove overlapping scheduled jobs.
  7. Measure outcomes. Track table size, file count, query scan, optimization cost, freshness, and failed writes before and after a layout change.

Review questions#

  • Who owns the object lifecycle and disaster-recovery copy?
  • Which workloads require time travel, replay, streaming checkpoints, or external-engine access?
  • Are clustering keys based on current workload evidence?
  • Can a compaction or vacuum operation collide with ingestion or a long-running reader?
  • Is the table's retention policy documented independently from the code default?

Official sources#

Referenced by