# Delta table and storage design

Table design is an operating decision: layout, retention, optimization, and ownership affect
cost and reliability long after the first write succeeds. Prefer Unity Catalog managed tables
unless an explicit interoperability or lifecycle requirement justifies external ownership.

## Design rules

1. **Choose ownership first.** Record whether Databricks or another platform owns the files,
   metadata, retention, and deletion lifecycle.
2. **Partition sparingly.** Do not copy a source partition scheme by habit. Use query patterns,
   data volume, and file size to justify physical layout.
3. **Evaluate liquid clustering for new large tables.** Choose clustering keys from selective,
   frequent filters and joins, then verify benefit from query evidence rather than assumptions.
4. **Control small files at the writer.** Tune ingestion cadence and write behavior before adding
   an endless repair schedule.
5. **Treat `VACUUM` as destructive.** Retention must cover the real rollback, streaming, clone,
   and concurrent-reader windows. Never shorten it merely to save storage.
6. **Use one optimization owner.** If predictive optimization owns `OPTIMIZE` and `VACUUM`, remove
   overlapping scheduled jobs.
7. **Measure outcomes.** Track table size, file count, query scan, optimization cost, freshness,
   and failed writes before and after a layout change.

## Review questions

- Who owns the object lifecycle and disaster-recovery copy?
- Which workloads require time travel, replay, streaming checkpoints, or external-engine access?
- Are clustering keys based on current workload evidence?
- Can a compaction or vacuum operation collide with ingestion or a long-running reader?
- Is the table's retention policy documented independently from the code default?

## Official sources

- Delta Lake on Azure Databricks — https://learn.microsoft.com/azure/databricks/delta/
- Liquid clustering — https://learn.microsoft.com/azure/databricks/delta/clustering
- Predictive optimization — https://learn.microsoft.com/azure/databricks/optimizations/predictive-optimization