0036 — Continue the lakehouse without metadata recovery
Cut the lakehouse over to Airflow without metadata backup and without keeping the legacy writer as a fallback.
Public edition condensed on 2026-10-21; original decision date and meaning retained.
Replaced by ADR 0037. Read it before applying this decision.
Context
ADR 0033 chose Airflow and the Spark Operator, and ADR 0035 accepted metadata loss during learning. Five consecutive shadow runs and controlled recovery checks passed, so the writer cutover was ready. The metadata was still disposable, and keeping both a legacy writer and a database recovery path added work without protecting anything valuable.
Decision
Proceed without Airflow metadata backup or restore, and accept total metadata loss. Do not keep the legacy writer after cutover.
- Stop the legacy writer before Airflow receives authoritative write access.
- Run one manual authoritative Workflow Run, and require it to pass Spark and read-only Trino validation.
- Remove the legacy writer configuration through reviewed Git.
- Start the Airflow schedule.
After that, repair failures on the Airflow and Spark Operator path.
This decision grants no live-operations authority. Git edits, Flux actions, workload removal, and storage deletion each remain a separate boundary. Authoritative Iceberg data and required evidence stay protected from control-plane cleanup. The learning deployment keeps its review date.
Alternatives considered
- Prove metadata recovery first: the metadata had no durable value.
- Keep a fallback writer: maintains two workflow paths.
- Delay cutover: unnecessary after the shadow and recovery evidence.
Consequences
Losing the CNPG volume can erase all workflow history, and in-flight work may need a fresh run. The removed legacy path cannot be restored from Git without reconfiguration.
ADR 0037 later refined the scheduled-observation requirement.