Picture an overnight load of web signups into Bronze. Near the end, one row carries a
null where the job expects a date, and the job fails. The morning dashboard shows
yesterday’s numbers, and someone starts reading a stack trace to find one record among
hundreds of thousands.
It happens because the pipeline has only two outcomes built in: succeed with everything, or fail with
everything. The usual patch, a try/except that logs and skips, swaps a loud failure for a
silent one: the row disappears and nobody can say how many went or why.
This post gives a way to decide, table by table, whether a bad row should stop the run, be dropped, or be quarantined (kept aside with the reason it failed). It covers the common ways to implement each and the work quarantine leaves you. It does not claim quarantine is always right; for some tables, stopping is.
Three outcomes for a bad row
| Outcome | What consumers see | Main risk |
|---|---|---|
| Fail the run | Stale but consistent data | One row blocks every other row |
| Drop the row | Fresh data, silently incomplete | No trail; producer never learns |
| Quarantine the row | Fresh data, known gaps | A queue someone must work |
Quarantine is not free. A table that loads without its bad rows is a partial load. For a signups feed that is usually fine; for a ledger that must balance, a partial load can be worse than no load.
A rule for choosing
If a bad row makes the other rows wrong, stop the run. If a bad row is only wrong itself, quarantine it and let the rest through.
Applied to typical tables:
| Table or check | Suggested outcome | Why |
|---|---|---|
| Events, signups, clickstream | Quarantine | Each row stands alone |
| Bronze to Silver cleaning | Quarantine | External sources you do not control |
| Ledgers, payroll, reconciliations | Fail the run | Totals must be complete |
| Missing or retyped columns | Fail the run | Affects every row, not one |
| Too many rows rejected | Fail or alert | A broken feed, not a bad row |
| Contract tests in CI | Fail | Block the change before merge |
The last data row matters: quarantining one row in a million is a different event from quarantining half the batch. Whatever you choose per row, decide separately what share of rejects means the feed itself is broken.
The ways teams implement this
These all work, and the right one often depends on the platform you already run.
| Approach | Outcomes offered | Watch out for |
|---|---|---|
| Delta CHECK constraints | Fail the write | No per-row option |
| Databricks expectations | Warn, drop or fail per rule | Dropped rows counted, not kept |
| Pandera | Raise, or drop invalid rows | Keeping rows is your code |
| Great Expectations | Report failures, trigger actions | Splitting rows is your code |
| Hand-written split | Anything you build | Rules and reasons drift per pipeline |
Behind the table:
Delta CHECK constraints
fail the transaction when any row violates them.
Databricks pipeline expectations
log a count of dropped records, and the docs describe quarantine as a separate pattern you implement.
Pandera’s drop_invalid_rows
removes failing rows from the result.
Great Expectations
can return the indices or a query for unexpected rows. A hand-written WHERE split into good and
bad tables is easy for one or two tables.
The open question is consistency: the choice of outcome, the rules and the recorded reason often live in different places, and each new pipeline repeats them slightly differently.
How LakeLogic handles it
In LakeLogic the rules and the outcome sit in the table’s contract. Quarantine is on by default: rows
that fail a rule go to the bad output with an _lakelogic_errors list naming each rule they failed,
and the valid rows continue.
quality:
row_rules:
- name: email_format
sql: "email LIKE '%@%.%'"
- name: age_not_negative
sql: "age IS NULL OR age >= 0"
fail_pipeline_on_dataset_error: false # true: a failed dataset rule stops the run
quarantine:
include_error_reason: true
fail_on_quarantine: false # true: any quarantined row fails the run
Running the five-row signups file from the schema validation post through the CLI with LakeLogic 1.88 on Polars:
lakelogic run --contract contract.yaml --source signups.csv \ --output-good good.parquet --output-bad bad.parquet Run complete [bronze_web_signups] | src=5 rows=5 ok=2 qtn=3 60.00% Saved good records to good.parquet Saved quarantined records to bad.parquet
signup_id email age _lakelogic_errors SU-2 not-an-email 28 ["Rule failed: email_format (email LIKE '%@%.%')"] SU-3 sam@example.com -5 ["Rule failed: age_not_negative (age IS NULL OR age >= 0)"] SU-4 null 41 ["Rule failed: email_required (...)", "Rule failed: email_format (...)"]
The quarantined rows keep their original values, so a reviewer can see exactly what arrived. In a
Python pipeline the same split is result.good and result.bad from
DataProcessor(contract).run_source(path). In a scheduled pipeline the bad rows are written to
the contract’s quarantine.target; if none is set, LakeLogic logs that the quarantined
rows were not saved.
When a table should stop instead, the contract says so:
quarantine.fail_on_quarantine: truemakes the pipeline runner fail the run when a table quarantines any row, listing up to ten failing rules. Valid rows have already been written by then; what it stops is the rest of the run.quality.fail_pipeline_on_dataset_error: truestops the run when a dataset rule (such as a row count or uniqueness check) fails. Without it, the failure is logged and notified.- Without either, the runner still reports a table that quarantined every row, or more rows than it kept, as a partial result rather than a plain success.
Rejected-share limits across runs, and batch checks in general, are covered in Quarantine Bad Rows, Check Every Batch.
What quarantine still leaves you to do
- Own the queue. Someone has to read the quarantine output, contact the producer or fix the rule. An unread quarantine table is a slower way of dropping rows.
- Plan the replay. When a source is corrected, decide how fixed rows re-enter: reload the source, or re-run the contract over the quarantined rows and merge by key. Design this before you need it.
- Tell consumers about gaps. A dashboard built on a partial load should know it is partial.
The run counts (
ok=2 qtn=3) are the evidence to surface. - Watch column-level changes. With
schema_policy.unknown_fields: quarantine, an unexpected column quarantines every row in the batch, because the column is on every row. That may be what you want; know it before you set it.
When you do not need LakeLogic for this
- One or two tables: a hand-written good/bad split with a reason column is easy to maintain.
- Databricks declarative pipelines where dropping with a logged count is acceptable: expectations do it natively.
- Tables that must always be complete: a constraint that fails the write is simpler than quarantine.
LakeLogic helps when many tables need the same decision written next to their rules, with the reason recorded the same way on every quarantined row.
Key takeaways
- Choose the outcome per table: stop, drop or quarantine, by whether one bad row makes others wrong.
- Never drop silently; keep the row and the rule it failed.
- Set a separate limit for how many rejects mean the feed is broken.
- Give the quarantine an owner and a replay plan before turning it on.
See quarantine on real rows
The Data Quality and Trust notebook feeds clean and dirty records through a contract and shows each quarantined row with its reason. It runs in Google Colab.