Is a Schema Check Enough?
Where Schema Validation Stops and a Data Contract Starts

Picture a partner file of web signups landing in your Bronze folder. Your validation step passes: every column is present and every type is right. A week later someone asks why the welcome email bounced for a batch of new users, and you find "not-an-email" sitting in the email column, a negative age, and a new phone column nobody agreed to receive.

This happens because the check answered the question it was given (is this the right structure?) and nobody wrote down the other questions: which values are acceptable, what to do with a row that breaks the rule, and what to do when the producer changes the file.

This post separates those questions, shows which tools answer each one today, and explains where a data contract adds something a schema check does not. It does not argue that you should replace your schema validation; in many cases it is enough, and the post says when.

Four questions, not one

A validation step at the edge of a lakehouse can answer up to four different questions:

1. Shape: are the expected columns present, with the expected types and nullability?
2. Values: is each value acceptable (a plausible email, an age of zero or more, a status from a known list)?
3. Consequence: when a row fails, does the whole load fail, does the row vanish, or is it kept aside with a reason?
4. Change: when the producer adds or drops a column, what is the agreed behaviour, and who decides?

“Schema validation” usually means question 1, and many tools answer question 2 as well. Questions 3 and 4 are where teams most often have no written answer, and where the difference between a schema and a contract becomes practical.

Why the shape-versus-meaning line is too simple

It is tempting to say a schema cannot catch "not-an-email". That is not accurate. JSON Schema can restrict a string with a pattern, and dataframe libraries such as Pandera attach value checks to each column. A schema that only declares types is a choice, not a limit of the idea.

What a schema rarely carries is the policy around the check. It says what valid looks like; it usually leaves the consequence of invalid data, and the response to a changed file, to whatever code calls it. That code is where behaviour drifts between pipelines.

The ways teams answer these questions today

These all work, and many teams combine two of them.

Some detail behind the table. Pandera’s drop_invalid_rows removes failing rows from the returned dataframe; lazy validation separately reports a failure_cases table you can use to keep them. A dbt model contract checks column names and data types before the build, and dbt’s docs point to data tests for content. A Delta CHECK constraint fails the whole transaction when any row violates it. Databricks expectations let each rule warn, drop or fail, and log a count of dropped rows; keeping those rows is a separate pattern. Great Expectations can return the indices or a query for unexpected rows, which your pipeline then acts on.

None of these is wrong. The gap appears when the answers to questions 3 and 4 live in orchestration code, copied per pipeline, rather than next to the rules they apply to.

What a data contract adds

A data contract, as LakeLogic uses the term, is one YAML file per table that declares all four answers together: the fields, the value rules, what happens to rows that fail, and the schema change policy, plus ownership metadata. The LakeLogic runtime reads the file and applies it when it processes the data.

Here is a contract for the signups file. It is the file used in the run shown below.

version: "1.0.0"
info:
  title: Bronze Web Signups
  owner: marketing-data          # declared owner
dataset: bronze_web_signups

schema_policy:                    # question 4: change
  evolution: strict
  unknown_fields: drop

model:                            # question 1: shape
  fields:
    - name: signup_id
      type: string
      required: true
    - name: email
      type: string
      required: true
    - name: event_date
      type: date
      required: true
    - name: age
      type: int

quality:                          # question 2: values
  row_rules:
    - name: email_format
      sql: "email LIKE '%@%.%'"
      category: validity
    - name: age_not_negative
      sql: "age IS NULL OR age >= 0"
      category: validity
  dataset_rules:
    - name: has_rows
      sql: "SELECT COUNT(*) FROM source"
      must_be_greater_than: 0

quarantine:                       # question 3: consequence
  include_error_reason: true

A worked run

Suppose the file holds five rows: one clean, one with not-an-email, one with age = -5, one with no email, one with no age, and an extra phone column. Running it with LakeLogic 1.88 on the Polars engine:

from lakelogic import DataProcessor

result = DataProcessor("contract.yaml").run_source("signups.csv")
print(result.good)   # rows that passed
print(result.bad)    # rows that failed, with _lakelogic_errors

# result.good: 2 rows, columns signup_id, email, event_date, age (phone dropped)
SU-1   jane@example.com   2026-03-01   32
SU-5   lee@example.com    2026-03-02   null

# result.bad: 3 rows
SU-2   not-an-email      ["Rule failed: email_format (email LIKE '%@%.%')"]
SU-3   sam@example.com   ["Rule failed: age_not_negative (age IS NULL OR age >= 0)"]
SU-4   null              ["Rule failed: email_required (\"email\" IS NOT NULL)",
                          "Rule failed: email_format (email LIKE '%@%.%')"]

# log
Quality Check: has_rows | Result: 2 (expected > 0.0) | Status: PASS
Schema drift detected for 'Bronze Web Signups': missing=[], unknown=['phone']

Three things in that output answer the questions a schema check leaves open:

• Failing rows are kept with a reason. Each row in result.bad carries _lakelogic_errors, a list of every rule it failed. required: true on a field generates its own rule (email_required), so a missing email and a malformed one are reported separately.
• The new column follows a declared policy. With unknown_fields: drop, phone is removed from the output and the drift is logged. With unknown_fields: quarantine, every row in the batch went to quarantine with “Unknown fields present: phone”, because the column is present on every row. Choose that setting only if you want a surprise column to stop the whole batch for review.
• Batch checks run on the valid rows. The has_rows dataset rule measured 2, the count after the split. By default a failing dataset rule is logged and notified; setting quality.fail_pipeline_on_dataset_error: true makes it stop the run.

The same contract runs on the other LakeLogic engines (DuckDB, Spark, Snowflake, BigQuery). The run above was on Polars only; the exact text in _lakelogic_errors can differ by engine, so do not parse it.

What happens to the quarantined rows next, and when you should fail the run instead, is the subject of Fail the Run or Quarantine the Row?

What the contract declares versus what it proves

Be precise about what each part of the file establishes:

• Checked on every run: field presence and types, row rules, dataset rules, the unknown-field policy.
• Declared only: info.owner records who owns the table. It does not show that the team has accepted that responsibility, and it controls no access.
• Optional: lineage columns such as _lakelogic_source and _lakelogic_run_id are added only when the contract sets lineage.enabled: true.

A passing run proves the declared rules held. It does not prove the data is right in ways nobody wrote down.

When you do not need a data contract

• Validating API requests or events in an application. JSON Schema or Pydantic in process is the natural fit.
• Everything is dbt and failing the build is acceptable. Model contracts plus data tests cover shape and values.
• One table with hard limits. A Delta CHECK constraint is simpler if rejecting the whole write is what you want.
• A single Pandera pipeline where you already keep failure_cases and nobody else needs the rules.

A contract earns its place when several tables, often owned by different teams, need the same answers to all four questions written in one reviewable file instead of in each pipeline’s code.

Limits worth knowing

• A contract is another file to maintain. If nobody reviews it when the source changes, it drifts like any other code.
• Simple patterns are not full validation. LIKE '%@%.%' catches obvious junk, not every invalid address.
• Quarantine needs an owner. Kept rows help only if someone reads them and fixes the cause.
• Rules cost compute. Each row rule adds work to every run; keep expensive checks for high-risk tables.

Key takeaways

1. Split validation into four questions: shape, values, consequence and change.
2. Schema tools can check values too; the common gap is the policy around a failure.
3. Decide in writing what happens to a failing row and to an unexpected column.
4. Keep failed rows with the rule they broke, so the producer can fix the cause.
5. Treat an owner field as a declaration, not proof of accountability.