LakeLogic Core · open source

Governed pipelines,
defined in YAML.

Standardise ingestion, transformation, quality checks and materialisation with executable data contracts. Validate locally, check contract changes in CI, and run them with Polars, DuckDB or Spark.

Apache 2.0 · Python 3.10+ · Polars · DuckDB · Spark

Run the 5-minute quickstart View on GitHub

PyPI …

orders_contract.yaml

# Business expectations as code
model:
  fields:
    - name: order_id
      type: integer
      required: true
    - name: amount
      type: float
      required: true
quality:
  row_rules:
    - name: positive_amount
      sql: "amount > 0"

100Input

97Accepted

3Quarantined

Stop rebuilding the same controls in every pipeline. LakeLogic turns version-controlled contracts into executable validation, quarantine and materialization behaviour while complex transformations remain normal Python or SQL.

Start with the base package

Polars and DuckDB are included. Add the Spark extra when you need distributed lakehouse execution.

pip install lakelogic

pip install "lakelogic[spark]"

One contract, three control points

Move feedback closer to the change

01 · Develop

Validate representative data locally

Run schemas and quality rules with Polars or DuckDB before paying the startup cost of a remote Spark environment.

02 · Review

Check contract changes in CI

Use configured gates to identify breaking schemas, missing PII declarations and lineage changes before deployment. Some gates require comparison context.

03 · Execute

Retain failures with evidence

Separate accepted and failed records, preserve diagnostic context, and stop downstream work when dataset-level thresholds are breached.

Engine boundaries matter. The contract model is shared, but engines and catalog combinations are not identical. Review the capability matrix before choosing Spark, Polars, DuckDB, Delta or Iceberg.

See the pattern in a
working data-mesh example

RideFlow on Databricks

RideFlow is a fictional business used to demonstrate domain-owned data products on Databricks Unity Catalog, with the same contracts also running on Microsoft Fabric and Snowflake. The repository shows contracts, Bronze–Silver–Gold flows, quarantine, lineage, SCD Type 2 and cross-domain dependencies.

Inspect real YAML contracts and deployment definitions.
Trace accepted and quarantined outputs through the pipeline.
See where LakeLogic stops and Databricks infrastructure takes over.

Explore the reference implementation

Reference architecture

Domain autonomy with shared controls

Independent domain products use common contract machinery without claiming that YAML alone creates organisational ownership or platform governance.

Bronze Silver Gold

7Domains

3Layers

View architecture and code →

Faster onboarding

Infer a draft, then review it

infer_contract() can inspect a supported file or table sample and return a draft with fields, types and suggested rules. The draft remains reviewable YAML—not an invisible production decision.

PII detection and AI-assisted descriptions are optional capabilities with additional dependencies or configured providers.

draft_contract.py

from lakelogic import infer_contract

draft = infer_contract(
    "data/orders.csv",
    title="Orders Bronze",
    suggest_rules=True,
)

draft.show()
draft.save("contracts/orders.yaml")

Where LakeLogic fits

Start with pipelines.
Expand with evidence.

Reliable data products

Put schemas, quality rules, ownership metadata and service expectations beside the code that produces the dataset.

Domain-owned lakehouses

Reuse contract machinery across teams while domains retain responsibility for business meaning and product changes.

Deterministic AI boundaries

Let AI assist with drafts or proposed changes, then use explicit contracts, review and runtime gates to control what executes.

Documentation

Choose an engine
and build a contract

Apache 2.0

Read the source
and run the tests

From the blog

Go deeper on the design choices

Read all posts →

Architecture

Paying for Spark on Small Data? Choose the Engine Without Rewriting Your Rules

Where serverless Spark, Polars, DuckDB, dbt and Ibis fit, and one definition of the rules across engines.

Read → Reliability

Fail the Run or Quarantine the Row? Deciding What Happens to Bad Records

Stop, drop or quarantine: how to choose per table, and the work quarantine still leaves you.

Read → Contracts

Is a Schema Check Enough? Where Schema Validation Stops and a Data Contract Starts

Schema tools check values too. The gap is what happens to a bad row or a new column.

Read →

Run the contract before you trust the claim.

Open the quickstart, inspect the accepted and failed records, then decide whether LakeLogic fits your pipeline.

Run in Google Colab View the repository

Designing, reviewing and monitoring contracts across teams? The LakeLogic platform is built on Core.

Governed pipelines,defined in YAML.