Blog

Data engineering insights, cost optimization guides, and tutorials from the LakeLogic team.

Data Quality Management: Which Parts Belong in the Pipeline

Five jobs: define, check, act, observe, remediate. Which tool fits each.

→

Kafka to a Lakehouse: Measure Freshness, Throughput and Data Quality

A runnable Kafka-to-Delta notebook: backlog throughput, event latency and rejected records, measured separately.

→

Data Quality Checks in Polars: Five Ways to Do It, and When Each Fits

Polars expressions, Pandera, Patito, dataframely, GX and contracts, compared by what they do with bad rows.

→

Data Mesh Contracts: What One Domain Can Rely On From Another

A mesh splits the data, not the agreement. How to make the cross-domain interface checkable.

→

Paying for Spark on Small Data? Choose the Engine Without Rewriting Your Rules

Where serverless Spark, Polars, DuckDB, dbt and Ibis fit, and one definition of the rules across engines.

→

Fail the Run or Quarantine the Row? Deciding What Happens to Bad Records

Stop, drop or quarantine: how to choose per table, and the work quarantine still leaves you.

→

Is a Schema Check Enough? Where Schema Validation Stops and a Data Contract Starts

Schema tools check values too. The gap is what happens to a bad row or a new column.

→

Moving from Great Expectations to Data Contracts: What Maps, What Changes, When to Stay

What GX does well, how Expectations map to a contract, and when to stay.

→