Catalog-native dataset validation

Kedro pipelines read and write data through a catalog: a YAML file that maps a dataset name to a type, a path and some options. What the catalog could not do was say anything about what the data is supposed to look like. So a malformed column travels happily through the I/O layer and detonates somewhere in the middle of a node, three steps later, with a traceback pointing at the wrong place entirely.

Teams had four workarounds and none of them were good: hand-written hooks that nobody could discover, an unmaintained kedro-pandera plugin, function-level decorators that only fire during a pipeline run, or nothing. The gap I cared most about was notebooks. A data scientist loading a dataset interactively got no checks at all.

The version I got wrong

My first proposal, KEP-7, put schemas on node signatures. You would annotate a node function's arguments with a Pandera schema and the framework would validate against the type hints it found. It reused machinery I had already built for parameter validation, and it demoed well.

In review, deepyaman pointed out the flaw. If two pipelines both consume the same dataset and annotate it differently, whichever one the framework resolved last would silently win. No error, no warning, just a schema quietly replaced by another one. He argued that a schema is a property of the dataset, not of any particular function that happens to read it, and therefore belonged on the catalog entry. noklam separately asked for a global kill switch and a programmatic API that an IDE could call for diagnostics.

They were right, and it was not a defect I could patch. The conflict was structural. I closed my own proposal and rewrote it. KEP-10 makes the conflict impossible by construction: one dataset, one validator, declared where the dataset is declared.

The design that passed

validator: becomes a reserved key on any catalog entry. Shorthand is a dotted path to a schema class; the long form takes options.

• Enforcement lives in the catalog's load()/save() funnel, not in a wrapper object around the dataset.
• Save-side validation runs before the write, so invalid data never reaches disk.
• A pluggable Validator protocol. One method, validate(data) -> data; any raise counts as a failure. Pandera is the reference adapter, at roughly 200 lines.
• A programmatic API returning a structured result with a machine-readable error type, so the VS Code extension can render diagnostics instead of guessing.

Why not a wrapper

This is the part I changed my mind about publicly, having already announced the opposite. A _ValidatingDataset proxy is the obvious implementation and it is a trap. It breaks isinstance(ds, AbstractVersionedDataset), which real user code checks. It has to fake internal flags like _EPHEMERAL and_SINGLE_PROCESS, delegate exists() and release()by hand, and survive being pickled by ForkingPickler so thatParallelRunner keeps working. Every one of those is a bug waiting for a user to find it.

Validating inside the funnel means the dataset stays exactly the object it always was. Nothing downstream can tell the difference.

The details that only show up in a real implementation

Writing the prototype surfaced a set of problems that no amount of design-doc review would have found.

• PySpark never raises. Pandera's PySpark backend does not throw on failure. It accumulates errors onto df.pandera.errors instead. A naive adapter would have silently passed every invalid Spark DataFrame it was given, which is worse than no validation at all.
• Pandera moved its namespaces. The top-level pa.DataFrameModel has been deprecated since 0.30 in favour of pandera.pandas, pandera.polars and pandera.pyspark. The adapter detects which one a schema came from.
• A protocol check that lied. isinstance() against a runtime_checkable protocol returns True for any class that merely defines the method, so passing a class where an instance was expected returned the uninstantiated class and silently dropped its options. Found it, fixed it, pinned it with a test.
• The Norway problem, still with us. In YAML 1.1 an unquoted on: key parses as boolean true, and it survives the OmegaConf round-trip intact. The config parser normalises it; the demo project exercises it deliberately.
• Failures are grouped and capped. A million bad rows should not produce a million lines of traceback, so failures are collected per check with a bounded sample of examples. The full Pandera report is preserved on __cause__ for anyone who wants it.

What it cost, and what I could prove

Validation sits on a hot path, and "it's probably fine" is not an answer you can put in a proposal. I committed three micro-benchmarks so the performance guidance is evidence rather than assertion, and enumerated twelve risks: hot-path cost, raw-access bypass, coercion silently changing data, hooks observing pre-coercion values, validators lost through environment overlays, degradation under a custom catalog class, thread races. Each has a stated mitigation.

There is also a kill switch, because anything that can reject your data at runtime needs one. A setting plus an environment variable read directly in kedro.io rather than in framework settings, so it still applies to catalogs that plugins rebuild inside hooks and to bare catalogs constructed in a notebook.

Where it stands

KEP-10 was accepted by Technical Steering Committee vote in June 2026. I ran the technical design session in July, and the implementation is landing as a series of reviewable pull requests against a tracked parent issue. Kedro has eleven enhancement proposals in total. I wrote two of them.

The demo project is the clearest statement of the point. A supplier feed arrives with a duplicate ID, a malformed rating and an invalid flag. With validation on, the run stops at the I/O boundary and reports all three failed checks at once, before anything is written. With KEDRO_DATASET_VALIDATION=0, you get what Kedro does today: ValueError: could not convert string to float: 'not-a-rating', raised deep inside a node, nowhere near the actual problem.