Observability for LLM pipelines

A Kedro pipeline is a graph of functions connected by a catalog. That model works beautifully for data: declare a dataset, get reproducible I/O. It had nothing to say about the artefacts a GenAI pipeline actually produces: traces, prompts, evaluation runs, the conversational back-and-forth of an agent.

So teams building LLM applications on Kedro were doing observability out of band: SDK calls scattered through node bodies, credentials threaded through by hand, no way to swap providers without editing code. Exactly the situation Kedro exists to prevent for data.

Making telemetry a dataset

I built out the dataset families for two observability providers so that tracing, prompts and evaluations are declared in the catalog like everything else, which means credentials come from the credentials file, the provider is swappable by environment, and none of it leaks into node logic.

• Langfuse. Trace datasets, created from scratch.
• Opik. Prompt, trace and evaluation datasets, the largest single addition at over 1,700 lines including its own documentation.
• AutoGen conversation tracing. Capturing multi-agent conversations via OpenTelemetry and routing them to either provider.
• A shared common layer, because two providers implemented independently is how you end up with two subtly different definitions of a trace.

The vector-store abstraction

The ChromaDB dataset had the same problem one level down. It exposed load_args/save_args, a shape borrowed from tabular datasets that fits a vector store badly, and one that would have to be reinvented for every new backend.

I replaced it with a handle model: an abstract base for vector-store datasets and aVectorStoreHandle that hands the node a live, configured client rather than pretending a vector database is a file you read and write. Roughly 950 lines added against 475 removed, with the test suite growing by 800 lines. A rewrite that made room for the backends that come next.

An MCP server that answers questions about Kedro

Separately, and mostly because I wanted to see whether the pieces fit: a retrieval-backed MCP server exposing Kedro's documentation to Claude and other MCP clients. A persistent ChromaDB vector store, sentence-transformer embeddings, header-aware chunking of the docs, and a Slack front-end over the same assistant.

The part I like is where its input comes from. It ingests llms-full.txt, the machine-readable documentation bundle that Kedro generates because I added the plugin that produces it. Upstream contribution first, retrieval layer on top of it second.

Why this matters for the framework

The argument I have been making, onthe Coffee Chats and at GOSIM in Paris, is that agents are the visible part of a GenAI system and the smallest part of the work. What decides whether one of these systems survives contact with production is the surrounding pipeline: preparing the data, connecting the steps, making the whole thing repeatable and observable. That is a problem Kedro already solved once, for data. This work is the argument that it transfers.