ALL SPEAKERS
AT A GLANCE
Role
Software Consultant
Organisation
TNG Technology Consulting GmbH
Category
Community
ABOUT THE SPEAKER
Konstantin Todte is a Software Consultant at TNG Technology Consulting in Munich. He develops and operates a production multi-agent chatbot built on LangChain, with tools connected via MCP, as part of a larger microservices platform. His current focus is quality control for that system: regression evals, contract tests for agent protocols, and LLM-as-judge monitoring with Langfuse. He holds an M.Sc. from TUM.
TALK
Testing the Untestable? Quality Control for Multi-Agent LLM Systems in Production
How do you test a system whose core is non-deterministic, whose tools
live on remote servers owned by other teams, and whose output is a
stream of natural language?
Our team runs AI Assist, the customer-facing AI assistant in BSH's Home
Connect app. It's a multi-agent system in production, and this talk is
about how we keep it from breaking. Most of the heavy lifting is still
done by classical testing: unit and integration tests for all the code
around the model, because that code is where most bugs actually live.
The interesting part sits on top. We built an eval layer with a small
golden dataset curated from real conversations, deterministic checks
that verify what the agent decided to do (which tool bundle it selected,
whether the answer contains what it must), and LLM judges for the softer
qualities that have no objective test.
We'll show you what worked, what we had to throw away, and which kind of
test catches which kind of failure, with regression stories from
production and our CI pipeline.
Key takeaways:
- Non-determinism changes where your tests belong, not whether you need
them. In our experience, the agent's decisions are stable enough to
assert on in CI, while the wording of its answers is not.
- Examples taken from real production traffic tend to surface failures
that hand-written test cases miss.
- LLM judges help us watch quality trends in production, and we
deliberately keep them out of the build gate.
Why attend:
If you run an LLM system in production, or are about to, you have
probably felt the gap between passing tests and actually trusting a
deployment. We'll share how we approach that gap for AI Assist: which
checks we gate our pipeline on, which ones we deliberately don't, and
what we're still working on, so you can decide what makes sense for your
own setup. Most useful if your team owns a tool-using LLM system and the
pipeline that ships it.
MORE VOICES
Other speakers at code.talks
See them live — and 100+ more.
One ticket, every track. Nov 4–5, 2026 at Kinopolis Hamburg.
Get your ticket





