ALL SPEAKERS

Community

Konstantin Todte

Konstantin Todte

Software Consultant

TNG Technology Consulting GmbH

AT A GLANCE

Role

Software Consultant

Organisation

TNG Technology Consulting GmbH

Category

Community

ABOUT THE SPEAKER

Konstantin Todte is a Software Consultant at TNG Technology Consulting in Munich. He develops and operates a production multi-agent chatbot built on LangChain, with tools connected via MCP, as part of a larger microservices platform. His current focus is quality control for that system: regression evals, contract tests for agent protocols, and LLM-as-judge monitoring with Langfuse. He holds an M.Sc. from TUM.

TALK

Testing the Untestable? Quality Control for Multi-Agent LLM Systems in Production

How do you test a system whose core is non-deterministic, whose tools
live on remote servers owned by other teams, and whose output is a
stream of natural language?

Our team runs AI Assist, the customer-facing AI assistant in BSH's Home
Connect app. It's a multi-agent system in production, and this talk is
about how we keep it from breaking. Most of the heavy lifting is still
done by classical testing: unit and integration tests for all the code
around the model, because that code is where most bugs actually live.
The interesting part sits on top. We built an eval layer with a small
golden dataset curated from real conversations, deterministic checks
that verify what the agent decided to do (which tool bundle it selected,
whether the answer contains what it must), and LLM judges for the softer
qualities that have no objective test.

We'll show you what worked, what we had to throw away, and which kind of
test catches which kind of failure, with regression stories from
production and our CI pipeline.


Key takeaways:

- Non-determinism changes where your tests belong, not whether you need
them. In our experience, the agent's decisions are stable enough to
assert on in CI, while the wording of its answers is not.

- Examples taken from real production traffic tend to surface failures
that hand-written test cases miss.

- LLM judges help us watch quality trends in production, and we
deliberately keep them out of the build gate.

Why attend:

If you run an LLM system in production, or are about to, you have
probably felt the gap between passing tests and actually trusting a
deployment. We'll share how we approach that gap for AI Assist: which
checks we gate our pipeline on, which ones we deliberately don't, and
what we're still working on, so you can decide what makes sense for your
own setup. Most useful if your team owns a tool-using LLM system and the
pipeline that ships it.

MORE VOICES

Other speakers at code.talks

Eric Evans photo
Hero
Eric Evans
Founder & Author ·
#Architecture
BIO
Eric Evans
Founder & Author ·

Eric has long worked in the development of software for complex domain problems. He wrote the book Domain-Driven Design and has been learning new and better ways to do it ever since.

More info →
Gergely Hodicska photo
Hero
Gergely Hodicska
Founder · Water Bear Consulting
#Culture
BIO
Gergely Hodicska
Founder · Water Bear Consulting

Gergely is a product engineering leader and coach with deep experience in building and scaling engineering organizations. He is a former VP of Engineering at Bitrise and Ustream, and has worked across engineering leadership, product development, organizational design, culture, change management, and coaching. He focuses on helping teams build better systems, improve collaboration, and create environments where people can grow and do their best work. He combines leadership experience with strong technical depth, including hands-on software engineering, architecture, developer tooling, and building systems used at significant scale.

More info →
Gitte Klitgaard photo
Hero
Gitte Klitgaard
Founder and Coach at Native Wired · Native Wired
#Culture
BIO
Gitte Klitgaard
Founder and Coach at Native Wired · Native Wired

Gitte Klitgaard is an agile coach, trainer, advisor and mentor solving problems in organisations. She uses strategy, accountability and responsibilty, collaboration, psychological safety, communication, setting good engineering practice, and what ever else is needed. She works with teams, leaders, and individuals on all organisational level. Gitte is authentic, she will cut to the chase, and help people become themselves thereby reaching success.

Gitte knows how combining human skills with good leadership skills is a necessity for an organisation to get stuff done. And getting the right stuff done. Her ability to align the individual well-being and growth together with company objectives is remarkable. One of Gitte’s key strengths is holding people accountable while fostering psychological safety.

Gitte has given talks and keynots at three continents, where she highlights and makes topics like mental health and psychological safety accessible. Other topics include courage, retrospectives, various agile topics, leadership, management, and her personal goal: bringing humanity back in the workplace.

Her community contributions include organizing coach camps for over a decade and actively engaging on social media. She creates safe and respectful environments at work and outside. She listens to and engages the more silent voices and minority groups. Gitte also contributes to the community by writing blogpost and on Bluesky as she shifted away from Twitter. She wrote a section about team safety in "The Art of Agile Development, 2. edition" by James Shore which was published in 2021.

In her spare time, Gitte collects LEGO and Yodas and keeps in touch with friends from all over the globe including some, she considers her second family.

Gitte is owner of Native Wired and has lead change at companies like IBM, LEGO, Spotify, and SVT.

More info →
Luca Rossi photo
Hero
Luca Rossi
Founder · Refactoring
#Culture
BIO
Luca Rossi
Founder · Refactoring

Luca is the founder of Refactoring.fm — a newsletter and community about running better engineering teams, that goes out weekly to more than 170,000 engineers and managers. He has written 300+ articles, and the Refactoring community includes some of the brightest minds in the industry. Before Refactoring, Luca has been Head of Engineering at Translated.com, and co-founder and CTO of Wanderio, where he grew the product from 0 to 25M customers.

More info →

See them live — and 100+ more.

One ticket, every track. Nov 4–5, 2026 at Kinopolis Hamburg.

Get your ticket

Let’s connect on LinkedIn.

Speakers, program drops and much more of the community.

A 2-day sweet spot conference for the entire dev team to stay future-proof in the age of AI.

© 2026 code.talks · alphalist GmbH · Hamburg

Let’s connect on LinkedIn.

Speakers, program drops and much more of the community.

A 2-day sweet spot conference for the entire dev team to stay future-proof in the age of AI.

© 2026 code.talks · alphalist GmbH · Hamburg