What if you didn’t have to read 2000 lines of code to make sure your agent doesn’t cause a regression incident?
You open a file to make a change, check out the git history and find that the last person to touch it left the company four years ago. A little worried, you trudge on, not knowing something downstream depends on this obscure piece of code behaving exactly the way it does right now. One or two people would know about it and would happily tell you if you asked, but you didn’t. How could you? You don’t even know the question. It’s tens of thousands of lines of code to read if you really wanted to understand everything, but nobody wants to read that much code. So you make the change as small as you possibly can, see the tests are green and, hoping that’s good enough, hold your breath and ship. And if that doesn’t scare you even a little, you’re probably one of the lucky few.
That dread has a name, and it isn’t shipping bad code. It’s the fear of shipping a regression your customers will notice and breaking your promise to them.
The promises we break
Every system you’ve ever maintained is a pile of promises - and most of them are promises nobody ever bothered to write down. It can be a behavioral contract, where a customer’s integration against your API relies on a minute detail or one service implicitly relies on another’s behavior. It can be a data contract, like replacing an empty array with null in a response. It can even be a customization promised to a customer, hidden in some unknowable configuration in the database.
“…all observable behaviors of your system will be depended on by somebody,” observes Hyrum’s Law.
And to make matters worse, your mental model of the system and how the system behaves for your customers are always different, and when they disagree, what you actually shipped always wins because the purpose of the system is what it does1.
Most brownfield work is straddling that gap and wishing in your heart-of-hearts that, when you add that “one small fix” to that piece of legacy code, you won’t break any promises hidden somewhere. Sometimes you just can’t ask for forgiveness.
So why don’t we just catch it?
Because there’s too much code, and nobody wants to read thousands of lines of code. It’s dense, it’s noisy, and reading it well is oftentimes slower than writing it.
But we’re in a new age - the AI writes everything, right? So you “write better specs.” What’s a better spec, though? A spec is an incantation you chant at a non-deterministic machine and cross your fingers. No matter how precisely you word the spell, and no matter how good the coding harness may be, the machine doesn’t owe you a good implementation.
“Sure, we need tests,” says spec-driven development, “so let the AI write the tests too.” But tests are also code so obviously nobody really reads tests either.
And when you review generated code? Let’s be honest about what actually happens there: you don’t read code, you scan it. You pattern-match across a diff, nod at the AI saying “I did everything!” and agree. Scanning is not the same as reading, and it’s definitely not the same as understanding, and a far cry from knowing all previously made promises survived.
And with that feeling of “can I really be sure about this?” you check the output and UI manually here and there and approve. Congratulations - you’ve reinvented manual QA, except as an amateur relearning how to do it without even realizing you are.
So here’s the whole idea in a nutshell:
What if the automated tests were readable?
Not “readable” the way a 500-line test file is allegedly readable. Readable the way a sentence is readable.
The tool for it is already well-established - Cucumber, a framework from 2008 - which lets every test be written in plain English:
Given a customer with an active subscription
And the [premium-routing] flag is enabled
When they request their order history
Then the response includes the [loyalty_tier] field
And a call is made to the billing service
You can read that in a couple seconds and say “yes” or, better yet “wait, why are we touching billing there?” and these kinds of surprises are the best part! Couple this with good old behavior-driven development and you have a nice throwback to the late 2000’s.
What’s so special about this then? Well, in our case, the AI owns the tests. You never write or maintain them. You only have to agree or disagree with them. It goes and nearly autonomously builds a whole swathe of test suites, covering the length and breadth of your system in readable, business-oriented tests that let you know whenever any promise is inadvertently broken.
We call it Telem, and it’s a set of a few dozen interconnected skills crafted with care that we hand to coding agents.
What you’re actually looking at
Let’s look again at the test we saw above. Let’s talk about some of its interesting properties.
Given a customer with an active subscription
And the [premium-routing] flag is enabled
When they request their order history
Then the response includes the [loyalty_tier] field
And a call is made to the billing service
You’ll notice the test treats the service as a black box. The tests only mention externalities - what goes in, what comes out, and what it does to the world around it. Naively that would mean these tests would take forever to run - we have to set up every single service, datastore and network destination the service ever talks to, but these tests are blazingly fast. How? Every external dependency is swapped for an in-memory fake that you can seed with data and make fail on command.2
That the tests are all blackbox means you can gut and rewrite the internals and the tests don’t so much as blink - no more keeping changes small just to feel safe. This is what automated testing should look like.
Another thing you’ll notice is that there’s lots of interactions in the validation part (Then). Telem has very strict validations. Service skipped a declared output? Test fails. Wrote an undeclared row to the database? Also fails. Hit a previously undeclared feature flag? Yup - that fails too.
You’re probably asking “How can this happen? I can’t necessarily predict what my code will or won’t do.” Well, what you might not be able to see from this example is that these tests are fully deterministic. Anything non-deterministic - the clock, randomness, the fire-and-forget write to S3 you forgot existed - gets faked. The payoff is the thing every engineer secretly wants and never gets: zero flakiness, enforced rather than merely hoped for.
More? Telem covers the behaviors you could have, not just the ones you happen to ship. Production runs one combination of feature flags; the system could be in any of them - so the suite tests every relevant permutation of feature flags and is clever about not trying those flags the test doesn’t care about so it doesn’t try to run 2^40 permutations (remember that “new feature flag found” failure we mentioned above - that’s how we keep it honest).
And did we mention it’s incredibly fast? That’s the whole ballgame. Slow tests are tests nobody runs.
With these capabilities, Telem serves as two types of harness:
- Test Harness - the rig that exercises your service in isolation and watches what it does, and it builds the harness backwards from existing implementation and forward as you add, change and remove code.
- Agent Harness - the rails a coding agent runs on, on top of the generic harness like Claude Code or Codex; in our case, this is the deterministic feedback loop that keeps it from building the wrong thing or declaring the work done before it actually is.
Case study: the gateway
Our customer-facing API gateway is effectively Promises Central. It’s the service at the edge, the one every customer request flows through, and functionally where most of our promises to customers live. About as high-stakes and as tangled as it gets. So we unleashed Telem on it.
With Telem at the helm, the AI generated over 24,000 scenarios - the endpoint’s behaviors across the full space of feature-flag combinations and user-observable states. The whole suite runs in under a minute on my personal MacBook. And here’s the counterintuitive part: because every test is pure and isolated, the work is embarrassingly parallel. Most test suites just shrug at extra hardware and stay slow; throw more cores at this one and the wall-clock drops.
But the number of tests is never the point. One of our Principal Engineers made a small, genuinely scary change: adding a field to a versioned API response. The kind of edit that used to come with a stomach-knot, because the blast radius was unknowable - including a nasty path where a managed request detours through a downstream service before coming back to be finalized. The tests said, precisely, that it behaved exactly as intended, detour and all. The stomach-knot was simply gone.
I made a small but risky change. These types of changes used to be very scary and hard to get right. These tests gave me the confidence that it behaves as I expected - including one especially tricky case [...]
And the part that still feels like it’s cheating: nobody wrote those tests, and nobody maintains them. The AI does. You review.
An unexpected twist: Telem as a Translator
Everything up to here is about safety - not breaking promises. But the deepest value of Telem is honestly somewhere else entirely.
Generating the suite turned out to be a system probe. Wherever we used it, Telem surfaced behavior nobody remembered was there - a buried “if this specific customer, do this other thing” branch - and the AI captured it, in plain English, in a feature file.
And that’s the thing. The code remembers, we don’t. Telem’s real power is that it acts as a translator - it turns that messy historical memory into readable prose an engineer can actually reason with and validate, while they’re in the middle of changing it. The suite becomes executable documentation of the system as it actually is - legible to humans and agents alike - instead of the system we all vaguely believe it to be. And your documentation gets deterministically revalidated in every CI run. That alone is a huge win.
One of our Engineering Managers gave feedback that hit the nail on the head:
[...] the agent came back with an implementation that made sense - not just code that technically passed.
Having this kind of documentation opens other doors, too. Hand the tests to the person who supposedly owns the endpoint and ask “is this how you thought it worked?” - you’ll both be surprised, reliably. Run it backwards and have the AI generate a spec from the scenarios, and you get a real document: what the business flow actually does, its request schema, validation rules, processing pipeline, the lot.
One of the things we did was ask the agent to teach us one of our billing system’s flows, based solely on the text of the tests. It created a set of HTML pages explaining the feature in descending levels of abstraction, creating intuition into each level, just as an engineer onboarding a new team member would.
It generated five levels, and they blew us away. You can go ahead play around with the first level, exploring its interactive aspects yourself.
The new shape of writing a feature
Now that the system’s behaviors are fully covered by tests, writing a feature becomes a twist on the classic red-green-refactor - with the labor rearranged. Write the spec however you like; the method doesn’t matter here.
- Red: The AI turns the spec into Cucumber scenarios that you read and make sure your intent was preserved.
- Green: Let the AI build the code. It’s free to add tests but never delete any. The whole time, the moment two features collide you see exactly what and exactly where, and steer toward the decision you actually want instead of finding out what the AI decided for you.
- Refactor: Go wild - the suite is your net.
- White: Your implementation now contains multiple promises you likely didn’t mean to make. Telem instructs the agent to build tests to match those implicit promises, forcing you to review what you might never have noticed.
Your role turns from author to reviewer - review the spec, review the tests, review the code - creating intuition for yourself as you build the mental model from the most abstract level to the least. You’ll be surprised, the agent will be surprised, good times will be had by all.
And now let’s pan back to you, that engineer from the opening - the one shrinking the change as small as it’ll go, holding your breath, shipping on a prayer. Take this. Make the change, the suite runs in under a minute, and it tells you in plain English exactly which promises you’re about to break. You’ll see the fear just… leaves.
Changing economics of software
None of this is a new idea. Exhaustive, readable, deterministic behavior tests - a plain-English scenario for every path through every combination of flags, kept honest as the code moves - would always have been the dream. We never built them because the mere thought was preposterous: tens of thousands of scenarios, hand-authored and maintained forever, for a single service. That’s not a sprint, that’s somebody’s career. So nobody did it, and that was the right call to make.
And then, more or less overnight, the price of that fell through the floor - by orders of magnitude. The suite that would have cost a career now costs a command you run before lunch and a few hundred dollars in tokens. It became a no-brainer.
Where we are and where we’re going
Telem is packaged behind a couple of commands - run /telem-bdd:backfill in a repo and an agent brings the whole approach over, leaving behind use case coverage and the tools and skills to keep it fresh. Run /telem-bdd:sync and your repo gets the latest and greatest, increasing coverage and quality by plugging holes and constantly bringing in better tools. Now, we’re rolling it out across our systems, the promise-bearing ones first. So far, so practical.
But here’s the part I can’t stop thinking about. For as long as any of us has been doing this, working on old code has carried the same fear in the back of our minds - of touching something we don’t fully understand, and the damage-controlling practices of changing as little as we can and hoping. We’ve treated it as the price of working on anything real - the background radiation of every brownfield codebase ever written.
But it was never a law of nature. Combining tools old and new, for the first time, we get to stop being scared of our own code.
