Faster coding is not the same as faster delivery.
Over the last year or so, my work at Forter has taken me from Toolchain, our shared tool-access layer for agents, through Agentbox, a sandboxed agent harness, to implementing agents like CI Failure Doctor and Grumpy Reviewer — and now into bringing those pieces together as an AI-assisted software development lifecycle.
These are not five unrelated AI experiments. They address different parts of the same problem: how do we turn an intention into a safe, useful change without a human having to manually carry it through every step?
That is also how I have come to think about engineering productivity. Writing code faster is useful. But if the result waits for context, fails CI, queues for review, or creates operational work after release, we have not necessarily delivered anything sooner.
Will Larson’s post about small changes describes a related challenge: more changes make review a bottleneck, and safely removing that bottleneck requires better verification. What I want to share is the broader system we have been building around that problem — and how I think we should judge whether it is working.
First, give agents access to the real company
Toolchain addresses a prerequisite that is easy to overlook in a coding demo: an agent needs access to the systems where the relevant information and actions actually live.
At Forter, Toolchain is our MCP server and central tool-execution layer. It exposes approved integrations such as GitHub, Glean, Snowflake, Elasticsearch and Slack to agents and IDEs, with shared management of access, connections and usage visibility. Instead of each new agent inventing its own integrations, it can use a common capability. (We wrote about the sprint that made this possible in Making Agents Easy.)
The productivity argument here is not “we added more tools.” It is that we should not have to solve access, context, security and observability again for every new workflow. Those are platform problems. Solving them once makes the next application cheaper to build and easier to understand when it fails.
Toolchain also gives us a system-wide view of the tool landscape: tool-level success and failure, agent consumption, and token-usage visibility. That matters because an integration being available is not the same as it being useful. We need to see where agents are repeatedly failing or spending effort, then improve the shared tool rather than compensate with another paragraph in a prompt.
Then, make execution repeatable
Agentbox tackles the next problem: moving from an agent that works in someone’s terminal to something we can run repeatedly inside an engineering workflow.
It provides a hardened runtime and a Python SDK, alongside a GitHub Action interface and reusable skills. The intent is to make the execution environment and its controls reusable, rather than rebuilding them with every agent.
For the original CI Doctor workflow, that meant separating what the agent could inspect and edit inside its container from the workflow steps that published comments and pushed eligible patches. The design also included restricted tools, restricted network access and startup checks for the sandbox controls — all the things that give us confidence we can leave the agent to do the job we intended.
Guardrails are not just constraints on automation. They make automation practical. If every run needs a person watching it, we have just built a different user interface for the same manual work.
CI Doctor: start with a real interruption
A failed build is a good place to apply this. There is a concrete trigger, useful evidence in the logs, and a limited set of possible next actions.
CI Failure Doctor investigates failed GitHub Actions runs, produces a structured diagnosis, and can apply eligible fixes. A clear separation between diagnosis and execution lets us tune “understand the failure” independently from “change the repository.” Its initial auto-fix scope was deliberately conservative: mechanical corrections such as formatting, dependency pins and unambiguous configuration fixes. Soon it was crushing flaky tests and highlighting infrastructure flakiness for us to fix.
CI Doctor is present in many repositories and has recorded at least 5,000 workflow runs. That is evidence of adoption and repeated use — not a claim that it fixed 5,000 failures or saved a known number of engineering hours. Just a large number of interruptions avoided.
The outcome I care about is whether the developer gets back to useful work sooner. That means measuring time to a correct diagnosis, time to a green build, whether the fix survives, and whether the agent creates extra work or repeated repair attempts. Counting invocations tells us where to look; it does not answer those questions. For this, we needed to add agent evaluation and learning to the agentic platform.
Grumpy: review needs its own feedback loop
Grumpy Reviewer applies the same approach to code review. It reviews a PR using shared Forter review skills and repository-specific conventions, then posts a review with inline findings. The interesting work is not making an agent produce comments — it is determining whether those comments improve the change.
We built an evaluation pipeline that connects reviews to what happened afterwards on GitHub. It looks at acceptance of findings, distinguishes silent code changes from findings left unchanged, searches for follow-up fixes as a proxy for missed issues, and compares Grumpy’s findings with those of other reviewers. These signals have limitations, but they give us a way to examine usefulness, noise and blind spots rather than just review volume.
Acceptance alone is especially easy to overinterpret. An author agreeing with a comment does not prove the comment was important; an author ignoring it does not prove it was wrong. A later fix touching the same file does not automatically prove the original reviewer missed a bug. These are signals for investigation, not ground truth.
That is the standard I want for agents: not evaluating the agents themselves, but evaluating the outcome of their work — and using the evidence we collect to improve the shared system as a whole.
The opportunity is in the waiting, not just the typing
AI has made the number of PRs Forter engineers create rocket — and our median first-human-review latency has risen with it, to 50 minutes. Our median Grumpy review duration is 2.5 minutes. This identifies a possible queue to improve, not an end-to-end speedup we have already achieved.
With Grumpy we are exploring risk-scoped automated approval. A historical fit test classified approximately 59% of 715 sampled PRs across five important repositories as low risk — PRs that could have been approved by the AI reviewer. That is simulated eligibility, not measured safe approval in production. It can only be understood by tracking historical rework and incident metrics, and monitoring for impact.
A modelled reduction in review waiting is not the same as recovered engineering hours: people often do other work while a PR waits. It may still reduce lead time, context switching and coordination, but we need to measure those effects rather than relabel waiting as labour saved.
There is a second trap. If PR volume grows tenfold and we remove 60% of human reviews, humans still face four times the original review load. A percentage improvement can look impressive while the queue gets worse. We need to track absolute human demand and total delivery time as well as automation coverage.
AI SDLC: connect the pieces before building another platform
This leads to the work we are doing now on AI SDLC: connecting context, execution, validation, review and feedback into one coherent flow.
We are building the ability to assign a ticket to an agent and have that wake a resumable agent fleet which can produce a plan, stop for approval, implement and validate the change, open a reviewed PR, and eventually feed the outcome back into durable learning. Human approval gates and human-controlled merge are explicit boundaries.
One decision I made was to compose the working pieces before building a new orchestrator or control surface. Toolchain, Agentbox, Grumpy, evaluations and learning should become reusable parts of the path, not separate successes that every consumer has to glue together differently. The design also makes runtime convergence on Agentbox explicit, rather than treating another parallel execution stack as the answer — reusing, improving and consolidating on what is already working well for Forter.
That is the progression I see across this work:
- First, make capabilities accessible.
- Then, make execution repeatable.
- Apply them to concrete bottlenecks.
- Measure whether they help.
- Finally, connect the useful parts into a lifecycle that other teams can adopt.
These are not the only bets
We have many talented engineers and analysts across our teams, and once tools like Toolchain and Agentbox were made available to them, a host of agents have been created to help automate KTLO, analysis, risk detection, onboarding and support. All of these improvements make us faster: we are making fewer mistakes, making decisions with richer data, and handing off more work to agents.
Measure whether the roadmap actually moves faster
The missing link in many productivity conversations is the work we intended to deliver.
We should record the expected duration of a roadmap item when we commit to it, then compare that expectation with the time to a usable outcome and the time to full delivery. “We planned for two months and delivered in two weeks” is a much more meaningful story than “we merged more PRs” — provided we explain the scope, staffing, dependencies and definition of done. That is an illustrative example, not a result I am claiming here.
For each initiative, I would keep a lightweight record:
- What outcome did we choose to invest in?
- What duration and engineering effort did we expect, and when did we record that expectation?
- When did work start, when did users first benefit, and when was the agreed scope complete?
- What changed in scope, staffing or dependencies?
- What operational cost or rework followed delivery?
Forecast accuracy is not the same as speed. A padded estimate makes a good ratio without making customers any better off. We should keep the original forecast, explain changes, and examine actual lead times for comparable kinds of work. This only works if estimate-beating is not a target for individuals.
This gives the tools a test that matters. Did CI Doctor reduce time lost to failed builds? Did Grumpy reduce the review queue without adding noise? Did shared context and the AI SDLC reduce handoffs? Did all those improvements the team made using the tooling add up to earlier delivery of the roadmap work we actually wanted?
Faster cycles need operational evidence too
Agents are writing more of our code, which means we understand less of it by having written it. Understanding used to arrive as a side effect of authorship — you knew the system because you built it. Take the authorship away, and understanding has to come from somewhere else — or we end up scared of our own systems, and too scared to let agents change them.
Game days are how we keep that confidence and understanding. Quite a few teams at Forter already run them — our Competitive Game Day format heightens the intensity by turning them into competitions — and resilience exercises have covered availability-zone failures, Vault recovery and cross-region tokenization recovery. Those exercises tested assumptions about how systems actually behave under failure. They are evidence of resilience practice, not evidence that the AI tooling caused reliability improvements — but they are exactly the practice that keeps us confident enough to let agents change systems we did not write ourselves.
What I am proposing is to make that practice systematic: a monthly cadence of Game Days plus continuous incident tracking, connected to the delivery view. The monthly cadence is the next operating practice I am proposing here, not a claim that it has already been established across Forter.
Game days provide deliberate feedback; incidents show what happened outside the rehearsal. I would track the assumptions tested, recovery evidence, findings and their closure, alongside production incident severity, customer impact, detection and recovery time, and change-related failures. Counts need context: more findings in a game day can be good news, and fewer reported incidents can reflect changes in reporting rather than safer software.
My scorecard review made the same point about engineering measures: deployment growth can partly reflect better tracking, review-cycle time needs to be broken into its constituent waits, and incident counts need to be interpreted alongside the volume of changes. A dashboard is only useful if we understand what its numbers represent.
The system is the story
The story I want to tell is not that one agent makes engineers a fixed percentage more productive. It is that we have been building a system for removing repeated work and shortening feedback loops: shared access through Toolchain, repeatable execution through Agentbox, CI diagnosis through CI Doctor, review and evaluation through Grumpy, and now the work of connecting those capabilities through AI SDLC.
There is real adoption to report, concrete bottlenecks to investigate, and a set of design decisions and lessons worth sharing. There is also more measurement to do before claiming an organisation-wide productivity improvement.
The next step is to put the evidence together: faster delivery cycles, monthly exercises that challenge our operational assumptions, incidents that keep us honest about consequences, and roadmap expectations compared with actual delivery.
If those measures improve together, we have a credible productivity story. If the agents get busier but useful roadmap work does not arrive sooner, the cost shows up later in incidents and rework. Guess what? It means we have more engineering to do.
