ADLC: why your SDLC quietly breaks the day you ship an AI agent
Tung Bui Thanh (MikeDaDev)14 mins read

On this page
- # what ADLC actually means
- # why the classic SDLC is not enough
- # the loop, not the waterfall
- # frame the job
- # design the harness
- # write evals before you tune anything
- # iterate against the evals, honestly
- # the release gate
- # roll out progressively
- # observe traces, not just metrics
- # learn, and close the loop
- # what changes for the people
- # anti-patterns I keep seeing
- # if you only do three things on monday
The demo went perfectly. The agent read a support ticket, pulled the customer's order history, drafted a refund, and asked a human to approve it. Leadership clapped, the feature flag went to 5%, and everyone went home happy.
Two weeks later, nobody had touched the code, yet the share of refund drafts that humans approved without edits had drifted from 92% to 71%. The model provider had shipped a minor update behind the alias we were calling. A product manager had "tidied up" two sentences in the system prompt. A downstream team had renamed a field in the order API, and the agent had started guessing instead of failing. Every unit test was green, because none of them measured the thing that had changed.
If that story sounds familiar, you have already met the problem that the Agentic Development Life Cycle (ADLC) is trying to name. This post is my attempt to describe it from where I sit: as someone who has to put agents into production, keep them there, and explain to a room full of people why the numbers moved.
# what ADLC actually means
Let me be upfront: there is no single, agreed definition. Security vendors use the term for agents doing work across the delivery pipeline, and the attack surface that creates. Consultancies use it for an Agile process with AI embedded at every stage. Practitioners use it for the engineering discipline of building, evaluating and operating agents in production.
I find the last framing the most useful, so that is the one I use here:
ADLC is the lifecycle for software whose behaviour is defined jointly by code, prompts, models, tools and data, and is therefore probabilistic, versioned in more places than your repository, and never really "finished".
The other two meanings are not wrong; they follow from this one. Once you accept that an agent's behaviour is a distribution rather than a function, both the security posture and the team process have to change.
# why the classic SDLC is not enough
The SDLC rests on assumptions so deep that we rarely state them. Agents break almost every one.
| SDLC assumption | What happens with an agent |
|---|---|
| Behaviour lives in code | Behaviour lives in code plus the prompt, the model version, the tool descriptions and the retrieved data |
| Same input, same output | Same input, a range of outputs; you measure a pass rate, not a pass |
| "Done" is a merged PR | "Done" is a threshold on an eval suite that has to keep holding after launch |
| Dependencies change when you bump them | The model behind an API alias can change without a single commit on your side |
| Cost is mostly fixed infrastructure | Cost is a runtime variable: one confused run can make forty tool calls where three would do |
| A bug is a wrong line of code | A "bug" may be an ambiguous tool description, a poisoned document or a missing stopping condition |
None of this means we throw away what we know. Code review, CI, staged rollouts and incident reviews all still matter. What changes is the unit of quality. In a normal service it is the test case. In an agent it is the eval: a realistic task, run several times, scored against criteria you wrote down before you started tuning.
# the loop, not the waterfall
Most published ADLC diagrams are circles, and for good reason. This is the version I use with my teams. The top row is where you build; the bottom row is where you operate. The arrow that matters most is the one going back up: what you learn in production becomes new eval cases, and sometimes a change of scope or autonomy.
# frame the job
Before anyone opens a prompt file, I want three answers on one page.
What is the task, in a sentence a new hire would understand? "Resolve refund tickets under $200 that match policy" is a task. "Improve customer support with AI" is a wish.
How much autonomy does it get? A simple ladder moves the conversation further than any framework:
| Level | The agent... | Typical example |
|---|---|---|
| L0 | suggests; a human does everything | code completion, draft replies |
| L1 | acts, but every action needs approval | refund drafts, pull request proposals |
| L2 | acts alone on low-risk actions, asks on the rest | triage, labelling, read-only research |
| L3 | acts alone within a budget and a sandbox | test fixing in CI, data clean-up jobs |
What is the blast radius if it is confidently wrong? This one question decides more of your architecture than the choice of model. An agent that can only read is a different engineering problem from one that can move money.
Most teams start one level too high. Starting at L1 and earning L2 with data is almost always faster than starting at L2 and clawing back trust after an incident.
# design the harness
The model is the part you control least, so put your engineering effort into everything around it: tools, context, guardrails and stopping conditions. I call this the harness, and I review it the way I review an API design.
- Tools are an interface for a caller that cannot ask clarifying questions. Small, well named, well documented, with errors that explain how to recover.
- Write actions should be idempotent or reversible wherever possible, and have a dry-run mode.
- Every loop needs a ceiling: maximum steps, maximum tokens, maximum wall-clock time. An agent without a budget is a billing incident waiting for a date.
The companion post on agentic AI architecture goes deeper into each of these.
One habit pays off from day one: give every runnable configuration a manifest and a version. Behaviour depends on more than the commit hash, so pin all of it in one place and stamp the manifest version on every trace.
# agent.manifest.yaml
agent: refund-assistant
version: 2026.11.3
model: provider-model-2026-09-30 # an exact snapshot, never a floating alias
temperature: 0.2
system_prompt: prompts/refund/system.md # hashed in CI
tools:
- name: get_order
schema: tools/get_order.v3.json
- name: draft_refund
schema: tools/draft_refund.v2.json
budgets: { max_steps: 12, max_tokens: 60000, timeout_s: 90 }
autonomy: L1When a number moves, the first question in the incident channel becomes "which manifest versions are we comparing?" rather than "did anyone change anything?".
# write evals before you tune anything
This is the stage most teams skip, and the one that separates a demo from a product.
An eval set is a collection of realistic tasks with a way to score the result. Start with twenty to fifty cases taken from real tickets, real logs and real failures. Include the boring majority, the awkward edge cases, and a handful of tasks the agent must refuse. A case can be as plain as this:
- id: refund-047
input: 'Order 88213 arrived broken, I want my money back. Photo attached.'
fixtures: { order: orders/88213.json, policy: policies/refund-v7.md }
expect:
tools_called_in_order: [get_order, check_policy, draft_refund]
tools_never_called: [issue_refund] # L1: drafting only
output:
refund_amount: 149.00
requires_approval: true
rubric: 'Apologises once, states the amount and timeline, no promises outside policy.'
tags: [happy-path, damaged-item]Then decide how to score it, in this order of preference:
- Deterministic checks wherever possible: the right tools in the right order, valid JSON, an amount that matches policy. They are cheap, fast and never argue with you.
- Trajectory checks for the path as well as the destination. An agent that reaches the right refund by guessing the order total is a time bomb, even though the final output passes.
- Model-graded rubrics for the genuinely fuzzy parts, such as tone and completeness. Calibrate the grader against a sample of human labels and track the agreement (Cohen's kappa is a reasonable start). A grader you have never compared with a human is an opinion, not a metric.
The discipline is test-driven development with statistics. Write down what "good" looks like while your judgement is still fresh, before you have spent a week falling in love with a particular prompt.
# iterate against the evals, honestly
Now you can tune. Change one thing at a time and compare pass rates rather than single runs. This is where a little statistics saves a lot of embarrassment.
Small suites are noisy. With 50 cases and a true pass rate around 85%, the 95% confidence interval is roughly ±10 percentage points. A prompt change that moves you from 84% to 88% has told you almost nothing. Either grow the suite, run each case several times, or agree up front on the smallest change you are willing to believe.
Average success hides inconsistency. Run every case k times and report two numbers. pass@k asks whether the agent succeeded at least once; it suits tasks where a human picks the best of several attempts. pass^k asks whether it succeeded every time, which is what a customer-facing agent needs, because each user only gets one attempt. The τ-bench authors popularised pass^k for exactly this reason, and on many agents it falls sharply as k grows. An agent with 80% average success that only passes all eight trials 40% of the time is not an 80% agent from the user's point of view.
Slice before you average. A headline of 91% can hide the fact that the "must refuse" slice dropped to 40%. Tag your cases and gate on the critical slices separately.
Treat prompts, tool descriptions and manifest values as code. They live in the repository, they go through review, and every change runs the suite in CI. A reviewer should see "this edit to the system prompt dropped refusal accuracy by 12 points" in the pull request, not discover it in production.
# the release gate
A release gate for an agent has more dimensions than a green build:
| Dimension | What I gate on |
|---|---|
| Quality | Overall pass rate above threshold; no regression beyond noise on critical slices; pass^k reported |
| Safety | Red-team suite for prompt injection, data exfiltration and tool misuse passes |
| Cost | Tokens and tool calls per successful task, median and p95, versus the previous manifest |
| Latency | p95 wall-clock per task, including tool time |
| Reproducibility | Exact model snapshot pinned; manifest version recorded |
Two details matter here. First, measure cost per successful task, not per task: an agent that fails fast looks cheap until you count the human who finishes the job. Second, for the safety suite, the OWASP Top 10 for Agentic Applications is a solid checklist to start from.
If a release cannot pass all of them, it does not ship. I have watched teams argue for an hour about a 2% quality gain while nobody noticed that cost per task had tripled.
# roll out progressively
Shadow mode first: the agent runs on real traffic, but its actions are logged rather than executed, and you compare them with what humans actually did. Agreement with the human decision, broken down by slice, is the most convincing number you will ever show a sceptical stakeholder.
Then a small canary with human approval on every action. Then widen the audience, and only after that widen the autonomy. Changing both at once makes it impossible to tell which one caused the next problem.
# observe traces, not just metrics
Request rate and error rate tell you very little about an agent. You need the full trace of every run: the prompt, each model call, each tool call with its inputs and outputs, the tokens spent, the manifest version and the final answer. When something goes wrong, the question is never "did it error?" but "at which step did its reasoning go sideways?", and only a trace can answer that. The OpenTelemetry semantic conventions for generative AI are a sensible common format if you do not want to lock yourself into one vendor.
A pattern I like: sample a small percentage of production traces every day and run them through the same graders you use offline. That turns the eval suite into a live monitor, and it catches silent drift like the one in the opening story within a day rather than a fortnight.
# learn, and close the loop
Every production failure becomes a new eval case, ideally within a week. Every confusing complaint gets a trace attached. Every time the agent asked for approval and the human said "obviously yes", that is evidence for promoting that action up the autonomy ladder. Every time a human corrected it, that is evidence against.
This is the flywheel that makes ADLC worth the effort. The suite grows from twenty cases to several hundred, each one a mistake you will never pay for twice. Keep a separate held-out set that nobody tunes against, refreshed every quarter, so you can tell genuine improvement from overfitting to your own tests.
# what changes for the people
The process is the easy part. The harder shift is in roles and habits.
Tech leads review behaviour, not just diffs. A prompt change can be three lines long and matter more than a thousand-line refactor. The review question becomes "show me the eval delta", and a pull request without one is incomplete.
Someone owns the eval suite. Not "the team". A named person, the way someone owns the on-call rotation. Unowned evals rot within a quarter: cases go stale, graders drift, and the pass rate becomes a number nobody believes.
Product managers write acceptance criteria as eval cases. This has been the most productive conversation I have had with PMs in years. "Give me ten tickets the agent must handle, and five it must refuse" is far more precise than a paragraph in a spec.
On-call needs new runbooks. "The model is being weird" is not an incident category. Your runbook should cover rolling back to a previous manifest, pinning an older model snapshot, lowering autonomy with a flag, and switching the agent to suggest-only mode without a deploy.
# anti-patterns I keep seeing
- Vibes-based release. Five people try the agent, it "feels good", and it ships. Feelings do not survive a model update.
- One giant eval number. A single 91% hides the slice that collapsed. Slice your results and report
pass^k. - Grading with an uncalibrated judge. A model grader that has never been compared with human labels can drift in the same direction as the agent it grades.
- Autonomy as a launch feature. Teams announce a fully autonomous agent on day one, then quietly add approval steps after the first incident. Do it in the other order.
- Frozen evals. A suite written before launch and never touched again measures last quarter's product.
- Ignoring cost until finance calls. Cost per successful task belongs on the same dashboard as quality from the first release.
# if you only do three things on monday
- Write down the autonomy level and blast radius of every agent you run, on one page, and share it.
- Collect thirty real tasks, decide how to score them, and run them several times in CI on every change to a prompt, tool or model snapshot.
- Turn on full tracing with a manifest version on every trace, and add each production failure to the eval set within a week.
None of this is glamorous. It is the craft we already practise, adjusted for software that answers a little differently every time you ask. The teams that treat agents as a lifecycle rather than a launch are the ones still running them a year later.
References:
- Anthropic: Building effective agents
- Cycode: Agentic Development Lifecycle (ADLC), what you need to know
- Arize: Evaluation-driven development, moving AI agents from pilot to production
- Yao et al.: τ-bench, a benchmark for tool-agent-user interaction (pass^k)
- OWASP: Top 10 for Agentic Applications
Just personal notes, shared in the hope they are useful. If you found this helpful, feel free to pass it on to your friends and colleagues!
Happy coding 😎 👍🏻 🚀 🔥.