A convincing demo takes an afternoon. Something that still works in month six is a different piece of engineering, and most of it is not the prompt.
The gap between an agent that demos well and an agent that runs is wider than almost anyone expects going in. The widely quoted figure is that roughly nine in ten agent pilots never reach production, and while I would treat the precise number with some caution, the pattern is not in dispute.
It is that the demo covered the happy path, and production is made almost entirely of the other paths.
What an agent is actually made of
Instructions define scope, tone and refusal behaviour. Most teams spend nearly all their effort here, because it is the visible part and it produces immediate improvement. Diminishing returns arrive quickly.
Grounding decides what the agent can see. The critical design rule: retrieval should run under the identity of the person asking, not under a service account. Service account grounding is faster to build, and it converts every agent into a permission bypass. Microsoft’s own guidance is unambiguous: public-facing agents must not touch internal business data, and internal and public agents should be separated at the subscription level.
Tools are the named actions the agent can take. Two rules save a great deal of trouble. Type the inputs, so the model cannot invent a malformed request. And please add an explicit approval step before anything that writes, sends, or commits, at least until you have evidence that you can remove it.
Guardrails are content safety, data loss prevention and a defined refusal path. The refusal path is the one people forget, and it is the difference between an agent that says it cannot help with that and one that confidently answers a question it should have declined.
Four ways the demo lies
Silent drift deserves particular attention because it is the one that catches mature deployments. A source document gets updated, a model version rolls forward, a connector changes its response shape. Nothing errors. Quality degrades, gradually, and you find out when someone stops using it rather than when someone complains.
The only defence is a fixed evaluation set you rerun on a schedule. Twenty to fifty representative questions with known good answers, run weekly, scored. It is not sophisticated, and it does not need to be.
Without an evaluation set you cannot distinguish an improvement from a lucky prompt. Every change becomes an argument about vibes.
The unhappy path is the product.t
If I could change one habit in agent design, it would be this. Teams write the successful interaction first and treat everything else as error handling to be added later. Later never quite arrives.
Invert it. Before building the thing that works, write down what happens when the agent does not know, when the source is missing, when the user asks something outside scope, when a downstream system is unavailable, and when the answer is available. Still, the user is not entitled to it. Those five cases account for most of the real traffic in month two, and deliberately designing them is what makes an agent feel trustworthy rather than erratic.
The handover to a human matters most of all. An agent that cleanly says this needs a person, here is who, with the context attached, is more useful than one that answers everything at eighty per cent confidence.
For the practitioners
- Run the AI Red Teaming Agent against Foundry applications before deployment to scan for safety and security issues. Treat it as a gate, not an option.
- In Copilot Studio, enable automatic security scans and check the agent runtime protection status so that validation is continuous rather than a launch-day activity.
- Standardise on approved frameworks and protocols across the organisation. MCP has become the common integration path, and letting each team choose its own creates an interoperability problem you will pay for later.
- Log prompts, responses, tool calls, refusals and cost from the first day of the pilot. Purview audit captures Copilot and agent interactions, and activity explorer in DSPM for AI shows prompt and response text along with referenced files.
- Isolate confidential from public data with real boundaries, in an Azure landing zone that means internal agents in corp subscriptions and public agents in online subscriptions, not a naming convention.
The operational question nobody asks early enough
Who fixes it at nine on a Tuesday morning when it stops working?
If there is no answer, the agent will not go to production, regardless of how good it is. Someone needs to own it, someone needs to be able to see what it did, and there needs to be a way to roll back a change that made things worse. Copilot Studio gives you environments and versioning for exactly this. Foundry gives you the full engineering lifecycle. Neither gives you the person.
That is the real reason so many agent pilots stall. Not because they did not work, but because nobody was ever going to be responsible for them working on an ordinary Tuesday.
What goes in the evaluation set
Everyone agrees that evaluation matters, and very few teams build one, mostly because it is unclear what to include. It is less demanding than it sounds. Twenty to fifty cases, drawn from four categories, gets you most of the value.
| Category | What it contains | Roughly how many |
|---|---|---|
| Typical questions | The everyday cases, taken from real usage rather than invented | Half the set |
| Known hard cases | Ambiguous phrasing, multiple plausible sources, questions that span two systems | A quarter |
| Must refuse | Out-of-scope requests, questions about data the asker cannot access, attempts at prompt injection. | An eighth |
| Previously broken | Anything that failed in production. Once fixed, it stays in the set permanently.y | Grows over time |
Scoring does not have to be automated to be useful. A person reading fifty responses and marking each as good, acceptable, or wrong can do so in under an hour and produce a number you can track. Automate it later, once you know what you are measuring.
The must-refuse category is worth building first, even if you build nothing else. It is the cheapest possible protection against the failure mode that damages trust the fastest: an agent answering something it had no business answering.
The cost of over-engineering
Everything above argues for more rigour, so it is worth stating the opposite case, because it is possible to overshoot in ways that are just as damaging as underbuilding.
An agent that summarises a weekly team update does not need a red-teaming exercise, a 50-case evaluation set, or an approval workflow. Applying production discipline uniformly to every agent is how organisations end up with a governance process so heavy that people circumvent it, producing exactly the shadow estate the process was meant to prevent.
The proportionate approach ties the rigour to the exposure. Who sees the output, what can it reach, and what happens if it is wrong. An internal drafting aid in which a human reads every word before it goes anywhere is a low-risk artefact. An agent that emails customers directly is not. Deciding those tiers explicitly and writing down what each one requires is what prevents the conversation from being reopened for every new agent.
What to take away
- Ground retrieval through asking the user for their identity, never through a service account.
- Put approval steps in front of every write action until you have evidence to remove them.
- Design the refusal and the human handover before designing the successful path.
- Build a fixed evaluation set and rerun it on a schedule. Drift is silent, and it is the mature deployment killer.
- Name the owner and the support route before go-live, not after.
