Your Next User Might Not Be Human

Autonomous software agents are starting to operate corporate systems on their owners' behalf. The awkward discovery is how little of that software was ever built to be used by anything other than a person.

For most of the history of business computing, one assumption sat beneath everything else, so obvious that it was rarely written down: at the far end of every transaction there would be a human being. Screens were designed for eyes, forms for fingers, workflows for the rhythm of an office worker's afternoon. Automation existed, but it was a servant that waited to be summoned. Somebody still had to press the button.

That assumption is now being quietly repealed. Given a goal, a set of credentials and access to a few tools, a modern software agent can decide which steps to take: searching for an appointment, comparing the options, holding a provisional booking and returning to its principal only for approval. Inside a company, it might assemble figures from several systems, draft a report and allocate follow-up work.

The intelligence involved is only part of the picture. The model does the reasoning; the surrounding software determines whether that reasoning has anywhere useful to go.

The consequence is not that people vanish from software. It is that a growing number of products acquire two classes of user with sharply different needs: the person who decides, and the machine that executes. Firms have spent the past few years asking how to add artificial intelligence to their products. The more searching question, and the one with real operational consequences, is whether an agent could operate the product they already sell, reliably and without doing anything expensive.

Garbage in, agency out

Humans are magnificent at coping with software that is quietly broken. A greyed-out button is understood to mean a permissions problem. Two fields with different names are recognised as containing the same thing. An experienced member of staff knows that a record must be moved into one status before another action becomes available, a rule that appears in no documentation and never has. This tacit knowledge is the dark matter of enterprise technology: invisible, uncosted and holding the whole structure together.

Agents have no reliable access to it, and this is where the comedy begins. A model that can draft a passable commercial contract is undone by a status field marked "Complete", which in the average organisation means one of four things depending on which department populated it and in which decade. The reasoning may be immaculate. The substrate is a spreadsheet that escaped in 2014 and was never recaptured.

Firms have been promised that artificial intelligence will finally deliver a return on their data. What they discover instead is that agents perform a merciless audit of it, and that the audit tends to happen inside live systems.

This changes what matters in software. For years, competitive advantage often accrued to whoever built the friendliest interface; design became the moat and the demonstration became the sales pitch. If a rising share of interactions begins through a machine rather than a mouse, the interface is no longer the whole product. Infrastructure long treated as plumbing becomes part of the experience too: APIs, permission models, data definitions and error handling.

The shape of those interfaces matters as much as their existence. An endpoint called updateRecord may be technically convenient, but it is semantically useless: it tells an agent very little about what is being done, why, or which rules apply. Actions named for the business, such as approving an application, withdrawing a booking or requesting further information, carry intent, and intent is what can be governed. Emerging standards such as the Model Context Protocol, alongside newer agent-to-agent approaches, are attempts to reduce the custom integration work traditionally required whenever one system must talk to another. The specific standards will evolve; the underlying principle is likely to endure. A system's capabilities should live in stable contracts rather than remain trapped inside individual screens.

The permission problem

Corporate security has spent decades learning to separate who somebody is from what they may do. Agents introduce a third and less comfortable question: on whose behalf, and for how long?

An agent configured by an administrator should not inherit an administrator's reach merely because it was created there. It needs to know which records it may see, which actions it may complete unsupervised, whether it may write as well as read, whether it may communicate externally, whether it may commit money and whether its access expires when the task does. Increasingly there is a further question, one that would have sounded baroque three years ago: may it hand work to another agent?

Least privilege is an old doctrine, but agents raise the cost of ignoring it, because the software is now choosing which action to attempt. That makes the placement of human approval a product-design problem rather than a matter of temperament. Sorting a list or preparing a draft warrants no ceremony. Rejecting an application, altering a customer's entitlement or moving money plainly does. Demanding confirmation at every step produces an assistant so tiresome that staff route around it, which is the expensive failure; demanding it only after an irreversible act is theatre. The useful distinction is between what an agent may do, what it may prepare but not submit, what it may do only with explicit approval, and what must remain with a person. Oversight of that kind belongs in the workflow, not in a prompt politely asking the model to check when unsure.

Failure deserves the same seriousness. "Something went wrong" is merely rude to a person, who can telephone a colleague. To an agent it is an instruction to do nothing intelligent: it cannot tell whether the request lacked a field, permission was refused, a duplicate already exists or a third-party service is simply having an afternoon. Each of those warrants a different response: correct, ask, retry later, reroute or escalate. A system that cannot distinguish between them will convert small errors into large ones with impressive efficiency.

Audit trails need the same care. Recording the final outcome is not enough; depending on the risk involved, teams may need to know:

• whom the agent was acting for;
• what information it accessed;
• which tools it called;
• what it changed;
• where approval was requested, and who granted it;
• what happened when a step failed;
• which model, agent version or workflow was involved.

When something goes wrong in a multi-step process, the forensic question is whether the problem came from the model, the data, the tool definition, the permissions or the underlying system. That question cannot be answered from the final response alone, and it is cheaper to design for before the first incident than after it.

A dividend in disguise

None of this is an argument for rebuilding everything around agents, an error that will no doubt be committed enthusiastically and at scale. Where rules are stable and every valid path can be specified in advance, conventional automation remains cheaper to run, easier to test and considerably more predictable. Agents earn their keep where the work contains genuine ambiguity: the cases that defeat rigid systems and currently consume a person's morning. A characteristic mistake of the next few years will be paying handsomely for an agent to do a job a scheduled script was already doing adequately, and without opinions.

Where agents do belong, autonomy is best extended by instalments. Let the system retrieve and summarise first, then recommend, then draft, then act within narrow limits, and widen its remit only when there is evidence that the earlier stages are dependable. A convincing demonstration and a dependable operational process are different objects, a distinction procurement committees have historically found elusive.

The consoling news for firms without an artificial-intelligence strategy is that the best preparation for agents looks almost nothing like one. Well-defined interfaces, consistent data, granular permissions, meaningful errors, documented business rules, reversible actions, dependable logs, automated tests and proper monitoring are not speculative bets on a machine-run future. They are the unglamorous hygiene that makes any platform cheaper to integrate, maintain and change, and they pay a return whether or not a single agent is ever deployed. They also reduce an organisation's dependence on knowledge that exists only inside one long-serving employee's head, an asset that has a habit of resigning.

The unglamorous audit

The most useful place to begin, then, is not with a strategy but with a task. Find some grubby, recurring piece of work that requires a person to gather information, interpret it and shuttle between several systems, and ask what an agent would actually need to finish it: where that information lives, how it would reach each system, which rules it must follow, where it could do real damage, which judgements must remain human, which actions can be reversed and how anyone would verify the result.

The exercise is useful largely because of what it reveals. The hardest part is almost never the language model. It is the condition of everything the model would have to touch, and the inconsistent records, undocumented logic and brittle processes that frustrate integrations today will constrain agents tomorrow, except that agents will arrive with rather more initiative.

Interfaces are not about to disappear. People will still need to explore, compare and decide, and good design will continue to matter for them. But more actions will begin outside the product altogether, and when they do, a system's APIs, permissions, data structures and error messages become part of its user experience whether anyone intended that or not.

Agent-ready software is not software with a chatbot bolted to the front. It is software that states plainly what it can do, applies its own rules consistently and can account for itself afterwards. Your next user may well still be a person. They simply may not be the one clicking the buttons.

At GearedApp, this is increasingly how we think about AI readiness: not as a feature to bolt on, but as another reason to get the fundamentals of a platform right.