Abstract
Automation is usually measured by the visible work it removes from an operation: fewer manual steps, shorter waits, less human intervention. That accounting is useful, but it leaves out a decisive part of the problem. When an organisation automates a process over many years, it changes more than who performs the work; it also changes who retains the knowledge needed to understand it when it stops behaving as expected. This essay places that loss of continuity between execution and understanding within a tradition running from Lisanne Bainbridge’s ironies of automation and Endsley and Kiris’s out-of-the-loop problem to recent work on meaningful human oversight, cognitive debt and intent debt. Against that background, it uses two formulations developed for this essay—the exception paradox and context continuity—to examine what happens when normal cases disappear from human experience while responsibility remains. The discussion begins with a fictional but technically plausible incident and moves from database triggers to today’s AI agent architectures. The argument does not challenge the value of automation; it challenges the assumption that keeping someone formally in the loop is enough when that person no longer has the context that makes genuine oversight possible.
The Monday when everything was green
At 9:17 on an otherwise ordinary Monday, Sarah opened the monitoring dashboard and took a few seconds to realise this was no display glitch: three hundred and eighty-six cases were in a state they should never have reached. They were neither blocked nor incomplete. They were marked as processed. That was all the more unsettling because the overnight batch, responsible for updating thousands of records before dawn, had finished at 4:06 with an immaculate ‘OK’. The scheduler was green, the database was responding normally and the integration console showed no alerts. Technically, it had been a perfect night.
Sarah was the business analyst and knew the procedure better than anyone from an operational perspective. She needed little time to recognise that the data was wrong; what she could not see was how it had got there. She posted a message in the project channel, attached a screenshot and called James, the technical lead. Two minutes later, the conversation no longer sounded like a routine query. The question that changed the mood of the morning was not who had made the mistake. It was something rather less comfortable: ‘Can we undo this?’
For the first half hour, nobody had a complete answer. The overnight batch ran a database package that selected cases deemed eligible to move to the next state. The package reported no errors and behaved as expected when rerun against test data. Then came the first clue: updating the main table fired an AFTER trigger, which placed certain state changes on an integration queue. An independent service consumed that queue, sent the event to an external system and confirmed receipt. Once the destination had accepted the message, correcting the source record was no longer enough; actions had occurred outside the original transaction that a simple UPDATErollback could not reverse.
By 10:03, the team had traced the technical chain. The reason behind it remained elusive. A condition inside a procedure was classifying cases as valid when, under the new business rule, they should have been stopped. The line of code had been there for years. Git showed when it had appeared; the issue tracker still held the associated ticket and even the names of those involved in the change. The ticket referred to an exception introduced for one particular processing cycle, but did not say whether it was meant to be temporary, precisely which assumption justified it or which other components depended on it. With extraordinary fidelity, the automation had preserved a decision the organisation no longer fully remembered: the person who had implemented it had left the company a few months earlier.
The scene is fictional, but hardly exotic. Anyone who has spent years working with enterprise systems will recognise its ingredients: processes that run for so long they cease to attract attention; people moving between projects; documentation that explains how to operate a mechanism and says much less about why it was designed that way; business rules that began as exceptions and eventually became part of the architecture. The incident matters not because an automation failed, but because, until that morning, it had perhaps worked rather too well.
What was beginning to break was not the ability to execute the process. That remained intact and, indeed, far exceeded what a human team could accomplish. Something else had been eroding: the continuity between the system’s behaviour and the memory of the people answerable for it. That shift—quiet while everything goes well, costly when an anomaly appears—is the thread running through this essay.
The system still knew how to do its job. The organisation was beginning to forget why it did it that way.
What we actually automate
Some of today’s enthusiasm for artificial intelligence rests on a reasonable intuition. If a job takes place entirely in front of a screen, its tools, inputs and outputs already inhabit a space that software can occupy. A model can read documents, query databases, write code, call APIs, classify cases or draft a report without leaving the digital environment. From that perspective, it seems logical to suppose that office occupations—and technical ones in particular—will be easier to replace than jobs that require physical action in the world.
The reasoning becomes less tidy when we step back from the individual task and look at the system around it. Processing a document is a task; deciding what to do when two sources supply conflicting data already belongs to the process. Determining the effects a decision might have, which permissions are acceptable, which exceptions deserve escalation and which consequences must be reversible belongs to the system. When that system produces an outcome affecting other people, another dimension emerges, one that does not disappear simply because the code wrote itself: somebody must be able to explain, correct or take responsibility for it.
An agent’s ability to write code, select a tool or propose a plan does not remove that layer of decisions. In a real project, someone—a person, a team or the organisation itself—still agrees to let the system act on particular data, with particular permissions and within particular limits. That responsibility usually becomes visible where the impressive demonstration ends and everyday operation begins: when a rule changes, an integration fails, a permission proves excessive or someone must decide whether an action can proceed without review. In these and other situations, AI may make construction much cheaper; it does not make those decisions unnecessary.
Measuring automation solely by the number of manual steps it removes can therefore be misleading. It is a reasonable measure of efficiency, but a poor measure of understanding. The interesting question is not just how many operations a person no longer performs, but how much knowledge remains accessible once those operations are no longer part of the team’s everyday experience.
The exception paradox
In 1983, Lisanne Bainbridge published a paper that still feels current more than four decades later, whenever another technological wave promises to remove people from ‘work’. In Ironies of Automation, she observed that automated systems tended to take over normal operation while leaving abnormal situations to humans. The irony was clear: the less an operator participated in day-to-day work, the harder it could become to intervene precisely when the system departed from its usual behaviour.[1]
In the mid-1990s, Mica Endsley and Esin Kiris experimentally studied the out-of-the-loop performance problem. The issue was not some general lack of operator competence, but something more specific: the loss of situation awareness and the longer time needed to recover control after moving from active participation to more passive supervision.[2] Parasuraman and Riley subsequently broadened the picture by examining the use, misuse, disuse and abuse of automation, including overreliance on systems that usually behave correctly.[3]
Artificial intelligence does not invalidate those observations; it carries them into a wider setting. Imagine a team handling one hundred cases a day: eighty follow predictable patterns, while twenty require judgement, negotiation or diagnosis. Automating the first eighty looks excellent on the business ledger: eighty per cent of the volume no longer needs human intervention. For the person still involved in the process, however, the balance has reversed. Almost everything now reaching them belongs to the difficult group.
I will call that inversion the exception paradox, without claiming to coin an academic term. Automating normality reduces the amount of human intervention but increases the proportion of anomalies in the work that remains. And anomalies bring an additional difficulty: they are infrequent. The professional who once travelled the system’s ordinary routes every day stops seeing them, noticing small deviations and keeping fresh a body of knowledge that was often never fully written down.
I do not present the exception paradox as a phenomenon separate from that tradition, but as a way of condensing it and carrying it into contemporary digital work. Bainbridge showed the irony of reserving abnormal conditions for the operator after automating normality; Endsley and Kiris described what can happen when that operator loses situation awareness through withdrawal from everyday practice. Here, the paradox names their organisational consequence: every advance in the automation of routine work increases the relative weight of exceptional cases in the work left to people.
Routine work, in that sense, served a purpose that rarely appeared on productivity dashboards: it kept understanding in practice. Someone handling ordinary cases saw how the data evolved, knew the sequence of states, recognised an odd result before it became a problem and learned, almost without intending to, which parts of the procedure deserved caution. Automation rightly removes much of that mechanical work, but it can take with it part of the informal mechanism through which an organisation retained its operational knowledge.
Human in the Loop, Out of Context
That is why I find the phrase Human in the Loop increasingly unconvincing when it is offered as a guarantee in itself. Knowing that a person exists somewhere in the loop tells us little about the quality of their involvement. A reviewer may be required to approve a decision while knowing nothing of where the data came from, how it was transformed, which tools were called before the result reached their screen or which irreversible actions have already taken place.
NIST—the US National Institute of Standards and Technology—addresses this problem in its AI risk management framework by calling for human responsibilities and oversight mechanisms to be explicitly defined and documented.[5] In 2026, Liming Zhu and his coauthors proposed a particularly useful distinction: AI can retain operative agency—the ability to execute and direct a task—while the person must retain evaluative agency, meaning the real capacity to understand, verify, redirect or replace the proposed solution.[8] Human presence alone does not guarantee that second capacity.
Concern about this asymmetry predates generative agents. Madeleine Clare Elish introduced the concept of the moral crumple zone to describe situations in which a human ends up absorbing moral or legal responsibility for an automated system’s failure despite having had limited effective control over it.[10] The image is uncomfortable precisely because it reverses the usual promise of Human in the Loop: a person can remain close to responsibility while being far removed from any real ability to intervene.
The debate has moved further in 2026. In a recent position paper, Margaret Mitchell, Avijit Ghosh and Samir Passi argue that certain agent designs do more than make human oversight difficult: prolonged use can degrade the very cognitive capacities that oversight requires, including judgement, operational knowledge and the ability to notice when something is going off course.[11] The argument connects almost directly with Bainbridge and Endsley, but applies to systems that no longer merely execute a fixed sequence.
This is where the title Human in the Loop, Out of Context belongs. It is not intended to establish a new, neatly bounded category, but to draw attention to a particular contradiction: we can keep a person in the diagram, even assign them formal responsibility for a decision, while distancing them from the history needed to judge it. The human remains in the loop, but is shown the final frame of a film they have never watched.
This does not require the supervisor to become the system’s author. They need not know every line of code or the internal workings of a language model; they need something much more practical: a way to establish where the result in front of them came from and what room they still have to act. If they must approve a decision, they should be able to reconstruct the data used, the relevant steps that preceded it, which parts were automatic and which effects have already occurred beyond their reach. Without that information, their signature may still be mandatory, but their capacity for control bears little resemblance to meaningful oversight.
Keeping the human in the loop is not enough. We must also preserve the context that makes their presence useful.
The debts missing from the balance sheet: automation, cognition and intent
Software engineering has discussed technical debt for decades: decisions that buy progress today at the cost of making tomorrow’s changes more expensive. Something similar is happening with automation, though the liability does not always lie in code quality. Workflows, scripts, bots, integrations and agents can grow faster than an organisation’s ability to inventory, test and document them, or even to know who is responsible for them. Recent professional discussion has begun using the term automation debt for this phenomenon.[9]
The metaphor is useful because it reminds us that the cost need not appear immediately. An automation can be cheap to create and very expensive to retire. It may start as a small fix, accumulate exceptions, become a critical dependency and reach a point where nobody dares touch it before finding the person who ‘knows how it works’. The liability is discovered late, usually when something that has quietly worked for years must be changed, audited or recovered.
Over this operational debt lies another, harder to see. In 2026, Margaret-Anne Storey proposed a model that distinguishes cognitive debt and intent debt alongside technical debt.[6] The first emerges as shared understanding of the system erodes; the second, when the goals, constraints and rationale behind its design are no longer adequately captured outside people’s heads. Thoughtworks has also adopted the notion of codebase cognitive debt to describe the widening gap between an implementation changing at ever greater speed and the team’s collective understanding of it.[7]
Let us return briefly to 10:03 on that Monday: the team had found the condition allowing the cases to advance, which solved part of the mystery, though not the most uncomfortable part. Knowing which statement had run did not explain why that statement existed.
Storey’s distinction is particularly useful here. Cognitive debt develops as shared understanding of how a system works erodes; intent debt develops when the goals, boundaries and rationale that guided its design are no longer clearly available. Storey frames this model around software health in the age of AI.[6] In this essay, I propose reading the same problem at the scale of the automated process: it matters not only that someone can understand the code, but that the organisation retains the reasoning behind the process’s decisions. While everything keeps working, that loss can go unnoticed for years; it becomes visible when a rule changes, an audit arrives or an exception occurs, and somebody needs a better answer than ‘we’ve always done it this way’.
Triggers are the devil’s handiwork
Years ago, I first heard a Spanish programmer’s saying: ‘Triggers are the devil’s handiwork.’ I cannot remember who said it, and I suspect that person had not invented it either... I have heard it many times since, usually after spending far too long discovering why an apparently simple operation had caused something nobody expected.
The saying is unfair to triggers. Used well, a trigger can safeguard an audit trail or respond very effectively to a state change. The problem arises when a substantial part of the logic is scattered across side effects that cannot be seen from where the operation begins. A developer runs an UPDATE; the change fires a trigger; the trigger inserts a record into another table; a process detects the insertion and calls a service; the service changes a state in another system… Viewed separately, each piece may be reasonable.
That kind of debugging teaches a lesson worth remembering now: systems do not become hard to understand only because they contain sophisticated algorithms. They can also become opaque through an accumulation of distributed causality. In those cases, reading code is not enough; you have to know how to follow events.
The incident at the beginning captures the problem well: the overnight package merely updated a state; the trigger turned that change into an event, and other components carried the consequence beyond the database. By the time the questionable condition was found, looking at the original statement was no longer enough. The chain had to be reconstructed.
That is why triggers provide such a useful precedent for what we are now building with agents. For decades, we have seen that an architecture can be deterministic and still make life difficult for anyone trying to explain why something happened. Artificial intelligence does not invent that difficulty; it adds, perhaps, new ways to amplify it.
When causality learns to speak
Imagine a contemporary version of the same chain: a document arrives in a mailbox; an event starts a workflow; the workflow passes the content to an agent that classifies it with a language model; the agent consults a document collection, decides it needs an additional fact, calls an API, updates a record and generates an instruction for a second agent; that agent then acts on another system… For the end user, all of this may collapse into a phrase as innocent as ‘the case has been processed’.
The resemblance to a cascade of triggers is obvious, but one difference should not be understated: in much traditional automation, enough patience could allow us to reproduce the execution path line by line. An agent architecture also involves probabilistic models, context retrieved at runtime, prompts, external tools and states that may change between one call and the next.
Observability must then tell a story, rather than merely confirm that a component finished without error. We need to preserve data provenance, relevant versions, tool calls, intermediate decisions and effects produced beyond the agent’s immediate reach. We also need to distinguish the parts of the chain governed by deterministic rules from those dependent on an inference that could have produced a different output in a different context. The more intelligence we move into the background, the more important it becomes to make its causality visible.
Site Reliability Engineering—SRE, the approach to service reliability popularised by Google—has long stressed that automation is a multiplier, not a guarantee of correctness. It can scale a good practice and a defect with equal efficiency.[4] The warning takes on a particular significance when a system does more than execute instructions: it selects tools, interprets information and decides which path to follow, as agentic AI does.
Speed, in that setting, cuts both ways. An operator might make a mistake on one case and notice something odd before the next; an overnight batch can change thousands of records before anyone opens their email; an agent with write permissions can string together actions that are each correct within its own context and still produce, in seconds, an overall outcome nobody anticipated. Automation does not require distrust of the machine, but it does require designs that recognise that errors, too, can travel at machine speed.
Automating without losing our memory: context continuity
The response cannot be to automate less on principle. That would be hard to defend after decades in which automation has removed repetitive work, reduced errors and made it possible to operate systems that could never be run manually. The question is how to automate without also destroying the conditions that will let us understand the system when its context changes.
Some of the properties involved are familiar—observability, traceability, reversibility and accountability—while others may receive less attention. Observability tells us what is happening; traceability reconstructs where each effect came from; reversibility forces us to consider, before automating, what can be undone and what becomes irreversible the moment it crosses into another system; accountability establishes who has authority to stop, correct or take responsibility for a decision.
I would add a fifth property, which I will call context continuity in this essay. I offer it as a design criterion, rather than an established category, bringing together a concern that recurs throughout the work discussed above: the person responsible should be able to retain or recover the history needed to understand the system. This does not mean producing more documentation out of obligation or filling repositories with manuals nobody reads. It means preserving usable reasons: which assumptions were taken as true, which exception prompted a rule, which risks were accepted, which decisions were considered temporary and which components were designed in reliance on them.
The idea has a practical consequence. Faced with a consequential automated action, an organisation should be able to answer more than ‘which process ran it’. It should know what started the action, which data and criteria were used, which version was active, which components made decisions and which effects have already passed beyond the point of reversal. And it should be able to do so without relying on the continued availability of the one person who was there when the solution was designed.
That test does not turn automation into bureaucracy. It allows autonomy to grow without understanding retreating at the same pace. If the only recovery strategy is to call whoever built the system five years ago, we do not have resilience. We have organisational memory lodged in a single person.
The work that remains
All of this brings us back to the starting point: what will happen to the work we currently do in front of a computer? It is hard to imagine that generating repetitive code, certain kinds of testing, document classification, report preparation or routine data transformations will retain exactly the same professional weight they had before AI became widespread. To deny that would be to confuse defending a profession with defending every task it has historically performed.
We should be equally wary of the reverse argument: if a machine can produce code, understanding systems must become less valuable… The opposite may happen. If software costs less to create, we will have more of it; if processes become cheaper to automate, automations will proliferate; if connecting agents to APIs, databases and enterprise tools becomes trivial, the number of relationships and effects somebody has to govern will rise. An abundance of construction may make understanding, rather than production, the scarce resource.
That should influence technical education. Learning to program does not lose its purpose because a machine can write code, any more than learning about databases lost its purpose when ORMs concealed much of the SQL used in everyday work. The centre of gravity shifts: memorising syntax may matter less than understanding transactions, consistency, concurrency, security, performance, integration or architecture, while knowing how to form a hypothesis when a system stops behaving as it should may be worth more than quickly producing a solution nobody will be able to explain two years from now.
The shift may even become intellectually demanding for those who remain ‘in the loop’. The greater the share of normality absorbed by automation, the greater the share of their work devoted to exceptions, ambiguous decisions and failures spanning several layers. Preparing people only to execute procedures amounts, in that setting, to training them for the part of the job most readily removed.
Back to 9:17
Sometime after midday, Sarah’s team managed to reconstruct the full sequence. The rule that had let the cases through was not absurd: years earlier, it had solved a real problem during a particular processing cycle, when some records arrived with a missing field and needed to proceed if they met other conditions. The exception was added, tested, proved useful and became part of the process; nobody could later establish when it had stopped being temporary. Nor had the trigger done anything unexpected: it responded to exactly the change it was designed to detect, the integration service consumed the queue and the external system accepted the message. Every component had honoured its contract. The incident could not comfortably be blamed on a defective part. The problem lay elsewhere: the original intent had gradually drifted away from the architecture that was still executing it.
It is hard to imagine a better metaphor for cognitive debt. The system had forgotten nothing. The organisation had forgotten.
With AI agents, that distance can grow without anyone making an irresponsible decision. An increase in the pace of change, the number of automations, the dependencies and the layers between an action and its consequences is enough. The exception paradox, the loss of situation awareness described by Endsley, Elish’s moral crumple zone and Storey’s cognitive and intent debts point, from different angles, towards the same underlying problem: retaining human responsibility is insufficient unless we also preserve the conditions needed to exercise it with understanding.
The debate about automation should therefore reach beyond calculating what proportion of tasks a machine can take over. That question will remain important for the economy, for businesses and for those of us trying to anticipate how our professions will change. But from the perspective of architecture, and from the experience of those who will have to maintain these systems when something goes wrong, another question emerges—perhaps less spectacular, and considerably less comfortable:
The real problem will not be deciding how much we can automate. It will be deciding how much we can automate without losing the human ability to understand what we have automated.