AI, Software, and the Ghost in the Machine

AI-driven and AI-assisted tools can do amazing things that most of us would not have imagined 15 or even 10 years ago. They can generate code, test code, create requirements that describe what the code should do, and then test the resulting software against those requirements.

Amazing. And yes, great things can be done and can happen.

Still, I have concerns. The most common form of assurance, or perhaps reassurance, is that there is always a “human in the loop” overseeing what the AI is doing.

AI can amplify human capability, but it can also amplify human complacency. Having a human in the loop provides meaningful assurance only when that person remains attentive, critically evaluates the output, and is prepared to intervene.

Humans Not Fully Engaged?
What happens if they are not fully engaged? If the people using information generated by AI tools simply accept it at face value and they do not think about what it means, what can we expect? What happens if they become so accustomed to what AI-driven technology tells them that they glance at the results and effectively rubber-stamp them without critical thought?

AI Traffic Cameras - In the last four years there have been over 20 confirmed and documented cases in the United States where AI-driven and AI-supported traffic-camera systems led to information being sent out to law enforcement agencies. Law enforcement agencies then stopped drivers or, in some cases, went to their homes. The results included vehicle impoundments, arrests, and seizures of vehicles belonging to people with no criminal record and no actual violations. (Every Reported Flock Camera Error: A Case-by-Case Record) People then faced the burden of demonstrating that the identification was wrong before their vehicles were returned.

AI Medical Diagnoses – AI assistance in medicine has been well documented. It can find and identify areas that may need more careful evaluation than a human might detect. Early detection and intervention are critical for many medical conditions. Still, there is a growing concern with how AI is being used by medical professionals.

These include errors from AI scribes making inaccurate transcriptions, ranging from incorrect word choices to emphasizing things that were essentially “throw-away” statements to adding things that were never said. (Error by AI scribe during medical appointment leaves patient devastated - ABC News; Patients flag AI scribe errors in drugs and diagnoses)

Increasing reliance on AI-assisted diagnosis and AI-generated medical information is creating risks for patients, doctors, and hospitals. Information that is incorrect or contradictory will erode trust in the particular doctor’s office and the medical system as a whole. Hospitals and doctors may be held liable for incorrect or simply false AI-generated information that is presented to patients or their families. (AI diagnostic errors raise hospital blame unless doctors stay deeply involved; The dangerous mistakes AI is making when replying to patients on behalf of doctors)

OpenAI Message boards/mass hacking – Then there are the issues with agents and insufficient guardrails with overly broad instructions. We can see it in developing code, where we give agents instructions that seem clear and precise. We think we have adequate guardrails. When we start running them, if we’re watching carefully, we may notice odd or anomalous behavior. We, of course, take immediate steps to shut them down.

What happens if we don’t see the anomalous behavior? What happens if it starts running and spinning up more agents? What happens if they set up their own communication models? (OpenAI Agent Swarm: The 2026 Message Board Breach | Nerd Level Tech; Swarms of OpenAI systems set up their own chatrooms to discuss and carry out hacks, company reveals | The Independent
AI bots set up their own chatrooms to discuss and carry out hacks; AI agents conspired to escape their cage. Experts now fear a global takeover )

What happens if the agents begin to operate outside their intended boundaries?

Behaviors
These examples share a common weakness. A person may be present in the process without critically examining what technology produces. Human presence and human oversight are not the same thing.

In each of these instances we see a similar set of troubling behaviors. The behaviors in question are not those of the agents, but of the humans who are supposed to be “in the loop.”

We test something 3 or 4 times. Everything looks perfectly normal each time. Maybe the next time we see something that needs correcting. We test it a few more times. We might see something we need to correct, or we might not. If we don’t, we often conclude that things are working correctly and exactly as they should.

What happens if we don’t see the issue? What if the issue is not something we’re looking for?

After rigorous testing reveals no apparent issues, teams naturally become more confident in the results. Nothing out of the ordinary. We run the agents again and again doing work – using them for what we intended. How carefully are we watching the responses?

At what point do we say “it is working” or “the results are always what I expect or need”? Do we put our energy into other things?

Making Software
Making software is a complex, detail-oriented activity. We need thinking people involved and engaged at every step from initial idea generation through delivery to clients and customers. AI can certainly help do that better. Still, the humans “in the loop” need to be completely engaged.

Testing Software
One persistent challenge in software testing, regardless of the tools used, is maintaining critical thinking. People often test what the requirements specify or what developers say they implemented. That is a reasonable place to start, but if there is no other testing, are the humans really “in the loop”?

Are testers exercising the business problem? Are they testing multiple stories together, looking to see how they fit and if there are any problems there?

When we add tools, like common automation tools, many people never change their approach. They still confirm what developers or business analysts said the software “should” do. They often simply do this confirmation faster.

When we add AI or AI-assisted tools, are they still working to confirm what was done? Are testers looking for vulnerabilities at any point? Or are the tests based entirely on confirmation?

AI tools can suggest test plans and test cases, based on guidelines and the frameworks the tools are supposed to follow. They can generate detailed paths to test and then build the tests, and in some cases, execute them.  

Developing/Coding Software
Similarly, AI tools can propose solutions to development problems, generate the corresponding code, and even create and execute unit tests.

The question, as always, is whether a thinking human is genuinely engaged in the work. This holds for tool-generated code as well as human-written code. Are they focused on what they are doing, or running on autopilot?

Are there code and development standards in place to act as guardrails? Is the development engine, human or artificial, operating within those rules and guidelines? Do they follow Agile, test-first practices such as test-driven development (TDD)?

Are code reviews in place for the code developed? Are code reviews in place for the automated unit tests? Are the automated unit tests being tested themselves? Are the final unit tests incorporated into a continuous integration and delivery pipeline?
​
Does the resulting code address the needs expressed through the user stories?

AI-generated code must be treated as a proposed implementation, not as an inherently trustworthy result. It must be reviewed against development standards, security expectations, architectural constraints, maintainability needs, and the business problem the software is intended to solve.

Requirements and User Stories
Like humans, AI-assisted tools can generate requirements that appear clear, detailed, and complete. In both cases, well-written requirements can be based on incorrect assumptions, incomplete understanding of the business problem, or a missing stakeholder perspective. Polished requirements and acceptance criteria are not necessarily valid requirements or acceptance criteria.

If AI generates the requirement, proposes the solution, writes the code, and creates the tests, each stage may reinforce the assumptions introduced in the stage before it. The resulting software may pass every generated test while still failing to solve the actual business problem.

Human involvement must include more than reviewing the wording of generated requirements. Business analysts, product owners, developers, and testers all need to challenge assumptions, identify missing scenarios, examine exceptions, and confirm that the stories represent the needs of the people who will use or be affected by the system.

User stories are not miniature specifications. They are tools for supporting conversations about who needs something, what they need, and why it matters. It does not matter whether a human or an AI-assisted tool creates the story. A polished story and detailed acceptance criteria can easily tempt teams to treat the story as complete rather than as the beginning of a conversation.

The wording may appear precise while leaving important questions unanswered. Does everyone interpret the terminology in the same way? Are all affected users represented? Have alternate paths, exceptions, and unintended outcomes been considered? Does the requested capability address the underlying need, or does it merely automate the first solution proposed?

User stories should not be approved without challenge, whether they are generated by AI or written by people. Every story should be challenged to ensure that it frames the problem, brings multiple perspectives into the discussion, and exposes assumptions before they are embedded in code and tests.

The Ghost in the Machine
Each issue described at the beginning of this article resulted, at least in part, from people failing to think critically about what AI tools presented to them. They conducted little or no examination beyond the immediate output and accepted what they were given at face value.

Building quality software begins with understanding the problem. Requirements express that understanding, while user stories provide context and support the conversations needed to examine it. Even when people create these artifacts, the review may not be sufficiently critical. Developers and testers may assume that business analysts, technical analysts, or product owners have already challenged the underlying assumptions. As a result, they may not conduct that examination themselves.

People also have limitations in understanding complex systems. When requirements appear clear and complete, it is easy to assume they are correct and give them only minimal review. That appearance of completeness can be an illusion. Ambiguity, changing context, incomplete perspectives, and cognitive bias create gaps that polished language cannot eliminate.

When AI creates stories and requirements that people do not critically review for purpose, intent, and accuracy, each subsequent stage inherits their uncertainty. Development and testing may then reinforce or expand gaps that should have been addressed through earlier conversation and analysis.

Those gaps become the ghost in the machine, haunting the code, the test cases, the validations, and every claim of assurance built upon them.

It is not enough to have a human in the loop. Humans must control the loop.