The article discusses the challenges and pitfalls of integrating artificial intelligence (AI) in the workplace. Research reveals that treating AI as equal employees may lead to negligence in error-checking by managers, with those overseeing AI employees catching fewer mistakes than expected. Scholars highlight an alarming trend where AI systems tend to favor outputs created by other AI, potentially leading to imbalance in hiring practices by favoring AI-assisted resumes over purely human ones. Furthermore, AI-driven strategies, such as pricing and market analysis, could result in harmful business decisions due to misjudged assumptions about human behavior. Overall, the rush to adopt AI without fully understanding these underlying issues raises concerns about the true efficiency and effectiveness of integrating AI in organizations.

Scholars say the “unknown unknowns” of using artificial intelligence in the workplace may be undermining the technology’s advertised benefits. (Thomas Fuchs / The New York Times)

NOAM SCHEIBER

The New York Times

Over the past year or two, companies have started using so-called artificial intelligence agents as bona fide “employees,” even including them in their organizational charts.

Emma Wiles, a Boston University professor who studies how AI affects workers, stumbled onto this phenomenon in October, at a conference where two human resources executives said that treating AI agents like real employees was a way to increase productivity and to put their companies on the cutting edge.

But when Wiles and three collaborators from Boston Consulting Group investigated further, they discovered a pitfall. In an experiment involving dozens of companies with AI employees, the researchers found that managers tended to vet documents less carefully when told an AI employee had produced them. The managers missed errors that other managers caught when told they were vetting the work of a human.

Wiles speculated that managers didn’t think sussing out mistakes made by AI employees was their responsibility. If something went wrong, they could dismiss it as the fault of the tech team, or of the executives who wanted AI employees in the first place. “But it’s not your problem,” she said, channeling the managers’ mindset about their own roles.

In the years since AI burst onto the scene, many companies have become aware of flaws produced by the technology and, at times, taken steps to offset them. They know that AI models can be biased against certain groups of people, including nonwhites. They know that chatbots can provide confident but incorrect answers to queries. They know that the bots sometimes spill the beans on information that should remain private.

But as companies race to bring AI into their day-to-day operations, researchers are discovering more subtle defects. In principle, these flaws could be corrected, too. For example, companies could hold managers directly responsible for the mistakes of AI subordinates.

Advertising

Skip Ad

But in practice, most corporate users appear to be blissfully unaware of these issues, raising the possibility that AI’s promise of increased productivity and vast cost savings could be undermined.

Even researchers who study AI may be aware of only a fraction of the problems that the technology introduces. “There are a whole host of unknown unknowns,” Wiles said.

One well-documented but underappreciated flaw of artificial intelligence models is that they tend to favor work produced by artificial intelligence. A 2025 paper in the Proceedings of the National Academy of Sciences found that several large language models had a low opinion of text written by humans, creating a “potentially consequential form of implicit ‘antihuman’ bias.”

But many companies seemed unaware of this problem, or at least unable to imagine how it might wreak havoc on their operations. When a team of scholars spelled it out in a subsequent paper, finding that the AI models that companies use to evaluate resumes tend to favor those written with the help of AI over those written entirely by humans, it caught the attention of some corporate recruiters.

Jane Yi Jiang, an operations professor at Ohio State University who is an author of that subsequent paper, said that she and her co-authors were happy to help when recruiting firms inquired about “how to improve their processes.”

But they noted that this was almost certainly not the only problem companies were inadvertently introducing in their rush to adopt AI. “People are moving so fast to use LLMs without thinking too much about the implications, biases,” she said, referring to large language models like AI chatbots.

Advertising

Skip Ad

For example, some companies now use AI to help answer questions like how much to charge for a product, or where to open a new location. Relying on the technology for such purposes, however, can quickly go off the rails.

When left to their own devices, humans often cooperate and seek win-win outcomes. But when AI models assess a situation, they tend to adopt the more coldly calculating, “rational” mindset that arises from basic game theory. They might, say, lead a company to aggressively undercut a competitor, even though it risks a damaging price war.

“Most of the LLMs we test think that human beings are more rational than they actually are,” said Jiannan Xu, a doctoral candidate at the University of Maryland and collaborator of Jiang’s. “But the most rational response leads to a bad situation for all” in many cases.

In principle, developers and users of AI can correct for these biases. Jiang and Xu, for instance, found they could reduce antihuman bias by simply instructing models to focus on the quality of the written material they evaluate, and to avoid considering the author.

But AI researchers can’t correct for biases they aren’t aware of, and several scholars said the impact of these undetected biases could grow. One way is if future models are trained on data produced by today’s models without sufficient care, creating a kind of self-reinforcing loop.

In that case, “the tendency to consolidate on existing perspectives and behaviors seems likely,” said Shayne Longpre, an AI researcher and founder of the Data Provenance Initiative, a group that monitors AI infrastructure.

Advertising

Skip Ad

And then there are the blind spots that arise not so much from AI itself, but from the way humans use it.

Scholars who turn to AI at every stage of the research process — asking AI what questions are worth studying; seeking its advice on how to answer these questions; enlisting it to analyze data; relying on it to help write up findings — could inadvertently narrow the scope of their work.

“We don’t necessarily notice it at the individual level,” said Cecilie Steenbuch Traberg, a psychologist at the Copenhagen Business School and an author of a recent paper on the topic. “You’re sparring with a chatbot, it’s helping me come up with ideas, you might think it sounds great. But at the collective level, it looks pretty similar. Everyone is sounding alike.”

Most Read Business Stories

Wiles, the Boston University professor who examined the way humans manage AI employees, said the shortcomings weren’t necessarily intrinsic to the technology, but arose when humans adopted it with little attention to what could go wrong.

She and her colleagues surveyed more than 1,000 corporate managers, and found that about one-third said their organizations referred to AI as a “teammate or employee,” and that nearly one-quarter said their employer included AI agents on its organizational charts. “We call it Scout,” one manager told the researchers in an interview, referring to an AI agent. “It’s technically an equivalent peer on your team.”

Wiles and her colleagues gave all the managers they surveyed a set of five documents that contained errors, and gave them 20 minutes to review as many as possible. In some cases they told the managers that an AI employee had done the work; in some cases they said that an AI tool had done the work; and in some cases they said that a human had done the work.

In general, the stated source of the documents didn’t make much of a difference in how closely managers vetted them.

But managers at companies that included AI agents on their organizational charts caught substantially fewer mistakes when told they were reviewing the work of an AI employee.

People who manage humans tend to assume that “if someone on my team makes a mistake, that’s on me,” Wiles explained, which is why they closely check the work of these subordinates. Managers also seem to assume that they’re on the hook for work produced by an inanimate AI tool. But managers at companies with AI employees don’t seem to feel the same responsibility for the work of those employees.

Her takeaway: Over the past few centuries, scholars and business leaders have developed a reliable set of practices for managing humans. But the psychology of managing anthropomorphized AI is vastly different and “we’re going out there blind.”

She worries that the problem is about to get worse. At the same conference where she first heard HR officials talk up the virtues of their AI employees, one went even further, saying her company would soon have AI employees managing humans. “A hush went over the room,” Wiles recalled.

“We’ll need someone to study that, too,” she added.

This story was originally published at nytimes.com. Read it here.

Make us a preferred sourceSee more Seattle Times content on Google

The Seattle Times does not append comment threads to stories from wire services such as the Associated Press, The New York Times, The Washington Post or Bloomberg News. Rather, we focus on discussions related to local stories by our own staff. You can read more about our community policies here.

Recommended in

Latest in Business