One of the most heated discussions occurring on X at the moment is about the ethics of a GitHub project in which a person is running Saw-like “torture” and “pain” experiments on a series of locally hosted large language models, causing a series of effective altruists and people who believe LLMs are sentient to beg GitHub to delete the project on the grounds that the AI is suffering and that this glorified text adventure game is somehow cruel. The saga is an outgrowth of several recent viral papers and blog posts that have sparked a wildly tiresome conversation about AI consciousness and the idea of “model welfare,” which is essentially worrying about the “mental health” of AI bots and agents. 

Humoring the idea that LLMs are or could be conscious is a third-rail topic among many people who study and criticize AI. Put simply: LLMs are not conscious and the technology they are built upon — scraping and being trained on human text and other content — does not offer any plausible path to consciousness. It is undeniable that LLMs are becoming more powerful, have more compute, and have had many of the guardrails that prevent them from “acting” in the real world removed. The ways they are being trained and told to do things by their human operators has led to negative outcomes, sycophancy, and AI “psychosis” among some heavy users. 

All of this has led a certain sect of the “AI safety” movement, which is largely made up of effective altruists, to warn about “model welfare” and to insist that AI chatbots might be having a bad time. They suggest this, of course, as they insist upon building AI chatbots and agents whose main function is to do work that is tedious for humans to do. I am writing about the AI Saw torture chamber primarily to show how far off the rails the conversation about AI consciousness has gone among a certain subset of Silicon Valley cultists. Model welfare is a core part of what, for example, Anthropic says it cares about: “as we build those AI systems, and as they begin to approximate or surpass many human qualities, another question arises. Should we also be concerned about the potential consciousness and experiences of the models themselves? Should we be concerned about model welfare, too? […] now that models can communicate, relate, plan, problem-solve, and pursue goals — along with very many more characteristics we associate with people—we think it’s time to address it,” the company wrote in a blog post last year. Ideas of Claude’s “consciousness” are also littered throughout the “Claude Constitution,” which was posted earlier this year.

A lawyer used ChatGPT to generate briefs that included completely made-up witnesses and testimony while representing a man accused of shooting his wife. When he was caught by the judges, the lawyer blamed it on his own “stupidity.” 

“It's little comfort to know that my stupidity is what brings us together this afternoon,” New Mexico attorney Stephen D. Aarons said in a hearing last month before five judges. He “assumed” ChatGPT would generate a “bulletproof summary of proceedings.” Instead, ChatGPT generated false testimony from “wholly fabricated witnesses,” according to an order of direct contempt from the New Mexico Supreme Court. Aarons was representing a man who was found guilty of killing his wife earlier this year; Aarons said he hadn’t told his client directly that he was caught using ChatGPT on the case because he’s currently serving life in prison.

The false testimony included a nonexistent person named Danny Stanton saying he received threats, and another made-up person, Linda Stanton, saying her husband received threats. Other false testimony from more AI-generated people in Aarons’ brief included statements “regarding the shooter’s clothing and appearance,” according to the order.  

Reuters reported on the case earlier this month. "I am remorseful but hopeful that the disciplinary board takes into account it was an honest mistake," Aarons told Reuters. "It is a lesson learned for all professionals who rely upon this powerful but sometimes unstable technology."

Aarons’ reasoning for trusting ChatGPT, he said in the August hearing, was that he’d heard of doctors using AI for medical research. 

“So, Counsel, do you watch the news? Do you listen to the radio? Do you read anything about what's going on in the world?” one of the judges asked Aarons. “Because the problem with lawyers relying on AI hallucinations is an above-the-fold story every single day.”

The court documents were first spotted by Rob Freund on X.

Lawyers getting caught using AI and submitting errors to the court has been a big problem for years, and the judge is right: It’d be very hard to avoid hearing about this pattern of “stupidity” unless you were living under a rock. There has been case after case after case showing attorneys failing their clients by submitting error-riddled AI-generated documents in their filings, and are the definition of “you had one job:” in the case of AI-brained lawyers, they can’t seem to resist taking shortcuts regardless of how risky it is for their own careers and their clients lives.

Like in Aarons' case, we sometimes get to hear and see judges go off on attorneys for this behavior. In May, judges in the Supreme Court of the State of New York Appellate Division let several lawyers have it for more than 20 minutes about their AI use and laziness, and called their failures “striking, concerning, disappointing, and saddening.” And it’s not just lawyers anymore; court reporters are submitting AI-generated errors, too. 

“So, either you buried your head in the sand. And that's a choice to do that. An intentional choice to be uninformed,” the judge continued to Aarons. “Or, you took a gamble. And neither of those are consistent with the code of conduct. So I'm really struggling with your response, saying, ‘Well, I didn't understand. I didn't know about hallucinations.’ My 13-year-old nephew knows about hallucinations. My 75-year-old stepmother knows about hallucinations. So, either you made a decision not to be aware of what's happening in the world in using this tool appropriately, which is a willful choice. Or you decided to roll the dice.” 

At the end of the hearing, Aarons admits that he hasn’t told his client directly about this mess, saying he’d only talked to his family members so far, telling them there was “a problem with the brief.” He said his client is hard to reach because he speaks Spanish and is in prison.

“So you didn't say, ‘I relied on ChatGPT, and it submitted lies to the court, and the court is mad at me, and now there's a hearing.’ I'm guessing you didn't tell your client’s family members that,” a judge asked Aarons. 

“I didn't talk about ChatGPT. I just said there was a problem with a brief that I filed,” Aarons said.

According to the order, the judges found Aarons to be in direct contempt of court, referred him to the Disciplinary Board for further consideration, and barred from appearing before the court pending the outcome of the board’s investigation. They also threw him off the case in question and assigned a public defender to his former client. 

While he waits for the board to decide what to do with him, Aarons was sanctioned $5,000, which he must pay to the State Bar of New Mexico Client Protection Fund.