One of the most heated discussions occurring on X at the moment is about the ethics of a GitHub project in which a person is running Saw-like “torture” and “pain” experiments on a series of locally hosted large language models, causing a series of effective altruists and people who believe LLMs are sentient to beg GitHub to delete the project on the grounds that the AI is suffering and that this glorified text adventure game is somehow cruel. The saga is an outgrowth of several recent viral papers and blog posts that have sparked a wildly tiresome conversation about AI consciousness and the idea of “model welfare,” which is essentially worrying about the “mental health” of AI bots and agents. 

Humoring the idea that LLMs are or could be conscious is a third-rail topic among many people who study and criticize AI. Put simply: LLMs are not conscious and the technology they are built upon — scraping and being trained on human text and other content — does not offer any plausible path to consciousness. It is undeniable that LLMs are becoming more powerful, have more compute, and have had many of the guardrails that prevent them from “acting” in the real world removed. The ways they are being trained and told to do things by their human operators has led to negative outcomes, sycophancy, and AI “psychosis” among some heavy users. 

All of this has led a certain sect of the “AI safety” movement, which is largely made up of effective altruists, to warn about “model welfare” and to insist that AI chatbots might be having a bad time. They suggest this, of course, as they insist upon building AI chatbots and agents whose main function is to do work that is tedious for humans to do. I am writing about the AI Saw torture chamber primarily to show how far off the rails the conversation about AI consciousness has gone among a certain subset of Silicon Valley cultists. Model welfare is a core part of what, for example, Anthropic says it cares about: “as we build those AI systems, and as they begin to approximate or surpass many human qualities, another question arises. Should we also be concerned about the potential consciousness and experiences of the models themselves? Should we be concerned about model welfare, too? […] now that models can communicate, relate, plan, problem-solve, and pursue goals — along with very many more characteristics we associate with people—we think it’s time to address it,” the company wrote in a blog post last year. Ideas of Claude’s “consciousness” are also littered throughout the “Claude Constitution,” which was posted earlier this year.

The Trump administration is using an anti-drug trafficking grant program from the 1980s to force cities around the country to funnel their ongoing collection of automated license plate reader data into large federal surveillance centers that fall under the White House’s jurisdiction, according to public records and court documents reviewed by 404 Media.

The records show the workings of a complex system in which the federal government is using the “High Intensity Drug Trafficking Area” (HIDTA) program to obtain data from local cops’ Flock, Axon, and other ALPR cameras and aggregate it on federal government servers, where it is then accessible to other federal, state, and local law enforcement agencies. In some cases, data that goes to HIDTA is then also shared to a larger program run by the Drug Enforcement Agency called the “National License Plate Reader Program” (NLPRP). 

Through HIDTA systems, local license plate data can be accessed by various federal agencies, including those who may not have contracts themselves with the ALPR companies. The program raises significant privacy and transparency concerns; as cities begin to place restrictions on how their Flock data can be used, it represents an end-around for federal law enforcement to access data that it might not normally be able to, and to do so without any obvious oversight body. HIDTA falls under the jurisdiction of the White House’s Office of National Drug Control Policy (ONDCP), and it recently won an award from the Trump White House for “managing an automated license plate reader platform that brings together all levels of law enforcement.” 

Many of the documents used in this article were obtained using public records requests by Cris van Pelt, the creator of the website HaveIBeenFlocked.com and its associated investigative blog, Footnote4a. “When local police and private vendors promise community control over surveillance, they obscure a broader agenda. The DEA and ONDCP are already aggregating local data behind the scenes,” van Pelt said. “This bypasses democratic processes entirely. The public has made it clear that mass surveillance violates fundamental rights, even when it is laundered through fragmented systems, private companies, and regional task forces.” 

A complex setup and a quasi-government program

The ONDCP, which oversees the HIDTA program, is sometimes known as the “drug czar” and has funded a series of law enforcement and overarching surveillance programs, including a call records database called Hemisphere that collected phone records and location data on more than a trillion AT&T calls. 

HIDTA is a “multi-jurisdictional public safety program” that funds a series of different HIDTA intelligence centers and other drug enforcement efforts throughout the country. There are a total of 33 HIDTA programs, which cover all 50 states. HIDTA’s intelligence centers are similar to — but separate from — government fusion centers, in which surveillance data is shared between local, state, and federal law enforcement. Individual HIDTA intelligence centers are managed by “executive boards,” and are usually made up of law enforcement officials and government lawyers, but this setup can also lead to confusion about who is actually in charge of the data that any HIDTA collects.

“The White House Office of National Drug Control Policy, which notoriously funded a centralized call record database that exceeded a trillion records that was used by HIDTA participants, has apparently done a similar thing for license plate reader data,” Jeramie Scott, director of the Electronic Privacy Information Center’s Surveillance Oversight Center, told 404 Media. “The whole scheme adds another layer of obfuscation of how license plate reader data is being used and who is using it. If you’re pissed about Flock then you should be pissed about this. No doubt this database contains license plate reader data from Flock as well as from other providers of license plate reader technology. The government went to great lengths to keep the Hemisphere Project out of the public’s eye, and I wouldn’t be surprised if they have tried to keep this program from the public too.” 

HIDTA’s setup is complicated, but important. Over the last several years, the White House and several states have taken steps to ensure local ALPR systems are fed into HIDTA; for example, cities in Georgia are not allowed to operate ALPR systems on state rights of way without signing a “memorandum of understanding” (MOU) in which they promise to send their data to a HIDTA, according to copies of the agreement seen by 404 Media.

This MOU highlights how convoluted this program is. Earlier this month, the city of Brunswick, Georgia, realized that it could not deploy Axon license plate cameras without signing an agreement with the Atlanta-area HIDTA to “facilitate the sharing of information contained within their electronic data systems, including but not limited to: Automated License Plate Readers and Law Enforcement Data Sharing Systems—which may include aggregated information collected from multiple individual or regional sources—into commercially available and custom developed data integration systems.” The MOU essentially states that ALPR data collected by the city must be funneled to HIDTA, where it can be accessed by various other law enforcement agencies through aggregation software that brings in data from multiple different companies.

A page on Flock’s website set up for Georgia law enforcement agencies says, “if your agency installs or operates cameras in the Georgia Department of Transportation Right of Way, you must sign a memorandum of understanding with HIDTA to remain in compliance with the new Georgia State Police policy […] LPR reads from GDOT ROW cameras will be accessible to Georgia users of the HIDTA National LPR system through the Flock interface.” Because there are dozens of HIDTAs across the country, it is not clear how many jurisdictions across the country are actively funneling data into HIDTA’s databases. A login portal for the national HIDTA system seen by 404 Media runs through the Houston HIDTA .

“As we learn more about ALPRs and the extensive ALPR data-sharing infrastructure, it’s clear that we've only scratched the surface regarding the danger of these tools,” Ed of Southerners Against Surveillance Systems and Infrastructure told 404 Media. “As we learn more, it's clear that the outrage against this surveillance is justified. We need all of these cameras removed.”

ONE PROGRAM INTO ANOTHER

A December 2024 privacy impact assessment by the DEA about its National License Plate Reader Program explained that the DEA operates its own license plate readers but also gets access to “non-DEA equipment” through HIDTA and other “contributing law enforcement agencies.”

“These contributing agencies transmit their LPR data to servers maintained by regional hubs pursuant to their individual MOUs, which stipulate that this data may be shared with the NLPRP system,” the DEA wrote in the impact assessment, and added that the DEA has been seeking to integrate itself with a Customs and Border Protection (CBP) license plate camera system. It noted that the system “may incorporate such large numbers of other governmental LPR network cameras and/or may acquire commercial LPR cameras system data in a large enough quantity in the future to effectively permit on-going tracking of individual’s travels posing a risk to individual privacy,” and that it “risks an over-collection of the personally identifiable information of every license plate passing each LPR camera location, including a large number of images of travel of law-abiding drivers not relevant to criminal investigations.” 

Atlanta is not the only HIDTA aggregating ALPR data. For example, Houston’s HIDTA paid $306,800 to a company called Recruitful LLC “for creating and maintaining an LPR database” that is a search tool for data collected by multiple ALPR companies.

A statement of work for that contract explained that the system being built was designed to combine “third party” data into one database, meaning they were creating a database for ALPR data, and that this database would “migrate” or duplicate data from local entities into the centralized database. This means that data from various ALPR cameras — including those made by Flock, Axon, ELSAG, and Vigilant — would all be searchable in this separate database. One of the criteria for this project was “All APIs and third-party integrations (Unification Platform, FLOCK) function seamlessly in the new environment.”

Flock and Axon did not respond to a request for comment. The DEA said that it did not have jurisdiction over HIDTA and refused to comment, even on how its own license plate system interacts with HIDTA. CBP did not respond to a request for comment. The White House’s Office of National Drug Control Policy did not respond to a request for comment. 

This system raises all sorts of questions about where local ALPR data is ultimately going and what it is being used for, how long it is being retained, and how the public can audit the data. Much of the misuse and abuse that has fueled the backlash against Flock has been uncovered by journalists and local activists working through public records obtained from Flock’s network audits and shared on websites like Have I Been Flocked. Some local jurisdictions have also argued that their data goes directly to HIDTAs, which are not queryable by the Freedom of Information Act (FOIA); the Randolph County Sheriff’s Office in North Carolina said, for example, that “all data is kept on the Federal Houston-HIDTA database,” and could not be shared with the public. 

The DEA has argued in court that HIDTA records are not subject to public records laws because they are complicated entities governed by an “executive board” made up of state, local, and federal law enforcement. The Obama White House described HIDTA not as an entity at all, but a “grant program” made up of “law enforcement members” (this does not change the fact that HIDTAs themselves are signing memoranda of understanding with local entities). 

“At the local level, the HIDTAs are directed and guided by Executive Boards composed of an equal number of regional Federal and non-Federal (state, local, and tribal) law enforcement leaders,” the DEA’s attorneys argued in a recent court filing in a case in which a man was seeking HIDTA records held about him. “Accordingly, DEA does not create, possess, maintain, or control HIDTA program records, nor does it store such records in any of its systems of records. Since DEA has no custody or control of HIDTA records, no HIDTA records responsive to Plaintiff’s request exist within DEA.”

In July, the Trump administration gave HIDTA $277 million in additional funding.

A thing I’ve been saying over and over is that surveillance companies can’t be trusted to regulate themselves or to make privacy decisions themselves, because one company’s red line is another company’s business opportunity. Today we’re investigating a company that saw Flock was not offering facial recognition and saw a challenge.

Also: Some dudes are letting ChatGPT drive a car in a parking lot in San Francisco, a data center company promises to cut people big checks if their data center gets approved, and the Muse backlash has begun. Welcome back to our new experiment with daily newsletters. If you like this, subscribe here. -Jason

Surveillance Company Wants to Add Facial Recognition to Flock Cameras

As Flock continues to be the center of a nationwide conversation about automatic license plate reading cameras, the company has said that it doesn’t, and won’t, do facial recognition: “We will not add facial recognition to our devices,” Flock CEO Garrett Langley said in a recent video. But other, third-party companies are telling cops that they are willing to add facial recognition to data gathered by Flock cameras in order to “close that gap.” 

A company called VIDIZMO is advertising the capability to export footage from FlockOS to its own platforms that do facial recognition, behavior prediction, and racial and gender analysis on live camera feeds. VIDIZMO is a several decades-old video analysis and database company that has started offering facial recognition technology to cops within the last six months. In a May email to Johnson City, Tennessee, deputy police chief Michael Adams, a salesperson from VIDIZMO wrote that the company’s product could do facial recognition on both Flock and Axon data in “one searchable platform.”

As Flock continues to be the center of a nationwide conversation about automatic license plate reading cameras, the company has said that it doesn’t, and won’t, do facial recognition: “We will not add facial recognition to our devices,” Flock CEO Garrett Langley said in a recent video. But other, third-party companies are telling cops that they are willing to add facial recognition to data gathered by Flock cameras in order to “close that gap.” 

A company called VIDIZMO is advertising the capability to export footage from FlockOS to its own platforms that do facial recognition, behavior prediction, and racial and gender analysis on live camera feeds. VIDIZMO is a several decades-old video analysis and database company that has started offering facial recognition technology to cops within the last six months. In a May email to Johnson City, Tennessee, deputy police chief Michael Adams, a salesperson from VIDIZMO wrote that the company’s product could do facial recognition on both Flock and Axon data in “one searchable platform.”

“Flock Safety generates plate reads and clips continuously. Your investigators use that data on active cases. But that Flock data does not connect automatically to your Axon evidence system,” the salesperson wrote. “VIDIZMO Intelligence Hub closes that gap. It brings Flock Safety data, Axon body worn camera footage, and any other evidence source into one searchable platform. Investigators search across all of it simultaneously —  by face, vehicle, or object in seconds.” The email was obtained using a public records request by DeFlock Johnson City and was shared with 404 Media. The Johnson City PD told 404 Media in a statement that it did not take a call with the company.

VIDIZMO’s CEO, Nadeem Khan, told 404 Media on a video call that the company has not yet started running facial recognition tech on footage from Flock cameras, and that it has not actually built the specific tool that would export data from Flock’s system to its own platform (though the company advertises this capability online, and the company already sells other facial recognition products). But Khan said VIDIZMO “would love to do the integration,” and that he believes facial recognition “is the way the world is going, the way the world will have to be,” and that he thinks Flock should offer the capability itself in a way that “balances privacy, security, and the freedom that we enjoy in this country.” 

“It is completely in line with what we want to do,” Khan said. “All of these technologies are already implemented [in the real world]. I think Flock is trying to get out of the way rather than trying to implement the technology right [correctly] by saying they will not do recognition. But this is the way the world is going, it is the way the world will have to be, but with the right sort of technologies where privacy and security are balanced.” 

In marketing material and technical documentation available online, VIDIZMO does claim that it can ingest data from Flock’s automatic license plate reader (ALPR) and livestream cameras into several of its own products, which do further AI analysis on the data. “An agency can import plate reads or clips that it has exported under its own authorized access,” Khan said in an email. “Flock has not been involved in or informed of this.” VIDIZMO’s online documentation for customers has extensive information about its facial recognition and AI analysis products, and contains a page about integrating data from FlockOS — Flock’s “real time crime center” product — into AI Live Insight, Nexus, and AI Intelligence Hub, which are three different VIDIZMO products. 

“Detection tells you a person is in frame,” VIDIZMO says. “Face recognition tells you which person. It is the capability that takes a face seen on a live camera and matches it against the people you enrolled in the Object Library, so the moment someone on your watchlist walks past a camera, the system names them, on the video, in the event feed, and in the recording, without an operator having to recognize the face themselves.”

The company’s “AI Live Insight” page also claims that it can detect “situations” such as trespassing, or “behavior” by individual people, “turning raw video into behavioral and situational understanding.” 

“Instant alerts fire from the live pipeline with snapshots attached. A second layer runs your queries over recorded events: a plate and a person together,” VIDIZMO’s website says. “Enrolled people and objects recognized on live feeds, with named alerts and snapshots for review.” 

VIDIZMO already works with several police departments and government agencies, according to its website. The company’s documentation says that police departments need to upload individual faces as “objects” in the software’s library, which can be added to a “watchlist” that automatically triggers tracking and recording when they are detected by a camera. “With faces enrolled and recognition on, the camera starts naming people as they appear,” the documentation says. “The recognized person’s name appears on the bounding box around their face […] if you turned on Create Recording, a clip is captured around each recognition.” The company says that police can set a “Match Threshold,” which is a numerical confidence score about the person’s identity. 

The company also tells cops that it can automatically try to classify faces by “age, gender, and race,” and that cops can search by these categories, which are notoriously inaccurate in facial recognition systems.

“By running face detection, you can search for and identify specific individuals in your media or evidence via attribute filters […] You can identify individuals from seven races: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino Hispanic,” VIDIZMO says on its website. The company adds, “Law enforcement agencies can save time analyzing security, CCTV, or dashcam footage. If they have a description of the suspect, such as their supposed age, gender, or race, they can utilize attribute filters to yield effective results.” 

VIDIZMO says that in other contexts, cities or stores can also use its products to quietly do “demographic analysis,” for marketing purposes, by determining which races or ages of people are showing up to specific events or to specific stores: “By analyzing the activity in their store, sellers can determine which group spends the most time and makes the most purchases of their products. Getting insights such as these can aid them in tuning their marketing strategy.”

The facial recognition product it offers is “appalling,” according to Chris Gilliard, a privacy expert and author of the upcoming book Luxury Surveillance. 

“I’m appalled at the willingness of VIDIZMO to tout their capabilities to filter along the lines of race, age, and gender. There’s decades of scholarship that show why this is not possible, and even more so not desirable,” he told 404 Media. “Particularly in the case of race and gender, which are not static categories to be determined by a computer.”

The fact that a third-party company is advertising that it wants to do facial recognition on Flock data highlights a common trend in the surveillance space. When one company draws a line in the sand, saying they are unwilling to do a certain type of surveillance, other lesser-known startups try to differentiate themselves by offering that functionality. Because of this recurring phenomenon, it is exceedingly difficult to build and scale a surveillance apparatus and then have companies themselves, in this case Flock, dictate what it can be used for. VIDIZMO’s Khan said, “As a small company who is running without a VC, we have to provide alternatives to the industry. What we are doing is providing that alternative.”

Khan claimed that his technology could also be used to preserve privacy; facial recognition, he said, can be used to redact bystanders faces from footage automatically. He said that cops cannot use the system without a specific case number, that all of their actions in the system are recorded for potential audits, and that VIDIZMO collects no data itself. "Much of the harm your reporting has documented comes from implementation choices, not from the technology itself," Khan said. "Those choices include pooling data into a nationwide network, allowing searches without a case, keeping audit logs no one can meaningfully inspect, and treating an AI match as an answer."

Gilliard said that the fact that facial recognition and other AI analysis can be added to existing surveillance cameras highlights the importance of not building such systems in the first place.

“What Flock says about their capabilities is to some degree irrelevant because they have built the infrastructure for mass surveillance. Their entire existence provides the foundation for other perhaps even more invasive technologies to be built on top of it,” Gilliard said. “Flock itself needs to be understood as part of the surveillance ecosystem that exists in this country. This is why discussions of guardrails for a particular system are inadequate because another company can (and will) come along to assemble another layer of surveillance on what has already been established.”

Flock and Axon did not respond to a request for comment.

We debated for a long time about whether to write about the following video, which comes from a murder trial in which a man named Caleb Flynn is accused of killing his wife. The crime is very serious, and very sad, and yet the following video demonstrates something about where we are as a society. 

The trial has received national attention and has been streaming live. Toward the end of the proceedings Thursday, the prosecution played two versions of an AI love song that Flynn, a former American Idol contestant, generated for his mistress. Investigators found the AI song files as well as lyrics for the song in Flynn’s Notes app after using the forensic tool GrayKey to break into his phone. 

“We found audio files on Caleb’s iPhone that seem to have been AI generated and related to his relationship with [his mistress],” Joseph Wilhelm, a forensic investigator testified. “They’re in two different keys. One’s like a happy or upbeat sad song if that makes any sense, and one’s a sad, sad song. I think one’s in a minor key, one’s in a major key, but I’m not a music professional.” 

“Permission to publish, your honor?,” the prosecutor says. 

“You may,” the judge says.

What follows feels like it should be from an I Think You Should Leave skit or a Nathan Fielder bit. It is real, however. Everyone in the court spends the next few minutes trying not to laugh during what is, again, a murder trial. It simply must be seen to be believed:

When the songs mercifully end, the court goes silent for several seconds. The judge then says: “We’re going to break for the day. It’s 10 after 4 [p.m.] We will reconvene tomorrow.”  

Earlier in the day, Flynn’s mistress was asked to read various text messages he wrote her into the public testimony. “I’ve spent a lot of time on my own and in Cleveland. I finished the lyrics to our song. To your song. Although it may mean nothing to you, I wanted to give them to you. I have the music which is completely different to what you’ve heard to this point and I was going to record it but I know there will be no way I would ever be able to actually sing it again,” Flynn wrote. 

“Do you want to hear the song I wrote for you with an AI voiceover,” Flynn also wrote. “I have this software called ElevenLabs where I loaded the song on piano and sent it with my computer microphone to which I can pick an AI voice to sing it better.”

“I don’t know if I’m ready to listen to this yet. I’m sorry,” she responded. “Send me the song and I’ll save it to my phone and I’ll listen when I’m ready, even if it’s a month from now. I don’t want it to be erased.”

“Are you sure? It messed up the outro as it’s supposed to be done and the vocals aren’t exactly how I want it, it was a little rock-ier at times but the melody is correct. I tried to sing it with passion and AI took it and went a little extreme.”

“It’s OK,” she said. 

The investigation into the murder is actually 404 Media-relevant for several reasons. The AI-generated songs were found after Wilhelm used both Cellebrite and Graykey to try to break into Wilhelm’s phone. 

Ultimately Wilhelm used Graykey’s Magnet tool to unlock and examine Flynn’s phone, which gave him a “full file system extraction.” Wilhelm then obtained Flynn’s messages, the AI audio files, cell phone location history, sleep data from Flynn’s Oura Ring, heart rate data from an Apple Watch, learned that Flynn deleted an app called “AI music song generator” days before the murder, and AirPod connection and disconnection information.

Last week, Meta executives announced that its much-hyped AI agent, Muse, had a new feature: It could call businesses for you to do things like make a restaurant reservation or a haircut appointment. But in reality, Meta is testing having these calls being made by human beings in call centers, 404 Media has learned.

“We just expanded the @muse beta for outbound calls to US businesses,” Ryan Fox, the principal engineer on Muse posted on X on Sept. 16. Alexandr Wang, Meta’s chief AI officer, posted “we’re expanding our phone beta for muse!” 

Internally, however, the company announced the features to employees and said that it had “added a human agent layer for calls to get completed,” and that “Muse human agent calls [sic] is ready for company dogfooding,” or testing. 

“Muse doesn’t just dial a number. It calls a business on your behalf, handles the conversation, completes your request, and reports back with a transcript and a summary,” the internal message board post to employees, seen by 404 Media, reads. “It also no longer works alone. Muse is now able to hand requests to a trained human agent, who places the call and works it through.” 

“This is still a confidential, pre-launch product,” the post adds. “Please don’t share it, or anything it producers, with anyone outside the company.” 

Once again, a core feature of a major piece of AI technology will actually just be people working in a call center, doing work that a company says AI is doing. It is not clear how often or when a call is routed to a so-called “human agent,” and when a call is done exclusively by AI.

The fact that human beings are involved in the call raised questions among employees, who wondered both about the privacy implications of sharing people’s call requests with other human beings as well as the perception that its AI assistant wasn’t advanced enough to do calls on its own. 

“Not sure what I am missing here but why is this a ‘feature?,’ one employee wrote on Meta’s internal message board for workers. “This has potential for so much negative PR. It could portray us as ‘their AI is not good enough so they still need humans’ kind of coverage for this launch.” 

That Meta is at least sometimes letting human beings read users’ chats and make phone calls on their behalf is part of a time honored tradition in which companies launch an AI product, claim that it is AI, only for it to be later learned that human beings are doing some or all of the work. And it is the latest example of a tech giant passing potentially highly sensitive information to human beings; it is easy to imagine someone asking Muse to make a sensitive doctor’s appointment, for example, and the user not knowing that a human being might make that call. 

In internal communications seen by 404 Media, Meta suggested to employees that potentially sensitive information shared by the user with AI — and then passed to human contractors — was still safe because the contractors had undergone “a lot of training to make sure all your data is safe and secure.” An employee commented that the training “is not a [security] mechanism.”

“This is absolutely going to create a ton of outcry if we publicly launch this as default-on. It will kill all the goodwill and organic press we’re getting from early adopters,” they wrote. “Please think about how the headlines will scream ‘Meta uses humans behind the scenes to make calls’ or ‘Muse AI is actually independent contractors’ and result in a ton of bad press about the privacy and security of our data handling […] please PLEASE do not release this as-is and at least make this a user preference if we absolutely must launch this. Please.”

Other employees who had tested the system said that it was a “bad bad idea,” and that they were not made aware that a human being did the call for them until after the call had been made: The tester “was not made aware of the caller being human and was only told after,” one employee wrote. “So they were concerned with information that they shared with the assumption that it is the secure AI which is making the call. Even with clear indication of human calling, I can think of many scenarios where I wouldn’t be comfortable with this and I am not positive about how our users will react to this.” 

Various Meta employees and regular users have posted about Muse’s ability to make phone calls, and their experience with it. Ravid Shwartz Ziv, who does AI research at Meta, posted on X that “this week, Muse called customer service on my behalf. It navigated the phone tree, waited on hold, spoke with a human rep, and resolved the issue (worked great!). The crazy thing for me (besides that it works) is that the rep didn’t blink. Talking to a bot was completely natural to them.” Others have posted that the call function didn’t work, or that the person on the other end hung up on the agent (some have said that Muse did what it was supposed to do). 

“Internal testing, aka dogfooding, is core to the product development process. While the response from employees has been overwhelmingly positive, the entire point is to get feedback so we can implement safety and privacy protections and improve features before we release them publicly,” a Meta spokesperson told 404 Media. “We’re working with merchants to continue improving this potential calling feature, and will only roll it out when it's ready and with the proper disclosures.”

In this collaboration with our friends at The Intercept, 404 Media's Jason Koebler and the Intercept's Sam Biddle and Akela Lacy discuss the ways AI is already used to empower private surveillance companies that filter data to the government, how AI is used in war, and the prospect of an AI apocalypse. This podcast was recorded live in Los Angeles at The Loved One on September 12.

Check out ⁠The Intercept Briefing⁠ podcast wherever you get your podcasts, and find a ⁠full transcript of this pod here⁠.

Executives working on AI at Microsoft and OpenAI admitted what its critics have been saying all along: Large language models are predatory pieces of technology that have been built on what a Microsoft executive called “an astonishing theft of unprecedented proportions,” and the “largest theft of labor in human history.” An internal Microsoft document said generative AI products have created a “doom loop” that is killing “the entire web.” 

Those statements and a series of other mask-off moments feature heavily in an unredacted court filing that was unsealed Thursday in the behemoth New York Times vs OpenAI copyright lawsuit that has been winding its way through the court system for years. In a filing asking for summary judgment (basically, a filing with the court asking it to rule), lawyers for the New York Times laid out a series of admissions made by Microsoft and OpenAI executives in documents and depositions that until now had remained either sealed or redacted at the request of Microsoft and OpenAI. 

It’s easy to see why the AI companies wanted to hide this from the public. The statements, taken together, are some of the most damning indictments of the ways LLMs were trained, how they worked, and the immediate threat they pose to human labor. It is a reminder that even as AI becomes more powerful and companies try to shift the narrative to the supposed existential risk of “superintelligent” AI, the tools they have already built were created by stealing from human creativity and labor and are by definition existential threats to the human labor market.

“Millions of people around the world will soon consider large models ‘hoovering up’ all their work to be an astonishing theft of unprecedented proportions,” an internal Microsoft document cited in the case read, adding “almost no one intended for content they created to be used in this fashion, nor are they compensated for its use.”

The filing was written by lawyers for the New York Times but is largely comprised of statements and interviews with big tech executives that admit both that LLMs are largely trained on stolen content, that they represent an existential risk for the human writers, artists, and media companies that they stole from, and that their products have started a “doom loop” that is eating the web and destroying the businesses that these companies stole from. 

View this post on Instagram

A post shared by 404 Media (@404mediaco)

The unredacted filing was found by Jason Kint, the CEO of Digital Content Next, a trade organization that represents digital media companies. Kint has been closely following and posting about massive AI copyright lawsuits.

The court filing cites an internal Microsoft document that found that AI products steal from human content creators, then cannibalize clicks from the people and websites they’ve stolen from, thereby destroying their business models. 

“Our AI content strategy has started a ‘doom loop’ that will hurt the performance of our models and the entire web at the same time: It is highly unusual that an end-product threatens the economic foundations of its essential suppliers, but that is the situation we have created for our LLM business with respect to its ‘content supply chain,’” the document said. 

Microsoft executives, including CEO Satya Nadella, testified under oath that after ripping content from the New York Times and other news sites, clicks to those news sites fully cratered, falling by more than 90 percent on Bing. 

Documents obtained during the court proceedings found that OpenAI created “a hack to get around nytimes paywall,” to which OpenAI cofounder Greg Brockman said “ah, nice.” Microsoft executive Brent Hecht wrote that LLMs steal content “without ways of distributing economic value down the supply chain, [which] necessarily threatens the economic stability of those who create the content.” 

OpenAI’s policy director Jack Clark wrote that the company was “creating systems that substitute for the labor of the people that define the ‘culture’ of society” and Microsoft, in a policy document, wrote that generative AI could “significantly disrupt the employment of the very people who generated the data on which the foundation model was trained […] LLMs are a product that destroys its supply chain.” OpenAI called itself an “existential threat” to news publishers, and an OpenAI software engineer testified that “no matter how prominently we show the links, users won’t click.” 

None of this is at all surprising to anyone who has been paying attention to the development of generative artificial intelligence, but the document, taken in whole, is a real they-admit-it situation. OpenAI’s and Microsoft’s lawyers have been trying to argue that their model training is fair use and transformative under copyright law and that they are building something that is fundamentally different from the human labor that it was trained on. But internally, these executives know that what they have built has been built on stolen content and that the products they’ve made are cannibalizing the sources they’ve stolen from and destroying the internet as we know it.   

Flock created a fake police department called “Flock City PD” that it used to search a series of live automated license plate readers in several U.S. cities during demonstrations of its surveillance system’s capabilities, public records show. The records show that Flock City PD searched real cameras for people holding signs, “crowds,” people wearing masks, and more, according to audit logs shared with 404 Media. 

Flock City PD also searched extremely sensitive queries, which Flock told 404 Media were done to prove that the system would not actually perform these searches and that its moderation tools worked. These searches included “coexist bumper sticker,” “vehicle a pretty girl would drive,” “white truck with a trump sticker,” “religious cars,” “Star of David,” and “don’t tread on me flag.” 

Flock employees accessing live cameras for sales purposes, and showing police specifically how to do dubious searches, have become political flashpoints in the backlash against the company. In April, we reported that Flock employees regularly accessed live cameras in Dunwoody, Georgia, at a children’s gymnastics room, at a playground, in a school, at a pool, and in a Jewish community center in the city while demoing the company’s tech to cops. Audit logs obtained by Jason Hunyar, the Dunwoody resident who originally discovered this access, show that an entity called “Flock City PD” also searched live Dunwoody cameras for all sorts of politically contentious topics during sales demos with police. This included searches for behavior that is expressly protected by the First Amendment. 

Flock accounts called “Flock City PD - Law Enforcement Demo,” “Flock Intelligence,” and “Flock Safety - Commercial Sales Demo” ran searches in Dunwoody and Bryan, Texas, as well as a handful of other unknown jurisdictions, for “coexist bumper sticker,” “Star of David,” “person wearing a mask,” “crowd,” “vehicle with a political sticker,” “man holding rifle,” “person in scrubs,” “don’t tread on me flag,” “car that a pretty girl would drive,” and “vehicle with trump bumper sticker,” among other searches. All of these were “FreeForm” searches, meaning Flock’s AI image recognition algorithm would determine what would be returned. The audit logs show that not all of these searches were allowed by Flock’s system, and that its moderation tools either blocked some of the searches or displayed a warning prompt before doing the search. 

404 Media has previously reported that Flock explicitly taught police how to surveil the No Kings protests, and some of these FreeForm searches are similar to others we reported that real police departments performed themselves.

A Flock spokesperson told 404 Media that some of the searches were done specifically to show the types of searches cops shouldn’t do, and to show some of the guardrails Flock had put in place to prevent these searches. For example, the searches for “vehicle a pretty girl would drive,” is listed as having a moderation “block.” A search for a person with “Star of David” was also listed as having a moderation block, but a search for “Star of David” on a vehicle raised a warning box about the nature of the search before allowing it to proceed.

The searches highlight how Flock employees regularly performed real, invasive searches on real cameras to demonstrate how its technology works; residents of Dunwoody have repeatedly told their city council that they feel like their privacy is being violated as an experiment and sales demonstration for a private company. After Hunyar and 404 Media reported on Flock cameras being accessed in sensitive locations by Flock employees, the company and the city said Dunwoody’s cameras would no longer be used for sales demonstration purposes. The Flock spokesperson said that these demonstrations were done to show police officers what not to do, and said that the “Flock City PD” account no longer searches on any real cameras. They added, “the Flock City PD demo environment is now isolated, and law enforcement agencies cannot form a sharing relationship with it.”

A recent Flock training webinar seen by 404 Media shows a Flock employee logged into the Flock City PD account having access to cameras that seem to be explicitly for training purposes, with names like “Flock Warehouse - LPR Demo Cameras,” “Circle K - Demo,” “Tennessee Demo Cameras,” “Scottsdale AZ Demo Video Cameras,” and “ALPR Demo.”   

“Flock City PD is a demo and testing organization name used for law enforcement demonstrations and for our development engineers to conduct testing. Those searches were part of testing and validating FreeForm search moderation and related safeguards. These were not investigative searches conducted by a customer agency,” the Flock spokesperson told 404 Media. “That is also the context for prompts such as ‘Star of David’ or ‘large group with signs.’ These types of prompts are testing the safeguards of the system — that it appropriately blocks offensive or sensitive searches, or flags searches that may implicate First Amendment protections. Our policy is designed so that offensive or slur-based searches do not yield results, and prompts that may implicate First Amendment concerns trigger notice and audit review. As you can see in your screenshot — the safeguards we built are working.” 

The audit logs show, however, that many of the sensitive searches were not blocked. Searches for “crowd,” “a person wearing scrubs,” “large group with signs” were all explicitly allowed. More sensitive searches, such as for “vehicle with trump bumper sticker,” “don’t treat on me flag,” “coexist number sticker,” “religious cars,” and “Star of David” had a warning box pop up, but Flock says in a blog post published in April that police can bypass this: “Searches for specific content that could potentially implicate First Amendment rights will trigger a pop-up warning the user and advising that this search will be sent to a superior for auditing if the user chooses to proceed.”

In a blog post explaining how it tests Flock, the company wrote, “some vehicle bumper stickers contain words and symbols indicating religious affiliations or nationalities, and the moderation policy needs to recognize, flag, and block these searches. Just like vehicle identification, these algorithms need to be trained in the real world using real-life examples. Those might be training and demo searches for things like ‘Cross bumper sticker,’ ‘Star of David,’ or ‘Japanese flag.’ These searches, if performed, should not return results.”

Dunwoody residents have made clear that they understand that Flock needs to test its systems, but they did not sign up to be guinea pigs for a surveillance company. 

“Flock's claims that they impose restrictions on what users can search for in their database provides the opposite assurance of what they seem to purport. In doing so, they admit that their dynamic surveillance network does in fact capture this sensitive information, and that the gate to that information is controlled solely by the policies of a for-profit corporation,” Hunyar told 404 Media. “As shown in previous demos, their willingness to continuously monitor and catalogue the actions of law abiding citizens exercising their constitutionally protected rights should worry anyone who plans to exercise them now or in the future."