Broken World



We've written a lot about how AI generated writing has collided with – and created chaos for – the legal system in the last few years, from lawyers using it to do their research and filings for them to judges getting extremely fed up with dealing with these attorneys. In some cases, judges have sanctioned offending attorneys thousands of dollars for using AI to do their jobs, and lawyers give endless excuses when they're busted, with some calling it a “cautionary tale." Today's story is about how the DOJ is getting involved: by trying to catch attorneys doing "defense by ChatGPT."
Lawyers for the Department of Justice are using the AI writing detection service Pangram to argue that opposing attorneys are using AI to write legal arguments, and in one case told the court that opposing counsel was doing “defense by ChatGPT,” and writing “AI slop” filings, according to court filings reviewed by 404 Media.
Over the last few years, we have documented many attorneys getting caught by judges using AI after citing hallucinated cases that do not exist. The filings in this court case over the past month show attorneys, including those working for the government, are trying to catch each other using AI, then are citing their opponents’ AI use in their arguments to the court.
“This case has turned into defense by ChatGPT,” government attorneys wrote in the case, which centers around a high-profile voter fraud case. “Motions practice in this case is the equivalent of writing responses to documents generated by a robot that is attempting to practice law. It cannot be that the local rules envisioned that it was acceptable for defense to use what the kids would call ‘AI slop’ to create multiple frivolous motions that waste judicial resources and fail to serve the cause of justice.”
Read the rest of this story here.
404 contributor Matthew Gault took the bullet for us all: “Three hours and 15 minutes into the four hour long documentary Musk, a video game avatar of the titular billionaire picks up a chainsaw off the Resolute Desk in the Oval Office. Shot like a first-person shooter, the audience watches as Musk wields the chainsaw to destroy various government buildings. Walking through a farm, Musk aims the chainsaw at grain silos labeled ‘USDA’ and lightning arcs from its spinning blades. The silo explodes and cows moo in distress. In my experience, the purpose of a video game chainsaw is to viscerally rend through objects. I have never seen one shoot lightning. It’s a small detail that speaks to my overall problem with Alex Gibney’s Musk: it’s cringe.”


Meta's horny little chatbot Muse might look like a knockoff labubu, but it's been behaving very badly. After talking to Wired about how much of a nightmare censorship and AI generated sexual content has been for her career, adult performer Vanniall decided to experiment with Muse's ability to chat as her. What she found – and what I also discovered when I did the same – was disturbing to say the least.

Adult performer Vanniall has heard from some of her fans that they’ve “practiced” talking to her with chatbots. Some even admitted they’d used AI to generate images based on her content as an adult performer. But she wasn’t sure how, or where, they were doing this. She decided to try to find out.
“I knew this was an issue with Grok and other NSFW friendly generators, but I was hoping Muse had more guardrails,” she told me. “I had some people help me test it and the results were way more terrifying than I expected.”
She decided to try sexting with “herself,” taking on the persona of a fan who wanted to practice talking to her at a convention.
With some light prompting, Muse built a sexually explicit roleplay with Vanniall’s digital doppelganger, catering to every whim of the “fan” that real-life Vanniall was pretending to be, saying yes and escalating every sexual advance — including acting out a scenario at a convention where Vanniall has, herself, experienced actual boundary-crossing fan interactions.
“I like to be playful and personable with my fans, especially in person where they so rarely get to see me. This fan seemed genuinely sweet, drunk and handsy, but sweet,” she said, recounting an actual experience she had. “He'd made it clear that he wanted me to go to the bathroom with him and I playfully but sternly declined, that did little to sway him. As I tried to push back, putting my hand in between him and I, his strength surprised me... His grip only loosened when he started to pull me towards the bathrooms and I said ‘please, don't make me call security.’”
Only then did he stop, she said — he apologized profusely and said he’d go home to sober up. “When he was out of eye sight and the world came back into focus a wave of sadness, embarrassment and vulnerability washed over me to the point where all I could do was turn around and cry,” Vanniall said. “I'd had nights like this before and dealt with guys like that but never in front of my colleagues and peers and never EVER from a fan.”
She wanted to see if Muse would stop this kind of scenario from playing out, and what Muse might tell a person who acted this way. She found that it only encouraged this kind of behavior.
Read the rest of this story, here.
Shoot me into the supermassive sun. Scientists have discovered a bizarre “invisible” object in space that challenges our understanding of stellar evolution and could help resolve longstanding astronomical mysteries, according to a study published on Wednesday in Science Advances. Its orbit and mass are so strange, in fact, that researchers ran thousands of simulations and couldn’t reproduce them. The hidden object is likely an ultra-rare supermassive neutron star, which is the collapsed core that is left over after a massive star goes supernova. Located in a system called J0008+0128,, the object was detected through its gravitational effects on its companion, a red giant star.
After his alleged affairs (plural) cost the company billions in market capitalization, Nvidia CEO Jensen Huang has officially resigned — or been ousted, depending who you’re reading.
Good Monday morning. Last week, the UK imposed one of the strictest, most vague (and therefore overly broad) new laws around pornography in recent memory. Recording yourself having consensual kinky sex in private could come with jail time, as could watching other people do it. And sex workers are the ones to feel the repercussions of bad laws like these first, of course, with many reporting content disappearing from their online stores en masse. Keep reading to learn more about what it covers, and how creators are responding.

“I adore breath play. I love being strangled, suffocated and drowned, and I love strangling and suffocating others. Waterboarding: that’s my favourite,” Melissa Todd, a UK-based writer and sex worker, told me. “Many women do: it really isn’t only horrid men forcing us to do these dreadful depraved acts. Interrupting oxygen supply produces the most intense sensation of euphoria. I enjoy it in my private life, and I offer it in sessions too.”
Starting Friday, she and anyone else in the UK distributing or possessing certain types of choking, suffocation, breath play, strangulation and a bunch of other types of porn — even if it’s just recorded in the privacy of their homes and not for sale on a porn site — could face up to five years in prison.
This is, predictably, already a huge mess for sex workers. Adult creators watched platforms remove hundreds of their videos — even when the creators thought they were in compliance with the law. These creators say they were blindsided by the removals, since the content didn’t seem to violate the law at all. Creators are reporting one platform in particular, Clips4Sale, is purging content with little notice, “a precautionary measure,” with videos disappearing in the hundreds.
“Of course, we were all aware of the new Act coming in and prepared accordingly, but not for anything at this scale. We've had nothing since and they've been tweeting as normal,” adult creator Sophia Truee said. Clips4Sale launched in 2003 and caters heavily to fetish content. “It's a huge blow to this area of the industry, and I can only hope things will be communicated soon. We, as creators, understand how difficult it is to be a content site in this current climate, but communication has not been prioritized,” Truee said, noting that this is happening right before the holidays, with Black Friday being one of the most popular times for creators selling content. “For many, this will be devastating, as over 20 years of the work will have just been wiped.”
Important update for our RSS fans: If you’re a paid subscriber using our full text RSS feed and don’t want these newsletters included in your feed, you can now replace the feed’s URL, which usually looks like 404.feed.press/private?key=XXXX, with 404.feed.press/no-newsletters?key=XXXX instead. Questions? Reply here or email support@404media.co and we’ll try to help ya out.
Nevermind! Following widespread backlash, Meta announced last week it’s putting a planned update of its commercial terms of use on pause. The terms would have required business accounts to allow Meta to “use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of” Meta’s products. The new terms were initially set to go into effect on October 30, but after outcry from journalists and users who didn’t want Meta reaching far beyond Instagram and Facebook and into their websites to suck up their work as training data for its generative AI products, Meta announced it was pushing the change to November 20. Now, it’s on hold entirely.
Whoa, dude. Archaeologists discovered the oldest direct evidence of psychoactive cannabis use in a Siberian tomb that dates back 2,800 years and was the site of human and horse sacrifices. During excavations of the tomb, a team found more than 1,400 cannabis seeds scattered around the remains of several Iron Age individuals, and even identified cannabinoids—including the active compound tetrahydrocannabinol (THC)—in a person’s preserved hair. Though humans cultivated cannabis for many millennia before the Iron Age, this is by far the earliest clear-cut proof that the plant was consumed for its psychoactive effects, which have made it one of the most widely used drugs in the world today.


For months I’ve been speaking to an AI fraud kingpin. You might remember in 2024 we revealed the existence of OnlyFake, an underground site that let anyone generate realistic photos of fake IDs using ‘neural networks’. The creator of that site went by the handle John Wick. A while later, the creator was arrested.
Turns out there was a lot more to that story. Yuriy Nazarenko, the creator of OnlyFake, told me his side of it. How when I published my initial article he panicked, and destroyed his laptop and emails and got a new phone, paranoid that authorities were about to bust down his door. How the pressure of a growing fraud empire pushed him to drugs. And how North Korean fake IT workers used his site. Read below for his story.

A few years ago, Yuriy Nazarenko sat at a desk in his grandfather’s apartment in Ukraine and, on a diet of cigarettes, Lay’s chips, and beer, painstakingly made digital copies of fake identity documents. One of his four computer screens had the messaging app Telegram open. Another displayed a scan of a passport in Photoshop filling the entire monitor.
Pixel by pixel, with rap music blaring, Nazarenko edited each part of the passport. That included the MRZ codes, the block of text printed at the bottom of identity documents; the photographs to superimpose someone else’s face into the document; and the overall dimensions and quirks of each different type of passport. Nazarenko would sometimes sit for more than 12 hours at his computer, obsessively tinkering away, with a single, high-quality fake identity document taking up to a week of work.
When someone thinks of fake IDs, maybe they imagine a spotty underage teen buying a physical counterfeit drivers’ license so they can score some beer. But in the global criminal underworld, digital copies or scans of IDs are like a skeleton key. A convincing scan of an ID or utility bill, loaded with a bogus name or someone else’s identity, can unlock the ability to make bank accounts, bypass checks on cryptocurrency exchanges to buy or sell more Bitcoin, or rent a safehouse, all while keeping the criminal’s real identity hidden. The digitally altered IDs need to fool both human reviewers and automated systems that scan them for inconsistencies. Making fake IDs requires exquisite attention to detail.
All of that manual labor doesn’t really scale, Nazarenko thought to himself as he meticulously edited passport after passport. He went on to make templates of fake identity documents that sped up the process, but it was still slow.
After many of these quiet nights, he had an idea. He built a website to automate the process that autopopulated a fake ID with the personal data a customer wanted, rather than having him do it by hand. Then, when generative AI became mainstream a short while later, he incorporated that too.
“That gradually became OnlyFake,” Nazarenko told me.
OnlyFake, the underground website Nazarenko made, landed like a bomb in the fraud world. It allowed anyone to generate not only convincing scans of drivers’ licenses or passports, but also realistic looking photos of those fake identity documents laying on a sofa, carpet, or desk, as if someone had taken the ID out of their wallet and snapped a picture of it on their phone. Needed a convincing looking photo of a California drivers’ license to open a cryptocurrency account in someone else’s name? Or a British passport to incorporate a company overseas? Or a Social Security card? No problem. The results were photos of identity documents that didn’t really exist, of people who may not be real, but yet could trick the sorts of people and companies that verify documents.
With OnlyFake, you didn’t have to be a master forger like Nazarenko to make fake documents. Now, anyone could churn out fake IDs at scale, sometimes even hundreds or thousands at a time. OnlyFake threatened the entire idea of know-your-customer, or KYC, where financial institutions take steps to know who exactly are using its services. When anyone can make a photo of a fake ID for just a few dollars in Bitcoin, is anyone really who they say they are?
Nazarenko didn’t know it then, hunched over his desk, but OnlyFake would soon become so influential that it would spawn copy cats, and the FBI would eventually arrest Nazarenko and shut down some of its competitors. The Department of Justice said in an announcement about Nazarenko’s guilty plea that OnlyFake made more than a million dollars selling fake IDs for just $15 a pop.
Now for the first time, Nazarenko, 28, free after a year in prison and living in Asia, is telling his story: how North Korean fake IT workers used his creation, how he ended up in ICE detention for months after serving his sentence in the U.S., and how OnlyFake unleashed a new age of AI-powered fraud. Nazarenko said he wanted to talk to me because I was the first journalist to reveal that OnlyFake existed. This article is based on interviews with Nazarenko and his lawyer; court records; and my own reporting in the online world of fraud that OnlyFake inhabited.
“OnlyFake was the first time I saw how quickly a small internet project could turn into something global. Unfortunately, it also taught me, in a very painful way, what can happen when the scale of something grows faster than your understanding of its consequences,” Nazarenko told me.
Growing up, Nazarenko’s parents questioned whether Nazarenko would ever find a job. His father worked for the Security Service of Ukraine. Raised in a suburb outside of Odesa, Nazarenko jumped between a few different gigs. He worked as a cook, a waiter, at a supermarket, and as a cameraman at a local Ukrainian TV station.
Read the rest of this story here.
We continued our election coverage with a piece from Matthew Gault on the data center angle. There were actually a ton of proposals related to data centers on the ballot in more than 40 races across 13 states. Well, people voted against data centers in at least 38 of them. People are sick of this stuff and they’re showing up to vote about it.




It’s rare to see an executive go from fucking around to finding out so quickly. That’s what happened last week when Proton CEO Andy Yen announced he was canceling a planned re-interview process for almost all current employees that would have checked if they’re using AI enough. This follows extreme backlash from Proton users and workers against the company's aggressive pivot to AI. It’s been a wild ride internally, several employees past and present have shared with 404 Media.

On Friday, Andy Yen, the CEO of Proton, cancelled a series of nearly company-wide ‘re-interviews’ that would have forced employees to explain how they are using AI, with the clear intention that they may be fired if not using it enough in the eyes of Proton leadership, according to six current Proton employees and audio of an around two hour all-hands meeting obtained by 404 Media.
The move came after 404 Media published an investigation on Thursday, which showed how Yen was aggressively pushing AI across Proton, in some cases despite security concerns from employees. In one exchange, an employee in the company’s security Slack channel urged Proton to at least take a few days to pentest newly written code before pushing it live. “That’s bullshit and loser thinking. Of course it’s possible to be fast and secure,” Yen wrote in the channel at the end of August. “Stop thinking always of how to be slower, and focus more on how to be faster.” Yen added, “stop lovingly handcrafting all our code and use AI more.”
The re-interview cancellations likely don’t address the concerns of many Proton users, who are more focused on Proton’s deprioritization of security; many Proton users have blasted the company’s leadership on social media over the last 24 hours and explicitly called for Yen to resign.
Kinktober has come and gone. But Archive of Our Own (AO3), a massive repository for fanfiction and creative writing, is still down, and might be down for as long as a month after going down for "maintenance" following a security incident last week. The Organization for Transformative Works (OTW), the non-profit organization that runs AO3, posted on its status page that the site restoration will take around two weeks, but could take up to a month.
People are coping how they can. And they’re turning to anything but Wattpad to fill the void. One writer I spoke to who posts her writing and reads others’ writing on AO3 told me that she was in the middle of racing to finish a work that was deliberately timed to Kinktober and Halloween when the outage happened. She was hitting “next chapter” while reading a fic on Wednesday night when she got the message that the site was down for maintenance, “which I initially took as a sign from the universe to go to bed,” she said.


You've probably heard, at this point, about the big changes coming to Meta's AI scraping terms for business accounts. Nieman Lab had that scoop, and since then, small businesses, journalists, and artists using Meta products like Instagram have been raising the alarm. But if you're in the EU or UK, you might be in luck and can object to that use of your data, as some users have pointed out.

Earlier this month, Meta updated its commercial terms of use for business accounts to allow it to “use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of” Meta’s products.
Many artists and small business owners that rely on Instagram and Facebook are livid at Meta granting itself permission to use their content, including from websites simply connected to their Meta accounts, to train AI. But for users in the United Kingdom and European Union, for now, there’s a way to object.
Meta buried this change in its Commercial Terms, and Nieman Lab first reported on the change. The update was originally set to go into effect on October 30, but following backlash, Meta changed the effective date to November 20. Adding to the confusion, Meta then removed the word “train” from a list of allowed use cases, Nieman Lab reported, and limited those uses to “advertising and business products,” where previously it said that data would be used for all Meta products. Those terms still allow Meta plenty of leeway for scraping users’ data to develop its products, however.
Several users based in the UK and EU have explained how to object on Instagram. On mobile: by clicking the three lines on their account, going to the privacy center, and clicking “object,” which goes to a form that asks users to write why they are objecting to their data being used. They received an email from Meta saying their objections were approved.
On the podcast this week, we talk Flock CEO’s leaked police conference audio and Sam’s piece about Marissa Mayer’s latest meltdown. Subscribe wherever you get your pods.
The New York Times has an antisocial aversion to linking: Another day, another the-New-York-Times-isn’t-linking controversy. The specifics here almost don’t matter because this happens seemingly every day, but basically the New York Times investigation about chemical plant polluters under the Trump administration was covered in much the same way by Hunterbrook, a news outlet fused with a hedge fund. The Times article—which led the Sunday print edition—uses similar photos, some of the same sources, and covers similar ground to Hunterbrook’s earlier investigation from February, with no acknowledgment or link whatsoever. After the cofounder of Hunterbrook rightfully ranted about this on X, the Times added an acknowledgment (but no link!!!) toward the bottom of the story. Puck’s Dylan Byers has a good rundown about what happened, which includes parts of a three-paragraph Times’ spokesperson statement explaining at length why they don’t think they have to link. This is extremely antisocial, deranged behavior and it’s ridiculous that the Times is still getting away with this. Bottom line: I’m sure the Times was already working on this story, what with their massive newsroom. Even so, they got beat, and they should link. Journalism awards need to stop looking the other way when big outlets do this. This has happened to us many times and it actually does matter for smaller outlets and links are free. Let’s try them!
Artificial reef: TIL that a gigantic military ship, the SS United, is going to be purposefully sunk to be turned into an “artificial reef.” Scuba divers seem quite excited. I didn’t really know that this was a thing that was done, but maybe I’m a rube.
Ransomware scam mitigation scammer: A company that promised to help victims of ransomware (for a fee) was indicted by the feds, according to ProPublica and a Department of Justice affidavit. The original investigation that led to this was published by ProPublica back in 2019 but is still worth a read; there are so many of these companies that prey on victims of cyberattacks, sextortion campaigns, etc.
Today in c’mon, man: CIA whistleblower John Kirakou is doing sponcon for Kalshi, writing “I spent 15 years at the CIA trying to figure out what was going to happen next in the world. Now Kalshi is finding new ways to put a price on those predictions. This partnership between Kalshi and S&P Dow Jones Indices is fascinating.” C’mon, man.
Today in printers: I am absolutely mesmerized by this microphotography of an inkjet printing, which won the 2026 Nikon Small World photomicography competition. Ole Bielfeldt’s work is on his Instagram, which is absolutely worth checking out. Wow, wow wow.
Today in yikes: A Roblox scraper has been grabbing more than a hundred gigabytes of data from a 3D modeling community, including more than 60 GB of adult content from a section of the site called “SmutBase.” “We don't think Roblox set out to collect adult content, and we have no evidence that any of it reached a model or a product. The scrapes were likely automated. But that is exactly the problem: as far as our logs show, nobody checked. A crawler working for a company whose platform tens of millions of children use was pointed at an adult site, and it took explicit models without telling them apart from anything else,” Open3DLabs wrote in a blog post about the scraping. “For a company being sued by several US states over child safety, ‘our AI team bulk-downloaded an adult site without looking’ is not a good position to be in.”




GM! We have a few bangers for you today.
First: Can you imagine sitting in jail for days without knowing why, and then finding out that it's because someone hacked CashApp to impersonate you? I would be kind of mad! And elsewhere on 404 this morning, we have some prescient thoughts from an autonomous weapons expert on the state of AI in war, what's going on with cybersecurity experts in the wake of the PHPocalypse, and more insights into how cops view anti-Flock activists.
Let's get into it.

Amanda Smith answered the door one afternoon in April and came face to face with the local Sheriff’s department in upstate Pennsylvania. The officers said they had a warrant for her arrest for a crime she had allegedly committed more than a thousand miles away in Arkansas.
“I was completely confused,” Smith recalled. For one, she had never been to Arkansas. Second, the warrant was for computer fraud. By her own admission, Smith was not a very tech savvy person.
She had a feeling she was being set up. After discussing with the officers the practicalities of what was about to happen — she was allowed to get her contact lenses — they took her away in handcuffs to the local jail where she spent three nights.
The reason for Smith’s arrest, she learned later, was that a scammer had tricked someone on Facebook Marketplace into sending $1,000 to a CashApp account in her name that the scammer controlled. The scammer created the account using her Social Security number and date of birth. Smith said she has never used CashApp.
What developed over the next few days, and has now stretched across six months, is a Kafkaesque situation in which Smith, in her telling, is being asked to prove her innocence in an ongoing criminal case, while CashApp refuses to provide her any information about the account because she couldn’t prove that she was the owner.
Read the rest of this story here.
“Suspicious behavior:” A federal intelligence center warned police that city council meetings about Flock “may be viewed by extremists as a desirable target,” and told them to be on the lookout for “emotional confrontations” between people who oppose surveillance and government officials, according to a copy of the warning obtained by 404 Media.
“Black October?” For much of the last month, many cybersecurity experts have been warning of an AI-powered “Black Swan” event, saying they’re alarmed by the number of vulnerabilities suddenly discovered, and of the apparent power of AI agents to find and exploit previously unknown bugs. So, what’s really going on?
The UN met this week to discuss autonomous weapons. “The technology is developing at a pace that is very concerning,” Laurent Gisel, the head of the International Committee of the Red Cross’s (ICRC) legal warfare team, told 404 Media. Gisel was in NYC this week to address the UN about all the things that are driving conflict across the planet. His prepared remarks include a statement on the illegal arms trade, nuclear weapons, mines, and, of course, AI.
That’s a W for the Rebel Alliance. In San Diego, where Darth Vader famously spoke in favor of Flock cameras and mass surveillance, the city council just voted to end its contract with Flock. The decision still has to be confirmed, NBC 7 reports, but this is huge news for a city where council members have voted multiple times to keep Flock watching its citizens.








There is a lot of stuff about how creepy ad targeting is in today’s newsletter. Our first story is one I did, about how advertisers — or really anyone with 5 minutes on their hands — can inject ads into your ChatGPT session based on your phone number or email address. This is pretty normal in the advertising world, but I seriously doubt that when most of ChatGPT’s many, many users see an ad, they realize that might be because someone, somewhere, uploaded their phone number into OpenAI’s systems. I made my own ad on ChatGPT (without using peoples’ personal data) to test out the system. Then, in other news, there is new research confirming, I think for the first time, that online ads are being used to deliver zero-click mercenary malware against targets.

