A Florida police department claims that scammers used AI to clone a sheriff’s voice, with scammers increasingly using voice clones that trick people into thinking they’ve missed jury duty or have a warrant out for their arrest are increasing.
“They cloned his voice and was using Steve’s picture as well to say it was him making the phone call, and it was not,” Sheriff Chris Blackmon told News6.
💡
Have you been the target of deepfake video or AI voice cloning scams? I would love to hear from you. Using a non-work device, you can message me securely on Signal at sam.404. Otherwise, send me an email at sam@404media.co.
Impersonating a police officer is a third-degree felony in Florida. The Osceola County sheriff’s office did not immediately respond to a request for comment.
This particular scam cloning cops is spreading across several states, with reports of jury duty and arrest warrant scams targeting people in Georgia and Pennsylvania in the last week. These other reports don’t mention AI voice cloning explicitly, but note that the scammers sometimes have similar voices and tones as local police officers.
Voice cloning scams have been around for years, with scammers using commercially available AI voice impersonation tools to target the elderly, families, and businesses. Last year, 404 Media investigated real-time deepfake scams that incorporated voice and video to defraud people. In 2024, we reported on the ways scammers cloned celebrity voices and use AI videos in YouTube ads to scam people.
Scammers often impersonate company executives or leadership to trick employees into sending money, and voice cloning appears to be an increasing part of that. In February, scammers cloned a managing partner of a law firm with AI. Ultimately, a private bank ended up transferring $108 million.
This is Behind the Blog, where we share our behind-the-scenes thoughts about how a few of our top stories of the week came together. This week, we discuss a new pod setup, video games, and Elon's spending habits.
JASON: Exciting news: I am building a podcast studio / riso printing room / place to film videos in. I believe all of my colleagues and many people in my life have been laughing at me about this, but this also may just be the narrative I’ve created for myself to inspire me to prove the haters wrong. Since the beginning of 404 Media, I’ve worked out of my garage, but feel like I’ve been going a bit crazy sitting mostly in the same room since essentially the start of the pandemic. And so I found not a coworking space but a very affordable windowless box that I shall turn into a content palace.
I’m just going to try it for a few months to see if it makes sense or if I should return to working from the couch, but I’m quite excited about building it out. I only got it a few days ago and so at the moment it’s largely a bunch of boxes and random furniture I no longer had space at my apartment for. It’s got a drop ceiling with fluorescent lighting and landlord special laminate floors, but I also can do whatever I want to it, so I will likely do some sort of design on the walls, throw a rug down, whatever.
Police arrested a Chicago area high school maintenance worker who attempted to meet a child for sex after investigators posed as a 15-year-old boy using AI generated images.
Christopher Thomas Mertens started talking to who he thought was a child through a dating app in September. According to an affidavit written by the department’s FBI Task Force Officer McKailey Scott, the investigation was done by the Livonia Undercover Resistance to Exploitation Unit (LURE), which identifies and prosecutes child sex offenders. Scott set up an account on the queer-focused dating app Taimi with “an AI generated image of a teenage-appearing male with the name ‘Caleb,’ and no age listed on the profile.
Scott wrote that an unknown user messaged “Caleb,” and they moved to Whatsapp. After a few minutes of talking, the man asked Caleb for their age, and when Scott — posing as Caleb — said he was 15, the man said “Im interested” and “Im hard now.”
The investigator used a combination of tools, including checking the man’s phone number using Thomson Reuters CLEAR and running images of his face through Clearview AI to identify him as Mertens. For days, Caleb and Mertens exchanged sexually explicit text messages and Mertens made plans to drive to meet Caleb on his day off, believing Caleb’s (non-existent) parents weren’t home. Cops used Flock to look up and track Mertens’ car and see that he was driving to the meetup spot, but Mertens was also texting Caleb the entire time to let him know he was on the way and when he arrived. When he got out of the car, Livonia police took him into custody.
Two hours after being booked, Mertens told jail attendants that he wanted to speak to Scott. When Scott got to the jail, Mertens admitted everything, including that he works at a high school in Illinois. The complaint doesn’t mention which high school.
Investigators have been generating images of children for years. One of the earliest examples of these is Sweetie, a computer generated child developed by Dutch children’s rights organization Terre des Hommes in 2013 that the organization used to talk to men online. Today, police departments at the U.S. border use AI-generated online personas designed to interact with and monitor protestors and activists, according to documents obtained by 404 Media.
In August, a 24-year-old Connecticut man was arrested after soliciting an undercover police officer who used what was reported as “age regression technology” to appear as a 14 year old in videos. He was charged with enticing a minor and criminal attempt to commit risk of injury to a child.
In 2025, a 45-year-old man was arrested after investigators used AI generated images of children to catch him distributing child sexual abuse material. He was recently sentenced to 12 years in federal prison.
And in 2024, the state of New Mexico sued Snapchat after investigators set up a fake account, including AI generated images of children, that the platform allegedly recommended to “dangerous accounts, including ones named ‘child.rape’ and ‘pedo_lover10,’ in addition to others that are even more explicit,” according to the New Mexico Department of Justice.
AI generated child sexual abuse material is a rapidly growing problem for law enforcement. The National Center for Missing and Exploited Children has reported a massive rise in generative AI child abuse material, with criminals using it to extort and entice children, and children using it to bully and harrass other children.
Livonia police did not respond to a request for comment.
Sorry for the late newsletter today, we’re still a little sleepy. To ease you into this afternoon, I have an interview with the guy who coined the term “smutstep,” and has become a sensation among a very specific section of dark romance BookTok — as well as a lot of haters. I thought he had some interesting things to say about the content creation game and not taking yourself too seriously while respecting your own craft and audience. Enjoy.
Last week, a video came across my feed of a man saying horny lines over a dubstep beat. “On your knees,” he snarled. As the beat dropped, he growled, “Mmm, now look at me.”
“Instagram reels just showed me a guy who makes 'smutstep' for the booktok crowd and it is the funniest thing I have ever watched,” the person reposting the video on X wrote. The replies are predictably harsh: “I think this is actually the worst thing I have ever heard.” “Who the FUCK is demanding this 😭”
People demanding this is why it exists. DARQ DAGR, whose real name is James, is a dubstep musician and voice artist who’s been working in the voice acting world for years. But smutstep is what’s made his work go viral, in large part thanks to the people who hate it so much. Smutstep emerged as a reaction to BookTok, a subculture within TikTok built by book lovers and their various followings. Within BookTok, the romantasy book genre thrives, with series like The Fourth Wing and A Court of Thorns and Roses (ACOTAR) gaining massive readerships for their blend of elven high fantasy and dragon-based erotica. The romantasy genre is not a quaint fandom. It’s a billion-dollar industry.
Like so many online communities devoted to things a little horny and misunderstood, the wider world tends to scrutinize and discredit romance readers, especially when Fae fucking is involved. James, who wants people to know he does read the books too, saw a way into two niches that could compliment each other.
“I think what I like about the whole culture — the BookTok culture and dubstep culture — it's just letting your freak flag fly in a way,” he told me in a call. “It’s accepting and embracing what you're into within a safe and healthy space. I think that's a really beautiful thing.”
Sentry mode? The “security protocols” on a Tesla stopped a Homeland Security Investigations agent from placing a tracking device on it, according to a warrant reviewed by 404 Media. On Halloween of 2025, an HSI Special Agent filed a more than 60 page application to place a tracking device on the Tesla of a suspected drug trafficker in Milwaukee. Agents had been watching alleged drug dealer Sam Stair — a local landlord they believed was using his rental properties as the front for a drug trafficking operation — for months when they asked a judge to let them install a tracker on his car. According to the warrant application, Stair went everywhere in a white Tesla and the agents wanted to see where it was when they couldn’t keep eyes on it.
FROM AROUND THE WEB
Ghosting much... SpaceX will testify before the New York City Council this week about AI safety risks. This is after Elon and co ditched a hearing on Oct. 5.
Last week, a video came across my feed of a man saying horny lines over a dubstep beat. “On your knees,” he snarled. As the beat dropped, he growled, “Mmm, now look at me.”
“Instagram reels just showed me a guy who makes 'smutstep' for the booktok crowd and it is the funniest thing I have ever watched,” the person reposting the video on X wrote. The replies are predictably harsh: “I think this is actually the worst thing I have ever heard.” “Who the FUCK is demanding this 😭”
People demanding this is why it exists. DARQ DAGR, whose real name is James, is a dubstep musician and voice artist who’s been working in the music industry and voice acting world for years. But smutstep is what’s made his work go viral, in part thanks to the people who hate it so much – and the ones who demand more of it.
Smutstep emerged as a reaction to BookTok, a subculture within TikTok built by book lovers and their various followings. Within BookTok, the romantasy book genre thrives, with series like The Fourth Wing and A Court of Thorns and Roses (ACOTAR) gaining massive readerships for their blend of elven high fantasy and dragon-based erotica. The romantasy genre is not a quaint fandom. It’s a billion-dollar industry.
Like so many online communities devoted to things a little horny and misunderstood, the wider world tends to scrutinize and discredit romance readers, especially when Fae fucking is involved. James, who wants people to know he does read the books too, saw a way into two niches that could compliment each other.
“I think what I like about the whole culture — the BookTok culture and dubstep culture — it's just letting your freak flag fly in a way,” he told me in a call. “It’s accepting and embracing what you're into within a safe and healthy space. I think that's a really beautiful thing.”
James and I talked about modern artistry in a world where interactivity and engagement rule, why he doesn’t take all of this too seriously, and the value of escapism in a time when everything else sucks. And yes, I ask him to do the growl.
This interview has been edited for length and clarity.
SAM: I will admit just off the bat, I'm not in the BookTok world personally.
DARQ DAGR: Uh-huh.
I have nothing against BookTok. Beautiful community. But I can't get into it. I've tried like, ACOTAR. I haven't found my romantasy, I suppose. So I might ask you some dumb questions, but hopefully you can also serve as a little bit of an explainer for those of us who are not super in that world. Tell me how you got into this?
Yeah. So the thing is, it wasn't like when I was a kid, I was like, when I grow up I wanna growl. You know? I've been a trained professional vocalist for a really long time. I've done metal and I've done theatre, I've done pop, I've done rock. The reason I'm in Nashville right now is because I spent eight years in a country band that was touring the world. I just kinda do everything there is to do with my voice. I've been a professional performer and vocalist for a really long time. My wife is really into the BookTok world. And so I was exposed to the community and the books through her.
The first video that went viral for me was when I had just left my country band. And all I wanted was for the way I was presenting myself and expressing myself to be true to the things that I was into, which is dubstep. I had fallen in love with dubstep. I think it's the most beautiful community in the world. And then, I've always been a huge fantasy fan. I've always been really into Lord of the Rings and The Witcher and all that stuff. So getting into romantasy was not a big step for me, because I'm already a fan of fantasy. Making that video was me kind of saying, ‘I feel like a lot of these BookTok boys are getting the growl wrong.’ And then I did a growl that I did from back in my metal days. And it just blew up and it created this whole community of people who were really into the growl. But I was not the first person to do the growl.
There's another creator who was doing it long before me. His name's Michael Ayok. I can't take credit for being the first to BookTok growl, but I definitely get a lot of recognition for it.
Obviously I wanted to be producing dubstep music, but my community all wanted me to growl, so it was like, how do I make this make sense? And so I just started growling in my dubstep and inadvertently created a subgenre of dubstep called smutstep. It's all community driven though. Honestly, even the name smutstep was a comment on one of my videos.
Somebody had requested that I growl ‘sit down’ and I did a video growling it that did really well. So I was like, well, let's just make a dubstep drop out of this. It's community driven, and it's fun. I'm glad that people are liking it.
I think the biggest issue though is that... I'm having fun. I do not take this seriously. Again, it wasn't a dream of mine to grow up and growl at people. I took two very niche communities — the dark romance BookTok niche community, and the dubstep community — that are not designed to be exposed to general culture. So there's a lot of hate that comes with that and a lot of negativity. People react very strongly to what I do. And I think that that's as an artist, I think that's effective art. You want people to be moved to an extreme. You know? But you cannot control what that extreme is gonna be.
There are people that treat me like Marilyn Manson in the nineties. His art is meant to be taken seriously. He is making a social commentary where he's putting a mirror up to society and showing how ugly it is. Whether you love that or you hate it, it's meant to be taken seriously. Smutstep is not meant to be taken seriously. If you enjoy it, that's awesome. You know what I mean? I'm having fun with it. I love the community I've created around it. We're all having fun. But it's not meant to be taken as seriously as so many of the haters take it.
Yeah, it's broken containment from these specific niche communities, into the wider world. Which is the curse and blessing of social media, I guess. Something breaks out and becomes viral and then everybody's aware of it, even people who maybe are not the audience.
I sent you the link to the video that's got a million views now on X, and all the comments and replies are really rude. So just be warned. People are like, ‘I'm never gonna unmute this. I would never unmute this to save my life,’ that kind of thing. And like you said, it’s not that serious. If it's not your thing, that's okay. Move on.
But what do you think about those comments? The ones that are like, ‘this is gigacringe.’
That's been a lot of what I deal with on other platforms. Facebook and TikTok are way better. On Instagram, l have my comments limited to people who follow me because it's just so much hate.
People are crazy on Instagram. People will say the most sideways shit on Instagram.
Yeah, and that's that's the thing. Like obviously, as much as we're all just having fun, of course it does mentally take a big toll on you.
You said this is community driven. Were people waiting to have this sort of thing enter their already-existing BookTok community, or where do you think it comes from? Why are people resonating with this?
Men in BookTok is not a new thing. Making it dubstep is my brand. But for me, smutstep is like the KISS makeup. It's a gimmick. Everybody needs something that makes them stand out. So right now, smutstep is what makes me different, it's what makes me unique.
Most of my content is based off of people making requests in the comments. I think that may be why it's taken off the way it has. Obviously it has to do with me, but it has so much to do with the people who are enjoying it. This is something that I think in a lot of ways is actually a much more modern take on entertainment. From its inception, it’s community demand, and giving people what they want within what I wanna do. I'm not doing something I don't wanna do. It's like, all right, you guys want this? Well, I'm gonna make it, but I'm gonna go the dark romance route and make it super aggressive, because I want to make dubstep out of it, and people are like, ‘That's fucking dope.’
Are you familiar with like the romantasy theme park videos?
No. No.
Oh my god. You would get a kick out of this, and probably your wife would also, but it's part of SeaWorld. You know how around Halloween, theme parks will do like horror nights, and characters come out and chase you? It's that, but it's BookTok boys.
I'll send you a link. But it has gotten a really big reaction from wider culture, where people are like, ‘this is so cringe and weird.’ The execution is really cringe and weird, but it is responding to something people wanted, which was: romantasy in a more interactive way. Is there anything that people are demanding from you more than other stuff?
There's a book series called The Fourth Wing and there's a dragon in it called Tairn. Somebody said that Tairn should be Scottish, because his name is Tairneanach, which is a Scottish name. So I did a growl, but in the Scottish accent.
I do another series with masked growls. I'm supposed to do one today, if the mask gets here in time. It’s a Scream mask 'cause we're in spooky season, and people have been wanting that for as long as I've been doing this.
I think what I like about the whole culture — the BookTok culture and dubstep culture — it's just letting your freak flag fly in a way. It’s accepting and embracing what you're into within a safe and healthy space. I think that's a really beautiful thing. Maybe it's just as I get older, but I'm just so disillusioned with reality and I constantly want more escapism for myself, and want to give others more escapism as well. So that's kind of my goal as an artist is just escapism.
That’s a huge appeal of fantasy and romance in general. I'm also a big Lord of the Rings fan. That type of fantasy I can get with. I wonder if Lord of the Rings launched today, if we would be calling it romantasy. I guess there's not enough sex in it though.
The thing about romantasy is, there does need to be pretty pretty graphic scenes in it. But Lord of the Rings has none of that. I do read the romantic stuff too. I don't wanna be somebody who just does it for the views. I do read both.
Do you think people have picked up on the smutstep genre since you started? Are there new people doing it?
I haven't noticed, but that doesn't mean someone's not out there trying and just hasn't had a video hit yet. And to be honest, I'm not surprised. I think that it's one of those things... It's human nature to have an aversion to something new that they don't understand. It’s still in the phase where I think it's more ‘what the fuck’ than something that is socially acceptable. Which again, it was never meant to be broadly socially acceptable. It was meant to be niche-acceptable, which it already is. I don't think there's a lot of people trying to do it because I think people are too afraid of having to deal with the same stuff I deal with.
I would be remiss in letting you go if I didn't make you do a growl. Would you be willing to offer us a growl?
I can. It's not going to be that loud, because I don't have a mic, but I can do one. Ready?
[Growls]
Did you hear it? Did it pick it up?
It was like a growl in the distance. I'll practice mine. I'll do it on the podcast maybe and you can critique my growl.
Good Monday morning! Today's big story is an investigation from Joseph about cops and an AI tool that takes data from sites like Airbnb and Zillow and identifies specific properties. The internet is full of photos of the interiors of people's homes, from real estate and rental listings to short-term stays, and that data is in the hands of law enforcement.
And in other news, a very interesting interview with the whistleblower subject of Sorkin's new movie, some big legislative impact, and what caught my eye over the weekend around the web. Let's get into it.
A screenshot from a Raven promotional video on LinkedIn.
An AI company now lets cops take a photo and nearly instantly figure out what specific home it was taken in, using only clues in the photo itself, and is clearly pulling data from sites like Zillow and Airbnb.
The tool, called Raven, shows the increasing power of AI capabilities available to local law enforcement agencies. 404 Media previously reported that both the Los Angeles Police Department and Miami-Dade Sheriff’s Office (MDSO) have bought access to Raven, which was previously called GeoSpy. Now, Raven, made by a company called Graylark Technologies, has added features that let cops analyze the interior of homes.
In a post on LinkedIn earlier this month, Graylark CEO Daniel Heinen wrote the company was introducing “the ability to geolocate interior images across millions of short-term rental units worldwide.” An accompanying demo video shows someone taking a photo from X of a washer/dryer unit inside a building. They upload it to Raven, select “Find Property: Interior and exterior property geolocation,” and then choose a region to search.
After a couple of seconds, the map interface zooms in on an area in Charlotte, North Carolina. On the right hand side, the screen displays a list of results. The top one reads, “Renovated! Walk Uptown/located in 2 Hot Spots!” That result directly quotes the language on a specific Airbnb listing in Charlotte, and the photos in the Airbnb listing show the same property.
Spoilers ahead for “The Social Reckoning.” Did you catch this week’s interview episode? Jason talked to former Facebook employee Frances Haugen, who became one of the most important tech whistleblowers in 2021 when she exposed how Facebook’s algorithm was pushing people toward hate groups, showed Instagram was making teen girls feel worse about themselves, and showed that the company knew about its detrimental impacts on society. Haugen’s story is the basis for Aaron Sorkin’s new movie “The Social Reckoning,” which follows her evolution from model employee to leaker to formal whistleblower. Jason and Frances discuss what the movie got right about journalist-source relationships and how AI is supercharging privacy concerns.
This is Behind the Blog, where we share our behind-the-scenes thoughts about how a few of our top stories of the week came together. This week, we discuss AI and spirituality.
JASON: I think we’ve generally carved out a pretty good lane for ourselves in the broader AI debate, writing about its current capabilities, its current harms, and the fact it was trained on stolen content without saying that it will never be able to do anything of use. We are often tinkering with AI so that we can write about it from an informed perspective, and so every once in a while, while I’m working on a story I will see something where it feels like AI could possibly be useful for some specific part of the reporting.
Howdy, it's hump day and we're rolling along. Today we've got an update on a very strange case involving a man whose image was resurrected from the dead via AI and made to appear in front of a judge to announced "forgiveness" of his murderer. 404 Media contributor Matthew Gault had a long conversation with the man's sister, who wrote the words and generated the video, and she said that swaying the judge was the entire point.
A man convicted of manslaughter in Arizona will be resentenced because an AI-generated video of his victim speaking from beyond the grave was ruled to have carried “undue emotional weight” as an impact statement. An appellate court in Arizona ruled that the manslaughter charge will remain, but the judge must reconsider the length of the man’s prison term because of AI.
In 2021, Gabriel Horcasitas shot and killed Christopher Pelkey during a road rage incident. A jury found him guilty of manslaughter. During Horcasitas’ sentencing hearing, Pelkey’s sister Stacey Wales played an AI-generated video of her brother as part of her victim’s impact statement. In the video, which she scripted with her own words that she made an AI generated likeness of Pelkey say, the AI avatar of Pelkey forgives his killer and said they “could have been friends” in another life.
“I loved that video,” the judge said while handing down the maximum sentence of 10 and a half years.
Wales told 404 Media that her intent with the video was to sway the judge. “Isn’t that what you want human connection to be?” she said. “Do you ever come out of a movie and say: ‘Well, that was too emotional. That was too powerful. Human connection is powerful, and trying to convey that to somebody that's never known another person, this person on the earth, to convey their lifetime of 37 years down to a two-hour sentencing event is hard to do, and I think that victims and victims' families should be afforded every opportunity.”
A flock of new laws incoming. Several recently introduced pieces of legislation would, if passed, dramatically curb federal agencies’ access to Flock cameras, with the lawmakers referencing reporting by 404 Media in their explanations for why the legislation is needed. “At a time of growing concern about the unchecked power of artificial intelligence, Flock is eviscerating the very notion of privacy by installing tens of thousands of cameras in communities across America without their consent,” Bernie Sanders said.
This is Behind the Blog, where we share our behind-the-scenes thoughts about how a few of our top stories of the week came together. This week, we discuss inbox slop, Claude cults, and more.
JOSEPH: This is something I’ve been thinking about for a while. Jason and Emanuel covered iLands, the AI agents that email people offering to do work. My thing isn’t exactly the same, but similar.
For months, I’ve been noticing that many people email me ‘tips’ (I put in quotation marks because they’re sometimes not tips, but more, my Instagram account was banned, please help me) that are CLEARLY written by AI. I can tell this because the AI responses seem to keep using the same or similar words that it believes would apply to journalists, or are how journalists speak, but as a real, human journalist for more than ten years, I know that journalists don’t actually speak like that.
Student journalists meet the moment, as we’ve seen most recently this week. As the Cornell University sexual abuse case highlights the importance of student journalism, public records reveal how schools try to censor on-campus news outlets. Claire Woodcock talked to students and legal experts about how censorship from above impacts their work. “One of the biggest things we’ve certainly found with censors over the years is they want to be called everything but a censor,” one expert said. “What they’re doing is everything but censorship.”
Student journalists, their advisors, and legal advocates say colleagues and administrators are attempting to undermine student reporting on Jeffrey Epstein, Gaza, and other issues that public institutions would rather not have their reputations connected to.
Public records obtained by 404 Media show administrators taking meetings with parents and alumni when they take issue with student reporting and asking advisors to compromise their professional ethics. They also show escalating tensions between student newsroom staff and administrators, prompting more legal aid requests to advocacy groups.
This comes as student journalists at The Cornell Daily Sun earned recognition for breaking the “Cornell Seven” story regarding the lawsuit filed in New York’s trial court last month by a Jane Doe accusing current and former Cornell University students of drugging and sexual abuse. The student journalists told The Cut this week that a vague university and campus police statement from nearly two years ago prompted more questions than answers. Such is the basis for most accountability journalism on college campuses.
The cases we reviewed involved the phase of the reporting process that comes later, where a factual dispute the newsroom has defended or a complaint where the person doesn’t bother to explain what is allegedly wrong with the students’ reporting.
A lawyer used ChatGPT to generate briefs that included completely made-up witnesses and testimony while representing a man accused of shooting his wife. When he was caught by the judges, the lawyer blamed it on his own “stupidity.”
“It's little comfort to know that my stupidity is what brings us together this afternoon,” New Mexico attorney Stephen D. Aarons said in a hearing last month before five judges. He “assumed” ChatGPT would generate a “bulletproof summary of proceedings.” Instead, ChatGPT generated false testimony from “wholly fabricated witnesses,” according to an order of direct contempt from the New Mexico Supreme Court. Aarons was representing a man who was found guilty of killing his wife earlier this year; Aarons said he hadn’t told his client directly that he was caught using ChatGPT on the case because he’s currently serving life in prison.
The false testimony included a nonexistent person named Danny Stanton saying he received threats, and another made-up person, Linda Stanton, saying her husband received threats. Other false testimony from more AI-generated people in Aarons’ brief included statements “regarding the shooter’s clothing and appearance,” according to the order.
Reuters reported on the case earlier this month. "I am remorseful but hopeful that the disciplinary board takes into account it was an honest mistake," Aarons told Reuters. "It is a lesson learned for all professionals who rely upon this powerful but sometimes unstable technology."
Aarons’ reasoning for trusting ChatGPT, he said in the August hearing, was that he’d heard of doctors using AI for medical research.
“So, Counsel, do you watch the news? Do you listen to the radio? Do you read anything about what's going on in the world?” one of the judges asked Aarons. “Because the problem with lawyers relying on AI hallucinations is an above-the-fold story every single day.”
Lawyers getting caught using AI and submitting errors to the court has been a big problem for years, and the judge is right: It’d be very hard to avoid hearing about this pattern of “stupidity” unless you were living under a rock. There has been caseaftercaseaftercase showing attorneys failing their clients by submitting error-riddled AI-generated documents in their filings, and are the definition of “you had one job:” in the case of AI-brained lawyers, they can’t seem to resist taking shortcuts regardless of how risky it is for their own careers and their clients lives.
Like in Aarons' case, we sometimes get to hear and see judges go off on attorneys for this behavior. In May, judges in the Supreme Court of the State of New York Appellate Division let several lawyers have it for more than 20 minutes about their AI use and laziness, and called their failures “striking, concerning, disappointing, and saddening.” And it’s not just lawyers anymore; court reporters are submitting AI-generated errors, too.
“So, either you buried your head in the sand. And that's a choice to do that. An intentional choice to be uninformed,” the judge continued to Aarons. “Or, you took a gamble. And neither of those are consistent with the code of conduct. So I'm really struggling with your response, saying, ‘Well, I didn't understand. I didn't know about hallucinations.’ My 13-year-old nephew knows about hallucinations. My 75-year-old stepmother knows about hallucinations. So, either you made a decision not to be aware of what's happening in the world in using this tool appropriately, which is a willful choice. Or you decided to roll the dice.”
At the end of the hearing, Aarons admits that he hasn’t told his client directly about this mess, saying he’d only talked to his family members so far, telling them there was “a problem with the brief.” He said his client is hard to reach because he speaks Spanish and is in prison.
“So you didn't say, ‘I relied on ChatGPT, and it submitted lies to the court, and the court is mad at me, and now there's a hearing.’ I'm guessing you didn't tell your client’s family members that,” a judge asked Aarons.
“I didn't talk about ChatGPT. I just said there was a problem with a brief that I filed,” Aarons said.
According to the order, the judges found Aarons to be in direct contempt of court, referred him to the Disciplinary Board for further consideration, and barred from appearing before the court pending the outcome of the board’s investigation. They also threw him off the case in question and assigned a public defender to his former client.
While he waits for the board to decide what to do with him, Aarons was sanctioned $5,000, which he must pay to the State Bar of New Mexico Client Protection Fund.
Abusive, sexually explicit deepfakes have only become easier to make and more widespread in the last nine years. Since we first covered deepfakes when they appeared in 2017, they’ve never been the stuff of dark web sites or shadowy networks. They're found on social media networks, through basic web searches, and on easily downloadable apps and accessible forums.
The most prominent hosts of non-consensual, AI-generated imagery are supported by some of the biggest web infrastructure providers in the world, getting their hosting, email, advertising, and content management systems from these companies. A new study examines how providers like Cloudflare, Google, Proton, Namecheap, and WordPress help non-consensual imagery sites thrive. Despite years of knowledge about these wildly popular sites, big tech companies continue providing infrastructure services that prop up abuse imagery, even though their own terms of service often prohibit illegal and harmful content, the study claims.
Hany Farid, professor in Computer Science at Dartmouth College, Sophie Nightingale, a senior lecturer in psychology at Lancaster University, and Lancaster’s Sarah Morgan, who acts as the project coordinator on Nightingale’s “Protecting Ordinary People from Deepfake Harms” fellowship published their paper, “The Backbone of Abuse: How Infrastructure Providers Enable the Proliferation of AI-Generated Non-Consensual Intimate Imagery” in Stanford’s peer-reviewed Journal of Online Trust and Safety on Wednesday.
In a six-week period between February and March, they identified 400 URLs based on keyword searching and Google Alerts, then narrowed the sites down to 88 that were actively hosting non-consensual intimate imagery. Most of the content on these sites was of female celebrities, actresses, pop singers, K-pop idols, and women working in politics or activism. From there, they used open-source web-analysis tools like WHOIS to see what infrastructure providers the sites used.
“Five players emerged as dominant infrastructure providers: Cloudflare, Google, Namecheap, WordPress, and Protonmail,” the researchers wrote in their paper. Cloudflare provided “the lion’s share of services,” they wrote, by providing website hosting, content delivery network, and domain-name server services, as well as analytic tools. Google provided SSL certificates and advertising space for the majority of the sites studied, while Namecheap was the dominant provider of domain registrar services, WordPress provided the majority of their content management system services, and Proton provided mail servers.
Each of these service providers forbid customers and users from using their services for illegal activity. Publishing or threatening to publish AI-generated non-consensual sexual imagery is a federal crime in the U.S. and elsewhere, and sexual abuse imagery in general is illegal in many countries.
Cloudflare, Proton, and Namecheap did not respond to 404 Media’s requests for comment about these findings.
A spokesperson for Google told 404 Media in a statement: “Without the specific domains from the report, we can’t investigate these claims. We have strict policies against non-consensual explicit content — including AI-generated imagery — across all our products. We make it easy for people to quickly remove this content on Search, continuously update our systems to limit its visibility, and we prohibit monetizing or promoting non-consensual and sexually explicit content."
The Google spokesperson added that its advertising platform prohibits monetizing or promoting non-consensual and sexually explicit content, and treats this content as egregious violations, blocking ads on the sites or suspending the advertiser accounts involved. They also noted people can request the removal of non-consensual explicit images from Search, and pointed to Google’s recently-updated removal request process. Google has also updated its ranking algorithms by promoting non-explicit content where possible, they said, and demotes sites in search results that have a high number of removals against them.
A WordPress spokesperson said WordPress is “open-source software, not a hosting provider,” and wrote in a statement: “WordPress.org does not host websites or provide hosting services to sites that use WordPress, and we do not have access to or control over content published on independently hosted WordPress sites.” They noted that WordPress’s software is distributed under the general public use license, allowing anyone to use it for any purpose. “We take the issue of non-consensual intimate imagery seriously, but describing WordPress as providing services or infrastructure to these sites conflates the software a site uses with the services that host and operate it,” the spokesperson said.
In response to WordPress’s statement, Farid noted that while WordPress.org is not a host, WordPress.com is a hosting service. “It's operated by Automattic, which Matt Mullenweg also runs, and it hosts millions of sites. Any NCII site on WordPress.com is squarely within scope,” he told 404 Media. “Automattic provides services to self-hosted sites. Jetpack and the WordPress.com CDN serve images from i0.wp.com/i1.wp.com for sites that enable it, meaning the intimate images themselves can be served from Automattic infrastructure even when the site is hosted elsewhere. That's infrastructure by any definition.”
Farid also noted that WordPress.org maintains “an ongoing service relationship with self-hosted sites,” he said, with core updates, plugin and theme distribution, and security patches coming from WordPress.org servers. Farid mentioned Automattic’s ongoing legal battle with WP Engine, during which, in 2024, Mullenweg announced WP Engine was blocked and then unblocked from accessing WordPress servers; this “showed the project can and will cut a specific operator off from those services,” he said.
“So ‘no access to or control over’ is not entirely accurate,” Farid said.
These providers do draw their own moral and legal lines on what kinds of customers they’ll tolerate and when they cooperate with authorities. After initially refusing to do so, Cloudflare dropped hate speech and doxing site Kiwifarms from its protection services in 2022, and terrorist manifesto host 8chan in 2019. In 2018, Cloudflare banned sex work harm reduction social network Switter from using its services, citing anti-trafficking legislation FOSTA/SESTA, and in 2017, it Cloudflare stopped services to neo-Nazi site The Daily Stormer. In 2023, victims of the sex trafficking ring Girls Do Porn demanded Cloudflare stop providing services to sites hosting their abuse; Cloudflare claimed it doesn’t have “control over the content of websites using those services,” doesn’t have “the ability to alter or remove content on them,” and doesn’t “have knowledge of the people or entities who post any specific content to such websites.”
Aside from the infrastructure issue the researchers were studying, they found that 312 of the 400 sites they initially identified were unrelated to deepfakes, like gambling or random travel or cooking SEO-slop sites, but were using non-consensual imagery keywords as a way to rank higher in search results. This shows how incredibly popular the marketplace is for AI-generated abuse imagery, and how sites are able to manipulate Google search to appear higher in results if they use related terms.
Morgan told me that while journalists can expose site administrators (like in the case of MrDeepfakes.com, whose owner was identified by a massive joint investigation by Bellingcat, the CBC and TjekDet), and legislation can act as a deterrent, the research group wanted to focus on infrastructure providers as the “distribution supply” for these sites. “Creators are going to create and people are going to demand. We can’t stop that. But we can call for providers to cut the distribution supply and stop enabling these sites. This content needs preventing from being shared in the first place. All parts of the ecosystem have to work together to play a part,” she said.
The researchers recommend in their paper that these providers stop supplying services to sites hosting non-consensual imagery.
“We aren’t saying that infrastructure providers are knowingly facilitating this content — or that they are aware of what’s on every site that uses their provisions — but it’s in providers’ power to vastly improve their moderation and cease services to sites as soon as they are identified as hosting this content. And to check sites that are reported to them,” Morgan said. “A gold standard response in our eyes would be a commitment from infrastructure providers to working with NGOs and governments across the globe who are searching for solutions that will lead to verifying URLs and sharing these with participating platforms so they can be blocked.”
This is Behind the Blog, where we share our behind-the-scenes thoughts about how a few of our top stories of the week came together. This week, we discuss some small changes, an AI song, and internet soup.
JOSEPH: We’re making a couple of changes that might sound boring to someone who doesn’t read or listen to our work, but I’m actually legit excited about them, and I think they will be better for all of you too. I’ll talk about one now and maybe the other next week.
When I first started writing about online cultures and subcultures almost 10 years ago, one of the first scholars I read on the subject of internet anthropology was Professor Sherry Turkle. Her earlier books, including The Second Self (published in 1984) and Life on the Screen (1995) became something like a model for my own writing and this kind of reporting— approaching the internet with an ethnographer's mindset and taking nonjudgemental documentation of how people are behaving and feeling in a given space, even if that space is digital.
Through talking directly to people in their own environments, whether they were in MUDs and early virtual chat spaces or Second Life and massive role-playing games, her work revealed endlessly interesting firsthand perspectives on lifestyles that people still found subversive, marginal, and at times perverse. The people she interviewed often said that digital worlds offered support, companionship, sexual satisfaction, gender play and freedom that the quote-unquote "real" world did not. These were topics traditional academia wasn't yet touching: cyber-sex, intimate relationships unfolding in chatrooms, embodiment in online worlds that felt as real to their inhabitants as the world their physical selves sat in behind a keyboard.
What struck me about those books — and later, in Alone Together, Reclaiming Conversation, and The Empathy Diaries — was often how fundamental the human needs for connection and vulnerability are, despite the technology being used to fulfill those needs.
Fast forward to today. Something has shifted. “In 1984, I called the computer a second self,” Turkle writes in her latest book, Artificial Intimacy: Who We Become When We Talk to Machines. “Computers provoked an explosion of talk about what was essential about being a person and about how new computational ideas might illuminate our thinking about the mind. With the digital twin, the metaphor is turned on its head. It’s the agent who is in charge and imagines a version of you. Now you’re the second self. So when you are given time to get ‘back to your life,’ it’s been diminished.”
Professor Turkle is a licensed clinical psychologist, the founding director of MIT's Initiative on Technology and Self, and the Abby Rockefeller Mauzé Professor of the Social Studies of Science and Technology at MIT. Artificial Intimacy: Who We Become When We Talk to Machines comes out on September 29.
The technology today is wildly different than it was in the 80s, but as Professor Turkle explores in our conversation for the 404 Media podcast, the need for empathy, vulnerability, and to feel understood remains central to the human experience. With increasing reliance on conversational “companion” chatbots like ChatGPT and Replika, we risk misinterpreting machine programming as care, and risk misattributing human qualities like empathy and emotional intelligence onto unfeeling, incapable robots. The result, she argues, is a population forfeiting its empathy skills at an alarming rate, trading difficult feelings and conversations with humans for convenience and instantaneous feedback.
“When people turn to a chatbot, they don’t have to dread the fatigue of genuine give-and-take. And they don’t fear judgment or abandonment. But when we talk to chatbots, we lose the skills required to interact with humans, so chatbots start to feel like a better option. With an alternative, disappointment with people feels unnecessary,” Turkle writes.
What happens when people stop caring that there’s not a human on the other line? We got into that on the podcast.
“It's a product whose fundamental directive is to keep your attention. And so it matters very much that we sort of become glommed onto it as though it were a person when it isn't,” Professor Turkle said in our conversation. “But more than that, I think that it's very important to ask, what is the end game of this?” She referred to the interview Mark Zuckerberg gave last year when he said Americans can only have three real friends, and intended to let AI fill in the demand for more.
“Now that to me is a very dark endgame,” she said. “And so people who say, ‘What the hell? I kind of like it.’ Well, let's just think about where we're moving towards. Because it's not just a little entertainment. It's a corporate project to get us involved in certain kinds of relationships that are going to make tons of money and to keep us at our screens all day talking to sycophantic, seemingly loving entities that don't know who we are and don’t care about us.”
In our wide-ranging conversation, we talk about how we got here, the concept of “deskilling” one’s own capacity for human connection and vulnerability, whether there’s a way back out of this mess for us, and what happened when she asked a chatbot to roleplay her own legacy (spoiler alert: nothing good).
This is Behind the Blog, where we share our behind-the-scenes thoughts about how a few of our top stories of the week came together. This week, we discuss just doing it, machines that eat the internet, and an eavesdropping watch.
EMANUEL: It is a foundational, defining feature of 404 Media, but I sometimes forget that when I’m having trouble with a story, I need to just do it.
This is most of all true when it comes to reporting. I spent a couple of weeks looking into AI generated music on Spotify appearing under the names of real artists. It’s an issue I and many other publications have reported on in the past, but whenever I look into it I am shocked by how prevalent the issue is. If you talk to any musician or spend some time clicking around music platforms yourself, you’re going to quickly find hundreds of examples of this abuse of the digital music distribution system.
After Ed Sheeran dropped Macklemore from his tour over the singer’s support for Palestine, people flooded an online contest to open for Sheeran’s Mexico City show with fart videos, bouncing WWE butts, and troll posts from usernames like “EdSheeranSupportsAndFundsGenocide.” The posts appeared live on the contest page, where people can vote for submissions they want to see win, and went up seemingly without any moderation.
Guitar Center started promoting the “Open for Legends” contest on September 14, the same day Macklemore announced he had been removed from the last 10 shows of the U.S. leg of Sheeran’s tour. Macklemore had called for a “free Palestine” on stage at his previous shows and made statements in support of Palestinians in Gaza and the West Bank, Rolling Stone reported. “YOU COULD OPEN FOR ED SHEERAN,” a poster for the contest says.
The New York District Attorney announced on Monday that it has seized 12 websites hosting AI-generated non-consensual intimate imagery, primarily of celebrities and public figures.
The DA’s office said it believes that this marks the largest seizure of such sites in history. The people using and running these sites who are now under investigation used AI tools to turn approximately 1,200 people’s photos into “hyper-realistic” images and videos of them engaged in sexual conduct, according to the DA’s office.
The images allegedly depicted “politicians, first ladies of multiple countries, primarily public-facing individuals, including actors, politicians, athletes, musicians, social justice advocates, and social media influencers.
New York has had laws specifically criminalizing sexually explicit deepfakes in place since 2023. Bragg referenced this legislation when asked what law the office is acting on.
“While the victims of these sites are largely celebrities, we know that anyone can be a victim in these crimes, particularly in domestic violence cases, many of our intimate violence cases involve threats to at least images and videos, and we know that deepfakes are being used in the same way,” District Attorney Alvin Bragg said in the press conference Monday. “So, whoever you are, the message to survivors is the same: You are not alone. Help is available. If you let us know that an image has been posted without your permission, without your consent, we can take action as long as that website is accessible from Manhattan.”
Bragg declined to answer whether images of minors were included in these websites, saying, “The investigation is ongoing.”
In June, the U.S. Departments of Justice and Homeland Security seized two domains that were being used to publish “thousands of digitally forged images and videos depicting famous women as nude and sometimes engaged in sexual activity, without their consent,” and featured topics like “rape,” “forced,” and “degradation,” according to a press release. Images on those sites allegedly depicted politicians, first ladies of multiple countries, royalty, journalists, television presenters, athletes, entertainers, and others.
And in May 2025, the massive deepfakes hosting site and forum MrDeepfakes.com shut down following an investigation by the Canadian Broadcasting Company, Der Spiegel, and Bellingcat.
Earlier this month, the first man in the country to be convicted of violating the Take It Down Act was sentenced to 15 years in prison, after investigators found more than 3,000 real and AI-generated abuse images on his devices.
Slightly altered images of women who are public figures are taking over social media sites like X.com, 404 Media reported last month. These images are often not completely nude AI images of women, or face-swapped sexual material, but depict women as wearing more revealing clothing than they did in the real photo or video, or with digitally altered bodies.
The DA’s office is encouraging victims of deepfakes to contact its Cyber Crime Bureau at (212) 335-9600. Bragg said targets of deepfakes might not know that something can be done, and wants them to know there’s a “cop on the beat.”
“Our message to the people who are creating and publishing this content: What you are doing is a crime. We are investigating, and support survivors in taking down this content and seeking accountability for folks who break the law,” Bragg said.
Connexion
