The Mythcreant Podcast 582 – Shy Girl and AI Writing
In an unusual twist for the Mythcreants podcast, we’re talking about current events (gasp)! Don’t worry, we won’t make a habit of it. But when we heard that a Big Five novel had been pulled over accusations of LLM generated text, we had to look into it. Honestly, the story is way weirder than much of the coverage suggested.
Transcript
Generously transcribed by Lady Oscar. Volunteer to transcribe a podcast.
Chris: You’re listening to the Mythcreant podcast, with your hosts, Oren Ashkenazi and Chris Winkle.
[Intro Music]Chris: Welcome to the Mythcreant Podcast. I’m Chris…
Oren: …and I’m Oren.
Chris: [pretentiously lofty voice] On this podcast, we give advice. Not business advice, not process advice, just craft advice. Advice that lifts you up like hope, that lulls you like peace. That pulls you forward like ambition. It’s not words, it’s education.
Oren: God, I took psychic damage hearing that. Oof! [Chris laughs] Oof, Chris, you gotta warn me next time!
Chris: I’ve now learned too much about what AI-generated prose sounds like, or AI-generated text.
Oren: You’re absolutely right. What a great idea!
Chris: Sorry about giving you that psychic damage. I’ll be sure not to do it again.
Oren: Three seconds later… [both laugh]
Chris: Three seconds later, on this podcast…
Oren: So we’re doing something that’s almost current events on the podcast today. Kind of unusual. We’ll see how quickly we can actually get this out.
Chris: Current events, and I also wanna kind of broaden this into talking about, what does AI-generated fiction look like? What patterns are we seeing? What are some potential ways to recognize it, knowing that there’s still a lot of uncertainty here, and it could totally change with every new version or between different LLMs. But a little bit more useful stuff too.
Oren: Since we’re gonna be referencing the novel Shy Girl a lot, can I give a brief timeline of events?
Chris: Yeah. Let’s give you the timeline of events around this Shy Girl controversy.
Oren: I don’t have exact dates for most of these events. I’m just gonna give you the rough order they happened in, and they happened over the course of the last few months. So, the novel Shy Girl is self-published by an author who at the time went by Mia Ballard. It gets a little bit of traction online. It eventually reaches about 4,900 Goodreads ratings, which is not bad, not a runaway success, but better than a lot of authors do. People start to wonder if it’s LLM generated. There are Reddit threads. The YouTuber frankie’s shelf makes a nearly three hour video dissecting it.
This guy named Thaddeus Mackleroy – I believe that is how it’s pronounced – I’ve seen some MickIllroy, but Mackleroy appears to be the more common version. [Transcriber’s note: The name is spelled “McIlroy”; this is a discussion about pronunciation.] He hears about this, and he decides to investigate. As you will see later, he is not our friend, but for now, he knows people who run an AI detection company, which he sends the manuscript to, that he has acquired through ~means~, and they come back claiming that 78.4% of it is AI generated. Not that they have a 78.4% certainty, but that 78.4% of the document is AI generated. That is how he phrases it.
So he goes to the New York Times with the story. They then go to Hachette and Ballard for comment – Hachette being the company that is publishing it through their imprint Orbit – and Hachette/Orbit decides to pull the book. They make some vague statements about doing their own investigation. Very unclear what that means.
Ballard makes like one statement in an email to the New York Times claiming that she’s innocent, she didn’t do this, but also makes a vague allusion that her editor is responsible, with the exact quote being, “All I’m going to say is please do your research on editors before trusting them with your work.” She then claims she can’t say any more because she is pursuing unspecified legal action. Her minimal online presence vanishes, and then McIlroy publishes a blog post explaining all of this on LinkedIn, which is partly how I was able to put together this timeline. I also looked at, of course, the New York Times article and what Hachette said. And I wanna talk about that blog post later, but it’s probably not the most important thing in this story.
Chris: [laughing] To be clear, we don’t have access to any more information than the stuff we’re telling you that is freely available online, and by looking at the text of the book, and nothing is 100% for certain unless we traveled back in time and transported ourselves to see Mia Ballard and what she did to put this book together.
Oren: I realized, I think in that timeline that I put together, I forgot to mention the most important part, which is that Hachette/Orbit decided to pick up the book after it achieved its modest self-publishing success.
Chris: She got indie publishing success, and then was picked up by a trad publisher that did publish her in the UK, I believe, right before pulling it. But it had not been published in the US yet.
Oren: Most important part of the timeline, I obviously forgot it.
Chris: [laughs] But then completely pulled the book after the New York Times started investigating. Again, what I said is, we don’t know anything for 100% certain. At the same time, that said, I personally think that we can be all but certain that large portions of it were AI generated.
Ballard kind of admitted it when she said, “Oh, you ought to be careful who your editor is.” Look, it is not normal for an editor to contribute that much text to your book, and when I say that much text, I’m not even talking about the 78%, and we’ll talk more about what its narration looks like. It is very uniform. There are some places that are perhaps a little bit more quirky and human, which might reflect places where Ballard actually did some work herself. But this is not a situation where the AI style is isolated to little bits here and there, or in a few places that an editor might have done something.
All of us who are writers who care about our stories, I cannot imagine sending my work to an editor and then getting back something that was changed enough… That would be ghost writing, not editing.
Oren: Here’s the thing, I was talking to some other editors about it, and they were like, “well, surely she would’ve reviewed the changes that any editor made.” And from the way that I was looking at this book and how many just like incredibly obvious formatting mistakes it has, I wouldn’t be surprised if the author reviewed no changes. I still don’t believe this “the editor did it” story.
Chris: That’s what they all say. Every single time an author is caught using AI to generate text, they all claim it was somebody that helped them or they hired or something. It’s not very believable. At the very best, it would mean horrible negligence on their part, and also that they don’t really care about writing their own work anyway, because somebody else would be writing their work for them. That’s kind of the only way that this works, because this work, it’s very kind of monotonous and repetitive in the way that it’s narrated, so it’s everywhere. It’s the entire thing.
That’s just not how editing works. Generally, editors, if you get them to do heavy editing, they will supply more suggestions. Changes are tracked. They are making an effort to adhere to your voice and keep things consistent with the way that you normally narrate. If you had sort of AI software that was making constant suggestions, that would basically rewrite every single paragraph…but it’s just not feasible. Not if you’re the type of writer who actually writes your own work.
Oren: I have seen accounts from authors who claim that they hired an editor, quote, unquote, off of some race-to-the-bottom website, and what they got back was just their manuscript had been fed through an LLM.
Chris: You get what you pay for, folks.
Oren: It’s not impossible that that happened. It does strain credulity that the author would then go on to publish that. And even if that is what happened, I’d say they are equally culpable at that point. They had a responsibility to check their work.
Chris: Just wanna say what I think the biggest tell of Shy Girl is, because even things that seem nonsensical or inhuman can just be bad writing. We see unpublished manuscripts, and writers do a lot of weird things, a lot of nonsensical things. When people are trying to pound out thousands of words at a time some weird stuff ends up in there by accident. Some awkward phrasing ends up there, and you could look at any of that and call that inhuman if you wanted to. That doesn’t necessarily mean it’s AI generated just because it doesn’t seem to quite make sense.
However, the thing that is really the big tell for this one is the contrast between the fact that for most of the narration, the spelling and grammar is just fine, but the formatting and production quality is incredibly sloppy. Because the cover image of the original indie book was stolen, and it was like blown up, so it’s poor quality, but also the formatting of the book is just extremely sloppy. There are breaks in the middle of paragraphs.
I’ve seen indie authors, they don’t have a lot of money, they’re trying to put their book out. So they don’t necessarily know what good design is. They could try to do something themselves, and it might not turn out right. But a normal writer, even a really bad writer, who has actually spent all of that time writing, loves their book. That is the difference. This book is their baby. If they got a paperback copy, they would wanna open it up. They would wanna be really proud of it. If they see things like that, they would fix them, almost certainly. And if they’re so neglectful, then it’s weird that there aren’t any spelling errors.
Oren: Yep. That is very strange.
Chris: Because if they’re not neglectful, they’re not gonna be polishing…because it takes polish to get the spelling and grammar everywhere to be that smooth. And if they’re so sloppy in the formatting, they’re probably not hiring a professional copy editor to look at it. So how did it get that smooth, if they’re that neglectful?
Oren: You would expect way more typos that a basic spell check wouldn’t pick up, like incorrect words that are spelled correctly, but are the wrong word. That’s the sort of thing I would expect. The manuscripts that I edit are often extremely rough, and they can be rough in weird ways. So I am super hesitant to say that a certain thing is AI. I’m gonna be honest, if the author had not basically acknowledged that it was true, I would still be hesitant to say this was AI, just because I’ve seen so many weird things that human writers do, but there is a lot. There is a lot here.
Chris: But I think it is worth knowing… We also have some personal experience with plagiarizers who have taken our material from our blog and done various stuff with it. And one of the things that you get to learn, if this is done to you, [laughing] is that plagiarizers are just super, super lazy. Because that’s why they’re plagiarizing. And they don’t cut those big corners just to spend their time carefully formatting things, and carefully marketing things and packaging things, typically. So this kind of sloppiness of the production is what I would expect with a plagiarized work. They’re plagiarizing in order to not spend time, so they’re not gonna spend time. So that’s pretty par for the course with them too. And that’s something to be aware of.
The spelling and grammar being pretty good really suggests some automation happening there. That sort of is a little helpful because it gets out of some of the subjectivity that can be true, because a bad writer can create anything, or it will cover a lot of things that are characteristic of AI-generated fiction. It got that from actually good works. It got that from strong written works, and the most notable one being the em dash, which a lot of people came forth to defend. And the reason it’s so attached to em dashes is because professional writers are a lot more likely to use em dashes. And so the AI associates em dashes with strong writing. So if you would tell it to give you strong writing, it’s like, oh, well that means em dashes.
Oren: Obviously, that’s what everyone else is doing.
Chris: [laughing] Right? It’s a funny thing, because people seem to not even be able to tell it to get rid of the em dashes. It keeps using them. It can’t not use them.
Oren: So what are the specific signs in Shy Girl that we were looking at?
Chris: We did watch the three hour video by frankie’s shelf. And it was a good video. I’m gonna give Frankie credit for that. If you’re thinking, “Oh, I don’t wanna watch three hours, but I would like to just come in at the end of the video and watch some of it,” you can do that, but it’s not gonna have the same effect. And the reason is because Frankie spends those three hours just showing you the patterns, because this is about the repetition, the thing that really kind of emphasizes it. It’s the frequency of repeating the same things over and over again through the entire narration that really makes it stand out as possibly generated work.
And if you don’t read the book, or watch three hours of Frankie describing this repetition, it can be hard to understand what’s so unnatural about it. But basically it uses some words with strange frequency, like in particular “sharp,” and “heavy,” and “humming.” It has a lot of similes that are either slightly awkward, or just overly dramatic. It uses tons of adjectives to tell readers what to feel. And some of those adjectives are just like similes that are literally true, [laughing] which I have also seen authors do. I have seen authors write similes that are literally true, and therefore not similes, before. This is all something an actual writer could do.
Then I think personally, the thing that can really stand out as an editor is just the fact that the narration never changes to suit the moment. It feels melodramatic, because it has just exactly the same amount of drama throughout the entire thing.
Oren: Which is funny, because this is supposed to be a super dark book about torture and abuse.
Chris: But it never gets more dramatic. Or less dramatic. Everything is equally dramatic. It has like one little backstory summary near the beginning, but it doesn’t seem to pause to do a deeper dive to set up later events where it can be faster, for instance. It’s just the same amount. And when the character has what should be a serious emotional blow, it never stops to do more internalization, to explore the effect that it has on her in a deeper way. So even though there is internalization, even though there are emotional reactions, it ends up feeling shallow, because it never stops to lay more groundwork, give more background, dive deeper into her head. It just continues forward at the same kind of not-very-fast, kind of melodramatic, pace.
And that kind of monotony of the narration kind of reminds me of, honestly, AI-generated voice narration that I heard recently. It was like somebody had picked a setting where they wanted the voice to sound kind of bizarrely dramatic, and then that sort of dramatic style was applied uniformly to the narration, even in situations where it just did not fit. Like the intro to the audio book that was just saying here’s what the title is was in the dramatic voice. And the opening had a character who was bewildered and found themselves a new world and was saying casual modern day phrases in what I imagine Phantom of the Opera voice would be [laughing] – not actually being that familiar with that story – just like uniformly everywhere. It’s not what – it’s not normal. A new writer could totally do that. And then Frankie brings up some places where people just don’t talk like that, like an eviction notice that says, “pay rent or quit,” even if that’s technically accurate.
When I did the intro to this podcast, that was an imitation of AI style, like a little section. When I hear it, it starts to have a very distinctive voice that you could recognize. It has a lot of short, succinct fragments that are meant to add drama. And the pacing could almost feel stilted, and I can kind of recognize it because I automatically start reading it in that voice that I used at the beginning of the podcast – this sort of stilted, overly dramatic… So in this case, here’s just like one paragraph:
[dramatic voice] “Rent. Bills. A parade of due dates I couldn’t outrun. My brain gnawed at the problem, like it could chew its way to a solution, turning it over and over until every angle was frayed. Borrowing money felt like begging. Job applications felt like flinging darts in the dark, the targets moving further away with every throw. What I needed wasn’t politically correct – em dash – I needed something that was immediate. Radical.” [/dramtic voice]Again, it’s very dramatic. Those kind of one-word sentences throughout the paragraph, the em dash… The other thing about the em dash, it is a very dramatic punctuation mark. It adds a lot of drama to a phrase, so it also can make things feel melodramatic. But that kind of slightly stilted, overly-dramatic phrasing with a lot of repetition – which is not normally bad – in the way things are phrased, is kind of what it sounds like.
Oren: The first really interesting feeling that I had, looking at this whole thing, this whole scandal, was relief? Because if this is representative of where the AI slop book makers are, then the solution is don’t read really crappy books, like even if you can’t recognize that it’s AI or not. And I was already doing that.
We should point out that we don’t know that. This could just be a particularly bad example, and that’s why it got caught. But my first thought was, okay, I think who this really indicts is Hachette, who was willing to publish this because it got a modest amount of social media traction. And it’s like, I don’t know, Hachette, try not publishing crappy books. I’m not a businessperson. Maybe publishing crappy books is where the money is now. It just seems like maybe that’s the solution to this.
Chris: It’s interesting, because certainly LLMs by themselves cannot write a good novel. I feel like somebody who tried to put more of their own legwork in mixed with LLM-generated text might be able to go further, but at the same time, we kind of know that the nature of this is A, people are more likely to choose generative text if they are lazy and don’t wanna do the work themselves, and that people tend to quickly become dependent on LLMs in many cases, and over time are likely to cede more ground to them, and also let their guard down. Because AI makes mistakes. Unfortunately, it makes them just infrequently enough that people can’t maintain their vigilance when it comes to checking it.
Oren: And that’s sort of what I’ve been bracing for, because my experience with LLM-generated text is primarily nonfiction, as far as I know, because I run into it a lot on blog posts and social media. And the way I know this is that often the person who does it will admit that they did that. They’ll either brag about it, or they will admit it under pressure. Which I’ve always wondered why they don’t lie, but often they don’t. And the characteristic of these are that it has grammatically correct sentences, each of which is a proper sentence, but that it doesn’t add up to anything. And that they tend to be very long, and they tend to have a lot of organizational formatting that doesn’t mean anything.
But of course all of that is things that a person could do. A person could write a really long Reddit post with a bunch of bolded headings for no reason that never gets to a point. So that makes it kind of impossible to tell. It’s just that it’s bad, and that’s the more useful metric at this point. So that’s why I was expecting that to be the big scandal, and I suspect we will probably still get there. There will probably be someone who is willing to put in the work to make an AI-generated book not so obviously terrible, and then they’ll eventually get caught out, and that’ll be a big scandal. I’m sure that’s gonna happen eventually.
Chris: I’m sure that an actual human working with an LLM could go farther than just the LLM. But I do think that would mean very bad things for that writer long-term. Because we’ve seen people’s skills degrade, have a hard time remembering the story if you don’t write it yourself, all sorts of things. I think it would just be very easy to become dependent. And also a lot of the things that Frankie felt that were bad about this novel was just the fact that the similes weren’t there. And they either were too literal, or, just, I don’t know what that’s supposed to convey.
If you’re a person who is going over and has to check all of that LLM text, the idea of not letting any of that through seems unlikely to me. Eventually, you’re gonna start kind of accepting more of it and just letting it… And, as I see more about what AI-generated text looks like, there are more commonalities. I also think that if there are becoming more books that are written this way, people are more and more gonna recognize its voice, because it has its own voice. Even if we can’t identify – because other people might start copying it inadvertently, because there is style pollution. If you read a book with a certain style, you’re more likely to sort of subconsciously start copying that style. That’s how it goes. But at best that’s gonna be the general background style against which if anybody wants their style to stand out, and they’re writing to stand out, they’re gonna have to deviate from that.
A really funny anecdote – there’s an NYT article by Sam Kriss on what AI writing looks like that is quite informative – that apparently if you ask AI to write science fiction, it has an uncanny habit of naming the protagonist Elara Voss, specifically, and male characters are usually named Kael.
Oren: God dammit, Kale!
Chris: [laughing] Like K-A-E-L. And so, according to Sam Kriss, there are now hundreds of self-published books on Amazon featuring Elara Voss or Elena Voss.
Oren: Yeah, of course there are. That cannot be a surprise. That’s just exactly what we all knew was going to happen.
Chris: [laughing] I just find that very funny. So if you read sci-fi and the main character’s name is Elara Voss, or Kael, just run in the other direction.
Oren: And I do think it’s important to emphasize that there are no good guys in this story. It’s been pointed out that Hachette either knew or should have known that there was something fishy going on with this book. Because presumably they would’ve looked into its social media history, since that was in theory the reason they got it. And they would’ve noticed that there were a whole bunch of people asking, “Is this book AI generated? And also it’s really, really bad.” They at the very least were negligent. And I mean that in the colloquial sense. I’m not suggesting they were legally negligent, not a lawyer.
But also I wanna take this moment to talk about our good friend Thaddeus McIlroy, who is the one who brought this all to light. Because his blog post about all this is bizarre. First, he mostly wrote this blog post to complain that the New York Times didn’t give him enough credit. And if he was anyone else, I would agree with him, because the New York Times barely mentions him. But you then start to wonder, okay, Mr. McIlroy, why did you launch this investigation in the first place? Because he is actually very upset at what happened, that Ballard lost her book contract and had to give up this pen name. Like, he thinks that was an unjust result.
Chris: Was he thinking, like, “see this book is AI, glory to the AI”? Is that what he was doing?
Oren: I don’t know. He’s really unclear why he decided to do this. He has a section where he says, “Trade publishers are (somewhat) quietly adopting AI tools themselves, dancing delicately around whether this conflicts with their public-facing ‘principles.’ I was more than happy to use this story to highlight the conundrum.” So maybe he thought he was gonna pull a gotcha on Hachette, and they were gonna be like, “oh yeah, actually LLMs are fine”? Surely he didn’t think that was going to happen. If so, he’s extremely deluded. There is no way, especially not with a book this bad.
So then he’s upset that this is what happened. This obviously predictable result of his actions is what happened. This is a bit like a cop bemoaning that after catching someone for assault, that person went to jail. It was like, “Well, I didn’t want them to go to jail. All they did was beat up a bunch of people.” Well, okay, why did you catch them then? [Chris laughs] Remember, he’s claiming that these AI detectors, they work really well according to him, and this idea that they don’t work is like an outdated idea from when they were bad. And he lists some studies that claim that the AI detector he’s using has an error rate of 0.5%. That’s how accurate he thinks these things are, but he also thinks that we just have to accept that books are gonna be made with LLMs now, because there’s nothing to do about it.
What are you doing here? He is both bemoaning witch hunts, selling witch detectors, and also telling us it’s impossible to do anything about witches. The most bizarre thing I’ve ever seen. I’ve seen accusations that his blog post is AI generated by people who ran it through AI detectors, and maybe it is. Maybe he’s just bad at writing. I cannot believe what I read. As I was reading this, I was like, what was the purpose of this thing you did?
Chris: That sounds like somebody who thought that “here, if I reveal this book as AI, then Hachette will be forced to defend it and forced to defend its use of AI,” and thought it would go the other way. “If I bring AI out into the open, then it will cause AI to be acknowledged and lauded instead of canned.”
Oren: Instead of the obvious thing that was going to happen. We do know that a lot of AI boosters are completely disconnected from reality, so maybe that’s what’s going on.
Chris: Or just surrounded by other AI boosters, so they don’t seem to know how many AI haters there are. We hear again and again from so many people that they’re surprised when they do an AI thing and it’s controversial, and it’s like, who are you hanging out with that you didn’t know that?
Oren: It’s just so weird that the people who are selling us the product which is supposed to be the solution to the problem, do not want us to use their product to solve the problem.
Chris: Well, maybe that could be one reason, if they’re not reliable.
Oren: I don’t think they’re reliable. And he doesn’t think they’re reliable, because he’s like, well, you gotta presume innocence. And it’s like, do you? With an error rate of 0.5%? I don’t know, man. That’s your study, not mine. I’m just saying you seem to think it is.
Chris: Before we go, I’d love to talk about, just go over quickly, some patterns of AI writing in general. It tends to have very specific tells that you can learn. Again, these things will be found in writing, especially strong writing, because that’s where they come from, unfortunately, and it could vary with different versions.
We talked about the em dash. Professional writers love it, but LLMs like to use it a lot. Sets of three. That’s another thing that culturally we just find pleasing. If there’s a set of three, that’s another thing a strong writer is likely to use, but LLMs use them a lot. Repeating similar phrasing, using the same phrase is a way – that’s actually a lyrical technique that makes the words, you know, the style more rhythmic. But of course LLMs have picked up on that, so they have a lot of repeating phrasing. And then there’s specific imagery and words that LLMs tend to like, like “quiet” and “silence” and “humming,” and apparently they love ghosts, and phantoms, and the word “delve.”
Oren: Okay. [laughs]
Chris: The other thing that really stands out, that is very present in Shy Girl, and I think that there is a name for this, but I cannot find it anywhere. It’s a specific literary technique of basically pairing a tangible thing with an intangible concept, a concept that is often kind of emotionally loaded. For instance, in this case, when we say the main character’s mother is about to abandon her, we say that her suitcase is big enough to hold forever. So the forever would be like the intangible concept in the suitcase holding forever, and it kind of creates contrast, and can be kind of evocative, which is why this is a well-liked literary technique.
A lot of the writing tends to feel very, like, kind of flowery, and it gives it the illusion of having emotional depth. Even though it doesn’t really, if you actually keep reading it and look deeper, try to put it together after it’s done a paragraph of this. But the interesting thing about this is it’s a replacement for something LLMs seemed to be having trouble doing, which is sensory description. If we were to describe the suitcase physically, or like how it smelled or felt, that would basically put more burden on accuracy. It would really require knowing what a suitcase looks like, or feels like, or smells like in order to do that. Whereas if we just say it’s big enough to hold forever, you could associate forever with all kinds of tangible things as an artistic strategy, and so it gives the LLM more leeway. But eventually you’ll see that these intangible concepts are kind of nonsensically paired with things, or they’ll be mixed metaphors, or they’re just not used in as thoughtful a way as a human would do it. So that’s another thing that’s very common.
Oren: With that list, I think we’re going to have to finally call this extra-long episode to a close.
Chris: If you would like to support some actual human writers, [laughs] you can always become our patrons. Just go to patreon.com/mythcreants.
Oren: And before we go, I want to thank a couple of our existing patrons. First, there’s Ayman Jaber. He’s an urban fantasy writer and a connoisseur of Marvel. Then there’s Kathy Ferguson, professor of political theory in Star Trek. We’ll talk to you next week.
[Outro Music]This has been the Mythcreant Podcast, opening and closing theme, “The Princess Who Saved Herself” by Jonathan Coulton.