Problems with Using AI as a Content Generator

The following is a guest post by Jay Dixit, founder of Socratic AI

Most people treat AI like a question-answering machine: Ask a question, get an answer. What is chili crisp, and why is Gen Z so obsessed with it? What’s the best starting guess for Wordle? What’s the best time of day to post on LinkedIn? (My own real queries from today.)

There’s nothing wrong with using AI to get answers to your questions. But there’s another mode of interacting with AI that many people never consider — one I find much more useful for my creative process.

Here’s what I do instead: I flip the script and let the AI ask the questions for a change. Instead of prompting AI, I get the AI to prompt me.

You may have heard Jeremy call this technique the “reverse interview,” and he’s previously written about it as a tool for reflection. You might use reverse interviewing, say, to conduct a soul-searching interview about what you want from your career.

I take that same role reversal and apply it to the writing process. I call it the “Socratic interview.” It’s the foundational technique I teach for using AI as a thinking partner instead of a content generator. It’s also what inspired the name of my company, Socratic AI, and my upcoming masterclass with Narratively Academy. (See below for details, including an exclusive discount for Wonder Tools readers.)

The Socratic interview works for any writing task — a first-person narrative, a Substack post, a pitch deck, a talk for SXSW, even a speech for my best friend’s wedding. I use two versions of the technique: one to help me figure out what to say, and one to figure out how to say it.

Socratic Interviewing Level 1: Excavating raw material

How to use AI to help you surface memories, examples, and stories

I sometimes meet people who use an even more basic version: the adversarial interview. “Play devil’s advocate. Pressure-test my assumptions. Poke holes in my argument and reveal gaps in my logic.” Maybe call that the starting level.

What I find much more helpful is using the Socratic interview as a tool to access my own creativity.

As a writer, I’ve always had plenty of ideas, insights, and stories. I know they’re in there somewhere. But staring down the blank page is hard. What’s much easier is answering when someone asks me a direct question.

So I use Socratic interviewing to draw out the ideas, memories, stories, and examples I have in my head but haven’t gotten down in writing. For me, it solves the blank page problem — without ever using AI to generate prose.

Let’s say I need to write a groomsman speech for my best friend’s wedding. If I were to use ChatGPT as a ghostwriter the way most people do, it would output something lazy and trite. Look what happens when I ask it to generate a draft.

“That’s not luck, that’s character”? Pure slop.

Socratic interviewing is a different process altogether. Here, I give Claude a bunch of messy context about the writing task and ask it to help me remember the adventures and funny moments I’ve had with Tim over the years.

Generating answers is AI’s default mode, but it also excels at asking evocative questions to jog your memory and get your creative juices flowing. You just need to flip it into that Socratic mode.

I can ask for questions in a batch or, if I want it to feel less like a writing assignment and more like a conversation, take them one at a time. Either way, instead of sweating over a blank page, I’m now in a playful conversation about funny memories from my formative years.

Once I’ve answered these questions, the task ahead of me changes. I’m no longer struggling to compose some perfect sentence to somehow encompass the totality of our friendship. All I have to do is look at my own stories and start choosing and shaping my favorite moments.

---------

Sponsored Message

I open Shortform for books I haven’t had time to read, books I finished years ago, and books I’m not sure are worth my time.

Take What the Most Successful People Do Before Breakfast by Laura Vanderkam (screenshot above). The Shortform guide surfaces the essential ideas fast. It’s particularly useful for nonfiction books, biographies, self-help, and other genres where you’re reading for tangible takeaways.

I appreciate that smart humans write Shortform’s guides. In an era when AI-generated summaries are everywhere, there’s a difference. These feel like notes from thoughtful readers.

Shortform now covers podcasts and long articles too, not just books. If your reading list grows faster than you can get through it, Shortform is one of the more practical tools I’ve found for making progress on it.

Wonder Tools readers get a free trial + 25% off the annual plan 👇

----------

Socratic Interviewing Level 2: Defining your purpose and strategy

How to use AI to clarify your writing goals

Just as important as figuring out what you want to say is figuring out how you want to say it.

Like all art, writing is about making a thousand tiny decisions. It’s hard to get those decisions right if you don’t even know you’re making them. If the only directive I have in mind as I’m typing is “I need to make this good,” that doesn’t help me figure out how much explanation is sufficient, which words match the register I’m aiming for, and which details will evoke the experience I’m trying to evoke in the reader.

The solution is to define my strategy before I start writing. The best way I know to write something good is to clarify what I’m trying to accomplish, keeping a sort of design brief in mind (or in a separate doc) as I write. If I’ve clearly defined what experience I’m trying to evoke and what elements of craft I want to deploy to get there, I can make those decisions more intentionally.

“Here’s what I’m going for here. I want these opening pages to be super suspenseful. I want the reader to immediately root for the protagonist, and the scene to be fast-paced and easy to visualize, nothing vague or abstract.”

The thing is, I don’t figure this stuff out by staring harder at the draft itself. So before I start writing, I have AI interview me about that too. The questions it asks force me to clarify my purpose and strategy for the piece before I even start drafting.

I might use a prompt like this:

I’m writing a short satire piece for The New Yorker Shouts & Murmurs. Before I draft, ask me a series of questions about my goals for the piece: who my reader is, what kind of experience I want to create for the reader, and what craft choices I plan to use to achieve the desired effect.

Fleshing out my writing goals through Socratic questioning has a dual benefit. First, it clarifies my own thinking, so I’m writing with purpose instead of brain-dumping whatever’s in my head. Second, it gives the AI a clear set of criteria. So when I ask it for feedback — “How’s this paragraph?” “Is this section working?” — Claude already knows what I’m trying to achieve. That way, it can assess how my writing succeeds against those specific standards rather than some abstract ideal of what good writing looks like. Instead of generic notes — “This is a little casual,” “consider tightening” — now I get feedback against my own criteria: “You said you wanted this to be visual, but you’re doing a lot of explaining.”

My 6-Week Course for Writers Who Actually Want to Finish

Socratic interviewing is just one piece of how I use AI in my writing process (and never to generate writing). If you’re interested in learning how to use AI not just as a thinking partner, but as an accountability coach to keep you on track with your writing, I’m teaching a six-week class that covers the whole system.

The class is called The Socratic AI Intensive, taught in partnership with Narratively Academy, and it starts May 18. It’s built around how to use AI as a taskmaster, project manager, and accountability coach to help you set a goal, stay on task, and actually finish what you start. All without ever letting AI generate a single word.The accountability coaching is just one piece of the Socratic AI system. The class also covers a set of advanced principles I’ve never seen taught anywhere else — the signature techniques of Socratic AI I’ve developed over three years of working with AI in my own writing process as a thinking partner instead of a content generator. Here’s a taste of what we cover:

* Getting non-sycophantic feedback. As I’ve said ad nauseam, I don’t use AI to generate prose. Nor do I allow AI to rewrite my prose when it inevitably tries to jump in and “smooth and refine” my drafts. But I do use AI as a thoughtful first reader to get an external perspective on how my writing is coming across. I ask for high-level feedback on how to make a piece better — what’s repetitive, what’s vague, what’s unclear, where it gets boring, where it needs to get more concrete. Then I do the rewrite myself based on feedback. The AI’s feedback is even more specific and useful if you’ve already gone through the Level 2 process I described above: defining for the AI what you’re trying to achieve and how.

* Name what you suspect the problem is. When asking for feedback, it also helps to let the AI know where to focus its attention. If you simply say, “How’s this section working?” the AI will do its best to be helpful. But things work even better if you flag your specific concern: “Is this paragraph sappy/melodramatic?” “Does this headline sound defensive?” “Is it clear why she didn’t just walk back in and apologize, or do I need to spell out her motivation?” When you tell the AI what to focus on, you’ll get a better, more targeted answer.

* Socratic revision. As William Zinsser says, “Writing is rewriting.” Writing never comes out quite right the first time. What makes good writing good is revising and rewriting early drafts until they get good. To flip AI into an iterative, Socratic mode, I use one magic phrase: “iterate and improve.” My favorite Socratic prompt: “Don’t rewrite this for me. Ask me questions to lead me to my own creative insights, and give me ideas so I can iterate and improve.” The AI gives you feedback. You’re always the one writing.

* Make every conversation Socratic. AI’s default mode is to spit out answers. But AI is more helpful when it has relevant context. The problem is the context you don’t realize you’re leaving out. So even when you’re not doing a Socratic interview, you can still flip the script. In practice, I try to make all my AI conversations Socratic. Here’s a single sentence you can paste at the end of any prompt: “Before we proceed, ask me a question or two to make sure you understand what I want.” One sentence, and Claude stops guessing what you need — it asks. It’s the easiest way to break out of the vending-machine pattern.

In the class, I teach all of these techniques in detail and show you how to use them for your own writing. Five class sessions over six weeks. We’ll cover project planning, daily accountability, Socratic interviewing, iterative revision with feedback, and advanced workflows for working with AI on your actual files and projects.

The structure: Five weekly live sessions of two hours each, a system for using AI as an accountability coach, and a private community for support between sessions. You bring your writing project: a novel, a book proposal, a screenplay, a collection of essays, whatever you’ve been wanting to finish. You set an ambitious but achievable goal. By the end of six weeks, you’re done.

Here’s what you get:

* A set of tools and techniques for how to use AI to set milestones, break your project into daily tasks, and keep you on track

* A cohort of fellow writers doing the work alongside you

* Weekly live sessions with me

* A Slack channel for questions, technical help, and shared progress

This is the class where you actually finish your draft. By the end of six weeks, the project you committed to is done.

It’s $795, and Wonder Tools readers get $100 off with code WONDERTOOLS. Sign up at socraticai.co/intensive.

Jay Dixit is the founder of Socratic AI, where he teaches writers, educators, and knowledge workers how to use AI as a thinking partner instead of a content generator. He was previously Head of Community for Writers at OpenAI and has taught writing at Yale. His journalism has appeared in The New York Times, Rolling Stone, and Psychology Today.

Thank you Lucy Gray, Andrew Nelson, and others for tuning into the live video with Jay Dixit!

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit wondertools.substack.com/subscribe