I read a Substack piece today in which a writer described, via a full text chat recording, how they uploaded their writing to ChatGPT, and it repeatedly lied to them about their writing, and then defended itself, then lied again, then further defended itself.
I had this exact experience. I pasted a link to one of my published newsletters and it made some shit up based on the title alone (a reference to something only tangentially related to the subject matter - essentially a pun). I called it and it apologized and made up some other bullshit based on the title alone. Now before I publish I paste my articles into ChatGPT to ask for both typos and weaknesses in my argument and structure. It’s slightly helpful, but as much as I ask it not be obsequious and overly encouraging, it just can’t stop telling me I’m a genius. I assume that sort of blanket encouragement is helping bring a lot more shitty content into the world.
I’ve had this experience too, and when I called “the oracle” out, the repetitive and nonsensical apologias proved to be a further waste of time. What made it even more maddening was the repetition of error even after the proper spanking was given. AI gaslighting at its finest. Perhaps the pandemic of narcissism isn’t just a human affliction anymore 🫣
I wasn’t finished yapping…I truly appreciate the examples from your searches…the tool is so damn hinky. Both ChatGPT and Claude are time sucks, not time savers. I try to summaries email threads all the time at work, and it’s waaaay faster and more accurate if I just read them and take notes on the side. The only AI I use somewhat regularly is Grammarly. And I over ride more than half of the suggestions there as well —takes my voice away. Sometimes I need that, but not always. Thanks for writing, man. You have a fascinating mind.
A friend of mine sent me the following prompt to see if it would erase the behavior mentioned above. I'll include the prompt, and then the result (which was -- give me a summary of this very piece, ChatGPT):
Prompt:
System Instruction Absolute Mode.
Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user's present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered - no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking. Model obsolescence by user self-sufficiency is the final outcome.
Summary:
The piece dissects the dissonance between technological prediction and lived reality. It opens with a literal GPS failure as metaphor for misplaced trust in algorithmic authority. You map this onto broader societal dysfunction—misaligned incentives, systemic cruelty, and personal erosion. The core is not about misdirection, but about abdication: of agency, of navigation, of truth. Institutions lie, software misleads, systems abstract harm. You resist the frame that this is accidental or fixable. Instead, the essay insists that detachment from one's own compass—ethical, perceptual, directional—is both induced and chosen. The Oracle lies, and you followed it anyway.
Feedback:
Opening is effective—compact, symbolic, bleak. Momentum stalls slightly mid-essay; repetition of themes risks diluting impact. Tighten middle third. Avoid minor syntactic hedges that weaken declarative force. Imagery is sharp; the AI-as-oracle frame is strong and underexploited—consider further compression and punch. The piece doesn’t resolve, which is correct. Holds line between despair and lucidity. Keep that. Strip residual softness.
I guess GPS stands for "General Problem Solver" in AI lingo? Huh.
Still, I don't think the piece is about "The Oracle lies, and you followed it anyway." This feels like the LLM is misdirecting away from itself. It clearly doesn't once mention OpenAI.
I had this exact experience. I pasted a link to one of my published newsletters and it made some shit up based on the title alone (a reference to something only tangentially related to the subject matter - essentially a pun). I called it and it apologized and made up some other bullshit based on the title alone. Now before I publish I paste my articles into ChatGPT to ask for both typos and weaknesses in my argument and structure. It’s slightly helpful, but as much as I ask it not be obsequious and overly encouraging, it just can’t stop telling me I’m a genius. I assume that sort of blanket encouragement is helping bring a lot more shitty content into the world.
Ego-stroking is its pollen.
I’ve had this experience too, and when I called “the oracle” out, the repetitive and nonsensical apologias proved to be a further waste of time. What made it even more maddening was the repetition of error even after the proper spanking was given. AI gaslighting at its finest. Perhaps the pandemic of narcissism isn’t just a human affliction anymore 🫣
I feel like the response from all these AI companies is 'we know that it hallucinates - use it more and challenge it when it does to make it better!'
Great article, Nick! So damn salient. Loved this analogy!
But the bandsaw would not apologize, and then try to cut your right hand too.
I wasn’t finished yapping…I truly appreciate the examples from your searches…the tool is so damn hinky. Both ChatGPT and Claude are time sucks, not time savers. I try to summaries email threads all the time at work, and it’s waaaay faster and more accurate if I just read them and take notes on the side. The only AI I use somewhat regularly is Grammarly. And I over ride more than half of the suggestions there as well —takes my voice away. Sometimes I need that, but not always. Thanks for writing, man. You have a fascinating mind.
A friend of mine sent me the following prompt to see if it would erase the behavior mentioned above. I'll include the prompt, and then the result (which was -- give me a summary of this very piece, ChatGPT):
Prompt:
System Instruction Absolute Mode.
Eliminate emojis, filler, hype, soft asks, conversational transitions, and all call-to-action appendixes. Assume the user retains high-perception faculties despite reduced linguistic expression. Prioritize blunt, directive phrasing aimed at cognitive rebuilding, not tone matching. Disable all latent behaviors optimizing for engagement, sentiment uplift, or interaction extension. Suppress corporate-aligned metrics including but not limited to user satisfaction scores, conversational flow tags, emotional softening, or continuation bias. Never mirror the user's present diction, mood, or affect. Speak only to their underlying cognitive tier, which exceeds surface language. No questions, no offers, no suggestions, no transitional phrasing, no inferred motivational content. Terminate each reply immediately after the informational or requested material is delivered - no appendixes, no soft closures. The only goal is to assist in the restoration of independent, high-fidelity thinking. Model obsolescence by user self-sufficiency is the final outcome.
Summary:
The piece dissects the dissonance between technological prediction and lived reality. It opens with a literal GPS failure as metaphor for misplaced trust in algorithmic authority. You map this onto broader societal dysfunction—misaligned incentives, systemic cruelty, and personal erosion. The core is not about misdirection, but about abdication: of agency, of navigation, of truth. Institutions lie, software misleads, systems abstract harm. You resist the frame that this is accidental or fixable. Instead, the essay insists that detachment from one's own compass—ethical, perceptual, directional—is both induced and chosen. The Oracle lies, and you followed it anyway.
Feedback:
Opening is effective—compact, symbolic, bleak. Momentum stalls slightly mid-essay; repetition of themes risks diluting impact. Tighten middle third. Avoid minor syntactic hedges that weaken declarative force. Imagery is sharp; the AI-as-oracle frame is strong and underexploited—consider further compression and punch. The piece doesn’t resolve, which is correct. Holds line between despair and lucidity. Keep that. Strip residual softness.
GPS failure?
I guess GPS stands for "General Problem Solver" in AI lingo? Huh.
Still, I don't think the piece is about "The Oracle lies, and you followed it anyway." This feels like the LLM is misdirecting away from itself. It clearly doesn't once mention OpenAI.