#33 | When AI goes off the rails

TL;DR: Oh, brilliant! AI loses 25% of its capability when you chat normally. Here’s how to work with AI’s actual behavior efficiently.

Hi :),

Imagine you’re having a normal conversation with ChatGPT.

First response? Brilliant. Exactly what you needed.

Second exchange? Still solid.

By turn four, something shifts. AI starts suggesting things you already rejected, going off on weird tangents, or acting like you’re discussing something completely different.

Next thing you know, you’re bubbling your lips like a horse in pure disbelief.

But as it turns out, AI was just… being itself.

Sound familiar?

I bet you’ve experienced this pattern:

AI starts strong but then gets progressively weirder. You end up repeating yourself multiple times, and responses get wordier and less helpful as conversations continue.

There has likely been a moment when you just thought, “Screw this, I’ll start a fresh chat.”

You may think you’re speaking to someone who is slowly tuning out. Someone who nods along but clearly missed the plot three exchanges ago.

So, what’s happening here?

​Researchers​ finally tested what we all experience daily with ChatGPT & Co. (Microsoft study: LLMs get lost in multi-turn conversation)

They took tasks that work perfectly in single prompts and split them into natural, multi-turn conversations.

You know, the way real humans actually talk to AI.

The results? AI performance drops 25% when you chat normally instead of giving perfect, comprehensive, and complete instructions upfront.

From 90% accuracy down to 65%. A quarter of all capability… gone.

This happened with every model they tested: Perplexity, GPT, Claude, Copilot, Gemini—the whole gang.

And the patterns are boringly predictable:

• AI becomes verbose and unfocused
• Jumps to conclusions before you’re done explaining
• Makes incorrect assumptions about missing details
• Struggles to recover after early misunderstandings

Everything we’ve been frustrated about? It’s now documented behavior.

To top things off, the study used controlled “sharded instructions,” which means real conversations are probably even messier (Great!).

Recap: Users often experience performance drops when chatting naturally. Better performance occurs with clear, complete instructions upfront. ​See summary here​.

Being honest about the results + fixes

Before you get excited about “fixing” this, a few thoughts:

Strategic fixes might feel unnatural. You’ll have to change how you naturally communicate with AI. Some days, that won’t feel worth it.

AI will keep improving eventually. In my opinion, all limitations are temporary. Time spent optimizing could be better invested elsewhere. (Though nobody knows when “eventually” arrives.)

Competitive advantage? Also temporary. Once everyone learns these tricks, you’re back to baseline. We’re talking months, not years, before this becomes standard.

Most sophisticated users already know this stuff. If you’re reading AI newsletters, you probably recognize some of these patterns already.

Anyway, we can still look at some workarounds, can’t we?

Working with AI’s documented patterns

Research suggests these approaches may help—when effort’s worth it:

1. Front-load your context. Give AI everything upfront instead of revealing details incrementally. Performance can degrade as conversations get longer.

2. Use strategic resets. Know when starting fresh is better than fighting context drift. A new chat may save more time than explaining to the AI to stick to that silly rule for the third time.

3. Catch premature conclusions. Tell AI explicitly: “Don’t respond yet—I have more context coming.” Stop it from jumping to answers like an eager intern.

4. Watch for verbosity creep. Long, unfocused responses signal a conversation is derailing.

5. Calibrate expectations. Different task types fail differently. Creative brainstorming handles multi-turn chats better than analytical problem-solving.

Hands-on test: I’ve tested front-loading context across dozens of conversations.

Success rates improve, but need 30-50% more upfront thought. This is worthwhile for complex projects but likely excessive for quick questions.

What does this mean strategically?

Some conversation failures might be fundamental to how current AI models work.

Thus, we’re designing around AI’s nature—temporarily.

Strategic implications:

• If you’re training teams, front-loading context becomes a priority skill.
• If you’re building client workflows, strategic resets prevent embarrassing conversations that go nowhere.

Furthermore, if you choose AI tools, conversation memory features (ChatGPT is good here) matter more than raw capabilities.

However, you could also raise the following counterargument:

Why optimize for AI limitations when you could focus on higher-value activities or wait for better models?

Valid point.

My take: Try one technique for a week. If it saves frustration and time, keep it. If not, wait for AI to improve. ​Well, Claude 4 is here!​ Yay!

Your next move

AI conversation failures aren’t disappearing anytime soon.

Knowing why they happen changes every interaction—slightly.

Try this:

• Front-load context on your next complex AI request instead of building it piece by piece.
• Give this initial request also a name, e.g., Base1.
• Add reminders to bring the AI back on track, e.g., ‘Reload/Review Base1’

Notice the difference. You’ll probably save some frustration.

Whatever you decide to do, at least now you know why those chats go sideways sometimes or often.

Also, most people are still figuring this out. Understanding the flaws gives you a slight edge, for now.

Happy Friday,

Mark
The AI Learning Guy
👋⚡😎

P.S. Got your own weird AI conversation patterns? Hit reply. I’m collecting stories for future deep dives.

Interesting reads and books

1. ​LLMs get lost in conversation​
2. ​Study summary – Key points​
3. ​Introducing Claude 4​
4. Best AI Books 2025. ​View on Amazon*​.

Note: No single website has all the answers. This list serves as a starting point for those who want to explore or satisfy their curiosity about AI. Links: Links with * are affiliate links. See disclosure below.

​Affiliate disclosure: To cover the cost of my email software and the time I spend writing these emails, I sometimes link to products. Please assume these links are affiliate links. If you choose to buy through my links, a big THANK YOU – it will make it possible for me to keep doing this.