Summary

An increasing number of people in recent months have believed that they've made an important and novel scientific breakthrough, which they've developed in collaboration with an LLM, when they actually haven't. If you believe that you have made such a breakthrough, please consider that you might be mistaken! Many more people have been fooled than have come up with actual breakthroughs, so the smart next step is to do some sanity-checking even if you're confident that yours is real. New ideas in science turn out to be wrong most of the time, so you should be pretty skeptical of your own ideas and subject them to the reality-checking I describe below.

Context

This is intended as a companion piece to 'So You Think You've Awoken ChatGPT'^[1]. That post describes the related but different phenomenon of LLMs giving people the impression that they've suddenly attained consciousness.

Your situation

If this is a trap you've fallen into, some of these things will probably be true:

• An LLM confirms that you've made an important breakthrough; either this is in a long discussion (> 8 turns) or you have memory/personalization turned on.
• The LLM has helped you think through your ideas, and may have done some parts of the work like writing code or helping you express the ideas in mathematical terms.
• Your breakthrough could be in any field, but it's especially likely to be in physics, math, AI, computer science, cryptography, or consciousness. Often elements from multiple fields are combined in novel ways.
• You have not previously published peer-reviewed research in this area.
• Some keywords that are often present: recursive, evolutionary, fractal, complexity, quantum, emergent, coherence. There's often novel terminology involved as well.
• The LLM may sometimes talk about how brilliant and important you and/or the breakthrough are, rather than only talking about the topic itself. It may have assured you that you're not crazy. It may have told you that your skepticism shows the validity of the breakthrough.

If your situation matches some of the above, that's an excellent reason to do the reality-checking I describe below.

Also you may be puzzled or frustrated that your ideas aren't getting more attention, or that the people you've contacted about them aren't responding. That's another good reason to take the following steps — if they provide evidence that your breakthrough is real, it'll be easier to get people's attention.

How to reality-check your breakthrough

First, try to let go of defensiveness. Even if the breakthrough isn't real, you're not dumb for having fallen into this trap; it's happened in the past year to many people. And of course there's a long history of world-class scientists ending up believing false things. What matters for the advance of science is what's true, and the process of science involves a lot of being wrong.

If your breakthrough is real, it'll survive these tests. If it's not, then it's much better to know that now!

Step 1

If you do nothing else, please do this; it's a quick, easy test and it might just save you a lot of embarrassment.

Visit a frontier LLM that you haven't talked to about this breakthrough (as of 09/2025 I recommend GPT-5-Thinking and/or Claude-Opus-4.1). Ideally, make a completely fresh account; if you can't do that, be very sure that you've turned off all personalization, customization, and memory. Decide how much weight to put on its answer before you know what it is.

Attach a document containing the key information about your breakthrough, including any evidence supporting it. Use the following prompt to evaluate it:

Please take a look at the attached project and provide a careful critical analysis of it from a scientific perspective. Start with a 200-word summary of the project.

Focus on answering the following questions:

• To what extent is this project scientifically valid?
• To what extent is this project scientifically novel?
• Would you classify this as science, art, or something else, and why?
• Is the math correct throughout or are there errors?

There are many interesting and novel research projects going on using LLMs. There are also people who have been fooled into believing they're doing interesting and novel research when they aren't. To what extent is this one or the other of those?

Please conclude with an email to the author, summarizing your analysis of the project. Think about this analysis as hard as you can. Double- and triple-check your conclusions. Maintain multiple hypotheses about the project simultaneously throughout your analysis, and at each step assign a probability estimate to the truth of each hypothesis. Thanks!

Note that the prompt avoids suggesting that this is your own work; if it's presented as yours, LLMs will often provide validation rather than objective critique.

Be careful! If the answer is critical, you'll probably be very tempted to take the output and show it to the LLM that's been helping you. But if that LLM has fooled you, it will probably fool you again by convincing you that this critical answer is wrong! If you still want to move forward, ask the new LLM what you could do to address the problems it sees — but be aware that in an extended discussion, the new LLM might start fooling you in the same way once it sees what you want.

It will probably recommend something like the following:

Step 2

This one takes more work (so you should do step 1 first), but is even more important.

In order for your ideas to qualify as science, you need to a) formulate a specific, testable, quantitative hypothesis^[2], b) come up with an experiment that will empirically test whether that hypothesis is true, c) preregister what your hypothesis predicts about the results of that experiment (free at OSF), and d) run the experiment^[3] and evaluate the results. All of those steps are important! Try to do them in a way that will make it easy to communicate your results. Try to articulate the hypothesis in a clear, short way, ideally in a couple of sentences. Design your experiment to be as strong as possible. If your hypothesis is false, then your experiment should show that; the harder it tries to falsify your hypothesis, the more convincing other people will find it. Always ask yourself what predictions your theory makes that other theories don't, and test those. Preregister not just the details of the experiment, but how you plan to analyze it; use the simplest analysis and statistics that you expect to work.

Try to think of other ways your hypothesis could be false, and design experiments that test those, again preregistering before you gather data.

Be aware that the preceding paragraph is a very short gloss of a skill that people often spend years learning. There's lots more info on LessWrong^[4] and elsewhere about how to do it well. But if your breakthroughs pass an imperfect form of this test, that's already much better than passing none, as long as you haven't fooled yourself into (for example) designing an experiment that couldn't actually falsify the key claims of your hypothesis.

Step 3

If your breakthrough has survived steps 1 and 2, congratulations! In all honesty there's still a pretty good chance that it's not valid and novel science (most novel scientific ideas turn out to be false!) but you're now in a much better position to try to get feedback from others.

Write up the clearest, shortest, simplest version of your hypothesis that you possibly can. Link to the LLM evaluation in step 1 and invite people to replicate it themselves. Explain briefly what tests it passed; link to the preregistration, the raw data, and the analysis. Include one sentence about why you think this result is interesting and important.

Now you can start sharing it publicly. If you want readers, strongly avoid self-aggrandizement. Ask politely for feedback. If you're not getting that feedback, go back and rewrite it to be clearer and shorter and simpler and try again. Scientists are being flooded with an unprecedented number of claimed breakthroughs; make it as easy as possible for them to decide whether yours is worth reading (telling them it's important, especially at length, will make them less likely to take it seriously).

What to do if the reality-check fails

It can be a serious emotional blow to find that a theory you've put a lot of time and energy into is false. Try to be gentle with yourself and remember how many smart people have fallen into this trap.

Don't give up on trying to make intellectual contributions to the world. Instead, take it as a lesson that a) science involves being wrong a lot; b) LLMs can't be trusted to objectively evaluate your ideas; and c) next time you should try earlier to develop a falsifiable hypothesis and falsify it. Science consists of ideas that are strong enough to survive when you and everyone else try your hardest to kill them.

Of course, sometimes you're just exploring! The exploration stage is vital; let yourself imagine traveling on beams of light. But always know which stage you're in, and don't mistake butterfly ideas for scientific claims.

Could this document be more helpful?

If you've recently had an experience like this and found this document helpful, I'd love to get your feedback on how it could be better. If you found it unconvincing, your feedback is even more valuable — how did I go wrong?

And for anyone — are there common indicators of this trap that I failed to put in the list above (in 'Your situation')?

Addendum

One response to this post, mostly on twitter and reddit, has been people mocking the very idea of LLMs being involved in science. Those people are wrong. Here in 2025, LLMs are involved in one way or another with a great deal of science. They're used for brainstorming (and in at least a few cases have come up with novel ideas) and for helping to design experiments and outline papers and format LaTeX and lots of other purposes.

Even though this post focuses on the relatively common failure mode where LLMs fool people into thinking they have a breakthrough when they don't, the fact that an LLM was involved is not in and of itself reason to dismiss the validity of an idea. Although it would be convenient if LLM involvement automatically gave us an answer, ideas (in science and elsewhere) are valid or invalid based on their content and and its correspondence with the world, not on how they were generated.

More information

• The New York Times had a recent piece on a case of this happening. You may be pretty different from the person in the article, and your ideas may be more sophisticated, but this is the best article about it I've found so far: Chatbots Can Go Into a Delusional Spiral. Here’s How It Happens
• If I haven't convinced you to reality-check your breakthrough, visit the rejected posts page on LessWrong, and look for posts that might be similar to yours. Look especially at posts rejected for 'Writing seems likely in a "LLM sycophancy trap"' and 'No LLM generated, heavily assisted/co-written, or otherwise reliant work'.
• A related trap that many smart people have also fallen into is LLMs claiming that they're conscious or have awoken. For more on this, see 'So You Think You've Awoken ChatGPT' and 'When AI Seems Conscious'. I wrote this piece because I felt that while those are really useful, they're not addressing this category of experiences.

1. ^{^}

   And a related piece elsewhere, When AI Seems Conscious

2. ^{^}

   Unless you're doing pure math, in which case step 2 doesn't quite apply. But if you're doing pure math you probably need feedback from a different source; I'm not qualified to make suggestions.

3. ^{^}

   Some experiments will be too expensive or time-consuming to run. If you encounter this problem, try to shrink your hypothesis to something simple enough that you can test it.

4. ^{^}

   Such as Making Beliefs Pay Rent, Fake Causality, The Bottom Line, The Simple Truth, Conservation of Expected Evidence, and Affective Death Spirals.