As regular readers are aware, I do a lot of informal lit review. So I was especially interested in checking out the various AI based “deep research” tools and seeing how they compare.

I did a side-by-side comparison, using the same prompt, of Perplexity Deep Research, Gemini Deep Research, ChatGPT-4o Deep Research, Elicit, and PaperQA.

General Impressions

The Deep Research bots are useful, but I wouldn’t consider them a replacement for my own lit reviews.

None of them produce really big lit reviews — they’re all typically capped at 40 sources. If I’m doing a “heavy” or exhaustive lit review, I’ll go a lot farther. (And, in fact, for the particular project I used as an example here, I intend to do a manual version to catch things that didn’t make it into the AI reports.)

On the other hand, a manual lit review can take me a really long time, and the AIs work a lot faster (2-15 minutes depending on the tool). It might be worth using Deep Research tools as a first step before doing my own in-depth version, or using them as a quick “better-than-nothing” option for times when I don’t expect to have time to do a real lit review.

I tend to think that even as AIs get better at “knowledge work”, there will still be value in learning things yourself (so you make connections and notice what you want to do next) and in seeking out information curated and interpreted by a particular human whose perspective you value. I don’t feel superfluous yet.

In general, I found that none of the Deep Research tools had a problem with overt hallucination (that is, making claims about the contents of sources that weren’t in the actual sources.)

Most of the variation was in completeness, relevance, and source quality — how many relevant answers to my question did they include, how much of the info I asked for was present, and how many of the sources were actual research papers rather than commercial or public-facing websites like WebMD.

The better tools (ChatGPT, Elicit, and PaperQA) took longer to complete compared to the weaker tools (Perplexity & Gemini) but since no tool took more than 20 minutes, I don’t think there’s much advantage in getting faster, lower-quality Deep Research reports.

Prompt

I used the exact same prompt for all Deep Research tools:

Which chronic or progressive diseases are known to have prodromes? By "prodrome", I mean a set of "preclinical", "subclinical", or "precursor" symptoms or clinical findings, which precede the onset of diagnosable disease by a year or more. For instance, Parkinson's disease has a prodrome that can precede diagnosis by twenty years, including loss of sense of smell, autonomic nervous system symptoms like constipation, and sleep disturbances like REM Sleep Behavior Disorder. What other diseases have (confirmed or suspected) prodromes or subclinical syndromes? What are the characteristic symptoms or findings? How often do the subclinical, preclinical, or prodromal symptoms progress to the diagnosable disease?

Perplexity Deep Research

Here’s the link to Perplexity’s results.

Completeness: C

Perplexity used 15 sources to come up with 8 prodromal conditions.

7/8 of those included information about the time between prodromal findings and disease diagnosis; none included information about the chance that a prodromal condition would progress to a disease diagnosis.

Relevance: C

Only 6/8 were examples of prodromes that occurred at least a year before diagnosis of a chronic disease.

Credibility: B

Not all the sources were published papers — they included popular sites like WebMD and Medical News Today — but all claims were correctly linked to the corresponding source.

Overall Grade: C+

Gemini Advanced Deep Research

Here’s the link to Gemini’s results.

Completeness: B-

Gemini used 38 sources to come up with 6 prodromal conditions.

5/6 of those included information about the time between prodromal findings and disease diagnosis; 3/6 included information about the chance that a prodromal condition would progress to a disease diagnosis.

Relevance: A

All 6 conditions were examples of prodromal findings occurring at least a year before diagnosis of a chronic disease.

Credibility: B-

Not all the sources were published papers — they included popular sites like WebMD, Medical News Today, Healthline, etc — and individual claims were not linked to sources, which made checking claims more difficult.

Overall Grade: B

ChatGPT-4o Deep Research

Here’s the link to ChatGPT’s results.

Completeness: A

ChatGPT used 33 sources to come up with 15 prodromal conditions.

All included information on time between prodromal findings and disease diagnosis, and all included information on the chance that a prodromal condition would progress to a disease diagnosis.

Relevance: A

All 15 conditions were examples of prodromal findings occurring at least a year before diagnosis of a chronic disease.

Credibility: A

Sources are research papers; claims are linked to specific sources.

Overall Grade: A

Elicit Research Report

Here’s the link to Elicit’s results.

Completeness: B+

Elicit used 40 sources to come up with 6 prodromal conditions, all neurological. All included information on the time between prodromal findings and disease diagnosis; 3/6 included information on the chance that a prodromal condition would progress to a disease diagnosis.

Relevance: A

All 6 conditions were examples of prodromal findings occurring at least a year before diagnosis of a chronic disease.

Credibility: A+

All sources were research papers. Elicit offers options to specify “inclusion criteria” about which papers should be included in a lit review; it also links claims to specific highlighted snippets in sources, to make spot-checking especially easy.

Overall Grade: A-

PaperQA

PaperQA is a beta product produced by FutureHouse; it doesn’t (yet) allow public sharing of query results.

Completeness: A-

PaperQA used 38 sources to come up with 11 prodromal conditions. 9/11 conditions included information on time between prodromal findings and disease diagnosis; 4/11 included information on the chance that a prodromal condition would progress to a disease diagnosis.

Relevance: A

All 11 conditions were examples of prodromal findings occurring at least a year before diagnosis of a chronic disease.

Credibility: A

Sources are research papers; claims are linked to specific sources.

Overall Grade: A

Final Thoughts: “Creativity”

If you do a Google Scholar search for “prodrome”, you’ll overwhelmingly see results on neurological and psychiatric conditions, because those are the contexts where people use that word to refer to precursor symptoms in a progressive disease. It’s standard to talk about a “schizophrenia prodrome.”

But I don’t care so much about the word “prodrome”; my interest is in finding chronic progressive diseases that are hard to treat by the time they’re diagnosed, where it might make more sense to study the prodromes to see if the underlying progressive dysfunction can be slowed/stopped.

ChatGPT Deep Research was the only AI “creative” enough to give examples of cancers here. It’s not normal to use the phrase “cancer prodrome”. But there are lots of types of precancerous abnormal cell proliferation that increase the risk of progressing to actual malignant cancers. Adematomatous polyps are a sort of “pre-colon cancer”, ductal carcinoma in situ is a “pre-breast cancer”, prostatic intraepithelial neoplasia are “pre-prostate cancers”, etc. There are lots of cancers where we routinely screen for abnormal cell proliferation starting in middle age, and might remove a precancerous growth to prevent it from progressing to cancer.

Now, that’s not a particularly useful idea for my particular application.1 But I was impressed by the ability of ChatGPT to look beyond standard keywords and understand the logic of the query. A fair number of human research assistants might not do that.

In general, I’ve found research tasks pretty hard to delegate — just because it’s a “generalist” skill doesn’t mean everyone literate can do it. If you routinely use research assistants, or if you have more things to research than you have time to look into, I would recommend using one of the better Deep Research tools pretty heavily.