Midjourney: “an artificially intelligent researcher, library, posthuman archivist, mapping the noosphere”

As regular readers are aware, I do a lot of informal lit review. So I was especially interested in checking out the various AI based “deep research” tools and seeing how they compare.

I did a side-by-side comparison, using the same prompt, of Perplexity Deep Research, Gemini Deep Research, ChatGPT-4o Deep Research, Elicit, and PaperQA.

General Impressions

The Deep Research bots are useful, but I wouldn’t consider them a replacement for my own lit reviews.

None of them produce really big lit reviews — they’re all typically capped at 40 sources. If I’m doing a “heavy” or exhaustive lit review, I’ll go a lot farther. (And, in fact, for the particular project I used as an example here, I intend to do a manual version to catch things that didn’t make it into the AI reports.)

On the other hand, a manual lit review can take me a really long time, and the AIs work a lot faster (2-15 minutes depending on the tool). It might be worth using Deep Research tools as a first step before doing my own in-depth version, or using them as a quick “better-than-nothing” option for times when I don’t expect to have time to do a real lit review.

I tend to think that even as AIs get better at “knowledge work”, there will still be value in learning things yourself (so you make connections and notice what you want to do next) and in seeking out information curated and interpreted by a particular human whose perspective you value. I don’t feel superfluous yet.

In general, I found that none of the Deep Research tools had a problem with overt hallucination (that is, making claims about the contents of sources that weren’t in the actual sources.)

Most of the variation was in completeness, relevance, and source quality — how many relevant answers to my question did they include, how much of the info I asked for was present, and how many of the sources were actual research papers rather than commercial or public-facing websites like WebMD.

The better tools (ChatGPT, Elicit, and PaperQA) took longer to complete compared to the weaker tools (Perplexity & Gemini) but since no tool took more than 20 minutes, I don’t think there’s much advantage in getting faster, lower-quality Deep Research reports.

Prompt

I used the exact same prompt for all Deep Research tools:

Which chronic or progressive d

...

AI Products/Tools