TL;DR We made a tool that allows you to interactively explore LessWrong, EA Forum and - potentially - other large collections of documents. It seems like this part of the Information Retrieval field is highly underutilized in practice, so I listed some ideas on where it could be used, including a project on AI Governance research. Some of those I already tried, and it looks promising. Please, let me know if you have ideas on how to apply this to some other fields, especially if you'd be personally interested in doing this.

Intro

While doing my PhD and generally looking for information on areas that I don't understand, I've met the problem that the normal search engines give a very small subset of the information on the topic. This leaves you very little idea of what you miss and even how much missing information is there in total. So, together with Denis Skripov (software development), Elena Plekhanova (ideation part) and Alexander Fedorenko (both) we created a tool for building an overview of large collections of documents. We applied this tool to LessWrong and EA Forum.

The goal of this post is to share the visualizations, to describe the ideas behind the tool, its possible applications, and, hopefully, to gather some feedback. Lots of research on similar topics have been done, so I expect that some of the ideas listed here reinvent the wheel or are known dead-ends. If you have some info on that, please share it in the comments or in PM.

If you're only interested in testing the tool, jump straight to the "Instructions" and "Demonstration on LessWrong" sections. In "Motivation and the Concept description" I give a summary of my views on this part of exploratory search. "Methods" describe the algorithms we used. And in the "Further Applications" I describe my estimates on where these ideas could be applied. Any feedback is highly appreciated, but the "Input you could provide" section gives some details on what would be the most valuable.

Please, keep in mind that the tool is only a proof of concept and that every aspect of it can be improved.

Motivation and the Concept description

Imagine you're a graduate student who just started working in a new field and wants to make some sense of it. Or you just want to learn some new big topic, such as "rationality techniques". There are two general ways to approach the problem. One way, you may start with what you alread...