Sorry for the confusion. The things I listed are not programming libraries, but text editor extensions. VSCode is a very fancy text editor, and the extensions I listed are part of the Foam ecosystem, which allows nice ordering of notes and navigation across them. So if you're interested, I'd suggest to check out those, as well as the recommended extensions list. May be helpful for your problem by itself.
Do you plan on developing a version of this which could be downloaded and used locally on one's own computer, or does it require too much computing power for that to be viable?
Partially, yes. Transferring the whole tool into a usable portable app is a lot of troubles, especially the visualization is not so transferable now. So, I mainly think about publishing a piece of code that could be integrated into some existing note systems. It would be relatively easy to extract the part that goes through all documents and runs the clustering and embedding. Then, let's say, it could add an yaml heading with tags to each of the notes in... (read more)
Thanks! I can upload the table with cluster and tag labels per post if needed. From this mapping, answers should be relatively straightforward.
TL;DR We made a tool that allows you to interactively explore LessWrong, EA Forum and - potentially - other large collections of documents. It seems like this part of the Information Retrieval field is highly underutilized in practice, so I listed some ideas on where it could be used, including a project on AI Governance research. Some of those I already tried, and it looks promising. Please, let me know if you have ideas on how to apply this to some other fields, especially if you'd be personally interested in doing this.
While doing my PhD and generally looking for information on areas that I don't understand, I've met the problem that the... (read 2730 more words →)
Great job! I'm going to add this functionality, probably the next couple of weeks. But that will work only for individual articles. If you want to do this massively, I can run the script on your spreadsheet and send you the results. Would you be interested?
Regarding posting it on the EA forum, I want to run some investigation to show insights, relevant to EA. But that requires time, so I postpone posting...