This sounds really cool and I intend to play around with it when I have time in a few days. First piece of feedback: I think it would be good to have easy to click examples on the welcome page, so we can get a feel for it without having to pick a post to test.
Also, a connected force graph (like in connected paper) would be appreciated.
Do you intend to make the corpus available / to make the code open source? If you put it on a public github I predict you might get some interesting contributions.
I think it would be good to have easy to click examples on the welcome page, so we can get a feel for it without having to pick a post to test.
Done! Chose some high-karma posts that seemed particularly interesting for now.
I do intend to open source the code, and I'll definitely try out the connected force graph.
For a while now, I've wanted something like Connected Papers for LessWrong & the broader rationalist community.
I built a working prototype that has, so far, been pretty useful to me. You can find it at: situate.info/rational
Usage
You give it a LessWrong or Substack URL and receive a list of the top 100 similar documents from the database.
Notes:
How it works
I scraped ~372k documents:
Each document was split into ~360-token chunks (heading-aware, sliding window, or whole-doc for short pieces), embedded with Voyage-3.5 (1024-d), then averaged into a single doc vector. Similarity is calculated via cosine distance for a total of 100 matches per query.
What's next
I'm excited to receive feedback/ideas from you all, for further iteration.
I have a personal setup of this tool that allows for open-ended custom queries (embedded on-demand), and BGE reranking for slightly improved accuracy. At the moment, this would be too expensive/complicated for me to implement at scale, but I do currently plan to build a polished setup for the ingestion -> chunking & embedding -> search pipeline, that I can open source for others to use.
A semantic search on LessWrong alone would probably be an easy project for the Lightcone team, so I expect that specific function to eventually become obsolete. However, I plan to keep ingesting information from many different corners of the internet, so I think building an effective third party system is still useful.
In general, I'm interested in working on interfaces that improve A) individual navigation and synthesis of information and B) collective epistemics (or both at the same time). I see it as a promising direction, especially amidst the many intense efforts to raise x-risk awareness that are likely to get somewhat tangled up in complicated debates. I will be sharing more ideas soon.
Also, I think it's really important that talented newcomers to AI safety "situate" new posts in relevant past context. This is a broad and rapidly-changing field of thought, with no standardized curriculum to get people up to speed. Many of the cutting edge discussions on this forum happen within a small bubble of longstanding members, and it takes a while before newcomers could comfortably follow along. I think it's worth investing in speeding up that process for any potential increase in high-quality/strategic thinking.
Any broad advice/direction is appreciated too!