LW Frontpage Experiments! (aka "Take the wheel, Shoggoth!")

RobertM

I am sceptical of recommender systems - I think they are kind of bound to end up in self reinforcing loops. I'd be more happy seeing a more transparent system - we have tags, upvotes, the works, so you could have something like a series of "suggested searches", e.g. the most common combinations of tags you've visited, that a user has a fast access to while also seeing what precisely is it that they're clicking on.

That said, I do trust this website of all things to acknowledge if things aren't going to plan and revert. If we fail to align this one small AI to our values, well, that's a valuable lesson.

[-]Tamsin Leake1y*186

I'm generally not a fan of increasing the amount of illegible selection effects.

On the privacy side, can lesswrong guarantee that, if I never click on Recommended, then recombee will never see an (even anonymized) trace of what I browse on lesswrong?

[-]Ruby1y90

Typo? Do you mean "click on Recommended"? I think the answer is no, in order to have recommendations for individuals (and everyone), they have browsing data.

1) LessWrong itself doesn't aim for a super high degree of infosec. I don't believe our data is sensitive to warrant large security overhead.
2) I trust Recombee with our data about as much as our trust ourselves to not have a security breach. Maybe actually I could imagine LessWrong being of more interest to someone or some group and getting attacked.

It might help to understand what your specific privacy concerns are.

[-]Tamsin Leake1y*112

I would feel better about this if there was something closer to (1) on which to discuss what is probably the most important topic in history (AI alignment). But noted.

[-]Ruby1y72

Over the years the idea of a closed forum for more sensitive discussion has been raised, but never seemed to quite make sense. Significant issues included:
- It seems really hard or impossible to make it secure from nation state attacks
- It seems that members would likely leak stuff (even if it's via their own devices not being adequately secure or what)

I'm thinking you can get some degree of inconvenience (and therefore delay), but hard to have large shared infrastructure that's that secure from attack.

[-]kave1y20

I am sad to see you getting so downvoted. I am glad you are bringing this perspective up in the comments.

[-]niplav1y136

A core value of LessWrong is to be timeless and not news-driven.

I do really like the simplicity and predictability of the Hacker News algorithm. More karma means more visibility, older means less visibility.

Our current goal is to produce a recommendations feed that both makes people feel like they're keeping up to date with what's new (something many people care about) and also suggest great reads from across LessWrong's entire archive.

I hope that we can avoid getting swallowed by Shoggoth for now by putting a lot of thought into our optimization targets

(Emphasis mine.)

Here's an idea^[1] for a straightforward(?) recommendation algorithm: Quantilize over all past LessWrong posts by using inflation-adjusted karma as a metric of quality.

The advantage is that this is dogfooding on some pretty robust theory. I think this isn't super compute-intensive, since the only thing one has to do is to compute the cumulative distribution function once a day (associating it with the post), and then inverse transform sampling from the CDF.

Recommending this way has the disadvantage of not being recency-favoring (which I personally like), and not personalized (which I also like).

By default, it also excludes posts below a certain karma threshold. That could be solved by exponentially tilting the distribution instead of cutting it off (, otherwise to be determined (experimentally?)). Such a recommendation algorithm wouldn't be as robust against very strong optimizers, but since we have some idea what high-karma LessWrong posts look like (& we're not dealing with a superintelligent adversary… yet), that shouldn't be a problem.

If I was more virtuous, I'd write a pull request instead of a comment. ↩︎

[-]dr_s1y30

Personalization is easy to achieve while keeping the algorithm transparent. Just rank your own viewed/commented posts by most frequent tags, then score past posts based on the tags and pick a quantile based on the mixed upvotes/tags score, possibly with a slider parameter that allows you to adjust which of the two things you want to matter most.

[-]the gears to ascension1y1011

Disappointing to see this is the approach y'all are taking to making ai tools for the site, but I guess it does make sense that you'd want to outsource it. I'd strongly appreciate a way to opt out of having my data sent off-site for this or any future reason.

[-]habryka1y51

I am pretty excited about doing something more in-house, but it's much easier to get data about how promising this direction is by using some third-party services that already have all the infrastructure.

If it turns out to be a core part of LW, it makes more sense to in-house it. It's also really valuable to have an relatively validated baseline to compare things to.

There are a bunch of third-party services we couldn't really replace that we send user data to. Hex.tech as our analytics dashboard service. Google Analytics for basic user behavior and patterns. A bunch of AWS services. Implementing the functionality of all of that ourselves, or putting a bunch of effort into anonymizing the data is not impossible, but seems pretty hard, and Recombee seems about par for the degree to which I trust them to not do anything with that data themselves.

[-]the gears to ascension1y20

I'd like to opt out of all analytics. I believe the GDPR requires you to implement this?

[-]habryka1y20

GDPR is a giant mess, so it's pretty unclear what it requires us to implement. My current understanding is that it just requires us to tell you that we are collecting analytics data if you are from the EU.

And the kind of stuff we are sending over to Recombee would be covered by it being data necessary to provide site functionality, not just analytics, so wouldn't be covered by that (if you want to avoid data being sent to Google Analytics in-particular, you can do that by just blocking the GA script in uBlock origin or whatever other adblocker you use, which it should do by default).

[-]the gears to ascension1y4-14

drat, I was hoping that one would work. oh well. yes, I use ublock, as should everyone. Have you considered simply not having analytics at all :P I feel like it would be nice to do the thing that everyone ought to do anyway since you're in charge. If I was running a website I'd simply not use analytics.

back to the topic at hand, I think you should just make a vector embedding of all posts and show a HuMAP layout of it on the homepage. that would be fun and not require sending data anywhere. you could show the topic islands and stuff.

[-]jefftk1y42

If I was running a website I'd simply not use analytics.

My bet is if you were running a website like this you'd see how useful analytics are for making complex websites better.

[-]the gears to ascension1y4-2

I have been employed making websites like this many times before. The analytics were extremely useful. It would have been much harder without it. Also, I have come to the opinion that one should not use analytics despite this.

[-]the gears to ascension1y86

I'd like a pure-recommender view, so that I can easily tell when I am looking at recommender vs frontpage posts. I would use that more than a mixed view.

[-]Ruby1y30

It's the plan to have that live, only reason we didn't deploy it on Thursday was we have to do a small bit of extra work to extend caching (to achieve acceptable performance) to the pure-recommender view. Probably have it up soon.

[-]habryka1y30

Oh, yeah, admins currently have access to a purely recommended view, and I prefer it. I would be in favor of making that accessible to users (maybe behind a beta flag, or maybe not, depending on uptake).

[-]Ruby1y20

See comment.

[-]Ruby1y5-1

We've had the choice of tabs up for a month now and the results so far are encouraging, or at least not discouraging. There are many users who are very pleased with the Recommendations, liking among other things that it brings to attention posts that otherwise get lost if you only see what's new. Clickthrough-rates are higher for people using the Enriched/Recommendations tab, although this is most certainly a selection effect on the kind of user who changes tab at all. Switching some people over automatically is motivated by wanting to get a better signal here before doing something like changing the global default.

The current recommendations still needs more work though. People are much less likely to click on recommendations of posts that they've already clicked on, but it's proving tricky to eliminate such recommendation entirely. Also the algorithm overwhelmingly recommends posts from the last year when we'd like to see it surfacing stuff from further back too. Still, Latest is overwhelming stuff from the Last week, so it's still an improvement over the counterfactual.

--

From when we started the project, we've settled on the "hybrid" list being likely optimal as the default list people look at. Many people want to "keep up with the latest" even if they're also interested in good posts from all time, so any recommended list of posts that's the default has to have a heavy latest component. We first tried making two calls to the Recommendations API, one with heavy recency bias, but it was hard to get it consisted, so we switched to just splitting the list between the usual Latest algorithm and new recommendations algorithm.

This has the advantage that is preserves some of the "common knowledge" aspect of the current algorithm where you know which posts other people are seeing too, and an author knows that if they get upvoted, their post will be visible automatically and transparently to many people. As discussed elsethread on this post, we want to have a pure-recommendations tab as well and have been waiting on a bit of coding to make that happen.

--

People often have the fear of goodharting on the wrong metric (like clicks) for recommendation algorithms. I think we do need to keep an eye on that, and I want to build more analytics tools for detecting drift here, and more talking to people. I think as we fix up more basic issues like excluding read content and getting it to even recommend posts from older than a year ago^[1], we'll put more attention on is the trend good.

^{^}
One guess I have is the algorithm is stuck for dumb "structural" reasons, in that it's been given recent data which is overwhelmingly of people reading recent content, so when it queries "what's good?" recent content comes out on top even without explicitly training that into the system.

[-]niplav1y40

I realized I hadn't given feedback on the actual results of the recommendation algorithm. Rating the recommendations I've gotten (from -10 to 10, 10 is best):

[-]Ruby1y41

I'd be interested in a comparison with the Latest tab.

[-]niplav1y20

[-]Jeremy Gillen1y20

I'm enjoying having old posts recommended to me. I like the enriched tab.

[-]momom21y10

When I vote, I basically know the full effect this has on what is shown to other users or to myself.

Mindblowing moment: It has been a private pet peeve of mine that it was very unclear what policy I should follow for voting.

In practice, I vote mostly on vibes (and expect most people to), but given my own practices for browsing LW, I also considered alternative approaches.
- Voting in order to assign a specific score (weighted for inflation by time and author) to the post. Related uses: comparing karma of articles, finding desirable articles on a given topic.
- Voting in order to match an equivalent-value article. Related uses: same; perhaps effective as a community norm but more effortful.
- Voting up if the article is good, down if it's bad (after memetic/community/bias considerations) (regardless of current karma). Related uses: karma as indicator of community opinion.

In the end, making my votes consistent turned out to be too much effort in every case for extensive calculations, which is why I came back to vibes, amended by implicit considerations of consistent ways to vote.

I was trying to figure out ways to vote which would put me in a class of voters that marginally improved my personal browsing experience.
It never occurred to me to model the impact it would have on others and to optimize for their experience.
This sounds like an obviously better way to vote.

So for anyone who was in the same case as me, please optimize for others' browsing experience (or your own) directly rather than overcalculate decision-theoretic whatevers.

[-][anonymous]1y10

Suggestion: A marker for recommended posts which are over x duration old. I was just reading this post which was recommended to me, and got half-way through before seeing it was 2 years out of date :(

https://www.lesswrong.com/posts/3S4nyoNEEuvNsbXt8/common-misconceptions-about-openai

(Or maybe it's unnecessary and I'll get used to checking post dates on the algorithmic frontpage)

[-]habryka1y20

We are experimenting with bolding the date on posts that are new and leaving it thinner on posts that are old, though feedback so far hasn't been super great.

^{^}

This is mostly about enabling recommendations for logged-out users requiring some more technical work.

^{^}

Since the dawn of LessWrong 2.0, posts on the frontpage have been sorted according to the HackerNews algorithm:

Each posts is assigned a score that's a function of how much karma it was and how it old is, with posts discounted over time. In the last few years, we've enabled customization by allowing users to manually boost or penalize the karma of posts in this algorithm based on tag. The site has default tag modifiers to boost Rationality and World Modeling content (introduced when it seemed like AI content was going to eat everything).

^{^}

We have Recent Discussion which is a pure chronological feed of posting and commenting activity that I find its a bit too much of a firehose with lots of low-interest stuff, so I don't look at it much.

^{^}

Since trying out the "subscribe to user's comments" feature that we shipped recently, I've found this to be an interesting way to discover posts to read. I'm motivated to read things people I like are discussing.

^{^}

For now, the tabs are only visible to logged-in users, though the frontpage redesign has been rolled out to everyone. Logged-out users see the contents of the "Latest" tab (which is what the previous frontpage showed under the "Latest Posts" section).

LESSWRONG
LW

LESSWRONG
LW

71

LW Frontpage Experiments! (aka "Take the wheel, Shoggoth!")

71

71

Update: June 20th

Update: May 13th

Original Post, April 22nd

Why algorithmic recommendations?

The Concrete Experiment

Preventing Bad Outcomes

Further cost of a recommendations (common knowledge, Schelling discussion, author incentive)

Thoughts?