I was recently surprised to notice that Anthropic doesn't seem to have a commitment to publish its safety research.^[1] It has a huge safety team but has only published ~5 papers this year (plus an evals report). So probably it has lots of safety research it's not publishing. E.g. my impression is that it's not publishing its scalable oversight and especially adversarial robustness and inference-time misuse prevention research.

Not-publishing-safety-research is consistent with Anthropic prioritizing the win the race goal over the help all labs improve s... (read more)

Do counterfactual mugging scenarios violate the 2nd Kolmgorov Axiom, or Normalization?

I'm thinking about, eg, this situation:
Imagine a superintelligence (like a version of Pascal’s demon) approaches you and tells you it has flipped a coin. If the coin landed heads, the superintelligence would never interact with you. If it landed tails, it will ask you to give $100. If you agree to give $100 in the tails scenario, the superintelligence will retroactively reward you with $10,000 if the coin landed heads.

There are 4 possible outcomes, right? {Heads+Would Pay, Heads+Would Not Pay, Tails+Would Pay, Tails+Would Not Pay} To obey normalization, P(H+WP, H+WNP, T+WP, T+WNT) = 1, right?

Any n-arity function can be simulated with an an (n+1)-arity predicate. Let a and b be constants. With a function, we can write the FOL sentence $a+b>1$, where $+$ is the default addition function. We can write the same as $\forall x,\left({+}_P\right(a,b,x)\to x>1)$ where ${+}_P$ is now a predicate that returns true iff $a$ added to $b$ is $x$.

The current state of the art for salary negotiations is really bad. It rewards disagreeableness, stubornness and social skills, and is just so unelegant.

Here's a better way of doing salary negotiation:

Procedure via a two-sided sealed-bid auction, splitting the difference in bids^[1]:

• Normal interviewing happens.
• Job-seeker decides on their minimum acceptable rate $r_{\text{min}}$.
• Employee-seeker decides on the maximum acceptable payment $r_{\text{max}}$.
• Both reveal $r_{\text{min}}$ and $r_{\text{max}}$, either first through hashsums of the numbers (with random text appended) and then the cleartext, or

Has anyone here investigated before if washing vegetables/fruits is worth it? Until recently I never washed my vegetables, because I classified that as a bullshit value claim.

Intuitively, if I am otherwise also not super hygienic (like washing my hands before eating) it doesn't seem that plausible to me that vegetables are where I am going to get infected from other people having touched the carrots etc... . Being in quarantine during a pandemic might be an exception, but then again I don't know if I am going to get rid of viruses if I am just lazily rinsi... (read more)

Billionaires read LessWrong. I have personally had two reach out to me after a viral blog post I made back in December of last year.

The way this works is almost always that someone the billionaire knows will send them an interesting post and they will read it.

Several of the people I've mentioned this to seemed surprised by it, so I thought it might be valuable information for others.

Linkpost: "Against dynamic consistency: Why not time-slice rationality?"

This got too long for a "quick take," but also isn't polished enough for a top-level post. So onto my blog it goes.

I’ve been skeptical for a while of updateless decision theory, diachronic Dutch books, and dynamic consistency as a rational requirement. I think Hedden's (2015) notion of time-slice rationality nicely grounds the cluster of intuitions behind this skepticism.

I don’t think we are that far off from turning a book into a movie using AI. I just read H. P. Lovecraft’s At the Mountain of Madness, which is about discovering the remains of an ancient alien civilization in Antartica. It would make an amazing movie, and Guillermo del Toro has been trying to do it, but since such a project is so expensive, he hasn’t been able to put the pieces together. An AI could make movies that just wouldn’t get made otherwise. And maybe it could make one for me on the fly. I could tell it that I love the style of The Road with elements of horror, and it would produce the movie just for me.

** Progress Report: AI Safety Fundamentals Project ** This is a public space for me to keep updated on my AI safety fundamentals project. The project will take 4 weeks. My goal is to stay lean and limit my scope so I can actually finish on time. I aim to update this post at least once per week with my updates, but maybe more often.

Overall, I want to work on agent foundations and the theory behind AI alignment agendas. One stepping point for this is Selection theorems; a research program to find justifications that a given training process will result in a ... (read more)

About 1 year ago, I wrote up a ready-to-go plan for AI safety focused on current science (what we roughly know how to do right now). This is targeting reducing catastrophic risks from the point when we have transformatively powerful AIs (e.g. AIs similarly capable to humans).

I never finished this doc, and it is now considerably out of date relative to how I currently think about what should happen, but I still think it might be helpful to share.

Here is the doc. I don't particularly want to recommend people read this doc, but it is possible that someone wil... (read more)

Some cities have dedicated LW/ACX Discord servers, which is pretty neat. Many of the cities hosting meetups over the next month are too small to have much traffic to such a server, were it set up. A combined, LW meetup oriented Discord server for all the smaller cities in the world, with channels for each city and a few channels for common small-meetup concerns, seems like a $20 bill on the sidewalk. So I’m checking whether such a thing exists here, before I start it.

Some time back, Julia Wise published the results of a survey asking parents what they had expected parenthood to be like and to what extent their experience matched those expectations. I found those results really interesting and have often referred to them in conversation, and they were also useful to me when I was thinking about whether I wanted to have children myself.

However, that survey was based on only 12 people's responses, so I thought it would be valuable to get more data. So I'm replicating Julia's survey, with a few optional quantitative questi... (read more)

When thinking about deontology and consequentialism in application, I was trying to rate morality of actions based on intention, execution, and outcome. (Some cells are "na" as they are not really logical in real world scenarios.)

In reality, to me, it seems executed "some" intention matters (though I am not sure how much) the most when doing something bad, and executed to the best ability matters the most when doing something good.

It also seems useful to me, when we try to learn about applications of philosophy from law. (I am not an expert though in neith... (read more)

In some ways it doesn't make a lot of sense to think about an LLM as being or not being a general reasoner. It's fundamentally producing a distribution over outputs, and some of those outputs will correspond to correct reasoning and some of them won't. They're both always present (though sometimes a correct or incorrect response will be by far the most likely). 

A recent tweet from Subbarao Kambhampati looked at whether an LLM can do simple planning about how to stack blocks, specifically: 'I have a block named C on top of a block named A. A is on tabl... (read more)