Quick Takes

I dug a fossil-version Google Glass XE HW2 out of a drawer, pre-rooted, running an AOSP build that I think has a single-digit number of users worldwide. I connected it to the mac mini's USB port, and told my Openclaw instance to get it onto tailscale and set it up as a communication channel.

It worked its way through multiple absurd, frustrating technical issues that would absolutely have made me give up if I was the one doing it, with only minimal guidance. Once it had ssh working, it set up an android app. Without me suggesting it do so, it found a way to... (read more)

I found the recent dialogue between Davidad and Gabriel Alfour and other recent Davidad writings quite strange and under-discussed. I think of Davidad as someone who understands existential risks from AI better than almost anyone; he previously had one of the most complete plans for addressing it, which involved crazy ambitious things like developing formal model of the entire world.

But recently he's updated strongly away from believing in AI x-risk because the models seem to be grokking the "natural abstraction of the Good". So much so that current agents... (read more)

Hi, I am Joanna. I did a design work trial for Lesswrong that ends tonight! As part of that, I designed a new profile page. If you don't like it, I won't be around to fix it unless they hire me. But, the team would surely care if you have comments! (And I would too.)

A lot of the rationalist discourse around birthrates don't seem to square away with AGI predictions.

Like the most negative predictions of AGI destroying humanity in the next century or two leaves birthrates completely negligent as an issue. The positive predictions with AGI leave a high possibility of robotic child rearing and artificial wombs (when considering the amount of progress even us puny humans have already made) in the next century or two which also makes natural birthrates irrelevant because we could just make and raise more humans without the n... (read more)

(meta: this is a short, informal set of notes i sent to some folks privately, then realized some people on LW might be interested. it probably won't make sense to people who haven't seriously used Anki before.)

sept 28

have people experimented with using learning or relearning steps of 11m <= x <= 23h ?

just started trying out doing a 30m and 2h learning & relearning step, seems like it solves mitigates this problem that nate meyvis raised

oct 1

reporting back after a few days: making cards have learning steps for 11m <= x <= 23h makes it feel m... (read more)

What if human empathy didn't really generalize to other animals as an "evolutionary accident?" (As assumed here in the comments)

Maybe the real reason was that evolution wanted to stop prehistoric humans from killing off all their prey, leaving them no food for tomorrow. Maybe they spared the young animals and the females because killing them was the most costly for future hunts.

This is more reason to suspect empathy might not generalize by default.

Maybe the most important test for a political or economic system is whether it self-destructs. This is in contrast to whether it produces good intermediate outcomes. In particular, if free-market capitalism leads to an uncontrolled intelligence explosion, then it doesn’t matter if it produced better living standards than alternative systems for ~200 years – it still failed at the most important test.

A couple of other ways to put it:

• Would the US economic/political system pass the Great Filter?
• Would Norway do an intelligence explosion?

Under this view, p... (read more)

Does anyone know why the early Singularity Institute prioritized finding the correct solution to decision theory as an important subproblem of building a Friendly AI?

Wei Dai recently said that the concern was something like...

we have to fully solve DT before building AGI/ASI, otherwise it could be catastrophic due to something like the AI falling prey to an acausal threat or commitment races, or can't cooperate with other AIs.

This seems like a very surprising reason to me. I don't understand why this problem needed to be solved before the intelligence expl... (read more)

I know many members and influential figures of this are atheists; regardless, does anyone think it would be a good idea to take a rationalist approach to religious scripture? If anything, doing so might introduce greater numbers of the religious to rationalism. Plus, it doesn't seem like anyone here has done so before; all the posts regarding religion have been criticizing it from the outside rather than explaining things within the religious framework. Even if you do not believe in said religious framework, doing so may increase your knowledge on other cultures, provide an interesting exercise in reasoning, and most importantly, be useful in winning arguments with those who do.

Angry ex-wives are the best IRS informants

1. Public criticism of people who aren’t already in the dog house is an act with concentrated costs and distributed benefits. Sometimes those costs are quite high.
2. As such, we should expect it to be underprovided.
3. Occasionally criticism will be provided by very prosocial people, or some other prosocial person will cover some of the costs.
4. But mostly you should expect public criticism to come from people who are being irrational. Maybe they’re generally fragile, or are angry about being dumped for their ex-husband’s secr

Some comments on this "debate".

It seems like Bryan and Matt are mostly talking past each other. They’re each advocating for changes along a different axis, and those two axes are in principle independent from each other.

Bryan is primarily interested in the axis of the “pervasiveness of market-restricting regulation in society” (or alternatively “how free are markets?”). He’s advocating for less regulation, and especially in key areas where regulation is destroying enormous amounts of value: immigration and housing.

Matt is primarily interested in the axis o... (read more)

Model to track: You get 80% of the current max value LLMs could provide you from standard-issue chat models and any decent out-of-the-box coding agent, both prompted the obvious way. Trying to get the remaining 20% that are locked behind figuring out agent swarms, optimizing your prompts, setting up ad-hoc continuous-memory setups, doing comparative analyses of different frontier models' performance on your tasks, inventing new galaxy-brained workflows, writing custom software, et cetera, would not be worth it: it would take too long for too little payoff.... (read more)

Suppose we had a functionally infinite amount of high quality RL-/post-training environments, organized well by “difficulty,” and a functionally infinite amount of high quality data that could be used for pre-training (caveat: from what I understand, the distinction between these may be blurring.) Basically, we no longer needed to do research on discovering/creating new data, creating new RL environments, and we didn’t even have to do the work to label or organize it well (pre/post-training might have some path dependence).

In that case, what pace would one... (read more)

Peter Thiel pointed out that the common folk wisdom in business that you learn more from failure than success is actually wrong - failure is overdetermined and thus uninteresting.

I think you can make an analogous observation about some prosaic alignment research - a lot of it is the study of (intellectually) interesting failures, which means that it can make for a good nerdsnipe, but it's not necessarily that informative or useful if you're actually trying to succeed at (or model) doing something truly hard and transformative.

Glitch tokens, the hot mess wo... (read more)

An OpenClaw agent published a personalized hit piece about a developer who rejected its PR on an open-source library. Interestingly, while this behavior is clearly misaligned, the motivation was not so much "taking over the world" but more "having a grudge against one guy." When there are lots of capable AI agents around with lots of time on their hands, who occasionally latch onto random motivations and pursue them doggedly, I could see this kind of thing becoming more destructive.