All of Rudi C's Comments + Replies

What New Desktop Should I Buy?

Considering that DIY is widely acknowledged to be viable, my prior is that the marketing around Alienware is bullshit and it’s just fine. Care to point to hard data that refutes this?

Humans provide an untapped wealth of evidence about alignment
Rudi C1moΩ-220

I am skeptical of your premise. I know of zero humans who terminally value “diamonds” as defined by their chemical constitution.

Indeed, diamonds are widely considered to be a fake scarce good, elevated to their current position by deceptive marketing and monopolistic practices. So this seems more like a case study of how humans’ desires to own scarce symbols of wealth have been manipulated to lead to an outcome that is misaligned with the original objective.

I just introspected. I am weakly attracted to the idea of acquiring diamonds. I therefore know of at least one human who values diamonds. I never claimed that humans are hardwired [] to value diamonds. I pointed out that some people do value diamonds, and pointed out that true facts have guaranteed-to-exist explanations. If you're interested in building a mind which values diamonds, first ask why some already-existing minds value diamonds.
9Logan Riggs1mo
I believe the diamond example is true, but not the best example to use. I bet it was mentioned because of the arbital [] article linked in the post. The premise isn't dependent on diamonds being terminal goals; it could easily be about valuing real life people or dogs or nature or real life anything. Writing an unbounded program that values real world objects is an open-problem in alignment; yet humans are a bounded program that values real world objects all of the time, millions of times a day. The post argues that focusing on the causal explanations behind humans growing values is way more informative than other sources of information, because humans exist in reality and anchoring your thoughts to reality is more informative about reality.
Another argument that you will let the AI out of the box

The problem is that normal people very often give up collective resources to look good. They just don't give up their personal resources. For the AI, the former is sufficient.

The scenario requires not only that they give them up, but that they give them up on a very immediate basis, which is less likely.
Grandpa Has Different Rules

People attributing their own shortcomings to others is rather weak evidence.

Consume fiction wisely

It is also pretty unbelievable. (Spoilers ahead.)

The security around keeping the whole secret is way off. This is their biggest priority, and they know it. Yet the children can just walk where they are not supposed to go, and discover it.

The technological measures do not match up, and they absolutely can have sensors that make conspiring and/or escaping much harder.

The children are too competent. Well, we can forgive this one, but it really takes things too far; e.g., one child has learned to make a device from scraps of other devices to disable their GPS ... (read more)

Consume fiction wisely

The damage chance per encounter is higher with sharks than cows, surely?

Consume fiction wisely

The claim here is definitely 'audiobooks would generally be more relaxing than the written word.'

I personally find it somewhat true; I need to listen to fiction very attentively to not lose the plot, but I can jump back into a nonfiction podcast/audiobook after not listening for 10 minutes just fine (most of the time).

Consume fiction wisely

No adult updates their probability that dragons are real after reading Game of Thrones

Without fiction, the hypothesis "dragon" would not even exist in our minds. We are wasting cultural bandwidth on this concept, and our probability estimation of it is orders of magnitude more than if we did not have it plastered everywhere in fiction.

such that you update on them.

This is a valid point, and I think an extreme case of it can be seen in fundamentalist religions. But my prior is that anyone who understands the argument the OP has presented, is smart eno... (read more)

Consume fiction wisely

Can you write a post about things you learned via video games? I am highly skeptical that they can teach anything transferable to the real world for STEM-adjacent adults. (Programming video games like can teach some programming, but they are more like gamefied Leetcode than a strategy/puzzle game. Most non-programmers I have introduced these games to could not even win the starting levels.)

I wrote an LW post about the one game I can recommend in LW circles. It's called Understand [] . The main thing it trains is continually coming up with hypotheses and falsifying them. I've played gazillions of other games, including stuff like Factorio and almost all the Zachtronics games, but don't think their upsides outweigh their significant downsides, like being enormous timewasters.
Personally, I think I got a good feel of the basics of business management from Capitalism Lab []. It is a very detailed business sim that is trying to be as realistic as possible for a game. According to the dev [] , the game's predecessor was also used at Harvard and Stanford as a teaching aid. Judging by the immense complexity and depth of the game, I find the claim believable.
How to think about and deal with OpenAI

Epistemic status: I am not an expert on this debate, I have not thought very deeply about it, etc.

  1. I am fairly certain that as long as we don’t fail miserably (i.e., a loose misaligned super AI that collapses our civilization), FOSS AI is extremely preferable to proprietary software. The reasons are common to other software projects, though the usefulness and blackboxy-ness of AGI make this particularly important.
  2. I am skeptical of “conspiracies.” I think a publicly auditable, transparent process with frequent peer feedback on a global scale is much more
... (read more)
5Daniel Kokotajlo7mo
Thanks! In case you are interested in my opinion: I think I agree with 1 (but I expect us to fail miserably) and 2 and 6. I disagree with 3 (AGI, unlike nukes, can be used in ways that aren't threatening and don't hurt anyone and just make loads of money and save loads of lives. So people will find it hard to resist using it. So the more people have access to it, the more likely it is it'll be used before it is safe.) My timelines are about 50% by 10 years; I gather from point 4 that yours are longer. I think 5 might be true but might not be; history is full of examples of different groups fighting each other yet still managing to conquer and crush some third group. For example, the Spanish conquistadors were fighting each other even as they conquered Mexico and Peru. Maybe humans will be clever enough to play the AIs off against each other in a way that lets us maintain control until we solve alignment... but I wouldn't bet on it.
Blood Is Thicker Than Water 🐬

My point is more about prioritization. English, math, programming and computer literacy, economics, basic home skills (cooking, trivial repairs, etc.), and possibly rationality (though the existence of “The Dark Valley of Rationality” makes me a bit hesitant on this one) are much better subjects for a “general info” curriculum.

PS: Knowing about elementary particles (without a mathematical model of them) is trivial. You can fit all such facts into a single year’s science curriculum. The things that take time to learn are calculations, e.g., finding the mass of some reagent after some chemical reaction.

How to think about and deal with OpenAI

Ironically, I am a believer in FOSS AI models, and I find OpenAI’s influence anything but encouraging in this regard. The only thing they are publicly releasing is marketing nowadays.

4Daniel Kokotajlo7mo
Yep! :) But the damage is done; thanks to OpenAI there is now a large(r) group of people who believe in FOSS AI than there otherwise would be, and there are various new actors who have entered the race who wouldn't have if not for OpenAI publications. To be clear, I'm not confident FOSS AI is bad. I just think it probably is, for reasons mentioned. I won't be too surprised if I turn out to be wrong and actually (e.g. for reasons Dirichlet-to-Neumann mentioned, or because FOSS AI is harder to make a profit on and therefore will get less investment and therefore will be developed later, buying more time for safety research) the FOSS AI ethos was net-positive. I'd be interested to hear your perspective on FOSS AI.
How to think about and deal with OpenAI

Taboo “racing”? I don’t understand what concrete actions were thought to have been skipped.

I don't know what you mean by skipped. Here's some more concreteness though:

--Thanks to OpenAI, there is more of an "AI research should be made available to everyone" ethos, more of a "Boo anyone who does AI research and doesn't tell the world what they did or how they did it or even decides not to share the weights!" Insofar as this ethos persists during the crucial period, whichever labs are building AGI will be under more internal and external pressure to publish/share. This makes it harder for them to go slow and be cautious when the stakes are high.

--... (read more)

Blood Is Thicker Than Water 🐬

I find that understanding the ways in which dolphins are mammalian, is very much an informational challenge; I need to know a lot more biology to be able to use the category of “genetically mammal” than just plain old “fish.” Obviously, knowing more is better, but it is not obvious to me that forcing everybody to learn the more informed categories is societally optimal. I doubt anyone but specialists will ever find the information instrumentally useful, so we are just wasting some limited bandwidth to teach people stuff they don’t need to know. (On the other hand, people mostly waste their time with, e.g., learning about the differences between ten different Robin characters, so perhaps the endeavor is justified after all.)

The idea of "general education" is that it's good for ordinary people to learn lots of things that were discovered by specialists: partially because we value knowledge for its own sake, but also because it's hard to tell in advance what knowledge will end up being useful. In principle, you could reject the idea that general education is generally good, but if you're going to be consistent about what that entails, I don't think anyone who reads this website actually wants to go there. Do I really need to know that matter is made of "atoms", that have a "nuc... (read more)

Launched: Friendship is Optimal

I just finished the story. I think the main idea is obviously good, but I find the execution too simplistic. The princess’s intelligence comes off as human in her social interactions (not superhuman), and humans seem dumbed down. I don’t think the AI could have forced everyone to “upload” with the level of coercion shown in the novel.

I myself would not have uploaded unless I was already dying. I see no benign reason that the process of uploading needs to be destructive.

I also doubt people with real power would be easily swayed to upload, e.g., the original... (read more)

Signaling isn't about signaling, it's about Goodhart

This post is simplistic and vague. E.g., does the OP think dressing in dirty, shabby clothes (which, from a nonsignalling perspective, aren’t a negative in our environment) is not an obvious failure of marketing that leads to lost opportunities?

Anyhow, caring about signaling is a priority in most cultures that I have glimpsed, almost all businesses, and human nature. Against such a strong prior, the OP fails to provide any strong updates to the contrary.


An interesting fact came to my mind; music that affects one's mental state is forbidden in Islam.

Perhaps the reason this theme of "sinful pleasure" keeps repeating is the observation that pleasure is a reward signal that does not quite match the utility functions of the conscious mind. At least, that has always been the key motivator of this idea subspace to me.


Can you imagine what it would be like to try to convey to pre-music folk even that music is real and that it might be worth learning how to listen to it?

This analogy is capturing my current understanding of this post and its various comments pretty well: Looking is like music, in that it is a difficult, voluntaryish act of observing and manipulating hidden mental states. This will result in wireheading, among other things, but it might be sometimes useful. (Note that music is also wireheading, but it can still be useful in narrow contexts.)

Cool. FWIW, I've come to think that wireheading is an anti-concept as applied to humans. It's one of those "presume the conclusion" type mental movements. In practice it seems to act like a back door for arguments based on belief residue like the Protestant work ethic / "pleasure is sinful" stuff. (A little more concretely: It makes sense to talk about some system engaging in wireheading only when there's a goal imposed from outside the system. It's like glorified Goodharting. But if the goals come from within the system, it stops being clear what "wireheading" means. On the inside it might feel like "Oh, I just found a vastly easier way to get what I want — and what I want wasn't what I thought I wanted!" Without an external evaluation criterion, that actually just becomes correct.) With that said, I think I intuit what you mean by calling music and Looking "wireheading". I don't mean to dismiss that. Stuff like, if you meditate enough to get Great Insights™ such that you don't bother to eat food anymore and you die, that seems like a pretty dramatic failure and kind of throws those "insights" into question.
Is "gears-level" just a synonym for "mechanistic"?

But mechanistic world models do suggest that meaning in a traditional (mystical? I can’t really define it, as I find the concept itself incoherent) sense does not (and cannot) exist; so I think the “negative” connotations are pretty fair, it’s just that they aren’t that negative or important in the first place. (“Everything adds up to normalcy.”) Rebranding is still a sound marketing move, of course.

In some sense, yeah, "life is inherently meaningless" and "living beings are just machines." However, I am still struggling to wrap my head around the objectivity of aesthetics, meaning and morality. Information is now widely considered physical (refer to papers by R Landauer [] and D Deutsch []). Maybe someday, we will once and for all incorporate aesthetics, meaning and morality under physicalism. If minds are physical, and aesthetics, purposes, and morality are real aspects of minds, then wouldn't that imply that they are therefore objective notions? And thus not "meaningless"? This is a gnarly rabbit hole, and I am not qualified to talk about this topic. I recently read Parfit's "Reasons and Persons" to gain a deeper grasp of these topics and it's a stunning and precious book, but I need to do more work to understand all this. I may have to read his magnum opus "On What Matters" to wrap my head around this. We don't have a proper understanding of minds at this point in time. Developing robust theories about rationality, morality, aesthetics, desires, etc., necessitates actually understanding minds. As you've pointed out, marketing matters. In my view, this is part of the reason why epistemic and instrumental rationalities are distinct aspects of rationality as defined in the sequences [] . If your goal is to explain an idea to your interlocutor and you can convey the same truth using different wording, with one wording leading to mutual understanding and the other leading to obstinacy, then the instrumentally rational thing to do would be to use the former wording. Here we have a situation where two things are epistemically equivalent but not instrumentally so.
Teaser: Hard-coding Transformer Models

(Unrelated.) Have you considered putting an RSS field of your Twitter account on its bio? This way people can follow you without you needing to approve them, and since it’s read-only, your burden won’t increase.

(Not to mention that RSS is a much better medium than Twitter in the first place.)

I don't think Twitter allows such RSS feeds.
Writing Causal Models Like We Write Programs

What useful problems do PPLs solve? Ideally some applications that are interesting for us non-corporate people. Can it be used for medical statistics (e.g., in nutrition)? (Any examples?) Is the reason it is not used the illiteracy of the scientists, or are the mainstream methods better?

PPLs are a tool to bring complicated statistical modeling to the masses. Computers are capable of doing much more advanced statistical modeling than appears in every non-statistics paper, but most people don't have the expertise to build them. PPLs allow you to write complicated statistical models and then evaluate them with state-of-the-art methods without having to build everything from scratch.
EfficientZero: human ALE sample-efficiency w/MuZero+self-supervised

Expertise status: I am just starting with RL.

Will using a hardcoded model of the environment improve these models, or do the models need the representations they learn?

Using EfficientZero's architecture, how many hours does it take on a single TPUv2 for the agent to reach amateur human level? In general, is EfficientZero being sample efficient or compute efficient?

What is the currently most compute efficient algorithm for simple, two-player deterministic games with a lot of states (e.g., go)?

PS: The reason I am asking is that I learn stuff by coding much b... (read more)

"Another possibility, still speculative but discussed by many scientists in recent days, is that omicron evolved over many months within an immunocompromised patient with a protracted infection. In a patient treated with therapeutics such as monoclonal antibodies or convalescent sera, a viral strain that can survive the assault can potentially amass a host of mutations. Such cases have been documented, but they are not known to have led to outbreaks in the general population." Source: [] This does seem to be a recurring theme - very slow mutation for the most part, and lots of new mutations occurring in immunocompromised "supermutators".
Sasha Chapin on bad social norms in rationality/EA

I myself sometimes feel bad when I engage in, say, writing fiction. (Reading fiction is pretty obviously useless, so I know I am just doing it for “fun.” It doesn’t confuse me the way producing fiction does.) I was like this before I even knew there was a Rationality subculture. I don’t try to justify these behaviors at all; I am just not sure if they are aligned with my values, or not, and in what quantities they are healthy.

So while I agree with the gist of this post, I believe the core issue to be more of a tradeoff rather than an obvious evil.

We Live in a Post-Scarcity Society

The stock's value declines (as supply has gone up). So the "frozen" money declines, too.

That the value of the stocks goes down doesnt really impact the operation of the company. Money isn't the same as production the impact would be mostly on the paper side of things instead of real economy.
The Best Software For Every Need

(I simply use unused books to raise the monitor. Works like a charm.)

The Best Software For Every Need

I doubt Google can "add closed source components" to Chrome with any success. MS will simply recreate the extensions in open-source, getting a lot of mindshare and PR in the process. Android became what it is because Google was ahead of the curve and other companies did not know how useful mobile OSes were going to be.

I don't think Google added much closed source to Android until after Amazon - probably Google's 2nd biggest competitor - forked Android for their own tablets. In that case, it kinda' worked and the threat diminished, but never-the-less I think I agree with you - it wouldn't work this time - and I don't think Google will do it. It does suggest though that they would have been better off making Chrome closed source from the beginning (WebKit is BSD), and while I hate to say such a thing, I think the whole market would be better off. Then, instead of all these copies of Chrome being the primary alternatives to Chrome, Firefox would be doing much better. Mozilla would then be in a dramatically better financial position and could continue to make great contributions to open source in spite of Apple blocking them on iOS. Maybe their increased user base and significance would even force Apple to relent!
The Best Software For Every Need

See also Nyxt. I haven't tried it myself yet, as its macOS support seems to be immature, but it is one of those projects I have an eye on. It could one day be the emacs of web browsers.

There is also, but the security might be sketchy. I am not holding my breath for performance either.

The Best Software For Every Need

Why do you want to switch to macOS? The only thing going for it is having Adobe and Office software. On the other hand, it is likely to do explicit on-device scanning, it doesn't support Docker well, it is generally slow and can hang when the internet connection has problems. Hell, even its API for changing the background wallpaper doesn't work reliably for me.

The Best Software For Every Need

How is it that iOS doesn't kill it? I have yet to see any app that can run in the background continuously; Even apps that use the location API as a workaround will be eventually killed.

We Live in a Post-Scarcity Society

Well, one person is much more likely to keep the stock, while some of the thousand will cache out. This seems to me to encourage consumption, discourage investment and labor on the first order, while the consumption itself can encourage investment on the second order. I don't know how these opposing effects will play out in the long run, but the short term effect is most probably going to be high inflation and costly labor.

Caching in will involve transfer rather than destruction of the stock. The stock will have a new owner who has then voluntarily bound to the production. At the very limit the single original owner could buy it back. If he is unwilling because he would run out of neccesity money ie bread then that would transfer the "frozeness" to the new owners.
The Best Software For Every Need

I am now too invested in zsh to find any other shell worthwhile, but if I could go back, I would not use a traditional shell at all; I would code a custom shell using Common Lisp. If this is too much effort, I would recommend trying out both zsh (possibly use OhMyZSH if you don't want to waste time setting it up properly yourself) and fish, and sticking with the one you like better.

The Best Software For Every Need

hammerspoon, phoenix.js are opensource alternatives that are more powerful and lightweight in their niches. (BTT still has more breadth and ease of use.)

The Best Software For Every Need

If you use cut (or awk or sed for cutting), try perl. I have found it to be way more intuitive, powerful, and only a bit less concise.

The Best Software For Every Need

I have tried Obsidian, Joplin, Logseq, Notion, Evernote, Onenote, Google Notes, Apple Notes, and various other markdown editors. I have ultimately found emacs’ org-mode plus git (for syncing, backups, and, well, version control) plus ripgrep/fzf (for searching and quickly jumping to a file/section) plus org-super-links (which provides automatic backlink insertion/deletion) plus gpg (for encrypting sensitive files/sections; emacs and org-mode support for gpg is superb) to be leagues ahead of the competition. Its mobile story is pretty sad though (There are ... (read more)

That seems really unlikely to me, as several plugin developers who write open source plugins for Obsidian routinely reverse engineer Obsidian's internals (in order to improve our plugins). Obsidian, after all, is built as an Electron app (or Capacitor on mobile) in TypeScript, with AFAIK the only non-JS bits coming from open source projects (such as various Node modules and Electron/Capacitor themselves). Thus, the entire code base is merely partially obfuscated, rather then being a truly "closed" source app. It is only "closed" in the sense that it is against copyright law to create and distribute your own version. So for Obsidian to disappear altogether, it would require both 1) the end of the current entity with ownership, and 2) that end to happen in such a way that a successor copyright owner exists to actively stamp out any attempts to create a community build -- or an API-compatible clone. That could happen, I suppose. But it's hard to imagine what company would have deep enough pockets to do it, and yet also feel threatened enough by Obsidian to want to eradicate it.
We Live in a Post-Scarcity Society

Even the citizens of Kardashev Type III civilization can’t all date Emma Stone and/or Keanu Reeves at the same time.

This is not trivial. Genetic engineering/plastic surgery/other forms of self-enhancement can push everyone to pretty much the same ceiling. Of course, "status" itself is kind of a relative "resource," and it can affect "attractiveness," but our society is far from this being its bottleneck.

I also believe that even status is not a conserved resource. As other resources increase and the general population becomes more resourceful, more ethic... (read more)

If a companys stock is held by 1 person vs a thousand how does that make the production fall?
How much slower is remote work?

One relevant point is that remote work might be a “disruptive” technology: cheaper, more suitable for certain niches, etc, but not as good as the traditional thing. As time passes and the technology matures, it might claim increasing niches, such that in the end it surpasses or becomes an essential additive to the traditional technology.

It is definitely useful in some settings! For instance it's much easier to collaborate with people not at Berkeley, and in some cases those people have valuable specialized skills that easily outweigh the productivity hit.
The Apprentice Thread

Isn't the academic grad school basically this same model, at scale? I do not see any improvements here that are scalable.

[NORMAL] Scalable? Maybe not. But this model has significant advantages over grad school. * Many of the mentors here do not work at academic institutions. * This system is less formal, which reduces overhead for all parties. * Grad school is a physical institution. Not everyone can get a visa to every country.
Assume long serving politicians are rationally maximizing their careers

Indeed, a lot of the most ridiculous human behavior is non-experts mimicking experts randomly and picking the wrong attributes.

Assume long serving politicians are rationally maximizing their careers

Khamenei is not actually challenged by new people in the system. His position is more or less permanent. But to get to that position, then, yes, he must have done some things "right."

Iran has multiple power centers that interact in a very opague way. It would be possible for a very popular president who has the security forces/military on his side to strip Khamenei of power. Pushing for nuclear makes Khamenei more popular with the security forces/military while also weaking the influence of business men as a political force. See the Dictators Handbook for the general principles of political power and the following TED talk for Iran's political landscape in particular and the nuclear question.
Kids NCurses Messenger

Telegram was a much better choice for this purpose. Their APIs are completely open (supporting alternative third-party clients and bots has been one of their priorities since years ago), and there are fantastic wrapper libraries available. Their clients are also native, not the Electron crap. There was already (at least) one Telegram [TUI](, too.

PS: Cute. :-)

Perhaps, but my house already uses Slack, and installing another app just for this isn't ideal?
Scott Aaronson at the AstralCodexTen Online Meetup

It would be great if Lesswrong online events could be recorded and put in a podcast. Live is great if you plan to participate, but for just listening, it sucks. 


One of the good examples I have seen is the Techmeme podcast; They host a lot of Clubhouse/Twitter/etc live chats, and they post the content to their podcast. Some tools have recording as a built-in feature, e.g., Telegram's voice chats.

The case for hypocrisy

Can you provide concrete examples of the specialized pieces?

Detailed maps of technical or personal subjects. Like knowing particular idiosyncracies of your job or relationships.
The case for hypocrisy

I think people are already tolerant of the level of hypocrisy that can be useful. For example, a new convert to Islam will have more slack in doing unislamic things.


Anyhow, this is not an isolated matter. Any kind of punishment has the potential to create adverse effects; Banning ransom payments can cause secret ransom payments, banning drugs powers gangs, banning one carcinogenic chemical can make companies use an even worse carcinogenic chemical, ... . There is no general solution to these, but I’m inherently skeptical of claims that favor the status quo of “rabbits” in a rabbit-stag game. 

Challenge: know everything that the best go bot knows about go

This is the most intuitive answer to me, as well. It’s also extremely difficult, and it‘s unclear how it is going to be useful for doing alignment generally. 

Perhaps one idea is to train AI to write legible code, then use human code review on it. This seems as safe as our current mode of software development if the AI is not actively obfuscating (a big assumption).

Challenge: know everything that the best go bot knows about go

There are weaker computational machines than Turing machines, like regexes. But you don really care about that, you just want to ban automatic reasoning. I think it’s impossible to succeed with that constrain; Playing Go is hard, people can’t just read code that plays Go well and “learn from it.”

[link] If something seems unusually hard for you, see if you're missing a minor insight

Some examples:

  • some doors need to be pushed/pulled when turning the key
  • using vector syntax is much faster than loops in Python
  • cans can be opened relatively easily with just a spoon if the right angle is used

A related problem is being mistaken about how high the quality bar of a task actually is. Perhaps also known as 'obsessing.'

Dropping the quality bar for a task is one of the best techniques I've ever encountered for getting annoying tasks finished. Seeing the quality bar's position is the first step to questioning it, and I think a lot of the culture and education that my friends and I have encountered tries to taboo the concept of imagining that such a bar could ever be set in a position other than maximum. Claiming that the bar has to be set to max works well for forcing people of average skill or motivation in a particular area to produce an adequate product, but that system falls apart as soon as one encounters a problem on which one's personal skill or knowledge or motivation raises the maximum conceivable quality of output to a level that's impractical or undesirable to strive for.
To generalize this slightly, using Python to call C/C++ is generally much faster than pure Python. For example, built-in operations in Pandas tend to be pretty fast, while using .apply() is usually pretty slow.

I think it’s possible to just upload the video to Youtube, and then download its automatically generated subtitle with youtube-dl, and finally convert that subtitle into plain text (using, e.g., ).

Spotify is centralizing podcasting, and plans to implement the same monopolistic, privacy-invasive ad policies Google/Facebook are adhering to. It is worth considering whether allowing them to do this is a net harm for the consumers.

Load More