Quick Takes

One common source of Illusion of Transparency problems is different senses of what's a normal range. The countercharm for this is using actual numbers and not relative comparisons. For instance, I talk to a lot of people who run conferences and meetups. Most of them anchor the size of an event off what they've seen or run. So someone might say "yeah, that was a really big convention last week!" to mean an EAG with almost two thousand people. Then someone else might say "yeah, the convention next week is looking like it's going to be a small one" and they're talking about DEF CON where twenty thousand would be a low turnout. For another example, I like hiking. Sometimes I say I'm going on a short hiking trip, and someone reasonably assumes I'll be back in a few hours, when in fact I'm bringing a tent and three or four days of food. But I'm not that into hiking; on the trail I regularly meet people who enjoy hiking the Long Trail (~270 miles) as a short backpacking trip because it's not the Appalachian Trail! (~2,200 miles.)

I predict that "dewebiffying" or "de-IoT-iffiyng" is going to become a big trend over the next few years.

"Fake cookies", i.e. sugar free cookies with artificial sweetener, are great (Write-up longer than necessary because I'm not spending enough effort making it short) I like sweets, but if I have too much sugar I get short-term addicted. I.e. I get sugar cravings, and my mood is worse. I can get over it (and have done so many times) by avoiding sweets for a few days but it's unpleasant. In the past my main strategy was to avoid getting into the addicted state, by not having enough sweets to trigger it. I can have some, just not too much. Eating fruits also helps, since this some what satisfy the craving without reinforcing it. I.e, if I have sweets, it's easer to stop and less likely to trigger the addiction, if I end the "meal" with a fruit. This hasn't been enough of a problem for me to put much agency into solving it. Or maybe it's more correct to say that it's only been a noticeable problem at times when I don't have spare agency to throw at it. E.g. if I'm feeling down for other reasons, it would be nice to use sweets as mood enhancer, but also if I'm already down then don't want to have to add dealing with sugar cravings on top of what else is going on. In this situation, me being prone to sugar addiction, becomes a noticeable problem, but I'm also not in a state where I'm great at thinking about solutions. Recently I got into sugar craving state, and I was also down for unrelated reasons, and really did not have the spare spoons to deal with it. So I decided to try "fake cookies". For some reason they don't sell them in the local stores, here but you can get them on amazon. And it just worked. Fake cookies satisfy the addiction without reinforcing it. It's a bit embarrassing that I hadn't tried this before. I think the reason is a mix of some naturalistic fallacy, and that things I tasted previously that contains artificial sweetener where not tasty. Also, I didn't actually expect it to work, but I still should have tried. Things that I've tasted before

I'm worried that people won't reflect well and so (1) they'll decide to use the cosmic endowment for something mediocre and/or (2) they'll have a catastrophic failure in some of {decision theory, bargaining, dealing with threats, reasoning about simulators, acausal trade (including ECL etc.)}. Some angles: * Current AI systems are way better at science/engineering than philosophy. Maybe even ASI won't solve philosophy (in particular, maybe there's not a sufficiently good training signal to train super-philosophers via ML). * By default almost everyone is and will be overconfident in their intuitions on philosophy/metaethics/axiology. By default almost nobody will "suspect that they are confused about morality, and [be] eager or willing to apply reason and deliberation to find out what their real values are, or to correct their moral beliefs." * Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. And maybe it's necessary. * It may be natural for powerful people to use AIs for instrumental stuff (acquiring power, inventing technology, building industrial capacity) but not terminal stuff (axiology, etc.). Some angles: * If they asked an enlightened philosophy ASI about its views and it replied straightforwardly, they would think it sounded crazy. * In general the ASIs will say crazy-sounding things about cosmology, religion, etc. * They will have seen other people "reflect" with "help" from ASIs and they think those people now have crazy views, talk about living in a simulation, etc. and they don't want to end up like that. * @habryka probably disagrees. Idk, it's so easy for me to imagine that Elon (or almost anyone else) would listen/defer to AI on the instrumental stuff but not reflect on philosophy/axiology. * Maybe there will have been a period where superpersuaders went around getting people to adopt new ideologies—maybe even by saying true but filtered things—and so people develop stro

https://rikiyat.github.io/blog/weight-space-programming/ Based only on raw weights, Astra 6 is able to tell what a model was trained to do and directly edit, without environmental feedback, those model weights to drastically increase performance on various tasks. RSI probably immediately falls out of these capabilities, and we do not know if OpenAI has RL'd for this. RSI does not fall out of these capabilities, but we have no reason to believe that these were well elicited.[1] If these capabilities can be improved through RL, then I think we have reason for concern. 1. ^ These capabilities seem to be emerging on the frontier, so they might be elicited: https://x.com/alexkrentsel/status/2117376683724660898?a=zL0qV4at-5PBm8oi_HY&s=46

I have been running a reading group/crash course on AI alignment-related stuff for (mostly) philosophy grad students in my department. The goal was to provide some of the background (from LessWrong etc.) that a philosopher who is new to the field couldn't find by looking at the usual philosophical venues, especially stuff that will naturally get philosophers excited. Here is the reading list we've been using (to which I'm still making small updates as we go).

So, the Hackening. I have yet to hear any sensible-sounding takes on what to do about it, at a personal level. Mostly people repeat the usual shibboleths about refreshing your passwords, which doesn't seem to engage at all with any model of the new threat regime. Anybody have actual models about what the new threats will look like, how well various actors will keep up with them, and what to do defensively at a personal level?