Shortform Content [Beta]
Moloch is not about coordination failures.
Moloch is about the triumph of instrumental goals.
Coordination *might* save us from that. Or not. "it is too soon to say"
I basically agree, but I do assign it to Moloch. *shrug
I would love to see examples of contributions with actual steelmanning instead of just seeing people who pay lipservice to it.
Yes, though I assume the best test for whether you really steelman someone would be if you can take a break and ask her whether your representation fits.
The way I see "Politics is the Mind Killer" get used, it feels like the natural extension is "Trying to do anything that involves high stakes or involves interacting with the outside world or even just coordinating a lot of our own Is The Mind Killer".
From this angle, a commitment to prevent things from getting "too political" to "avoid everyone becoming angry idiots" is also a commitment to not having an impact.
I really like how jessica re-frames things in this comment. The whole comment is interesting, here's a snippet:
... (read more)Basically, if the issue is adversar
"Back to the Future: Curing Past Suffering and S-Risks via Indexical Uncertainty"
I uploaded the draft of my article about curing past sufferings.
Abstract:
The long unbearable sufferings in the past and agonies experienced in some future timelines in which a malevolent AI could torture people for some idiosyncratic reasons (s-risks) is a significant moral problem. Such events either already happened or will happen in causally disconnected regions of the multiverse and thus it seems unlikely that we can do anything about it. However, at least one pure theoret... (read more)
When somebody is advocating taking an action, I think it can be productive to ask "Is there a good reason to do that?" rather than "Why should we do that?" because the former phrasing explicitly allows for the possibility that there is no good reason, which I think makes it both intellectually easier to realize that and socially easier to say it.
Something that I've been thinking about lately is the possibility of an agent's values being partially encoded by the constraints of that agent's natural environment, or arising from the interaction between the agent and environment.
That is, an agent's environment puts constraints on the agent. From one perspective removing those constraints is always good, because it lets the agent get more of what it wants. But sometimes from a different perspective, we might feel that with those constraints removed, the agent goodhearts or wire-heads, or otherwise fails... (read more)
Side note, which is not my main point: I think this also has something to do with what meditation and psychedelics do to people, which was recently up for discussion on Duncan's Facebook. I bet that mediation is actually a way to repair psychblocks and trauma and what-not. But if you do that enough, and you remove all the psych constraints...a person might sort of become so relaxed that they become less and less of an agent. I'm a lot less sure of this part.
There's a concept I want to think more about: gravy.
Turkey without gravy is good. But adding the gravy... that's like the cherry on top. It takes it from good to great. It's good without the gravy, but the gravy makes it even better.
An example of gravy from my life is starting a successful startup. It's something I want to do, but it is gravy. Even if I never succeed at it, I still have a great life. Eg. by default my life is, say, a 7/10, but succeeding at a startup would be so awesome it'd make it a 10/10. But instead of this happening, my brain pulls a ... (read more)
I started writing on LW in 2017, 64 posts ago. I've changed a lot since then, and my writing's gotten a lot better, and writing is becoming closer and closer to something I do. Because of [long detailed personal reasons I'm gonna write about at some point] I don't feel at home here, but I have a lot of warm feelings towards LW being a place where I've done a lot of growing :)
I'm glad about your growth here :)
I’ve noticed myself using “I’m curious” as a softening phrase without actually feeling “curious”. In the past 2 weeks I’ve been trying to purge that from my vocabulary. It often feels like I'm cheating, trying to pretend like I'm being a friend when actually I'm trying to get someone to do something. (Usually this is a person I'm working with it and it's not quite adversarial, we're on the same team, but it feels like it degrades the signal of true open curiosity)
I went through something similar with phrases like "I'm curious if you'd be willing to help me move." While I really meant "I hope that you'll help me move."
My personal experience was that shifting this hope/expectation toba real sense of curiosity "Hmm, Does this person want to help me move?" Made it more pleasant for both of us. I became genuinely curious about their answer, and there was less pressure both internally and externally.
[Online dating services related]
The incentives of online dating service companies are ridiculously misaligned with their users'. (For users who are looking for a monogamous, long-term relationship.)
A "match" between two users that results in them both leaving the platform for good is a super-negative outcome with respect to the metrics that the company is probably optimizing for. They probably use machine learning models to decide which "candidates" to show a user at any given time, and they are incentivized to train these models to avoid matches that cause users to leave their platform for good. (And these models may be way better at predicting such matches than any human).
I don't know much about their business and customer modeling specifically. In other subscription-based information businesses, a WHOLE LOT of weight is put on word of mouth (including reviews and commentary on social media), and it's remarkably quantifiable how valuable that is. For the cases I know of, the leaders are VERY cognizant of the Goodhart problem that the easiest-to-measure things encourage churn, at the expense of long-term satisfaction.
I keep seeing people say that the Self-Indication Assumption implies that given two possible theories with equal posterior probability of being true, SIA says the one that implies more observers is by default more likely to be true. But this would only be true if possible-universes were equally distributed by observer count. But they're not. Universes, even the set of possible universes, fall under either a Normal Distribution, or a Power Distribution. Either distribution implies that universes with more observers are less likely even though each one has m... (read more)
Posterior Probability doesn't take SIA into account. So the theories would be equally likely before applying SIA. Then, applying SIA, theory that predicts 2Y observers would become twice as likely. But then applying a type of "Universe Indication Assumption", where universes with twice as many observers are intrinsically a third less likely, the theory that predicts Y number of observers becomes more likely.
Is society just a tool to get Kegan 3 frames to want to LARP Kegan 4 and Kegan 5 frames?
I feel like I only wrote half that comment. Here's the rest.
That kind of compartmentalization is not something that comes naturally to people without systems in place to push them to it. In a traditional society, there's just sort of one social sphere (attempts at secret groups for ritual purposes notwithstanding) that overlaps with everything and you can bring your whole self all the time everywhere and people will expect you to do that. It's only that we ask more of people in our modern world because compartmentalization works well as a bridge to help pe... (read more)
I was initially extremely disappointed with the reception of this post. After publishing it, I thought it was the best thing I've ever written (and I still think that), but it got < 10 karma. (Then it got more weeks later.)
If my model of what happened is roughly correct, the main issue was that I failed to communicate the intent of the post. People seemed to think I was trying to say something about the 2020 election, only to then be disappointed because I wasn't really doing that. Actually, I was trying to do something much more ambitious: solving the ... (read more)
(Datapoint on initial perception: at the time, I had glanced at the post, but didn't vote or comment, because I thought Steven was in the right in the precipitating discussion and the "a prediction can assign less probability-mass to the actual outcome than another but still be better" position seemed either confused or confusingly phrased to me; I would say that a good model can make a bad prediction about a particular event, but the model still has to take a hit.)
(Taken from a comment)
One of the problem's with Rao's Gervais principle that I later realized(that I think Zvi's sequence shares to some degree) is that it doesn't distinguish between Kegan 4.5 Sociopaths, and Kegan 5 leaders. This creates the impossible choice between having freedom as a loser, meaning as a clueless, or influence as as a sociopath, pick one.
Similarly, Zvi's sequence gives the choice of truth as a simulacra 1, belonging as Simulacra 2, and influence as Simulacra 4.
Neither framing admits that it's possible to get to a stage of l... (read more)
Yes, I agree with that. Of course it's meaningful! It wouldn't be a reflection of reality if it wasn't. But meaningful isn't the same as complete or undistorted.
For example, I think it's meaningful (maybe not the most insightful thing that could possibly be said, but meaningful) to talk about the original Star Trek in terms of head, heart, and gut as reflected in the characters of Spock, McCoy, and Kirk. I don't think this covers everything that Star Trek is, or everything that those characters are, or everything that real people ca... (read more)
Is it possible to make an hourglass that measures different amounts of time in one direction than the other? Say, 25 minutes right-side up, and 5 minutes upside down, for pomodoros. Moving parts are okay (flaps that close by gravity or something) but it should not take additional effort to flip.
One question I have about both your solution and mine is how easy it is to vary the time drastically by changing the size of the hole. My intuition says that too large holes behave much differently than smaller holes and if you want a drastic 5x difference in speed you might get into this "too large and the sand sort of just rushes through" behavior.
A rough and dirty estimate of the COVID externality of visiting your family in the USA for Christmas when you don't feel ill [EDIT: this calculation low-balls the externality, see below]:
You incur some number of μCOVIDs[*] a week, let's call it x. Since the incubation time is about 5 days, let's say that your chance of having COVID is about 5x/7,000,000 when you arrive at the home of your family with n other people. In-house attack rate is about 1/3, I estimate based off hazy recollections, so in expectation you infect 5xn/21,000,000 people, which is about... (read more)
I recently realized, thanks to a FB comment by Paul Christiano, that this is thinking about things in kind of the wrong way. R is approximately 1 because society is tamping down infection rates when infections are high and 'loosening' when infections are low. So, by infecting people, you cause some chain of counterfactual infections that perhaps ends when society notices and tamps down infection, but also you cause the rest of society to do less fun interacting in order to tamp down the virus. So the cost of infecting somebody is to cause everybody else to be more conservative. I'm still not quite sure how to think about that cost tho.
[Epistemic status: conspiracy theory/raving of the mads]
We all know that GPD/standard of living all track with energy use, yet arguably the most convenient and widespread energy sources current are fossil fuels, which saw mass adoption with the start of the industrial revolution. Which happened, it can be said without hyperbole, eons ago [citation needed].
For some time, nuclear fission seems poised to replace fossil fuel, yet a series of unfortunate events permanently sour the public perception of this technology [Chernobyl, Fukushima]. Even in countries t... (read more)
Instrumental Rationality Mini-Retrospective
I promised several years ago to write a retrospective on Hammertime a year after it was released. I broke that promise but I wanted to take some time to do the work now, and to summarize my current beliefs about how much rationalist self-improvement affected my personal growth. I'd also like to estimate how it compares to other schools of self-improvement I've dabbled in.
First, I should mention that epistemic rationality has been directly useful in my career, although this is highly unlikely to generalize. At leas... (read more)
Fascinating! Definitely plan to check this out, thanks for the recommendations and detailed introduction.
A well-known analogy from Yann LeCun: if machine learning is a cake, then unsupervised learning is the cake itself, supervised learning is the icing, and reinforcement learning is the cherry on top. (Unfortunately it seems like I can't embed images into a shortform).
I think this is useful for framing my core concerns about current safety research:
- If we think that unsupervised learning will produce safe agents, then why will the comparatively small contributions of SL and RL make them unsafe?
- If we think that unsupervised learning will produce dangerous agen
I wrote a few posts on self-supervised learning last year:
- https://www.lesswrong.com/posts/SaLc9Dv5ZqD73L3nE/the-self-unaware-ai-oracle
- https://www.lesswrong.com/posts/EMZeJ7vpfeF4GrWwm/self-supervised-learning-and-agi-safety
- https://www.lesswrong.com/posts/L3Ryxszc3X2J7WRwt/self-supervised-learning-and-manipulative-predictions
I'm not aware of any airtight argument that "pure" self-supervised learning systems, either generically or with any particular architecture, are safe to use, to arbitrary levels of intelligence, though it seems very much worth som... (read more)
I found myself saying recently, "While this strategy does not in this case seem to have much causal connection to good outcomes, I feel like following the strategy in the past few months has been good for my soul."*
Humans don't have souls. I could imagine substituting, "This strategy has made me an easier agent to coordinate with and has moved me closer to the morality I was taught growing up, which has reduced my cognitive dissonance with my formerly more consequentialist actions. And it's an important part of the strategy that I don't alter it just becau... (read more)
Just wanted to add that "made me an easier agent to coordinate with" applies not only to coordination with other people, but also to coordination with your past/future selves. That is, what is "good for your soul" is good even when other people are not involved.
It may even be the more important aspect, because if you can't trust your future selves, how could other people? (Your deals with other people implicitly involve deals with your future selves.)