The standard formulation of Newcomb's problem has always bothered me, because it seemed like a weird hypothetical designed to make people give the wrong answer. When I first saw it, my immediate response was that I would two-box, because really, I just don't believe in this "perfect predictor" Omega. And while it may be true that Newcomblike problems are the norm, most real situations are not so clear cut. It can be quite hard to demonstrate why causal decision theory is inadequate, let alone build up an intuition about it. In fact, the closest I've seen to a real-worl... (Read more)
in this fictional world we're imagining, when psychologists have said that a device's accuracy was X%, it turned out to be within 1% of X%, 99% of the time.
99% of the time for me, or for other people? I may not be correct in all cases, but I have evidence that I _am_ an outlier on at least some dimensions of behavior and thought. There are numerous topics where I'll make a different choice than 99% of people.
More importantly, when the fiction diverges by that much from the actual universe, it takes a LOT more work to show that any lessons are valid or useful in the real universe.
Right, right. So there is a correlation.
I'll just say that there is no reason to believe that this correlation is very strong.
I once won a mario kart tournament without feeling my hands.
This is very likely my most important idea, and I've been trying to write about it for years, but it's too important to write about it badly, so I haven't ever mustered the courage. I guess I'll just do a quick test round.
It starts with this "hierarchy of needs" model, first popularized my Maslow, that we humans tend to focus on one need at the time.
I like to model this roughly as an ensemble of subag... (Read more)
A few months ago, Olivier Bailleux, a Professor of computer science and reader of my book on Bayesianism, sent me an email. He suggested to apply some of the ideas of the book to examine students. He proposed Bayesian examination.
I believe it to be a brilliant idea, which could have an important impact on how many people think. At least, I think that this is surely worth sharing here.
tl;dr Bayesian examinations seem very important to deploy because they incentivize both probabilistic thinking and intellectual honesty. Yet, as argued by Julia Galef in this talk, incentives seem critical to cha... (Read more)
This is really interesting, thanks, not something I'd thought of.
If the teacher (or whoever set the test) also has a spread of credence over the answers then a Bayesian update would compare the values of P(A), P(B|¬A) and P(C|¬A and ¬B) [1] between the students and teacher. This is my first thought about how I'd create a fair scoring rule for this.
[1] P(D|¬A and ¬B and ¬C) = 1 for all students and teachers so this is screened off by the other answers.
Applying economic models to physiology seems really obvious. For instance:
Yet when I run a google search for the obvious phrase ... (Read more)
Those are all reasonable questions to ask and points to raise, and I'm not going to go to bat defending any of the suggestions I made off the top of my head when writing the original question. The point of the original question was to see if anybody out there had publications asking/answering the sort of questions you pose, and it looks like the answer is "no".
For some of these questions, as you argue, it's possible that the lack of literature is because there really isn't anything interesting to be found. But at least some of thes... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post
This is a brief review of On the Chatham House Rule by Scott Garrabrant.
I tend to be very open about my thoughts and beliefs. However, I naturally am still discrete about a lot of things - things my friends told me privately, personal things about myself, and so on.
This has never been a big deal, figuring out the norms around secrecy. For most of my life it's seemed pretty straightforward, and I've not had problems with it. We all have friends who tell us things in private, and we're true to our word. We've all discovered a fact about someone that's maybe a bit embarrassing or personal, where
... (Read more)I can tell you that if you just want to walk into DeepMind (i.e. past the security gate), you have to sign an NDA.
The dominant model about status in LW seems to be one of relative influence. Necessarily, it's zero-sum. So we throw up our hands and accept that half the community is just going to run a deficit.
Here's a different take: status in the sense of worth. Here's a set of things we like, or here's a set of problems for you to solve, and if you do, you will pass the bar and we will grant you personhood and take you seriously and allow you onto the ark when the world comes crumbling. Worth is positive-sum.
I think both models are useful, but only one of these models underlies the em... (Read more)
People generally only discuss 'status' when they're feeling a lack of it
While this has been true for other posts that I wrote about the subject, this post was actually written from a very peaceful, happy, almost sage-like state of mind, so if you read it that way you'll get closer to what I was trying to say :)
In 2008, Steve Omohundro's foundational Basic AI Drives made important conjectures about what superintelligent goal-directed AIs might do, including gaining as much power as possible to best achieve their goals. Toy models have been constructed in which Omohundro's conjectures bear out, and the supporting philosophical arguments are intuitive. The conjectures have recently been the center of debate between well-known AI researchers.
Instrumental convergence has been heuristically understood as an anticipated risk, but not as a formal phenomenon with a well-understood cause. The goal of this pos
... (Read more)It seems a common reading of my results is that agents tend to seek out states with higher power. I think this is usually right, but it's false in some cases. Here's an excerpt from the paper:
So, just because a state has more resources, doesn't technically mean the agent will go out of its way to reach it. Here's what the relevant current results say: parts of the future allowing you to reach more terminal states are instrumentally convergent, and the formal POWER contributions of different possibilities are approximately proportionally related to their in
... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this postSangha: Part 1
In years past, the word “community” conjured for me images of many people talking to each other, as at a party or a bake sale. When I thought of “finding community”, I thought of looking for a group of people that would frequently interact with each other and also me. It didn’t really sound appealing — lots of chaos, too many people talking at once, constant misunderstandings, and so forth. But I knew that I felt worse and worse over time if I never saw other people. So I entered a “community... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post
Of course, I'm not expecting you to support the idea in the answers, but simply mentioning its conclusion:)
I did a bad job of saying that I'm trying to highlight the attentional failures involved specifically.
Yay learning all the things! Your reviews are fun, also completely understandable putting energy elsewhere. Your energy for more learning is very useful for periodically bouncing myself into more learning.
Suppose a general-population survey shows that people who exercise less, weigh more. You don't have any known direction of time in the data - you don't know which came first, the increased weight or the diminished exercise. And you didn't randomly assign half the population to exercise less; you just surveyed an existing population.
The statisticians who discovered causality were trying to find a way to distinguish, within survey data, the direction of cause and effect - whether, as common sense would have it, more obese people exercise less because they find physical activity less rewarding; o... (Read more)
There are two issues with it.
You can not figure out how something works by only looking at some aspect. Think of the blind people and elephant story.
But it still has a point because with a subsystem that makes predictions the understanding of a system by pure observation becomes impossible.
What it says on the tin.
I agree with the spirit of your point, but I think we would be better served by a category anchored by an example other than a modern gym.
To me the problem is that the modern gym is atomized and transactional: going to the gym is generally a solitary activity, even when you take a class or go with friends, because it's about your workout and not collaboratively achieving something. There are notable exceptions, but most of the time when I think of people going to the gym I imagine them working out as individuals for individual purposes.
Rationality tra... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post
Lately I've come to think of human civilization as largely built on the backs of intelligence and virtue signaling. In other words, civilization depends very much on the positive side effects of (not necessarily conscious) intelligence and virtue signaling, as channeled by various institutions. As evolutionary psychologist Geoffrey Miller says, "it’s all signaling all the way down."
A question I'm trying to figure out now is, what determines the relative proportions of intelligence vs virtue signaling? (Miller argued that intelligence signaling can be considered a kind of virtue signaling, but
... (Read more)I don't think so, jockeying can only get you so far, and even then only in situations where physical reality doesn't matter. If you're in a group of ~50 people, and your rival brings home a rabbit, but you and your friend each bring back half a stag because of your superior coordination capabilities, the guy who brought back the rabbit can say all the clever things he wants, but it's going to be clear to everyone who's actually deserving of status. The two of you will gain a significant fitness advantage over the rest of the members of the tribe, and so you will outcompete them.
Voting theory, also called social choice theory, is the study of the design and evaulation of democratic voting methods (that's the activists' word; game theorists call them "voting mechanisms", engineers call them "electoral algorithms", and political scientists say "electoral formulas"). In other words, for a given list of candidates and voters, a voting method specifies a set of valid ways to fill out a ballot, and, given a valid ballot from each voter, produces an outcome.
(An "electoral system" includes a voting method... (Read more)
As the author, I think this has generally stood the test of time pretty well. There are various changes I'd make if I were doing a rewrite today; but overall, these are minor.
Aside from those generally-minor changes, I think that the key message of this piece remains important to the purpose of Less Wrong. That is to say: making collective decisions, or (equivalently) statements about collective values, is a tough problem; it's important for rationalists; and studying existing theory on this topic is useful.
Here are the specific changes I'd ... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post
This post is eventually about partial agency. However, it's been a somewhat tricky point for me to convey; I take the long route. Epistemic status: slightly crazy.
I've occasionally said that everything boils down to credit assignment problems.
One big area which is "basically credit assignment" is mechanism design. Mechanism design is largely about splitting gains from trade in a way which rewards cooperative behavior and punishes uncooperative behavior. Many problems are partly about mechanism design:
I wrote a post inspired by / sorta responding to this one—see Predictive coding = RL + SL + Bayes + MPC
The Cybersecurity Dilemma: Hacking, Trust, and Fear Between Nations organizes its arguments in a fairly no-nonsense, premise –> premise –> conclusion manner that I thought would be good to summarize with a blog post. You can buy the book here.
Security dilemmas are dangerous because in seeking their own security, states often build capabilities and take actions that can directly threaten the security of other states: often creating impressions of imminent offensive intent and prompting escalation. This is especially true for cybersecurity for several main reason... (Read more)
The book does assume from the start that states want offensive options. I guess it is useful to breakdown the motivations of offensive capabilities. Though the motivations aren’t fully distinct, it matters if a state is intruding as the prelude to or an opening round of a conflict, or if it is just trying to improve its ability to defend itself without necessarily trying to disrupt anything in the network being intruded into. There are totally different motives too, like North Korea installing cryptocurrency miners on other countries’ comput... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post
Cross-posted from Putanumonit where the images show up way bigger. I don't know how to make them bigger on LW.
Imagine that tomorrow everyone on the planet forgets the concept of training basketball skills.
The next day everyone is as good at basketball as they were the previous day, but this talent is assumed to be fixed. No one expects their performance to change over time. No one teaches basketball, although many people continue to play the game for fun.
Geneticists explain that some people are born with better hand-eye... (Read more)
For the record, I didn't actually downvote you--just wanted to share why I suspect others did. I agree with your full reasoning and didn't mean to imply you thought that was the only significant difference. I mostly agree with what habryka is saying, though.
Over the past few days I've been reading about reinforcement learning, because I understood how to make a neural network, say, recognise handwritten digits, but I wasn't sure how at all that could be turned into getting a computer to play Atari games. So: what I've learned so far. Spinning Up's Intro to RL probably explains this better.
(Brief summary, explained properly below: The agent is a neural network which runs in an environment and receives a reward. Each parameter in the neural network is increased in proportion to how much it i... (Read more)(Click to expand thread. ⌘/CTRL+F to Expand All)Cmd/Ctrl F to expand all comments on this post