All of JamesFaville's Comments + Replies

MIRI announces new "Death With Dignity" strategy

There is another very important component of dying with dignity not captured by the probability of success: the badness of our failure state. While any alignment failure would destroy much of what we care about, some alignment failures would be much more horrible than others. Probably the more pessimistic we are about winning, the more we should focus on losing less absolutely (e.g. by researching priorities in worst-case AI safety).

The average North Korean mathematician

I feel conflicted about this post. Its central point as I'm understanding it is that much evidence we commonly encounter in varied domains is only evidence about the abundance of extremal values in some distribution of interest, and whether/how we should update our beliefs about the non-extremal parts of the distribution is very much dependent on our prior beliefs or gears-level understanding of the domain. I think this is a very important idea, and this post explains it well.

Also, felt inspired to search out other explanations of the moments of a distribu... (read more)

Generalized Heat Engine

This is super interesting!

Quick typo note (unless I'm really misreading something): in your setups, you refer to coins that are biased towards tails, but in your analyses, you talk about the coins as though they are biased towards heads.

One is the “cold pool”, in which each coin comes up 1 (i.e. heads) with probability 0.1 and 0 with probability 0.9. The other is the “hot pool”, in which each coin comes up 1 with probability 0.2

 random coins with heads-probability 0.2

We started with only  tails

full compression would require roughly 

... (read more)
Oh shit, that's actually a pretty serious oversight. I'm effectively missing negative signs all over the place. Thanks for catching it!
human psycholinguists: a critical appraisal

As far as I'm aware, there was not (in recent decades at least) any controversy that word/punctuation choice was associative. We even have famous psycholinguistics experiments telling us that thinking of the word "goose" makes us more likely to think of the word "moose" as well as "duck" (linguistic priming is the one type of priming that has held up to the replication crisis as far as I know). Whenever linguists might have bothered to make computational models, I think those would have failed to produce human-like speech because their associative models were not powerful enough.

The appearance of a disagreement in this thread seems to hinge on an ambiguity in the phrase "word choice."

If "word choice" just means something narrow like "selecting which noun you want to use, given that you are picking the inhabitant of a 'slot' in a noun phrase within a structured sentence and have a rough idea of what concept you want to convey," then perhaps priming and other results about perceptions of "word similarity" might tell us something about how it is done. But no one ever thought that kin... (read more)

human psycholinguists: a critical appraisal

This comment does not deserve to be downvoted; I think it's basically correct. GPT-2 is super-interesting as something that pushes the bounds of ML, but is not replicating what goes on under-the-hood with human language production, as Marcus and Pinker were getting at. Writing styles don't seem to reveal anything deep about cognition to me; it's a question of word/punctuation choice, length of sentences, and other quirks that people probably learn associatively as well.

Writing styles don't seem to reveal anything deep about cognition to me; it's a question of word/punctuation choice, length of sentences, and other quirks that people probably learn associatively as well.

But isn't it interesting that the way human linguists thought word/punctuation choice worked in humans failed to produce human-like speech, and yet GPT-2 successfully produces human-like speech? Yes, obviously, it's the babbler instead of the full brain. But that definitely lines up with my internal experience, where I have some 'conceptual realm' that han

... (read more)
Information empathy

Why should we say that someone has "information empathy" instead of saying they possess a "theory of mind"?

Possible reasons: "theory of mind" is an unwieldy term, it might be useful to distinguish in fewer words a theory of mind with respect to beliefs from a theory of mind with respect to preferences, you want to emphasise a connection between empathy and information empathy.

I think if there's established terminology for something we're interesting in discussing, there should be a pretty compelling reason why it doesn't suffice for us.

2Sunny from QAD3y
The context I see this being used in is when you need to accuse somebody of not exhibiting it. It seems like it would usually be too broad to say that somebody is acting as though other minds don't exist; I think this narrower term is more likely to come in handy (your first reason).
On AI and Compute

It felt weird to me to describe shorter timeline projections as "optimistic" and longer ones as "pessimistic"- AI research taking place over a longer period is going to be more likely to give us friendly AI, right?

Hopefully. Yeah, I probably could have used better shorthand.
(Why) Does the Basilisk Argument fail?

This approach can be made a little more formal with FDT/LDT/TDT: being the sort of agent who robustly does not respond to blackmail maximises utility more than being the sort of agent who sometimes gives in to blackmail, because you will not wind up in situations where you're being blackmailed.

Subjunctive Tenses Unnecessary for Rationalists?

The subjunctive mood and really anything involving modality is complicated. Paul Portner has a book on mood which is probably a good overview if you're willing to get technical. Right now I think of moods as expressing presuppositions on the set of possible worlds you quantify over in a clause. I don't think it's often a good idea to try to get people to speak a native language in a way incompatible with the language as they acquired it in childhood; it adds extra cognitive load and probably doesn't affect how people reason (the exception being giving them new words and categories, which I think can clearly help reasoning in some circumstances).

Advice Wanted; Reconcile with religious parent

I'm atheist and had an awesome Yom Kippur this year, so believing in God isn't a pre-req for going to services and not being unhappy. I think it would be sad if your father's kids gave up ritual practices that were especially meaningful to him and presumably to his ancestors. I think it would be sad if you sat through services that were really unpleasant for you year after year. I think it would be really sad if your relationship with your father blew up over this.

I think the happiest outcome would be that you wind up finding bits of the high holidays that

... (read more)
Wirehead your Chickens

I've seen this discussed before by Rob Wiblin and Lewis Bollard on the 80,000 Hours podcast (edit: tomsittler actually beat me to the punch in mentioning this).

Robert Wiblin: Could we take that even further and ultimately make animals that have just amazing lives that are just constantly ecstatic like they’re on heroin or some other drug that makes people feel very good all the time whenever they are in the farm and they say, “Well, the problem has basically been solved because the animals are living great lives”?
Lewis Bollard: Yeah, so I think this
... (read more)
Beyond Astronomical Waste

Nothing, if your definition of a copy is sufficiently general :-)

Am I understanding you right that you believe in something like a computational theory of identity and think there's some sort of bound on how complex something we'd attribute moral patienthood or interestingness to can get? I agree with the former, but don't see much reason for believing the latter.

4Jameson Quinn4y
I have no idea if there is such a bound. I will never have any idea if there is such a bound, and I suspect that neither will any entity in this universe. Given that fact, I'd rather make the assumption that doesn't turn me stupid when Pascal's Wager comes up.
A Rationalist Argument for Voting

I just listened to a great talk by Nick Bostrom I'd managed to miss before now which mentions some considerations in favor and opposed to voting. He does this to illustrate a general trend that in certain domains it's easy to come across knock-down arguments ("crucial considerations") that invalidate or at least strongly counter previous knock-down arguments. Hope I summarized that OK!

When I last went to the polls, I think my main motivation for doing so was functional decision theory.

Beyond Astronomical Waste

I feel like scope insensitivity is something to worry about here. I'd be really happy to learn that humanity will manage to take good care of our cosmic endowment but my happiness wouldn't scale properly with the amount of value at stake if I learned we took good care of a super-cosmic endowment. I think that's the result of my inability to grasp the quantities involved rather than a true reflection of my extrapolated values, however.

My concern is more that reasoning about entities in simpler universes capable of conducting acausal trades wi... (read more)

3Jameson Quinn4y
When you're faced with numbers like 3^^^3, scope insensitivity is the correct response. A googolplex is already enough to hold every possible configuration of Life as we know it. "Hamlet, but with extra commas in these three places, performed by intelligent starfish" is in there somewhere in over a googol different varieties. What, then, does 3^^^3 add except more copies of the same?

I actually like the idea of building a "rationalist pantheon" to give us handy, agenty names for important but difficult concepts. This requires more clearly specifying what the concept being named is: can you clarify a bit? Love Wizard of Earthsea, but don't get what you're pointing at here.

It is Fear and the many ways it is used in society and can make a potential problem seem bigger than it is. In the general things like FUD [,_uncertainty_and_doubt]; a concrete example of that being the red scare []. Often it seems to have an existence bigger than any individual, which is why it got made a member of the pantheon, albeit a minor one With regards to the Group, people have found fear of the Other easier to form []. Obligatory sociology potential non-replicability warning.
Is skilled hunting unethical?

I think normal priors on moral beliefs come from a combination of:

  • Moral intuitions
  • Reasons for belief that upon reflection, we would accept as valid (e.g. desire for parsimony with other high-level moral intuitions, empirical discoveries like "vaccines reduce disease prevalence")
  • Reasons for belief that upon reflection, we would not accept as valid (e.g. selfish desires, societal norms that upon reflection we would consider arbitrary, shying away from the dark world)

I think the "Disney test" is useful in that it seems like it depends mu... (read more)

Is skilled hunting unethical?

I don't think the vaccination example shows that the heuristic is flawed: in the case of vaccinations, we do have strong evidence that vaccinations are net-positive (since we know their impact on disease prevalance, and know how much suffering there can be associated with vaccinatable diseases). So if we start with a prior that vaccinations are evil, we quickly update to the belief that vaccinations are good based on the strength of the evidence. This is why I phrased the section in terms of prior-setting instead of evidence, even though I'm a li... (read more)

The way we test our heuristics is by seeing if they point to the correct conclusions or not, and the way that we verify whether or not the conclusion is correct is with evidence. A single example is only a single example, of course, but I don't see how the failure mode can be illustrated any more clearly than in the case of vaccines -- and precisely because of the strong evidence we have that our initial impulses are misdirected here. What kind of example are you looking for, if it's supposed to satisfy the criteria of "justifiably and convincingly show that the heuristic is bad" and "no strong evidence that the heuristic is wrong here"? I'll try to rephrase to see if it makes my point any clearer: Yes, of all things that children immediately see as bad, most are genuinely bad. Vaccines may be good, but sharing heroin needles under the bridge is bad, stepping on nails is bad, and getting a bull horn through your leg is bad. It's not a bad place to start. However, if you hear a mentally healthy adult (someone who was once a child and has access to and uses this same starting point) talking about letting someone cut him open and take part of his body out, my first thought is that he was probably convinced to make an exception for surgeons and tumors/infected appendix or something. I do not think it calls for anywhere near enough suspicion to drive one to think "I need to remind this person that getting cut open is bad and that even children know this". It's not that strong a heuristic and we should expect it to be overruled frequently. Bringing it up, even as a "prior", is suggesting that people are under-weighting this heuristic relative to it's actual usefulness. This might be a solid point if there were evidence that things are simple , and that children are morally superior to adults. However, children are little assholes, and "you're behaving like a child" is not a compliment. It might be a good thing to point out if your audience literally hadn't made it fa
Is skilled hunting unethical?

Thanks for the feedback Raemon!

Concrete Concerns

I'd like to see ["when predators are removed from a system, a default thing that seems to happen is that death-by-predator is replaced by death-by-starvation" and "how do you do population control without hunting?"] at least touched on in wild-animal-suffering pieces

I'd like to see those talked about too! The reason I didn't is I really don't have any insights on how to do population control without hunting, or on which specific interventions for reducing wild animal s... (read more)

Rationalist Lent

Why do you think we should be more worried about reading fiction? Associated addictiveness, time consumption, escapism?

I replied to Kaj above.
I chose to give up fiction for Rationalist Lent (before reading the comments and before seeing this exchange). I've been observing my fiction (particularly sci-fi) habits for the past year or so, and have tried to reduce/improve them, including making it a major focus both times I attended a CFAR workshop. This has been only marginally successful. The behavior feels addictive in both the sense that I lose sleep or forget other important things due to it, and in the sense that it's hard to stop. When I investigate the desire to read scifi, it seems to be connected to desires around intellectual exploration, a sense of wonder/excitement, and some sort of displaced social desire around "camaraderie in service of large-scale goals". This makes me think it's a pica, and that I would be better served by using that energy to get more of those things in the real world. So, all of the above, at least in my case.
What the Universe Wants: Anthropics from the POV of Self-Replication

What I'm taking away from this is that if (i) it is possible for child universes to be created from parent universes, and if (ii) the "fertility" of a child universe is positively correlated with that of its parent universe, then we should expect to live in a universe which will create lots of fertile child universes, whether this is accomplished through a natural process or as you suggest through inhabitants of the universe creating fertile child universes artificially.

I think that's a cool concept, and I wrote a quick Python script f... (read more)

Essentially, I read this as an attempt at continental philosophy rather than analytic philosophy, and I don't find continental-style work very interesting or useful. I believe you that the post is meaningful and thoughtful, but the costs of time or effort to understand the meanings or thoughts you're driving at are too high for me at least. I think trying to lay things out in a more organized and explicit manner would be helpful for your readers and possibly for you in developing these thoughts.

I don't want to get too precise about answerin... (read more)

That's your personal opinion about their works. Continental rationalists have been useful to so many. So you'd dismiss any work based on the return on investment ratio? Not all thoughts need to be explicit, especially when describing links between ideas. Have you ever watched the series Connections < (TV_series)>? If not I would highly advise watching it.

I'm downvoting this post because I don't understand it even after your reply above, and the amount of negative karma currently on the post indicates to me that it's probably not my fault. It's possible to write a poetic and meaningful post about a topic and pleasant when someone has done so well, but I think you're better off first trying to state explicitly whatever you're trying to state to make sure the ideas are fundamentally plausible. I'm skeptical that meditations on a topic of this character are actually helpful to truth-seeking, but I might be typical-minding you.

An Artificial paradise made by humans. (A bit Sci-fi idea)

I'm downvoting this because it appears to be a low-effort post which doesn't contribute or synthesize any interesting ideas. Prime Intellect is the novel that first comes to mind as discussing some of what you're talking about, but several chapters are very disturbing, and there's probably better examples out there. If you have Netflix, San Junipero (Season 3 Episode 4) of Black Mirror is fantastic and very relevant.

Well, yes, that's a low effort post. (Late night and I'm not someone with experience in this) Thank you! The idea got stuck in my head but I don't know how to well express it... I'm kinda sorry for this post.
The Loudest Alarm Is Probably False

I like this post's brevity, its usefulness, and the nice call-to-action at the end.

I found the last six paragraphs of this piece extremely inspiring, to the extent that I think it nonnegligably raised the likelihood that I'll be taking "exceptional action" myself. I didn't personally connect much with the first part, though it was interesting. Did you used to want to want your reaction to idiocy be “'how can I help'”, even when it wasn't?

I used to get mad when people didn't know things I thought of as basic knowledge or notice things I thought of as obvious, and thought that expressing that would make them remember or pay more attention as well as being personally satisfying. Expressing that as anger got me in trouble a few times and also didn't get them to notice related but non-identical things, so I tried different reactions before settling on sounding kind and concerned and trying to explain the problem. This slightly raised the likelyhood of most people doing better next time, and eventually got me a useful reputation of being kind which was useful. I'd say I've been maintaining that reaction as a deliberate mask a majority of the time for the last ~8 years. I never bothered trying to change my internal reaction, but I acted (in the sense of an actor on a stage) the way that got me the best results until it became an automatic reflex. Now I find when this happens my thoughts don't flow "What an idiot, crap can't say that out loud, right okay fake being a kind person, what would a kind person say?" They flow "Hrm, that wasn't what I wish they'd done. I wonder what I can do to make this less likely to happen again, possibly by making them feel good about doing what I'd rather they'd done instead?" I don't think I could pinpoint the moment things changed over, but it's very different when I pay attention to it. I wrote paragraphs three to nine ("I think I might have wanted. . ." through "What do I want to be?" partially because explaining what I think now is made easier by explaining what I thought then and what changed, largely because I feel very strange responding to a call for heroes and was more comfortable responding with the caveat that I am a highly noncentral example, and lastely because it's the kind of thing I wish someone else had written and I had read when I was younger. It may be that this section could have been written better- I'm working without an editor here as none of this
The essay "Interstellar Communication Using Microbes: Implications for SETI" has implications for The Great Filter.

The case against "geospermia" here is vastly overstated: there's been a lot of research over the past decade or two establishing very plausible pathways for terrestrial abiogensis. If you're interested, read through some work coming out of Jack Szostak's lab (there's a recent review article here). I'm not as familiar with the literature on prebiotic chemistry as I am with the literature on protocell formation, but I know we've found amino acids on meteorites, and it wouldn't be surprising if they and perhaps som... (read more)

Rationalist Politicians

Have a look at 80K's (very brief) career profile for party politics. My rough sense is that efective altruists generally agree that pursuing elected office can be a very high-impact career path for individuals particularly well-suited to it, but think that even with an exceptional candidate succeeding is very difficult.

Improvement Without Superstition

Upvoted mostly for surprising examples about obstetrics and CF treatment and for a cool choice of topic. I think your question, "when is one like the doctors saving CF patients and when is one like the doctors doing super-radical mastectomies?" is an important one to ask, and distinct from questions about modest epistomology.

Say there is a set of available actions of which a subset have been studied intensively enough that their utility is known with high degree of certainty, but that the utility of the other available actions in is un... (read more),,,,,,

4Zachary Jacobi4y
This formulation reminds me somewhat of the Bayesian approach to the likelihood of research being true from Ionnidis 2005 [] (Why Most Published Research Findings Are False).
Strategic High Skill Immigration

I don't have much of a thoughtful opinion on the question at hand yet (though I have some questions below), but I wanted to express a deep appreciation for your use of detail elements: it really helps readability!

One concern I would want to see addressed is an estimation of negative effects of a "brain drain" on regional economies- if a focused high-skilled immigration policy has the potential to exacerbate global poverty, the argument that it has a positive impact on the far future needs to be very compelling. So would these economic costs ... (read more)

I suspect high skill immigration directly helps probably with other risks more than with AI due to the potential ease of espionage with software (though some huge data sets are impractical to steal). However, as risks from AI are likely more immanent, most of the net benefit will likely be concentrated with reductions in risk there, provided such changes are done carefully. As for brain drain, it seems to be a net economic benefit to both sides, even if one side gets further ahead in the strategic sense: [] Basically, smart people go places where they earn more, and send back larger remitances. Some plausibly good effect on home country institutions too:,_human_rights_and_liberal_values [,_human_rights_and_liberal_values]
Motivating a Semantics of Logical Counterfactuals

(Disclaimer: There's a good chance you've already thought about this.)

In general, if you want to understand a system (construal of meaning) forming a model of the output of that system (truth-conditions and felicity judgements) is very helpful. So if you're interested in understanding how counterfactual statements are interpreted, I think the formal semantics literature is the right place to start (try digging through the references here, for example).

Fish oil and the self-critical brain loop

Muting the self-critical brain loop (and thanks for that terminology!) is something I'm very interested in. Have you investigated vegan alternatives to fish oil at all?

My impression is that algae oil is more similar to fish oil than flax, if you decide to experiment - it's where fish get their omega-3 from [].
A. No I have not. B. You need to check if it does something similar before assuming that it will. Then flax oil is an option. However there is a debate as to whether it is the same thing or causes the same effect or is absorbed the same way. (fish oil vs flax oil). I don't know your stance on krill oil either and I would look there too. C. If the problem is bad enough, reconsider your dietary restrictions, speak to a professional or experiment with your intake. I am pretty hesitant to stop taking the fish oil and run other tests given that I get 24-36 hours before I inevitably get bitten by a critical loop.
Open thread, Jan. 16 - Jan. 22, 2016

At what age do you all think people have the greatest moral status? I'm tempted to say that young children (maybe aged 2-10 or so) are more important than adolescents, adults, or infants, but don't have any particularly strong arguments for why that might be the case.

this may be an odd counter position to the normal. I think that adults are more morally valuable because they have proven their ability to not be murderous etc. Or possibly also to not be the next ghandi. Children could go either way.
It depends very much on the context. In many instances where we want to save lives QALY are a good metric. In other cases like deciding how should be able to sit down in a bus, the metric is worthless.
If you think in terms of QALYs, that could be one reason to prefer interventions targeted at children. Your average child has more life to live than your average adult, so if you permanently improve their quality of life from 0.8 QALYs per year to 0.95 QALYs per year, that would result in a larger QALY change than the same intervention on the adult. This argument has numerous flaws. One which comes to mind immediately are that many interventions are not so long lasting, so both adults and children would presumably gain the same. It also is tied to particular forms of utilitarianism one might not subscribe to.
I don't think children actually have greater moral status, but harming children or allowing children to be harmed carries more evidence of depraved/dangerous mental state because it goes against the ethic of care we are supposed to naturally feel toward children.