# All of LawrenceC's Comments + Replies

Instrumental Occam?

DanielFilan asked me to comment on this paragraph:

Yet, even in the relatively formal world of machine learning, the practice seems contrary to this. When you are optimizing a neural network, you don't actually care that much whether it's something like a hypothesis (making predictions) or something like a policy (carrying out actions). You apply the same kind of regularization either way, as far as I understand (regularization being the machine-learner's generalization of Occam).

AFAIK, it's not actually standard to regularize RL policies the same way y

2abramdemski2yExcellent, thanks for the comment! I really appreciate the correction. That's quite interesting.
How good a proxy for accuracy is precision?

I wonder if that's because they're using the ISO definition of accuracy? A quick google search for these diagrams led me to this reddit thread, where the discussion below reflects the fact that people use different definitions of accuracy.

EDIT: here's a diagram of the form that Elizabeth is complaining about (source: the aforementioned reddit thread):

Does it become easier, or harder, for the world to coordinate around not building AGI as time goes on?

Hyperbolic discounting leads to preferences reversals over time: the classic example is always preferring a certain $1 now to$2 tomorrow, but preferring a certain $2 in a week to$1 in 6 days. This is a pretty clear sign that it never "should" be done - An agent with these preferences might find themselves paying a cent to switch from $1 in 6 days to$2 in 7, then, 6 days later, paying another cent to switch it back and get th \$1 immediately.

However, in practice, even rational agents might exhibit hyperbolic discounting like preferences (though no prefere

How good a proxy for accuracy is precision?

I believe your definition of accuracy differs from the ISO definition (which is the usage I learned in undergrad statistics classes, and also the usage most online sources seem to agree with): a measurement is accurate insofar as it is close to the true value. By this definition, the reason the second graph is accurate but not precise is because all the points are close to the true value. I'll be using that definition in the remainder of my post. That being said, Wikipedia does claim your usage is the more common usage of the word.

I don't have a... (read more)

2Elizabeth2yLooks like what I'm calling accuracy ISO calls "trueness", and ISO!accuracy is a combination of trueness and precision.
2017 LessWrong Survey

I took the survey!

Open thread, August 14 - August 20, 2017

Why do you think this doesn't exist?

Open thread, August 14 - August 20, 2017

For what it's worth, though, as far as I can tell we don't have the ability to create an AI that will reliably maximize the number of paperclips in the real world, even with infinite computing power. As Manfred said, model-based goals seems to be a promising research direction for getting AIs to care about the real world, but we don't currently have the ability to get such an AI to reliably actually "value paperclips". There are a lot of problems with model-based goals that occur even in the POMDP setting, let alone when the agent's model of the ... (read more)

Game Theory & The Golden Rule (From Reddit)

The reason for this is because of the 5% chance for mistakes. Copycat does worse vs both Simpleton and Copycat than Simpleton does against itself.

0MrMind4yIt reads like the Sokal paper, but it's actually not. Simply the OP didn't bother to cover the inferential distances.
0entirelyuseless4yIt looks like something cut and paste but that is probably because common_law does not know how to use the formatting.
The dark arts: Examples from the Harris-Adams conversation

I think the term "Dark Arts" is used by many in the community to refer to generic, truth-agnostic ways of getting people to change their mind. I agree that Scott Adams demonstrates mastery of persuasion techniques, and that this is indeed not necessarily evidence that he is not a "rationalist".

However, the specific claim made by James_Miller is that it is a "model rationalist disagreement". I think that since Adams used the persuasion techniques that Stabilizer mentioned above, it's pretty clear that it isn't a model rationalist disagreement.

2DanArmak4yI agree, and I didn't mean to imply otherwise.
MILA gets a grant for AI safety research

Awesome! I heard a rumor that David Krueger (one of Bengio's grad students) is one of the main people pushing the safety initiative there, can anyone confirm?

1Vika4yYes. He runs AI safety meetups at MILA, and played a significant role in getting Yoshua Bengio more interested in safety.
0kvas_duplicate0.16361211296761184yI know that David Krueger is one of the people working with 80000 hours on helping people to get into the AI safety field. He also organized a related google group [https://groups.google.com/forum/#!forum/david-kruegers-80k-people] .
Book Review: Mathematics for Computer Science (Suggestion for MIRI Research Guide)

Thanks for the review! I definitely had the sense that Rosen was doing a lot of hand holding and handwaving - it's certainly a very introductory text. I've read both Rosen and Eppstein and actually found Rosen better. The discrete math class I took in college used Scheinerman's Mathematics: A Discrete Introduction, which I also found to be worse than Rosen.

At the time I actually really enjoyed the fact that Rosen went on tangents and helped me learn how to write a proof, since I was relatively lacking in mathematical maturity. I'd add that Rosen does cove... (read more)

Thanks Søren! Could I ask what you're planning on covering in the future? Is this mainly going to be a technical or non-technical reading group?

I noticed that your group seems to have covered a lot of the basic readings on AI Safety, but I'm curious what your future plans.

0SoerenE5yThere are no specific plans - at the end of each session we discuss briefly what we should read for next time. I expect it will remain a mostly non-technical reading group.
Ideas for Next Generation Prediction Technologies

I haven’t heard much about machine learning used for forecast aggregation. It would seem to me like many, many factors could be useful in aggregating forecasts. For instance, some elements of one’s social media profile may be indicative of their forecasting ability. Perhaps information about the educational differences between multiple individuals could provide insight on how correlated their knowledge is.

I think people are looking in to it: The Good Judgment Project team used simple machine learning algorithms as part of their submission to IARPA durin... (read more)

Stupid Questions December 2016

I'm just saying that you have an infinite sequence of spheres with the property X. You're saying that because the sequence is infinite I can't point to the last sphere and therefore can't say anything about it. I'm saying that because all spheres in this sequence have the property X, it doesn't matter that the sequence is infinite.

This isn't true in general. Each natural number is finite, but the limit of the natural numbers is infinite. Just because each of the intermediate shapes has property doesn't mean the limiting shape has property X. Notably, in this case each of the intermediate shapes has a non-zero amount of empty space, but the limiting shape has no empty space.

Stupid Questions December 2016

Maybe think about the problem this way:

Suppose there was some small ball inside of your super-packed structure that isn't filled. Then we can fill this ball, and so the structure isn't super-packed. It follows that the volume of the empty space inside of your structure has to be 0.

Now, what does your super-packed structure look like, given that it's a empty cube that's been filled?

EDIT: Nevermind, just saw that Villiam gave a similar answer.

"Flinching away from truth” is often about *protecting* the epistemology

I think they're equivalent in a sense, but that bucket diagrams are still useful. A bucket can also occur when you conflate multiple causal nodes. So in the first example, the kid might not even have a conscious idea that there are three distinct causal nodes ("spelled oshun wrong", "I can't write", "I can't be a writer"), but instead treats them as a single node. If you're able to catch the flinch, introspect, and notice that there are actually three nodes, you're already a big part of the way there.

The bucket diagrams are too coarse, I think; they don't keep track of what's causing what and in what direction. That makes it harder to know what causal aliefs to inspect. And when you ask yourself questions like "what would be bad about knowing X?" you usually already get the answer in the form of a causal alief: "because then Y." So the information's already there; why not encode it in your diagram?

Nassim Taleb on Election Forecasting

Thanks for posting this! I have a longer reply to Taleb's post that I'll post soon. But first:

When you read Silver (or your preferred reputable election forecaster, I like Andrew Gelman) post their forecasts prior to the election, do you accept them as equal or better than any estimate you could come up with? Or do you do a mental adjustment or discounting based on some factor you think they've left out?

I think it depends on the model. First, note that all forecasting models only take into account a specific set of signals. If there are factors influen... (read more)

Nassim Taleb on Election Forecasting

We have the election estimate F a function of a state variable W, a Wiener process WLOG

That doesn't look like a reasonable starting point to me.

That's fine actually, if you assume your forecasts are continuous in time, then they're continuous martingales and thus equivalent to some time-changed Wiener process. (EDIT: your forecasts need not be continuous, my bad.) The problem is that he doesn't take into the time transformation when he claims that you need to weight your signal by 1/sqrt(t).

He also has a typo in his statement of Ito's Lemma which might affect his derivation. I'll check his math later.

Celebrating All Who Are in Effective Altruism

Can you give a link to posts showing elitism in EA that weren't written in response to this one?

2Gleb_Tsipursky6ySure! http://effective-altruism.com/ea/op/eas_image_problem/ [http://effective-altruism.com/ea/op/eas_image_problem/]
Open thread, Dec. 21 - Dec. 27, 2015

Wait, how would you get P(H) = 1?

0[anonymous]6yFine. p(H) = 0.5, p(H|A) = 0.2, p(H|B) = 0.15, p(H|C) = 0.15 It's not really relevant to the problem.
An Introduction to Löb's Theorem in MIRI Research

This is several months too late, but yes! Gödel Machines runs into the Löbstacle, as seen in this MIRI paper. From the paper:

it is clear that the obstacles we have encountered apply to Gödel machines as well. Consider a Gödel machine G1 whose fallback policy would “rewrite” it into another Gödel machine G2 with the same suggester (proof searcher, in Schmidhuber’s terminology). G1’s suggester now wants to prove that it is acceptable to instead rewrite itself into G0 2 , a Gödel machine with a very slightly modified proof searcher. It must prove that G0 2

Rationality Compendium: Principle 1 - A rational agent, given its capabilities and the situation it is in, is one that thinks and acts optimally

No, it isn't. Being curious is a good heuristic for most people, because most people are in the region where information gathering is cheaper than the expected value of gathering information. I don't think we disagree on anything concrete: I don't claim that it's rational in itself a priori but is a fairly good heuristic.

Rationality Compendium: Principle 1 - A rational agent, given its capabilities and the situation it is in, is one that thinks and acts optimally

I agree denotationally, but object connotatively with 'rationality is systemized winning', so I left it out. I feel that it would take too long to get rid of the connotation of competition that I believe is associated with 'winning'. The other point that would need to be delved into is: what exactly does the rationalist win at? I believe by winning Elizer meant winning at newcomb's problem, but the idea of winning is normally extended into everything.

I think that Eliezer has disavowed using this statement precisely because of the connotations that peop... (read more)

1ScottL6yI added in a quote from Baron.
Rationality Compendium: Principle 1 - A rational agent, given its capabilities and the situation it is in, is one that thinks and acts optimally

I'm not sure if this is correct, but my best guess is:

It maximizes utility, in so far as most goals are better achieved with more information, and people tend to systematically underestimate the value of collecting more information or suffer from biases that prevent them from acquiring this information. Or, in other words, curiosity is virtuous because humans are bounded and flawed agents, and it helps rectify the biases that we fall prey to. Just like being quick to update on evidence is a virtue, and scholarship is a virtue.

1Lumifer6yThere are a couple of problems here. First is the usual thing forgotten on LW -- costs. "More information" is worthwhile iff its benefits outweigh the costs of acquiring it. Second, your argument implies that, say, attempting to read the entire Wikipedia (or Encyclopedia Britannica if you are worried about stability) from start to finish would be a rational thing to do. Would it?
Complex Novelty

Yes, I think he recognizes this in this post. He also writes about this (from a slightly different perspective) in high challenge.

The Error of Crowds

Results from the Good Judgment Project suggest that putting people into teams lets them significantly outperform (have lower Brier's scores than) predictions from both (unweighted) averaging of probabilities and the (admittedly also unweighted) averaging of probability estimates from the better portion of predictors. This seems to offer weak evidence that what goes on in a group is not simple averaging.

You have a set amount of "weirdness points". Spend them wisely.

That being said, I'm confident that I would pass ideological turing tests.

Cool! You can try taking them here: http://blacker.caltech.edu/itt/

State-Space of Background Assumptions

Wow, that was a long survey. Done! I'm not sure how good my answers were, like others mentioned a lot of the questions felt underspecified.

MIRI's Approach

Thanks Nate, this is a great summary of the case for MIRI's approach!

Out of curiosity, is there an example where algorithms led to solutions other than Bird and Layzell? That paper seems to be cited a lot in MIRI's writings.

It's cited a lot in MIRI's writing because it's the first example that pops to my mind, and I'm the one who wrote all the writings where it appears :-p

For other examples, see maybe "Artificial Evolution in the Physical World" (Thompson, 1997) or "Computational Genetics, Physiology, Metabolism, Neural Systems, Learning, Vision, and Behavior or PolyWorld: Life in a New Context." (Yaeger, 1994). IIRC.

MIRI's Approach

I'm not sure what you're looking for in terms of the PAC-learning summary, but for a quick intro, there's this set of slides or these two lectures notes from Scott Aaronson. For a more detailed review of the literature in all the field up until the mid 1990s, there's this paper by David Haussler, though given its length you might as well read up Kearns and Vazirani's 1994 textbook on the subject. I haven't been able to find a more recent review of the literature though - if anyone had a link that'd be great.

The Brain as a Universal Learning Machine

This was a great post, thanks!

One thing I'm curious about is how the ULH explains to the fact that human thought seems to be divided into System 1/System 2 - is this solely a matter of education history?

Making Beliefs Pay Rent (in Anticipated Experiences)

You're definitely right that there's some areas where it's easier to make beliefs pay rent than others! I think there's two replies to your concern:

1) First, many theories from math DO pay rent (the ones I'm most aware of are statistics and computer-science related ones). For example, better algorithms in theory (say Strassen's algorithm for multiplying matrices) often correspond to better results in practice. Even more abstract stuff like number theory or recursion theory do yield testable predictions.

2) Even things that can't pay rent directly can be logical implications of other things that pay rent. Eliezer wrote about this kind of reasoning here.

"Mystics exult in mystery and want it to stay mysterious. Scientists exult in mystery for a different reason: it gives them something to do."

Richard Dawkins, The God Delusion, on the topic of mysterious answers to mysterious questions.

[FINAL CHAPTER] Harry Potter and the Methods of Rationality discussion thread, March 2015, chapter 122

Here's a thing that's been bugging me for a while.

For Gryffindors there's "Gryffindorks". Are there any similarly good insults for the other three houses?

Open Thread, May 25 - May 31, 2015

I've noticed recently that listening to music with lyrics significantly hampers comprehension for reading texts as well as essay-writing ability, but has no (or even slightly positive) effects on doing math. My informal model of the problem is that the words of the song disrupt the words being formed in my head. Has anyone else experienced anything similar?

1bbleeker7yIt is why I used to doodle in class, especially history class, instead of making notes. If I tried to make notes, the words I was writing down interfered with the words I was listening to, so I'd miss what the teacher was saying next.
1polymathwannabe7yI haven't worked with math in years, but yes, lyrics disturb my reading. I now prefer to pair verbal music with non-verbal tasks, and non-verbal music with verbal tasks.
Nick Bostrom's TED talk on Superintelligence is now online

I'm not sure your argument proves your claim. I think what you've shown is that there exist reasons other than the inability to create perfect boxes to care about the value alignment problem.

We can flip your argument around and apply it to your claim: imagine a world where there was only one team with the ability to make superintelligent AI. I would argue that it'll still be extremely unsafe to build an AI and try to box it. I don't think that this lets me conclude that a lack of boxing ability is the true reason that the value alignment problem is so important.

2WilliamKiely7yI agree that there are several reasons why solving the value alignment problem is important. Note that when I said that Bostrom should "modify" his reply I didn't mean that he should make a different point instead of the point he made, but rather meant that he should make another point in addition to the point he already made. As I said:
Open Thread, Apr. 20 - Apr. 26, 2015

And then 7 days later, you die.

0ChristianKl7yThe prediction was only about giving the money. Not about it permanently staying with the oracle.
Book Review: Discrete Mathematics and Its Applications (MIRI Course List)

Wow. That's pretty impressive.

If you have a decent background in Math already, I've been told that Knuth's Concrete Mathematics might be more interesting (though it's really not appropriate as an introductory text). I've skimmed through a copy, and it seems to cover series and number theory at a much higher level, if that's what you're looking for.

0[anonymous]7yeditremove
Book Review: Discrete Mathematics and Its Applications (MIRI Course List)

In my experience there have been three kinds of books: easy books, which I can skim and then do the exercises for, medium books, which I can read carefully one or two times and then do the exercises for, and hard books, which I need to read multiple times + take notes on to do the exercises for.

In most cases I try to do a majority of the exercises either in the sections indicated by the research guide, or, in the case where the research guide doesn't offer any section numbers, the whole textbook.

Book Review: Discrete Mathematics and Its Applications (MIRI Course List)

Okay, cool! Word of warning, though, I don't think the MIRI list isn't really good for people just starting out. Most of the books assume a decent amount of mathematical background. They're also oriented toward a specific goal (and most people probably don't know half the stuff on the list).

If you insist on using the MIRI list, I recommend starting with either this one, the Linear Algebra Book, or the Logic and Computability book. They're well written and don't require much mathematical background.

Speaking of which, how much math background do you have?

1SanguineEmpiricist7yHow thorough were you? What chapters/sections did you do?
1SanguineEmpiricist7yWell over the last year I've been studying Feller Vol 1, Probability via Expectation, Papoulis's probability book , and Abbot, Bressoud's book, and Strichartz. I also collect a lot of math books so I know random stuff but I definitely just want to get the plumbing right. I should probably just stick with one of each, I did discrete a while ago but that was before I fixed a few things causing major productivity losses for me so i'm interested to redoing everything now my executive functions aren't depressed. I'm thinking about getting epp as opposed to rosen
2SanguineEmpiricist7yI always did mathematics heuristically so I just want to know something every one else knows. So I'm going to do the miri list too.
Why isn't the following decision theory optimal?

Actually, if you push the precommittment time all the way back, this sounds a lot like an informal version of Updateless Decision Theory, which, by the way, seems to get everything that TDT gets right, plus counterfactual mugging and a lot of experiments that TDT gets wrong.

5itaibn07yAre you implying that UDT is formal?
Book Review: Discrete Mathematics and Its Applications (MIRI Course List)

Yes, I think that's true. There are gaps, but they're mainly "trust me" results way out of the scope of the book, like the existence of NP-complete problems and so forth. He definitely doesn't have proofs that require large leaps in intuition.

[FINAL CHAPTER] Harry Potter and the Methods of Rationality discussion thread, March 2015, chapter 122

I think he's referring to the definition of ambition Quirrel uses in Chapter 70:

The Defense Professor's fingers idly spun the button, turning it over and over. "Then again, only a very few folk ever do anything interesting with their lives. What does it matter to you if they are mostly witches or mostly wizards, so long as you are not among them? And I suspect you will not be among them, Miss Davis; for although you are ambitious, you have no ambition."

1shminux7yI guess it depends on what one finds interesting. Clearly QQ/LV found taking over Magical Britain interesting enough.
How to debate when authority is questioned, but really not needed?

If you're looking for well-policed blogs, you can try Slate Star Codex and any of the other "rationality blogs" listed in the LW wiki.