Curious, what do you think now that GPT-4 is out?
When I first saw this post it was at -1 karma, which didn't make much sense to me, so I upvoted it back to zero. Can anyone who downvoted it share their reasoning?
if there is any way of fixing this mess, it's going to involve clarifying conflicts rather than obfuscating them
This immediately brought to mind John Nerst's erisology. I've been paying attention to it for a while, but I don't see it much here (speaking as a decade-long lurker); I wonder why.
Human/Machine Intelligence Parity by 2040? on Metaculus has a pretty high bar for human-level intelligence:
...Assume that prior to 2040, a generalized intelligence test will be administered as follows. A team of three expert interviewers will interact with a candidate machine system (MS) and three humans (3H). The humans will be graduate students in each of physics, mathematics and computer science from one of the top 25 research universities (per some recognized list), chosen independently of the interviewers. The interviewers will electronically communicate
I feel like he was falling into a kind of fallacy. He observed that a concept isn't entirely coherent, rejected the concept.
My go-to writeup on this is Luke Muehlhauser's Imprecise definitions can still be useful section of his What is Intelligence? MIRI essay written in 2013, which discusses the question of operationalizing the concept of "self-driving car":
......consider the concept of a “self-driving car,” which has been given a variety of vague definitions since the 1930s. Would a car guided by a buried cable qualify? What about a modified 1955 Studebaker
I agree with this comment, and I'm confused why it's so disagreed with (-6 agreement karma vs +11 overall). Can anyone who disagreed explain their reasoning?
Apparently Jeff Bezos used to do something like this with his regular "question mark emails", which struck me as interesting in the context of an organization as large and complex as Amazon. Here's what it's like from the perspective of one recipient (partial quote, more at the link):
...About a month after I started at Amazon I got an email from my boss that was a forward of an email Jeff sent him. The email that Jeff had sent read as follows:
“?”
That was it.
Attached below the “?” was an email from a customer to Jeff telling him he (the customer) takes a long
Where are you going with this line of questioning?
If it's high-quality distillation you're interested in, you don't necessarily need a PhD. I'm thinking of e.g. David Roodman, now a senior advisor at Open Philanthropy. He majored in math, then did a year-long independent study in economics and public policy, and has basically been self-taught ever since. Holden Karnofsky considers what he does extremely valuable:
...David Roodman, who is basically the person that I consider the gold standard of a critical evidence reviewer, someone who can really dig on a complicated literature and come up with the answers, h
Yeah, I agree that's a weird way to define "high-dimensional". I'm more partial to defining it as "when the curse of dimensionality becomes a concern", which is less precise but more useful.
in the minds of people like Eliezer Yudkowsky or Paul Christiano, we're more likely doomed than not
My impression for Paul is the opposite – he guesses "~15% on singularity by 2030 and ~40% on singularity by 2040", and has said "quantitatively my risk of losing control of the universe though this channel [Eliezer's list of lethalities] is more like 20% than 99.99%, and I think extinction is a bit less less likely still". (That said I think he'd probably agree with all the reasons you stated under "I personally lean towards those latter views".) Curious to k...
Is there something similar for the EA Forum?
I think your 'Towards a coherent process for metric design' section alone is worth its weight in gold. Since most LW readers aren't going to click on your linked paper (click-through rates being as low in general as they are, from my experience in marketing analytics), let me quote that section wholesale:
...Given the various strategies and considerations discussed in the paper, as well as failure modes and limitations, it is useful to lay out a simple and coherent outline of a process for metric design. While this will by necessity be far from complete, and w
IL's comment has a BOTEC arguing that video data isn't that unbounded either (I think the 1% usefulness assumption is way too low but even bumping it up to 100% doesn't really change the conclusion that much).
There's a tangentially related comment by Scott Alexander from over a decade ago, on the subject of writing advice, which I still think about from time to time:
...The best way to improve the natural flow of ideas, and your writing in general, is to read really good writers so much that you unconsciously pick up their turns of phrase and don't even realize when you're using them. The best time to do that is when you're eight years old; the second best time is now.
Your role models here should be those vampires who hunt down the talented, suck out their souls, a
What do you think about deep work (here's a semi-arbitrarily-chosen explainer)? I suppose the Monday time block after the meeting lets you do that, but that's maybe <10% of the workweek; you also did mention "If people want to focus deeply for a while, they can put on headphones". That said, many of your points aren't conducive to deep work (e.g. "If you need to be unblocked by someone, the fastest way is to just go to their desk and ask them in person" interrupts the other person's deep work block, same with "use a real-time chat platform like Slack to...
The lame answer: yeah, it does mess with deep work, and I'm not super sure how to balance them.
The spicy answer: I have an unwritten polemic entitled "Against Deep Work". I can't share it though since I have not written it. Fortunately, much of what I hope to say in that post is captured in Chapter 9 of the Lean Startup, which has a section that resonates a lot with my experience. I'll just go ahead and quote it because it's so damn good (starting on page 191 in the linked PDF).
...Imagine you’re a product designer overseeing a new product and you need t
At least speaking from my experience, one of the default ways the Lightcone campus team gets deep-work done is by working in pairs. I also think we would structure things probably somewhat differently if we were doing more engineering or math work (e.g. the LessWrong team tends to be somewhat less interrupt driven).
I've found that by working in pairs with someone, I end up with a lot more robustness to losing context for a minute or two, and often get to expand my metacognition, while still getting a lot of the benefits of deep work. It doesn't work for ev...
I'm curious if Eliezer endorses this, especially the first paragraph.
I'm curious how you think your views here cash out differently from (your model of) most commenters here, especially as pertains to alignment work (timelines, strategy, prioritization, whatever else), but also more generally. If I'm interpreting you correctly, your pessimism on the usefulness-in-practice of quantitative progress probably cashes out in some sort of bet against scaling (i.e. maybe you think the "blessings of scale" will dry up faster than others think)?
+1 for "quantity has a quality all its own". "More is different" pops up everywhere.
Carbon dating
You're gesturing in the right direction, but if it's the age of the universe you're looking for, you really want something like uranium-lead dating instead, which is routinely used to date rocks up to 4.5 billion years old with precision in the ~1% range. Carbon dating can't reliably measure dates more than ~50,000 years ago except in special circumstances, since the half-life of 14C is 5,730 years.
Awhile back johnswentworth wrote What Do GDP Growth Curves Really Mean? noting how real GDP (as we actually calculate it) is a wildly misleading measure of growth because it effectively ignores major technological breakthroughs – quoting the post, real GDP growth mostly tracks production of goods which aren’t revolutionized; goods whose prices drop dramatically are downweighted to near-zero, so that when we see slow, mostly-steady real GDP growth curves, that mostly tells us about the slow and steady increase in production of things which haven’t been revo...
This post is great, I suspect I will be referencing it from time to time.
I don't know if you meant to include the footnotes as well, since they aren't present in this post. For instance, I tried clicking on
After a week, you'll likely remember why you started, but it may be hard to bring yourself to really care[2]
and it just doesn't lead anywhere, although I did find it on your blog.
Would you say this is the same as babbling?
Can't wait!
Much ink has been spilled on the difficulty of trying to solve a problem ahead of time and without any feedback loop; I won't rehash those arguments at length.
Can you point me to some readings, especially alignment-related stuff? (No need to rehash anything.) I've been reading LW on and off since ~2013 and have somehow missed every post related to this, which is kind of embarrassing.
Just letting you know that you seem to have double-pasted the 3rd bullet point.
I think TurnTrout's Reward is not the optimization target addresses this, but I'm not entirely sure (feel free to correct me).
for instance, one might try recruiting John Carmack to work on AI safety [this strikes me as a good idea, hindsight notwithstanding], only to get him interested enough that he starts up an AGI company a few years later
Is this a reference to his current personal project to work on AGI?
Edit: reading a bit more about him, I suspect if he ever got interested in alignment work he'd likely prefer working on Christiano-style stuff than MIRI-style stuff. For instance (re: metaverse):
...The idea of the metaverse, Carmack says, can be "a honeypot trap for 'archit
It is hoped that this will allow for solutions to some of the problems which are inherent to the prevailing conception of physics while opening up new avenues of investigation and allowing us to talk about concepts like information. In future posts, I'll explain how it does this in more detail.
Could you at least give a "teaser preview" of what are the "problems which are inherent to the prevailing conception of physics" you mention here? Perhaps the Applications page regarding hybrid systems, and the remark in Q2 of the FAQ about how constructor theory let...
This is a really useful framing, it crystallized a lot of messy personal moral intuitions. Thanks for writing it.
Something something akrasia maybe? Or some of the other stuff in that wiki's "see also" section?
Maybe John Nerst's erisology is the "dual" to your essay here, since it's basically the study of disagreement. There's also a writeup in The Atlantic, and a podcast episode with Julia Galef. Quoting Nerst:
...By “disagreement” I don’t mean the behavior of disagreeing. I mean the plain fact that people have different beliefs, different tastes, and react differently to things.
I find this endlessly interesting. A person that disagrees with me must have a different mind in some way. Can that difference be described? Explained? What do such differences say about th
The 'Resources' section lists How to Talk So Kids Will Listen and Listen So Kids Will Talk [book] -- I also enjoyed weft's Book Review: How To Talk So Little Kids Will Listen, written by Julie King and Joanne Faber, daughter of Adele Faber, who co-wrote the former with Elaine Mazlich. Quoting weft:
...The core principles are the same, but the update stands on its own. Where the original "Kids" acts more like a workbook, asking the reader to self-generate responses, "Little Kids" feels more like it's trying to download a response system into your head via model
I'm guessing you're referring to Brian Potter's post Where Are The Robotic Bricklayers?, which to me is a great example of reality being surprisingly detailed. Quoting Brian:
...Masonry seemed like the perfect candidate for mechanization, but a hundred years of limited success suggests there’s some aspect to it that prevents a machine from easily doing it. This makes it an interesting case study, as it helps define exactly where mechanization becomes difficult - what makes laying a brick so different than, say, hammering a nail, such that the latter is almost
This reminds me of Eliezer's short story That Alien Message, which is told from the other side of the speed divide. There's also Freitas' "sentience quotient" idea upper-bounding information-processing rate per unit mass at SQ +50 (it's log scale -- for reference, human brains are +13, all neuronal brains are several points away, vegetative SQ is -2, etc).
Perhaps I'm missing something (I don't work in AI research), but isn't the obvious first stop Christiano et al's Concrete Problems in AI Safety? Apologies if you already know about this paper and meant something else.
I concur with your last paragraph, and see it as a special case of rationalist taboo (taboo "AGI"). I'd personally like to see a set of AGI timeline questions on Metaculus where only the definitions differ. I think it would be useful for the same forecasters to see how their timeline predictions vary by definition; I suspect there would be a lot of personal updating to resolve emergent inconsistencies (extrapolating from my own experience, and also from ACX prediction market posts IIRC), and it would be interesting to see how those personal updates behave in the aggregate.
I'm reminded of Sarah Constantin's Humans Who Are Not Concentrating Are Not General Intelligences. A quote that resonates with my own experience:
...I’ve noticed that I cannot tell, from casual conversation, whether someone is intelligent in the IQ sense.
I’ve interviewed job applicants, and perceived them all as “bright and impressive”, but found that the vast majority of them could not solve a simple math problem. The ones who could solve the problem didn’t appear any “brighter” in conversation than the ones who couldn’t.
I’ve taught public school teachers, wh
Just wondering -- did you ever get around to writing this post? I've bounced off many Yoneda explainers before, but I have a high enough opinion of your expository ability that I'm hopeful yours might do it for me.
You may be interested in Kevin Simler's essay A Nihilist's Guide to Meaning, which is a sort of graph-theory flavored take on meaning and purpose. I was pleasantly surprised to see how much mileage he got out of his working definition, how many examples of meaningful vs not-meaningful things it explains:
A thing X will be perceived as meaningful in context C to the extent that it's connected to other meaningful things in C.
Applied Divinity Studies wrote a related post that might be of interest: How Long Should you Take to Decide on a Career? They consider a modified version of the secretary problem that accounts for the 2 problematic assumptions you noted (binary payoff and ignorance of opportunity cost); you can play with the Colab notebook if you're curious. Interestingly, varying the parameters tends to pull the optimal starting point earlier (contra my initial intuition), sometimes by a lot. The optimal solution is so parameter-dependent that it made me instinctively wan...
I'm confused about your pushback to AllAmericanBreakfast's (great) feedback on your style, which I find antagonistic to the point that (like AAB) I'm not comfortable sharing it with anyone, despite broadly agreeing with your conclusions and thinking it's important.
I'm curious if you can explicate the thought process behind such a high estimate.
This feels like you're avoiding the least convenient possible world, even if that wasn't your intention.
It is worth remarking though, that even a nuclear rocket might learn something useful from practicing the gravity turn maneuver. Just because you have an easy time leaving Earth’s atmosphere and have no need of finesse, doesn’t mean your travels won’t land you on Venus someday.
I'm reminded of the career advice page on Terry Tao's blog. When I first found it many years ago as a student, I wondered why someone like Tao would bother to write about stuff like "work hard" and "write down what you've done" and "be patient" and "learn and relearn your field". Was...
This strategy reminds me of epistemic learned helplessness.
Relevant: Gwern's the Algernon argument.
Great post. I don't have much to add, but here are some related reads:
...At the same time, Bezos became enamored with
Curious, what do you think now that GPT-4 is out?