All of AnthonyC's Comments + Replies

I'd just sort of assumed the similar sound to zazen was intentional and that if I'd known Japanese it might make more sense.

I mean, yes, because the proposal is about optimizing our entire future light for an outcome we don't know how to formally specify.

Sometimes I see people use the low-info heuristic as a “baseline” and then apply some sort of “fudge factor” for the illegible information that isn’t incorporated into the baseline—something like “the baseline probability of this startup succeeding is 10%, but the founders seem really determined so I’ll guesstimate that gives them a 50% higher probability of success.” In principle I could imagine this working reasonably well, but in practice most people who do this aren’t willing to apply as large of a fudge factor as appropriate.

The last company I w... (read more)

In the ancestral environment, population densities were very low. My understanding is that almost everyone in your band would be at least somewhat related, or have an ambiguous degree of relatedness, and would be someone you'd rely on again and again. How often do we think interactions with true non-relatives actually happened? 

I'm not sure there's anything that needs to be explained here except "evolution didn't stumble upon a low-cost low-risk reliable way for humans to defect against non-relatives for personal advantage as much as a hypothetical intelligently designed organism could have." Is there?

Very little of the value I got out of my university degree came from the exams or the textbooks. All of that I could have done on my own. Much of the value of the lectures could have been replicated by lecture videos. The fancy name on my resume is nice for the first few years (especially graduating in the middle of the 2009 recession) and then stops mattering. 

But the smaller classes, the ones with actual back and forth with actually interested professors? The offhand comments that illustrate how experts actually approach thinking about their fields?... (read more)

Then if it can compute infinite sets as large as the reals, it can handle any set of cardinality beth-1, but not beth-2 or larger. But because the cardinality of the reals is itself formally undecidable by finite logic systems (or by infinite logic systems of size less than aleph-w), I think this doesn't give us much specificity about the limits of what that means, or where it falls on the kinds of arithmetical hierarchy schemas finite logic systems enable us to define.

For my own sanity this is about where I stop trying to delve too deep for understanding,... (read more)

As I understand it, that point about "somewhat arbitrary choices in how finite logic should be extended to infinitary" would also include, for every one of the infinitely many undecidable-by-a-non-hypercomputer propositions, a free choice of whether to include a proposition or its negation as an axiom. Well, almost. Each freely chosen axiom has infinitely many consequences that are the no longer free choices. But it's still (countably) infinitely many choices. But if you have countably infinitely many binary choices, that's like choosing a random real numb... (read more)

I can see how that might help me eat less, but unless you chose the seven very carefully to be potentially nutritionally complete, sustaining that seems like a path to the kinds of deficiencies that made the agricultural revolution cost humans half a foot of height for most of the last ten millennia.

Yes, this sounds completely right. One unusually good doctor I had told me, "In the right patient, any drug can have any effect." It took me another four years to solve that particular problem, ten years in total, and I'm still concerned that when I see my new PCP (previous one retired) he might try to change my meds that've been working for 5 years.

Most doctors are too cautious, for whatever (often justified) reasons, to just try things. Most really don't know how to respect what patients know about themselves or to interact with an actually-intelligent p... (read more)

It's been years since I've talked to anyone working on this technology, but IIRC one of the issues was that in principle you could prevent the lag from leading to bad data that kills you if the pump could also provide glucagon, but there was no way to make glucagon shelf-stable enough to have in a pump. Apparently that changed as of 2019/2020, or is in the process of changing, so maybe someone will make a pump with both.

Thanks! I've only just started reading and there's really good stuff here.

My own take: in order for the zeitgeist to be optimistic on progress, it has to seem possible for things to get better. And for things to get better, it has to be possible for them to be good. But in most forums, I find, it's almost impossible to call anything good without being torn to shreds from multiple sides. We've raised the bar beyond what mere mortals can achieve, and retroactively damn the past's achievements by applying standards that even now are very far from universal. I... (read more)

On muscle knots - whatever they are - it isn't just a difference in the experience of those who have them, but also those massaging them. For me they've always been obvious. When giving a massage, there are relaxed muscles, tense muscles, and knots, and these are three very different feelings regardless of whatever I'm massaging with fingers, palms, knuckles, elbows, or otherwise. It's very clear that a knot-feeling-recipient and I almost always agree on the locations of knots, or their absence (the exception seems to be knots that are "deeper" under an al... (read more)

Is there anything about the way schizophrenia is (or used to be) diagnosed that would make it harder for the congenitally blind to get diagnosed? I ask because I know someone, completely deaf from birth (and who only learned sign language as an adult, not sure if that makes a relevant difference in terms of language processing), who for a long time couldn't get treatment for (and never got a formal diagnosis of) schizophrenia on account of a lack of auditory hallucinations or hearing voices.

Out of curiosity, how many serious accidents have there been at that intersection at that time? Not that I'd expect that to change the reply you got.

I think stories like this are unfortunately common in many places. Odd-seeming divisions of power and responsibility with no repercussions for failing to act create bizarre-seeming planning decisions. I have a family friend who has spent a decade repeatedly trying to have the city remove or prune a tree in front of their house. It has regularly dropped large branches on their driveway (recently crushing the ro... (read more)

Oh for sure (although this isn't just a 'first time we discuss it' response, it happens multiple times with people we're fairly close with, people who we've talked to about it throughout over a year and a half of planning). Most of the people who don't have that reaction know at least one person who has already done it, so they have a reference point. Many who do have no reference point other than Christmas Vacation or that Robin Williams RVing movie. 

And for a lot of people it would be terrible! No arguments there, there are definitely tradeoffs. I f... (read more)

I've always loved to cook, but for a long time mostly neglected ingredient quality, and gradually gained weight. Cleaning up my diet was a gradual shift over the course of years, with a few big bursts of effort focused on improving specific aspects (most recently in January 2020, I eliminated most refined grains, refined sugar, and processed oils). I do have defined exceptions: I don't try to apply the same standards to food I don't make myself at home, for example. Now two years later I'm traveling full time and using a different set of grocery stores eve... (read more)

you can change the plan again!

This is one point that, while obvious, seems hard for a lot of people to notice in practice. As an example, my wife and I recently sold our house and are RVing full time. We've noticed (and are told this is common) that many times we had conversations that went something like this: 

"Oh, what's your destination?" 

"We don't have one, but for the next few months we're heading vaguely [direction]." 

"Oh, so you're going to [place]?" 

"No, it's not a trip/vacation, it's just life, and a few weeks after we ge... (read more)

Assuming the relevant area of science already exists, yes. Recurse as needed, and  there is some level of goal for which generic rationality is a highly valuable skillset. Where that level is, depends on personal and societal context.

real deontologists 

I think you're right in practice, but the last formal moral philosophy class I took was Michael Sandel's intro course, Justice, and it definitely left me with the impression that deontologists lean towards simple rules. I do wonder, with the approach you outline here, if there's a highest-level conflict-resolving rule somewhere in the set of rules, or if it's an infinite regress. I suspect the conflict-resolving rules end up looking pretty consequentialist a lot of the time.

It doesn't actually take much time or effort to think to yo

One thing that has long surprised me about the strict Kantian rule-following point of view is the seeming certainty that the rule needs to be a short sentence, on the length scale of "Thou shalt not kill." (And yes, I see it as the same sort of error that many people make who think there's a simple utility function we could safely give an AGI.) My POV makes more of a distinction on the lines of axiology/morality/law, where if you want a fundamental principal in ethics, one that should never be violated, it's going to be axiological and also way too complic... (read more)

Better hardware reduces the need for software to be efficient to be dangerous. I suspect on balance that yes, this makes development of said hardware more dangerous, and that not working on it can buy us some time. But the human brain runs on about 20 watts of sugar and isn't anywhere near optimized for general intelligence, so we shouldn't strictly need better hardware to make AGI, and IDK how much time it buys.

Also, better hardware makes more kinds of solutions feasible, and if aligned AGI requires more computational capacity than unaligned AI, or if bet... (read more)

I would offer that any set of goals given to this AGI would include the safety-concerns of humans.  (Is this controversial?)  

If anyone figures out how to give an AGI this goal, that would mean they know how to express the entire complex set of everything humans value, and express it with great precision in the form of mathematics/code without the use of any natural language words at all. No one on Earth knows how to do this for even a single human value. No one knows how to program an AI with a goal anywhere near that complex even if we did.&nbs... (read more)

The fact that the statement is controversial is, I think, the reason. What makes a world-state or possible future valuable is a matter of human judgment, and not every human believes this. 

EY's short story Three Worlds Collide explores what can happen when beings with different conceptions of what is valuable, have to interact. Even when they understand each other's reasoning, it doesn't change what they themselves value. Might be a useful read, and hopefully a fun one.

I think the observation that it just isn't obvious that ems will come before de novo AI is sufficient to worry about the problem in the case that they don't. Possibly while focusing more capabilities development towards creating ems (whatever that would look like)?

Also, would ems actually be powerful and capable enough to reliably stop a world-destroying non-em AGI, or an em about to make some world-destroying mistake because of its human-derived flaws? Or would we need to arm them with additional tools that fall under the umbrella of AGI safety anyway?

Scott Alexander's short story, The Demiurge's Older Brother, explores a similar idea from the POV of simulation and acausal trade. This would be great for our prospects of survival if it's true-in-general. Alignment would at least partially solve itself! And maybe it could be true! But we don't know that. I personally estimate the odds of that as being quite low (why should I assume all possible minds would think that way?) at best. So, it makes sense to devote our efforts to how to deal with the possible worlds where that isn't true.

In different ways from different vantage points, I've always seen saving for retirement as a point of hedging my bets, and I don't think the likelihood of doom changes that for me.

Why do I expect I'll want or have to retire? Well, when I get old I'll get to a point where I can't do useful work any more... unless humans solve aging (in which case I'll have more wealth and still be able to work, which is still a good position), or unless we get wiped out (in which case the things I could have spent the money on may or may not counterfactually matter to me, d... (read more)

Some one else already commented on how human intelligence gave us a decisive strategic advantage over our natural predators and many environmental threats. I think this cartoon is my mental shorthand for that transition. The timescale is on the order of 10k-100k years, given human intelligence starting from the ancestral environment.

Empires and nations, in turn, conquered the world by taking it away from city-states and similarly smaller entities in ~1k-10k years. The continued existence of Singapore and the Sentinel Islanders doesn't change the fact that ... (read more)

I think one part of the reason for confidence is that any AI weak enough to be safe without being aligned, is weak enough that it can't do much, and in particular it can't do things that a committed group of humans couldn't do without it. In other words, if you can name such an act, then you don't need the AI to make the pivotal moves. And if you know how, as a human or group of humans, to take an action that reliably stops future-not-yet-existing AGI from destroying the world, and without the action itself destroying the world, then in a sense haven't you solved alignment already?

Second, the line of argument runs like this: Most (a supermajority) possible futures are bad for humans. A system that does not explicitly share human values has arbitrary values. If such a system is highly capable, it will steer the future into an arbitrary state. As established, most arbitrary states are bad for humans. Therefore, with high probability, a highly capable system that is not aligned (explicitly shares human values) will be bad for humans.

I'm not sure if I've ever seen this stated explicitly, but this is essentially a thermodynamic argument.... (read more)

For one, I don't think organizations of humans, in general, do have more computational power than the individual humans making them up. I mean, at some level, yes, they obviously do in an additive sense, but that power consists of human nodes, each not devoting their full power to the organization because they're not just drones under centralized control, and with only low bandwidth and noisy connections between the nodes. The organization might have a simple officially stated goal written on paper and spoken by the humans involved, but the actual incentiv... (read more)

This depends on how far outside that human's current capabilities, and that human's society's state of knowledge, that thing is. For playing basketball in the modern world, sure, it makes no sense to study physics and calculus, it's far better to find a coach and train the skills you need. But if you want to become immortal and happen to live in ancient China, then studying and practicing "that thing" looks like eating specially-prepared concoctions containing mercury and thereby getting yourself killed, whereas studying generic rationality leads to the wh... (read more)

In some sense I would think it's almost tautologically true that faster capabilities research shortens the timeline in which alignment and strategy researchers do their own work. 

I think there's a really great core to this post, but I don't know what to do with it, or how to regard it.

For one, this problem seems to be largely inescapable, though of course we can work to reduce its perpetuation. Where, exactly, can I go to live on unstolen land? History is very long, and I don't know when the first moment was that our ancestors could be said to have any sort of right to the land they lived on (especially where said ancestors' descendants still exist and live there), which the other predators and megafauna they displaced did not have... (read more)

I agree. Why the ICC was only investigating Africa matters. One part is that the ICC doesn't have enough perceived legitimacy or authority for stronger countries to submit to investigations; they rely on their own institutions instead, and the ICC doesn't have the power to override that. I highly doubt anyone in 2002 honestly expected that, say, the US would support it investigating US military actions in the Middle East (and providing access to classified records for such an investigation), and letting them investigate and punish Bush and Rumsfeld for war... (read more)

Does what I want include wanting other people to get what they want? Often, yes. Not always. I want my family and friends and pets to get more of what they specifically want, and a large subset of strangers to mostly get more of what they want. But there are people who want to hurt others, and people who want to hurt themselves, and people who want things that reduce the world's capacity to support people getting want they want, and I don't want those people to get what they want. This will create conflict between agents, often value-destroying conflict, and zero or negative sum competitions, which I nevertheless will sometimes participate in and try to win.

Rome  itself is a possible analogous example, in that the failure of distribution networks caused critical food shortages. That, in turn, was a problem because the economic model of the empire depended on continuous expansion to sustain itself, and the heart of the empire was incapable of supporting itself with locally available resources. Rome went from a city of ~1 million to tens of thousands, and didn't reach that point again until the 1800s. Extend the analogy to a world with no more frontier to expand into (except the technological frontier) and... (read more)

Several years ago I had a conversation with someone that helped them predict other people's behavior much better thereafter: most people are not consequentialists, and are not trying to be, they mostly do what is customary for them in each situation for the relevant part of their culture or social group. 

Your discussion in the post seems premised on the idea that people are trying to reason about consequences in specific cases at all, and I don't think that's usually true. Yes, very few rules are truly absolute, which is why most people balk at Kant's... (read more)

Similarly, you can get a master's degree in 2 years of courses, but a PhD requires an extended apprenticeship. The former is to teach you a body of knowledge, the latter is to teach you how to do research to extend that knowledge.

I'd also add that in his first lesson on metarationality, the one that opens with the Bongard problems, Chapman explicitly acknowledges that he is restricting his idea of  a system to "a set of rules that can be printed in a book weighing less than ten kilograms, and which a person can consciously follow," and that meta... (read more)

1.2 is (probably?) an illusion. A universe of finite total energy and finite size should  be subject to the quantum mechanical version of the Poincare Recurrence Theorem. After some number of cycles it would return arbitrarily close to its previous Big Bang starting state. Might require vast amounts of time, but still.

This leaves me with 1.1, a guarantee that only this single universe will ever exist, or a coin toss between 2.1 and 2.2, which guarantee a wide variety of different universes ever existing, whether I'm in them or not. I choose 2. 

No... (read more)

A week ago I wrote "The Russian Armed Forces is among the three most capable militaries in the world". Since then, I have been astonished by the incompetence of the Russian Armed Forces.

Has this changed your ranking of world military capabilities?

But as they do, users will favour the products that give them the best experience 

This is one point I find difficult to believe, or at least difficult to find likely. Most people, who are not unusually savvy, already give much more credence to ads, and much less credence to the degree to which they are actually affected by ads, than they should be. Why should that reverse as ads get even better at manipulating us? Why should I expect people to start demonstrating the level of long-term thinking and short-term impulse control and willingness to l... (read more)

But now, perhaps, we feel the rug slipping out from under us too easily. Don’t we have non-zero credences on coming to think any old stupid crazy thing – i.e., that the universe is already a square circle, that you yourself are a strongly Ramsey lizard twisted in a million-dimensional toenail beyond all space and time, that consciousness is actually cheesy-bread, and that before you were born, you killed your own great-grandfather? So how about a lottery with a 50% chance of that, a 20% chance of the absolute infinite getting its favorite ice cream, and a

I typically dream in color, but have occasionally dreamed in black and white. I have also occasionally dreamed in cartoon-ish environments, or 90s/2000s computer graphics-like animation. These were very distinct experiences when they happened, there was never any doubt in my mind on waking that the dream I'd just had was different than usual for me.

I'm curious for anyone who has lucid dreams: is this a change you can make mid-dream?

I do have a growing sympathy with the idea that just because you have a case of COVID that necessarily means you need to stay in the leper colony for a while. BUT I'm not sure about where one draws the line on that. 

I for one would like to have at least a semi-quantitative answer to how much risk we're (socially, legally) permitted to expose each other to as part of normal life, instead of an inconsistent, ad hoc set of rules and expectations. 

For example, you can drive, but only licensed, and not when drunk: sensible. 

By comparison, you hav... (read more)

Whenever someone asks a poll question about how much people would be willing to pay to something, I wonder how much of the answer is liquidity constrained. People who live paycheck to paycheck (a majority of Americans) who literally wouldn't be able to pay a week's income for any new expense might just not engage with the counterfactual in quite the way that it was intended.

"Man whose roommates wear t-shirts in winter, 
would better calm down with the goddamn heater."

I agree with the idea behind the advice, but for me it would still be bad advice.

I was the kid who wore shorts in winter in middle school. I just feel warm (which makes me sleepy) at temperatures others find too cold. Then in college I had a roommate from Hawaii, and later married a woman with a circulatory issue, so I've always kept on running the heater and wearing t-shirts. (And upgrading the insulation when feasible).

Honestly there are winter days I... (read more)