# Shortform Content [Beta]

Douglas_Knight's Shortform

Robin Hanson tweets:

Many expect to see a correlation between cynicism, thinking that low motives drive behavior, and pessimism, thinking the future will be bleak. I defy this, being both cynical and optimistic. But I'm curious; is this correlation real, and if so why? If not, why expect it?

I am skeptical that most people have beliefs and, in particular, I'm skeptical that these terms are for describing beliefs.

Taking these definitions as given, what do they predict? An outside view extrapolation of the future from the present should not depend too mu... (read more)

mike_hawke's Shortform

Slightly inspired by this post from Julia Galef. I've selected the following posts because they are insightful specifically at the meta-level.

Richard Ngo's Shortform

It seems to me that Eliezer overrates the concept of a simple core of general intelligence, whereas Paul underrates it. Or, alternatively: it feels like Eliezer is leaning too heavily on the example of humans, and Paul is leaning too heavily on evidence from existing ML systems which don't generalise very well.

I don't think this is a particularly insightful or novel view, but it seems worth explicitly highlighting that you don't have to side with one worldview or the other when evaluating the debates between them. (Although I'd caution not to just average ... (read more)

Quinn's Shortform

# Rats and EAs should help with the sanity levels in other communities

Consider politics. You should take your political preferences/aesthetics, go to the tribes that are based on them, and help them be more sane. In the politics example, everyone's favorite tribe has failure modes, and it is sort of the responsibility of the clearest-headed members of that tribe to make sure that those failure modes don't become the dominant force of that tribe.

Speaking for myself, having been deeply in an activist tribe before I was a rat/EA, I regret I wasn't there to hel... (read more)

0Viliam2dBut what if that makes my tribe lose the political battle? I mean, if rationality actually helped win political fights, by the power of evolution we already would have been all born rational...

1. Evolution does not magically get from A to B instantly.

2. Evolution does not necessarily care about X for many values of X.

This can include: winning political fights, whether or not nukes are built and many other things.

AllAmericanBreakfast's Shortform

"The subjective component in causal information does not necessarily diminish over time, even as the amount of data increases. Two people who believe in two different causal diagrams can analyze the same data and may never come to the same conclusion, regardless of how "big" the data are. This is a terrifying prospect for advocates of scientific objectivity, which explains their refusal to accept the inevitability of relying on subjective causal information." - Judea Pearl, The Book of Why

Matthew Barnett's Shortform

NVIDIA's stock price is extremely high right now. It's up 134% this year, and up about 6,000% since 2015! Does this shed light on AI timelines?

Here are some notes,

• NVIDIA is the top GPU company in the world, by far. This source says that they're responsible for about 83% of the market, with 17% coming from their primary competition, AMD.
• By market capitalization, it's currently at $764.86 billion, compared to the largest company, Apple, at$2.655 trillion.
• This analysis estimates their projected earnings based on their stock price on September 2nd and comes u
Andrew McKnight's Shortform

I've been thinking about benefits of "Cognitive Zoning Laws" for AI architecture.

If specific cognitive operations were only performed in designated modules then these modules could have operation-specific tracking, interpreting, validation, rollback, etc. If we could ensure "zone breaches" can't happen (via e.g. proved invariants or more realistically detection and rollback) then we could theoretically stay aware of where all instances of each cognitive operation are happening in the system. For now let's call this cognitive-operation-factored architecture... (read more)

TurnTrout's shortform feed

Reading EY's dath ilan glowfics, I can't help but think of how poor English is as a language to think in. I wonder if I could train myself to think without subvocalizing (presumably it would be too much work to come up with a well-optimized encoding of thoughts, all on my own, so no new language for me). No subvocalizing might let me think important thoughts more quickly and precisely.

Interesting. My native language is Hebrew but I often find it easier to think in English.

2Matthew Barnett6dThis is an interesting question, and one that has been studied by linguists [https://en.wikipedia.org/wiki/Linguistic_relativity].
2TurnTrout6dI'm not sure how often I subvocalize to think thoughts. Often I have trouble putting a new idea into words just right, which means the raw idea essence came before the wordsmithing. But other times it feels like I'm synchronously subvocalizing as I brainstorm
Douglas_Knight's Shortform

What is calculus? Who invented it? I don't mean Newton vs Leibniz, but Newton vs Archimedes.

If it is the ability of calculate certain things, Archimedes calculated many of those things. If it is a single particular theorem, the obvious candidate is the Fundamental Theorem of Calculus, connecting tangents to areas, due to Isaac Barrows, Newton's mentor.

I sometimes see people claiming that Newton bequeathed us a black box which was a giant step forward and now people learn it in high school and can do everything Newton could do. This is wildly wrong, but it ... (read more)

Vanessa Kosoy's Shortform

I propose a new formal desideratum for alignment: the Hippocratic principle. Informally the principle says: an AI shouldn't make things worse compared to letting the user handle them on their own, in expectation w.r.t. the user's beliefs. This is similar to the dangerousness bound I talked about before, and is also related to corrigibility. This principle can be motivated as follows. Suppose your options are (i) run a Hippocratic AI you already have and (ii) continue thinking about other AI designs. Then, by the principle itself, (i) is at least as good as... (read more)

Showing 3 of 13 replies (Click to show all)

"Corrigibility" is usually defined as the property of AIs who don't resist modifications by their designers. Why would we want to perform such modifications? Mainly it's because we made errors in the initial implementation, and in particular the initial implementation is not aligned. But, this leads to a paradox: if we assume our initial implementation to be flawed in a way that destroys alignment, why wouldn't it also be flawed in a way that destroys corrigibility?

In order to stop passing the recursive buck, we must assume some dimensions along which our ... (read more)

2Vanessa Kosoy3moI think you misunderstood how the iterated quantilization works. It does not work by the AI setting a long-term goal and then charting a path towards that goal s.t. it doesn't deviate too much from the baseline over every short interval. Instead, every short-term quantilization is optimizing for the user's evaluation in the end of this short-term interval.
2Charlie Steiner3moAh. I indeed misunderstood, thanks :) I'd read "short-term quantilization" as quantilizing over short-term policies evaluated according to their expected utility. My story doesn't make sense if the AI is only trying to push up the reported value estimates (though that puts a lot of weight on these estimates).
Matthew Barnett's Shortform

I now have a Twitter account that tweets my predictions.

I don't think I'm willing to bet on every prediction that I make. However, I pledge the following: if, after updating on the fact that you want to bet me, I still disagree with you, then I will bet. The disagreement must be non-trivial though.

For obvious reasons, I also won't bet on predictions that are old, and have already been replaced by newer predictions. I also may not be willing to bet on predictions that have unclear resolution criteria, or are about human extinction.

Daniel Kokotajlo's Shortform

This article says OpenAI's big computer is somewhere in the top 5 largest supercomputers. I reckon it's fair to say their big computer is probably about 100 petaflops, or 10^17 flop per second. How much of that was used for GPT-3? Let's calculate.

I'm told that GPT-3 was 3x10^23 FLOP. So that's three million seconds. Which is 35 days.

So, what else have they been using that computer for? It's been probably about 10 months since they did GPT-3. They've released a few things since then, but nothing within an order of magnitud... (read more)

100 petaflops is 'only' about 1,000 GPUs, or considerably less if they are able to use lower precision modes. I'm guessing they have almost 100 researchers now? Which is only about 10 GPUs per researcher, and still a small budget fraction (perhaps $20/hr ish vs >$100/hr for the researcher).  It doesn't seem like they have a noticeable compute advantage per capita.

Daniel Kokotajlo's Shortform

When I first read the now-classic arguments for slow takeoff -- e.g. from Paul and Katja -- I was excited; I thought they described a serious alternative scenario to the classic FOOM scenarios. However I never thought, and still do not think, that the classic FOOM scenarios were very unlikely; I feel that the slow takeoff and fast takeoff scenarios are probably within a factor of 2 of each other in probability.

Yet more and more nowadays I get the impression that people think slow takeoff is the only serious possibility. For example, Ajeya and Rohin seem ve... (read more)

So there is a distribution over AGI plan costs. The max cost is some powerful bureaucrat/CEO/etc who has no idea how to do it at all but has access to huge amounts of funds, so their best bet is to try and brute force it by hiring all the respected scientists (eg manhattan project).  But notice - if any of these scientists (or small teams) actually could do it mostly on their own (perhaps say with vc funding) - then usually they'd get a dramatically better deal doing it on their own rather than for bigcorp.

The min cost is the lucky smart researcher wh... (read more)

Just discovered and read about Conflict vs Mistake Theory, in my own mind my summary would be : Mistake Theory is about the "mind", Conflict Theory is about the "heart".

I was also tickled by the meta-level problem.

Daniel Kokotajlo's Shortform

I used to think that current AI methods just aren't nearly as sample/data - efficient as humans. For example, GPT-3 had to read 300B tokens of text whereas humans encounter 2 - 3 OOMs less, various game-playing AIs had to play hundreds of years worth of games to get gud, etc.

Plus various people with 20 - 40 year AI timelines seem to think it's plausible -- in fact, probable -- that unless we get radically new and better architectures, this will continue for decades, meaning that we'll get AGI only when we can actually train AIs on medium or long-horizon ta... (read more)

Showing 3 of 5 replies (Click to show all)
3Conor Sullivan9dCan EfficientZero beat Montezuma's Revenge?

Not out of the box, but it's also not designed at all for doing exploration. Exploration in MuZero is an obvious but largely (ahem) unexplored topic. Such is research: only a few people in the world can do research with MuZero on meaningful problems like ALE, and not everything will happen at once. I think the model-based nature of MuZero means that a lot of past approaches (like training an ensemble of MuZeros and targeting parts of the game tree where the models disagree most on their predictions) ought to port into it pretty easily. We'll see if that's enough to match Go-Explore.

4gwern9dI have no good argument that a human-sized EfficientZero would somehow need to be much slower than humans. Arguing otherwise sounds suspiciously like moving the goalposts after an AI effect: "look how stupid DL agents are, they need tons of data to few-shot stuff like challenging text tasks or image classifications, and they OOMs more data on even something as simple as ALE games! So inefficient! So un-human-like! This should deeply concern any naive DL enthusiast, that the archs are so bad & inefficient." [later] "Oh no. Well... 'the curves cross', you know, this merely shows that DL agents can get good performance on uninteresting tasks, but human brains will surely continue showing their tremendous sample-efficiency in any real problem domain, no matter how you scale your little toys." -------------------------------------------------------------------------------- As I've said before, I continue to ask myself what it is that the human brain does with all the resources it uses, particularly with the estimates that put it at like 7 OOMs more than models like GPT-3 or other wackily high FLOPS-equivalence. It does not seem like those models do '0.0000001% of human performance', in some sense.
Matt Goldenberg's Short Form Feed

I just realized that humans are misaligned mesaoptimizers. Evolution "wanted" us to be pure reproduction maximizers but because of our training distribution we ended up valuing things like love, truth and beauty as terminal values. We're simply misaligned AIs run amok.

Yes, but people also constantly exchange increased reproductive capacity for love, truth, and beauty (the world would look very different if reproductive capacity was the only terminal value people were optimizing for).  It's not that reproductive capacity isn't a terminal value of humans, it's that it's not the only one, and people make tradeoffs for other terminal values all the time.

Gunnar_Zarncke's Shortform

I'm looking for a post on censorship bias (see Wikipedia) that was posted on here on LW or possibly on SSC/ACX but a search for "censorship bias" doesn't turn up anything. Googling for it turns up this:

Anybody can help?