Grue_Slinky

Some time ago, I mentioned in a comment some issues I had with the Goodhart Taxonomy:

1) “adversarial” seems too broad to be that useful as a category

2) It doesn’t clarify what phenomenon is meant by “Goodhart”; in particular, “regressional” doesn’t feel like something the original law was talking about, and any natural definition of “Goodhart” that includes it seems really broad

3) Whereas “regressional” and “extremal” (and perhaps “causal”) are defined statistically, “adversarial” is defined in terms of agents, and this may have downsides (I’m less sure about this objection)

After thinking about things more, I'd like to expand on these points:

I've sort of made by peace with (3): whatever Goodhart is, it certainly

... (read 995 more words →)

Replying toWe run the Center for Applied Rationality, AMA

We run the Center for Applied Rationality, AMA

How do CFAR's research interests/priorities compare with LW's Open Problems in Human Rationality? Based on Brienne and Anna's replies here, I suspect the answer is "they're pretty different", but I'd like to hear what accounts for this divergence.

Replying to[AN #76]: How dataset size affects robustness, and benchmarking safe exploration by measuring constraint violations

[AN #76]: How dataset size affects robustness, and benchmarking safe exploration by measuring constraint violations

Nitpick: "transfer learning" is the standard term, no? It has a Wiki page and seems to get a more coherent batch of search results than googling "robustness to data shift".

Replying toWhat are we assuming about utility functions?

What are we assuming about utility functions?

Whoops, mea culpa on that one! Deleted and changed to:

the main post there pointed out that seemingly anything can be trivially modeled as being a "utility maximizer" (further discussion here), whereas only some intelligent agents can be described as being "goal-directed" (as defined in this post), and the latter is a more useful concept for reasoning about AI safety.

Grue_Slinky6yQuick Take

In reasoning about AGI, we're all aware of the problems with anthropomorphizing, but it occurs to me that there's also a cluster of bad reasoning that comes from an (almost?) opposite direction, where you visualize an AGI to be a mechanical automaton and draw naive conclusions based on that.

For instance, every now and then I've heard someone from this community say something like:

What if the AGI runs on the ZFC axioms (among other things), and finds a contradiction, and by the principle of explosion it goes completely haywire?

Even if ZFC is inconsistent, this hardly seems like a legitimate concern. There's no reason to hard-code ZFC into an AI unless we want a

... (read 506 more words →)

Replying toTechnical AGI safety research outside AI

Technical AGI safety research outside AI

[copying from my comment on the EA Forum x-post]

For reference, some other lists of AI safety problems that can be tackled by non-AI people:

Luke Muehlhauser's big (but somewhat old) list: "How to study superintelligence strategy"

AI Impacts has made several lists of research problems

Wei Dai's, "Problems in AI Alignment that philosophers could potentially contribute to"

Kaj Sotala's case for the relevance of psychology/cog sci to AI safety (I would add that Ought is currently testing the feasibility of IDA/Debate by doing psychological research)

*begins drafting longer proposal*

Yeah, this is definitely more high-risk, high-reward than the others, and the fact that there's potentially some very substantial spillover effects if successful makes me both excited and nervous about the concept. I'm thinking of Arbital as an example of "trying to solve way too many problems at once", so I want to manage expectations and just try to make some exercises that inspire people to think about the art of mathematizing certain fuzzy philosophical concepts. (Running title is "Formalization Exercises", but I'm not sure if there's a better pithy name that captures it).

In any case, I appreciate the feedback, Mr. Entworth.

(8)

In light of the “Fixed Points” critique, a set of exercises that seem more useful/reflective of MIRI’s research than those exercises. What I have in mind is taking some of the classic success stories of formalized philosophy (e.g. Turing machines, Kolmogorov complexity, Shannon information, Pearlian causality, etc., but this could also be done for reflective oracles and logical induction), introducing the problems they were meant to solve, and giving some stepping stones that guide one to have the intuitions and thoughts that (presumably) had to be developed to make the finished product. I get that this will be hard, but I think this can be feasibly done for some of the (mostly easier) concepts, and if done really well, it could even be a better way for people to learn those concepts than actually reading about them.

Grue_Slinky6y*

(7)

A critique of MIRI’s “Fixed Points” paradigm, expanding on some points I made on MIRIxDiscord a while ago (which would take a full post to properly articulate). Main issue is, I'm unsure if it's still guiding anyone's research and/or who outside MIRI would care.

(6)

An analysis of what kinds of differential progress we can expect from stronger ML. Actually, I don’t feel like writing this post, but I just don’t understand why Dai and Christiano, respectively, are particularly concerned about differential progress on the polynomial hierarchy and what’s easy-to-measure vs. hard-to-measure. My gut reaction is “maybe, but why privilege that axis of differential progress of all things”, and I can’t resolve that in my mind without doing a comprehensive analysis of potential “differential progresses” that ML could precipitate. Which, argh, sounds like an exhausting task, but someone should do it?

What are we assuming about utility functions?

(5)

A skeptical take on Part I of “What failure looks like” (3 objections, to summarize briefly: not much evidence so far, not much precedent historically, and “why this, of all the possible axes of differential progress?”) [Unsure if these objections will stand up if written out more fully]

I often notice that in many (not all) discussions about utility functions, one side is "for" their relevance, while others tend to be "against" their usefulness, without explicitly saying what they mean. I don't think this is causing any deep confusions among researchers here, but I'd still like to take a stab at disambiguating some of this, if nothing else for my own sake. Here are some distinct (albeit related) ways that utility functions can come up in AI safety, in terms of what assumptions/hypotheses they give rise to:

AGI utility hypothesis: The first AGI will behave as if it is maximizing some utility function

ASI utility hypothesis: As AI capabilities improve well beyond... (read 546 more words →)

K-complexity: The minimum description length of something (relative to some fixed description language)

Cake-complexity: The minimum description length of something, where the only noun you can use is "cake"

I often hear about deepfakes--pictures/videos that can be entirely synthesized by a deep learning model and made to look real--and how this could greatly amplify the "fake news" phenomenon and really undermine the ability of the public to actually evaluate evidence.

And this sounds like a well-founded worry, but then I was just thinking, what about Photoshop? That's existed for over a decade, and for all that time it's been possible to doctor images to look real. So why should deepfakes be any scarier?

Part of it could be that we can fake videos, not just images, but that can't be all of it.

I suspect the main reason is that in the future,... (read more)

Grue_Slinky's Shortform

The Quantity/Quality of Researchers Has Drastically Increased Over the Centuries

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Non-anthropically, what makes us think human-level intelligence is possible?

For just about any academic field F and number X greater than ~100, there are a lot more F researchers than there were X years ago, and this remains true if you restrict to researchers above a given level of quality/competence Q.

This observation seems pretty straightforward/uncontroversial to me, and yet it has quite a few consequences that seem quite nontrivial. Before getting to those, allow me to explain the claim in more detail.

The Claim

For concreteness, let's let F be "maths" and let X be "519".

Well, the world population is about 15 times what it was in the year 1500, therefore there's 15 times as many researchers (at any given level of quality).... (read 1863 more words →)

What are concrete examples of potential "lock-in" in AI research?

The basic question I'm asking is the following:

Suppose you didn't know humans exist. Could you still deduce that general intelligence is possible, and if so how?

This may be a weird hypothetical to imagine, so here's my best attempt to give a story to aid the counterfactual imagination:

At some point, you gain the ability to observe an alternate universe, which you eventually call the "Earth-universe". The fundamental laws of physics are very different here, and you get to work studying it. At first it feels very weird and you have almost no way of making sense of the raw sensory data. Eventually you learn that there are these things "galaxies" and "stars", and... (read 671 more words →)

Distance Functions are Hard

I had some colleagues watch Ben Garfinkel's talk, "How sure are we about this AI stuff?", which among other things, pointed out that it's often difficult to change the long-term trajectory of some technology. For instance, electricity, the printing press, and agriculture were all transformative technologies, but even if we recognized their importance in advance, it's hard to see what we could really change about them in the long-term.

In general, when I look at technological development/adoption, I tend to see people following local economic incentives wherever they lead, and it often seems hard to change these gradients without some serious external pressures (forceful governments, cultural taboos, etc.). I don't see that many... (read 734 more words →)