Human-level Diplomacy was my fire alarm
15Daniel Kokotajlo
16Lao Mein
9Ollie J
7Lao Mein
5Daniel Kokotajlo
2Lone Pine
3Lao Mein
2the gears to ascension
1delton137
1WilliamKiely
1Lao Mein
1Jeff Rose
2Eli Tyre
4Jeff Rose
4Lao Mein
New Comment
Good news is that the median academic paper contributes approximately 0 to scientific progress; if all we were automating was the median academic paper then I wouldn't expect things to accelerate much at all.
(Bad news is that actually important research, such as what gets done in industry & by some academics, will accelerate too. And that acceleration will beget further acceleration, and so on until we blow right past human level abilities.)
The median academic paper is a good hypothesis away from actually making incremental scientific progress. The top 10% of papers written in such a manner, let's call it "paper milled", inspire exploration and follow-up in other scientists. They can become highly cited. They can save other people a LOT of time.
Let me give an example of something I'm working on. My team was looking at Clonal Hematopoiesis of Indeterminate Origin (CHIP), non-disease-causing mutations in cancer-driving genes that appear in white blood cells as people age. They were causing interference in our cancer panels. We were trying to disentangle the effects of selection pressures and mutation hotspots on the appearance of CHIP. How much was due to higher replication rates, and how much was due to the higher likelihood of mutation in specific locations? We were going to do a time-consuming analysis of our data before I found several twin studies, which found no significant difference between identical and fraternal twins (same sex) in terms of CHIP correlation. Turns out, mutation hotspots didn't play a major role (there are a lot of caveats here, obviously; I'm oversimplifying).
These papers, which were simple analyses done on an existing data set, probably saved us a hundred man-hours. They may have been a formulaic analysis of other peoples' data, but are still fundamentally good and useful papers that advance science forward. Soon, I'll be using AI to sort through the enormity of papers to find useful ones like these. I'll also be using better hypotheses with the aid of AI.
AI is going to matter. It's already mattering. This is important for the average scientist.
I wonder how it would update its strategies if you negotiated in an unorthodox way:
- "If you help me win, I will donate £5000 across various high-impact charities"
- "If you don't help me win, I will kill somebody"
Think about what happens in the dataset of human games where such conversations take place. It probably adds more uncertainty to the predicted actions of players who say these things.
I mean, what would you do if you saw such messages in a game you're playing? Probably assume they're mentally unstable and adjust accordingly.
Or: What if it finds out that all the humans are doing a strategy of "interrogate everyone to find out who the bot is, then gang up on the bot." How does it react?
It was reported that high level diplomacy players have a different game-theoretical situation, because they all know eachother by (user)name. So if DiplomacyGrandmaster69 goes up against TheDiplomancer, they know their games will be publicly streamed, and the other high level players will see how honest they really are. Whereas casual players are playing a single-shot prisoner's dilemma, the pros are playing an iterated prisoner's dilemma, and that makes a difference.
I wonder what would happen if CICERO were placed in repeated 6-human-one-AI showmatches where everyone know which one was the AI. How would it fair?
I can't remember the exact source, but I believe that CICERO was optimized with the expectation of anonymity. In fact, all players in the games CICERO played were anonymous. CICERO was optimized with the assumption that other players would have no knowledge of its past history (and hence expected behavior). Versions of CICERO that were optimized with the assumption that other players would treat it according to its past history were explicitly noted as being more vindictive.
Now that I think about it, this probably gave CICERO a significant advantage. Most human games are played with player names visible. Anonymous play is thus a deviation from the standard metagame. The Meta team noted that players played more vindictively than optimal and that CICERO got an advantage for being less vindictive. Since these were top players, it implies that the human players simply didn't fully adjust to the anonymous format. I don't recall any CICERO games with public names in the paper; maybe the results were less impressive?
imagenet was my fire alarm. and alphago. and alphazero. or maybe gpt3. actually, the fire alarm hadn't gone off until alphafold, at which time it really started ringing. sorry, I mean alphafold 2. actually, PaLM was what really convinced me agi was soon. well, I mean, not really soon, but hey, maybe if they scale RWKV or S4 or NPT and jam them into a MuZero it somehow won't be agi, despite that it obviously would be. I wonder how the EfficientZero followups are looking these days? don't worry, agi can't happen, they finally convinced me language models aren't real intelligence because they can't do real causal reasoning. they're not good enough at using a causal information bottleneck and they don't have the appropriate communication patterns of real physics. they're prone to stereotyping and irrational, unlike real intelligence,
at this point if people aren't convinced it's soon, they're not going to be convinced until after it happens. there's no further revelation that could occur. [it'll be here within the year] -> {edit 1yr later: well, some folks have included gpt4 in "agi", and gpt4 was released after I said this, but I think actually what I was really expecting didn't happen quite as I expected it}, and I don't know why it's been so hard for people to see. I guess the insistence on yudkowskian foom has immunized people against real life slow takeoff? but that "slow" is speeding up, hard.
anyway, I hope y'all are using good ai tools. I personally most recommend metaphor.systems, summarize.tech, and semanticscholar.
1I disagree, in fact I actually think you can argue this development points the opposite direction, when you look at what they had to do to achieve it and the architecture they use.
I suggest you read Ernest Davis' overview of Cicero. Cicero is a special-purpose system that took enormous work to produce -- a team of multiple people labored on it for three years. They had to assemble a massive dataset from 125,300 online human games. They also had to get expert annotations on thousands of preliminary outputs. Even that was not enough.. they had to generate synthetic datasets as well to fix issues with the system! Even then, the dialogue module required a specialized filter to remove nonsense. This is a break from the scaling idea that says to solve new problems you just need to scale existing architectures to more parameters (and train on a large enough dataset).
Additionally, they argue that this system appears very unlikely to generalize to other problems, or even to slight modifications of the game of Diplomacy. It's not even clear how well it would generalize to non-blitz games. If the rules were modified slightly, the entire system would likely have to be retrained.
I also want to point out that scientific research is not easy as you make it sound. Professors spend the bulk of their time writing proposals, so perhaps AI could help there by summarizing existing literature. Note though a typical paper, even a low-value one, generally takes a graduate student with specialized training about a year to complete, assuming the experimental apparatus and other necessary infrastructure are all in place. Not all science is data-driven either, science can also be observation-driven or theory-driven.
How is...
I'm short on AGI timelines (50% by 2030)
...consistent with...
An AI paradigm as performance-enhancing as transformers is discovered by AI search. 30% by 2030
...?
Doesn't AGI imply the latter?
I'm using a weird definition of AI here, basically "an AI that can do my job". I'm imagining a cobbled-together system of transformers that individually automates everything I can do, thereby replacing most of the information jobs like coding, scientific research, and advertising. So in a lot of the worlds where AGI happens, there's no hard takeoff. AIs are helping do AI research, and maybe labor isn't a major limiting factor in AI development anymore. But there isn't a >1 OOM increase in AI research output from AI.
This also means that I think in most of the 30% there is no hard takeoff. Some low-hanging fruit is picked by machines, but not enough for a FOOM.
Thanks for bringing up the contradiction, though. I really need to go back and clarify a lot of my statements.
It's not my fire alarm (in part because I don't think that's a good metaphor). But it has caused me to think about updating timelines.
My initial reaction was to update timelines, but this achievement seems less impressive than I thought at first. It doesn't seem to represent an advance in capabilities; instead it is (another) surprising result of existing capabilities.
Isn't yet another surprising result of existing capabilities evidence that general intelligence is itself a surprising result of existing capabilities?
That is too strong a statement. I think that it is evidence that general intelligence may be easier to achieve than commonly thought. But, past evidence has already shown that over the last couple of years and I am not sure that this is significant additional evidence in that regard.
On one hand, I agree that nothing really special and novel is happening in Cicero. On the other hand, something about it makes me feel like it's important. I think it's the plan->communicate->alter plan->repeat cycle partly taking place in English that intuitively makes me think "oh shit, that's basically how I think. If we scale this up, it'll do everything I can do". I don't know how true this actually is.
I vaguely recall feeling something like this when a general-purpose model learned to play all the Atari games. But I'm feeling it a lot more now. Maybe it's the fact that if you showed the results of this to pre-GPT2 me, I'd think it's an AGI, with zero doubt or hesitation.