hippke

Wiki Contributions

Comments

Why the technological singularity by AGI may never happen

I think this calculation is invalid. A human is created from a seed worth 700 MB of information, encoded in the form of DNA. This was created in millions of years of evolution, compressing/worth a large (but finite) amount of information (energy). A relevant fraction of hardware and software is encoded in this information. Additional learning is done during 20 years worth 3 MWh. The fractional value of this learning part is unknown.

Why the technological singularity by AGI may never happen

How can we know that "it is possible to train a 200 IQ equivalent intelligence for at most 3 MW-hr"?

Why the technological singularity by AGI may never happen

How did von Neumann come close to taking over the world? Perhaps Hitler, but von Neumann?

Why the technological singularity by AGI may never happen

Sure! I argue that we just don't know whether such a thing as "much more intelligent than humans" can exist. Millions of years of monkey evolution have increased human IQ to the 50-200 range. Perhaps that can go 1000x, perhaps it would level of at 210. The AGI concept makes the assumption that it can go to a big number, which might be wrong.

A closer look at chess scalings (into the past)

From what I understand about "ELO inflation", it refers to the effect that the Top 100 FIDE players had 2600 ELO in 1970, but 2700 ELO today. It has been argued that simply the level increased, as more very good players entered the field. The ELO number as such should be fair in both eras (after playing infinitely many games...). I don't think that it is an issue for computer chess comparisons. Let me know if you have other data/information!

Benchmarking an old chess engine on new hardware

I ran the experiment "Rebel 6 vs. Stockfish 13" on Amazon's AWS EC2. I rented a Xeon Platinum 8124M which benched at 18x 1.5 MNodes/s. I launched 18 concurrent single-threaded game sets with 128 MB of RAM for each engine. Again, ponder was of, no books, no tables. Time settings were 40 moves in 60s + 0.6 per move, corresponding to 17.5 MNodes/move. For reference, SF13 benches at ELO 3630 at this setting  (entry "64 bit"); Rebel 6.0 got 2415 on a Pentium 90 (SSDF Computer Rating List (01-DEC-1996).txt, 90 kN/move).

The result: 

  • 1911 games played
  • 18 draws
  • No wins for Rebel
  • All draws when Rebel played white
  • ELO difference: 941 +- 63

Interpretation:

  • Starting from 3630 for SF13, that corresponds to Rebel on a modern machine: 2689. 
  • Up from 2415, that's +274 ELO. 
  • The ELO gap between Rebel on a 1994 Pentium 90 (2415) and SF13 on a 2020 PC (3630) is 1215 points. Of these, 274 points are closed with matching hardware. 
  • That gives 23% for the compute, 77% for the algorithm.

Final questions: 

  • Isn't +274 ELO too little for 200x compute?
  • We found 50% algo/50% compute for SF3-SF13. Why is that?

Answer: ELO gain with compute is not a linear function, but one with diminishing returns. Thus, the percentage "due to algo" increases, the longer the time frame. Thus, a fixed percentage is not a good answer.
But we can give the percentage as a function of time gap:

  • Over 10 years, it's ~50%
  • Over 25 years, it's ~22%

With data from other sources (SF8, Houdini 3) I made this figure to show the effect more clearly. The dashed black line is a double-log fit function: A base-10 log for the exponential increase of compute with time, and a natural log for the exponential search tree of chess. The parameter values are engine-dependent, but should be similar for engines of the same era (here: Houdini 3 and SF8). With more and more compute, the ELO gain approaches zero. In the future, we can expect engines whose curve is shifted to the right side of this plot.
 

Benchmarking an old chess engine on new hardware
  • With a baseline of 10 MNodes/move for SF3, I need to set SF13 to 0.375 MNodes/move for equality. That's a factor of 30. Caveat: I only ran 10 games which turned out equal, and only at 10 MNodes/move for SF3.
  • Yes: Rebel6 at normal 2021 settings (40 moves in 15 min) can be approximately matched with SF13 at 20 kNodes/move. More precisely: I get parity between Rebel6 (128 MB) and SF13 (128 MB) for 16 MNodes/move vs. 20 kNodes/move (=factor of 800x). On my Intel Core-M 5Y31 (750 kNodes/s), that's 21s vs. 0.026s per move. Note that the figure shows SF8, not SF13.
  • I was contacted by one person via PM, we are discussing the execution setup. Otherwise, I could do it by the end of July after my vacation.
A closer look at chess scalings (into the past)

OK, I have added the Houdini data from this experiment to the plot:

The baseline ELO is not stated, but likely close to 3200:

ExperimentkNodes/moveELO dropELO calculated
4k nodes vs 2k nodes23031280
8k nodes vs 4k nodes42801583
16k nodes vs 8k nodes82371863
32k nodes vs 16k nodes162082100
64k nodes vs 32k nodes321792308
128k nodes vs 64k nodes641562487
256k nodes vs 128k nodes1281362643
512k nodes vs 256k nodes2561342779
1024k nodes vs 512k nodes5121152913
2048k nodes vs 1024k nodes1024933028
4096k nodes vs 2048k nodes2048793121
Baseline4096 3200
A closer look at chess scalings (into the past)

Mhm, good point. I must admit that the "70 ELO per doubling" etc. is forum wisdom that is perhaps not the last word. A similar scaling experiment was done with Houdini 3 (2013) which dropped below 70 ELO per doubling when exceeding 4 MNodes/move. In my experiment, the drop is already around 1 MNode/move. So there is certainly an engine dependence.

A closer look at chess scalings (into the past)

From what I understand about the computer chess community:

  • Engines are optimized to win in the competitions, for reputation. There are competitions for many time controls, but most well respected are the CCC with games of 3 to 15 minutes, and TCEC which goes up to 90 minutes. So there is an incentive to tune the engines well into the many-MNodes/move regime.
  • On the other hand, most testing during engine development is done at blitz or even bullet level (30s for the whole game for Stockfish). You can't just play thousands of long games after each code commit to test its effect. Instead, many faster games are played. That's in the few MNodes/move regime. So there's some incentive to perform well in that regime.
  • Below that, I think that performance is "just what is it", and nobody optimizes for it. However, I think it would be valuable to ask a Stockfish developer about their view.
Load More