(...) the term technical is a red flag for me, as it is many times used not for the routine business of implementing ideas but for the parts, ideas and all, which are just hard to understand and many times contain the main novelties.
- Saharon Shelah
As a true-born Dutchman I endorse Crocker's rules.
For my most of my writing see my short-forms (new shortform, old shortform)
Twitter: @FellowHominid
Personal website: https://sites.google.com/view/afdago/home
Posts
Wiki Contributions
Wow! This looks fantastic.
I missed this the first time around - and judging from the number of upvotes so did a lot of other people. A shame.
Here's to hoping more folks will stumble upon your sequence like I did.
As a complete noob in all things mechinterp can somebody explain how this is not in conflict with SAE enjoyers saying they get reconstruction loss in the high 90s or even 100 %?
I understand the logscale argument that Lucius is making but still seems surprising ? Is this really what's going on or are they talking about different things here.
Yamnaya ancestry (Indo-European steppe-pastoralists) make up a large percentage of European genetic ancestry. Modern Europeans are a mixture of three ancestral populations: steppe-pastoralists from the east (Yamnaya...), Western huntergathers, and Anatolian farmers. In some Northern Europeans, the fraction of farmer ancestry may be less a minority.
"Yamnaya–related ancestry is found in the DNA of modern Central, and Northern Europeans (c. 38.8–50.4 %), and is also found in lower levels in present-day Southern Europeans (c. 18.5–32.6 %), Sardinians (c. 2.4–7.1 %), and Sicilians (c. 5.9–11.6 %).[80][71][13]"
https://en.wikipedia.org/wiki/Yamnaya_culture#:~:text=Yamnaya%E2%80%93related%20ancestry%20is%20found,%25)%2C%20and%20Sicilians%20(c.
1Thank you for doing this work. This is is a valuable piece of evidence.
SLT-enjoyers everywhere rejoice
1This sounds genuinely worrying. Largest negative timeline update I've made in many months.
3
This summer, experience the world where everybody is the Man in Black.
11Thank you Lorxus, that's appreciated. I'm sure we can make good use of them.
Unfortunately, we get many more applications than we have spots so we have to make some tough choices. Better luck next time!
1
1I'm curious if these observations are related at all to the work by Mendel, Hanni and Vaintrob on SAE features, more discussion here.
I mostly regard LLMs = [scaling a feedforward network on large numbers of GPUs and data] as a single innovation.