for anyone not wanting to go in and see the Kafka, I copied some useful examples:
ANNA ROGERS: I was considering making yet another benchmark, but I stopped seeing the point of it. Let’s say GPT-3 either can or cannot continue [generating] these streams of characters. This tells me something about GPT-3, but that’s not actually even a machine learning research question. It’s product testing for free.
JULIAN MICHAEL: There was this term, “API science,’’ that people would use to be like: “We’re doing science on a product? This isn’t science, it’s not reproducible.” And other people were like: “Look, we need to be on the frontier. This is what’s there.”
TAL LINZEN (associate professor of linguistics and data science, New York University; research scientist, Google): For a while people in academia weren’t really sure what to do.
R. THOMAS MCCOY: Are you pro- or anti-LLM? That was in the water very, very much at this time.
JULIE KALLINI (second-year computer science Ph.D. student, Stanford University): As a young researcher, I definitely sensed that there were sides. At the time, I was an undergraduate at Princeton University. I remember distinctly that different people I looked up to — my Princeton research adviser [Christiane Fellbaum] versus professors at other universities — were on different sides. I didn’t know what side to be on.
LIAM DUGAN: You got to see the breakdown of the whole field — the sides coalescing. The linguistic side was not very trusting of raw LLM technology. There’s a side that’s sort of in the middle. And then there’s a completely crazy side that really believed that scaling was going to get us to general intelligence. At the time, I just brushed them off. And then ChatGPT comes out.
Plenty of linguists and connectionists thought it was possible, if only to show those damned Chomskyans that they were wrong!
To be specific, some of the radical linguists believed in pure distributional semantics, or that there is no semantics beyond syntax. I don't know anyone in particular, but considering how often Chomsky, Pinker, etc were fighting against the "blank slate" theory, they definitely existed.
The following people likely believed that it is possible to learn a language purely from reading using a general learning architecture like neural networks (blank-slate):
- James L. McClelland and David Rumelhart.
- They were the main proponents of neural networks in the "past tense debate". Generally, anyone on the side of neural networks in the past tense debate probably believed this.
- B. F. Skinner.
- Radical syntacticians? Linguists have failed to settle the question of "Just what is semantics? How is it different from syntax?", and some linguists have taken the radical position "There is no semantics. Everything is syntax.". Once that is done, there simply is no difficulty: just learn all the syntax, and there is nothing left to learn.
- Possibly some of the participants in the "linguistics wars" believed in it. Specifically, some believed in "generative semantics", whereby semantics is simply yet more generative grammar, and thus not any different from syntax (also generative grammar). Chomsky, as you might imagine, hated that, and successfully beat it down.
- Maybe some people in distributional semantics? Perhaps Leonard Bloomfield? I don't know enough about the history of linguistics to tell what Bloomfield or the "Bloomfieldians" believed in exactly. However, considering that Chomsky was strongly anti-Bloomsfield, it is a fair bet that some Bloomsfieldians (or self-styled "neo-Bloomsfieldians") would support blank-slate learning of language, if only to show Chomskyans that they're wrong.
https://www.gov.cn/zhengce/202407/content_6963770.htm
中共中央关于进一步全面深化改革 推进中国式现代化的决定 （2024年7月18日中国共产党第二十届中央委员会第三次全体会议通过）
（51）完善公共安全治理机制。健全重大突发公共事件处置保障体系，完善大安全大应急框架下应急指挥机制，强化基层应急基础和力量，提高防灾减灾救灾能力。完善安全生产风险排查整治和责任倒查机制。完善食品药品安全责任体系。健全生物安全监管预警防控体系。加强网络安全体制建设，建立人工智能安全监管制度。
I checked the translation:
(51) Improve the public security governance mechanism. Improve the system for handling major public emergencies, improve the emergency command mechanism under the framework of major safety and emergency response, strengthen the grassroots emergency foundation and force, and improve the disaster prevention, mitigation and relief capabilities. Improve the mechanism for investigating and rectifying production safety risks and tracing responsibilities. Improve the food and drug safety responsibility system. Improve the biosafety supervision, early warning and prevention and control system. Strengthen the construction of the internet security system and establish an artificial intelligence safety supervision-regulation system.
As usual, utterly boring.
You have inspired me to do the same with my writings. I just updated my entire website to PD, with CC0 as a fallback (releasing under Public Domain being unavailable on GitHub, and apparently impossible under some jurisdictions??)
I don’t fully understand why other than to gesture at the general hand-wringing that happens any time someone proposes doing something new in human reproduction.
I have the perfect quote for this.
A breakthrough, you say? If it's in economics, at least it can't be dangerous. Nothing like gene engineering, laser beams, sex hormones or international relations. That's where we don't want any breakthroughs. "
(Galbraith, 1990) A Tenured Professor, Houghton Mifflin; Boston.
Just want to plug my 2019 summary of the book that started it all.
How to take smart notes (Ahrens, 2017) — LessWrong
It's a good book, for sure. I use Logseq, which is similar to Roam but more fitted to my habits. I never bought into the Roam hype (rarely even heard of it), but this makes me glad I never went into it.
I use a fairly basic Quarto template for website. The code for the entire site is on github.
The source code is actually right there in the post. Click the button Code, then click View Source.
https://yuxi-liu-wired.github.io/blog/posts/perceptron-controversy/
Concretely speaking, are you to suggest that a 2-layered fully connected network trained by backpropagation, with ~100 neurons in each layer (thus ~20000 weights), would have been uneconomical even in the 1960s, even if they had backprop?
I am asking this because the great successes in 1990s connectionism, including LeNet digit recognition, NETtalk, and the TD-gammon, all were on that order of magnitude. They seem within reach for the 1960s.
Concretely speaking, TD-gammon cost about 2e13 FLOPs to train, and in 1970, 1 million FLOP/sec cost 1 USD, so with 10000 USD of hardware, it would take about 1 day to train.
And interesting that you mentioned magnetic cores. The MINOS II machine built in 1962 by the Stanford Research Institute group had precisely a grid of magnetic core memory. Can't they have scaled it up and built some extra circuitry to allow backpropagation?
Corroborating the calculation, according to some 1960s literature, magnetic core logic could go up to 10 kHz. So if we have ~1e4 weights updated 1e4 times a second, that would be 1e8 FLOP/sec right there. TD-gammon would take ~1e5 seconds ~ 1 day, the same OOM as the previous calculation.
One day in grade 8 for reasons forgotten, the Chinese teacher was talking about some things, and said that when the news of 9/11 came, she was in the teacher's office, the general feeling in the teacher's office among the teachers is that "Great! They deserved that."
The timing was probably unfortunate. It happened during a period of heightened tensions due to the Hainan Island incident. Though when was there a period of lowered tensions between China and America after 2000?
How much that should constitute a Bayesian update, is left as an exercise for the reader. Notes: We were notified of this post by a twitter mutual, and we were not looking about 9/11 or reading LessWrong usually, so this might be considered a quite random sampling of the population.