This is a special post for quick takes by Algon. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
In an Open Letter to Scott Alexander, Stephen Pinker claims RAND [1] found there is "no describable scenario in which an AI could conclusively pose an extinction threat to humanity."
This is at best sloppy research, and at worst actively misleading, as Cody Fenwick points out that Rand's conclusions significantly differed from what Pinker claimed.
RAND actually said AI plausibly could wipe out humanity in 2 of 3 scenarios they looked at, though they couldn't definitively assert that any of the three scenarios is likely or unlikely.
I can see where Pinker got his claim--Rand did not show there is obviously a describable scenario in which an AI wipes out humanity. But:
1) Rand didn't search very hard
2) RAND certainly doesn't find it implausible that AI wipes out humanity, which is what Pinker is trying to argue against.
Overall, I continue to be disappointed by Pinker's arguments.
[1] https://www.rand.org/content/dam/rand/pubs/research_reports/RRA3000/RRA3034-1/RAND_RRA3034-1.pdf
I think it's sort of correct to characterize the RAND paper as concluding that extinction risk is not likely, though they add some nice caveats.
But the bigger problem is that the RAND paper is very bad, with a total lack of imagination about what a powerful AI could do. (It doesn't even consider robots or drones.)
2Algon
I think it's weasly to characterise it that way, as the RAND report also goes out of its way to say that they don't conclude extinction risk is unlikely. Plus, as you say, it's not a good paper.
5hephaestus
In what way did RAND not search very hard? The authors were transparent about their research process and screened specifically for scenarios where it could be plausible that every last human would die. They consulted multiple experts on each topic (except for the malicious geoengineering case, which is a very niche subject for which there are only three published papers on intentional warming). They left out mirror life for another analysis, but the cyber-physical argument that they make for extinction methodology is valid.
2Algon
AFAICT, they don't even talk about plain old robots killing everyone. That's a pretty obvious way to exterminate an enemy, even if it does not look respectable.
For another, they seem to limit themselves to scenarios that only use existing tech in existing ways. Like, why assume the AI can't ramp up nuclear stockpiles massively? Or, more pertinently, dismiss nanotech, not on the grounds of it taking too long to create, but technical feasibility? AFAICT drexlerian nanotech is pretty feasible, let alone programmable biology.
1Marko Katavic
It has the smell of methodological rigour at the top - but at crucial points of concentration for the argument they make weird choices.
Consider bio risk - which is the one they couldn’t rule out. The reason they didn’t rule it in is primarily because they couldn’t find a plausible AI-initiated deployment mechanism with the current level of technology, yet they put this caveat in the deployment section:
“Without the means to interact in the physical world in a general way (e.g., using robotics or manipulating expert humans to perform certain tasks), it is unclear how an AI agent would carry out these steps.”
Which is disingenuous at best, given proven propensity and efficacy of AI’s to perform persuasion and manipulation. This is made worse by the fact that they consider exactly those vectors in the nuclear scenarios and find them plausible (but the nuclear scenarios suffer from being catastrophic and not extinction at the results level).
Question: if 10 members of the public read an article making a case for AI safety, how many of them would say it is less than 10 years to human-level AI? How many would say it is less than 20 years? How many would say human level AI would appear before 2100? How many would say after 2100? (Assume these are multiple choice questions).
Answer:
70% said within 10 years, 30% said within 20 years.
Context: I recently ran a paid survey on Prolific, asking people to read A Case for AI Safety and give us feedback. People's timelines surprised me, but not @plex, who won some bayes points off me. Plex thought this could be a neat calibration test for LW.
2
17yams
Fwiw I don’t take Prolific and similar services to be especially reliable ways to get information about this sort of thing. It’s true that they’re among the best low-medium effort ways to get this information, but the hypothetical at the top of this post implies that they’re 1:1 with natural settings, which is false.
2Algon
That sounds reasonable, but how do you know this? Also, any recommendations for better ways to get this information w/o being more than 2x as costly? (Cost $100 for 10 people, who spent 35 minutes reading and giving feedback).
6yams
answer is somewhat complicated and I'm not sure 'know' is quite the right bar
contractor verification is a properly hard problem for boring bureaucratic reasons; it's very hard to know that someone is who they say they are, and it's very hard to guarantee that you'll extract the value you're asking for at scale ('scalable oversight' is actually a good model for intuitions here). I have:
1. Been part of surveys for services like the above
2. Been a low-level contractor at various mid-sized startups (incl. OAI in 2020)
3. Managed a team of hundreds of contractors doing tens of thousands of tasks per month (it was really just me and one other person watching them)
4. Thought quite a lot about designing better systems for this (very hard!!!)
5. Noted the lack of especially-convincing client-facing documentation / transparency from e.g. Prolific
The kinds of guarantees I would want here are like "We ourselves verify the identities of contractors to make sure they're who they say they are. We ourselves include comprehension-testing questions that are formulated to be difficult to cheat alongside every exit survey. etc etc"
Most services they might pay to do things like this are Bad (but they're B2B and mostly provide a certification/assurance to the end-user, so the companies themselves are not incentivized to make sure they're good).
Feel free to ask more questions; it's kind of late and I'm tired; this is the quick-babble version.
EDIT: they're not useless. They're just worse than we all wish they'd be. To the best of my knowledge, this was a major motivator for Palisade in putting together their own message testing pipeline (an experience which hasn't been written about yet because uh... I haven't gotten to it)
2Mateusz Bagiński
What's your sample size?
3Algon
10
1Sodium
man you should probably get some more I can't imagine it'll be that expensive?
2Algon
$100 for what I already got. I could pay less, but I am not sure if that would make the signal/noise ratio too low to be worthwhile. Maybe @yams could tell us?
Random things I learnt about ASML after wondering how critical they were to GPU progress.
ASML makes specialized photolithography machines. They're about a decade ahead of competitors i.e. without ASML machines, you'd be stuck making 10nm chips.
They use 13.5nm "Extreme UV" to make 3nm scale features by using reflective optics to make interference patterns and fringe. Using low res light to make higher res features has been going on since photolithography tech stalled at 28nm for a while. I am convinced this is wizardry.
RE specialization: early photolithography community used to have co-development between companies, technical papers sharing tonnes of details, and little specialization in companies. Person I talked to says they don't know if this has stopped, but it feels like it has.
In hindsight, no-one in the optics lab at my uni talked about chip manufacturing: it was all quantum dots and lasers. So maybe
It's unclear how you can decrease wavelength and still use existing technology. Perhaps we've got 5 generations left.
We might have to change to deep UV light then.
Even when we reach the limits of shr
ASML makes machines for photolithography, somehow using light wi... (read more)
7PeterMcCluskey
SemiAnalysis has a report (partly paywalled) here about a potential competitor to ASML.
2noahzuniga
Seems like this competitor is just fraud, as reported here: I Think Substrate is Fraudulent: Part 1
Perhaps my favourite relation in physics is
t/T = (l/L)^{1-k/2}.
This says that for a bunch of particle in a potential V = a x^k, if you let the system evolve over time T forming a path which has size L in some sense, then there is another path which is a re-scaled version of the original one s.t. if it has size l, then the time taken to form this new path is t.
We can use this trick to create a bunch of “scaling laws” for simple physical systems. For example:
1) Let V = a x ^{-1} i.e. a gravitational potential. Then we have k=-1, so
t/T = (l/L)^{3/2}
(t/T)^{2} = (l/L)^{3}
I.e. Kepler’s third law. In fact, this is a more general result because it also applies to particles falling into a potential well. If you increase the distance the particle takes to fall in, you can use this equation to tell you how much the time increases, too.
2) V = a x^{2} i.e. the potential of an oscillator. Then we have k=2, so
t/T = (l/L)^{0}
That is, we can take an oscillation and increase the amplitude without changing the time it takes to complete the oscillation.
3) V = a x i.e. potential of a constant force. Then we have k=1, so
t/T = (l/L)^{1/2}
(t/T )^{2}= (l/L)
This reflects the fact that the position of a part... (read more)
5papetoast
That's really cool, is there a wiki page for this formula? [Edit: Mechanical similarity but it's a stub]
btw: LW supports latex
2Algon
Good job on finding the wikipedia page for this! I didn't know what it was called, I just read about it in Landau vol. 1 many years ago.
And if I had to typset the equations I probably wouldn't have written this quick take.
3papetoast
Good job to chatgpt actually, I just dumped your post to it lol. Also you can just ask LLM to typeset the math but keep other content verbatim and im certain it will one shot it
5gwern
It'll zero-shot easy cases, yeah.
And if you want to convert to HTML/Unicode for places where you don't have direct LaTeX support, you can also have a LLM do that, albeit there area lot of edge-cases and I don't think LLMs will usually use more exotic Unicode like FRACTION SLASH for things like '3/2' etc, so I have a big script for that: https://gwern.net/static/build/latex2unicode.py (Github).
2Algon
Oh, I didn't realize you could just paste latex into the LW editor and it autoparses, I thought you had to use ctrl+4 to summon a block in which you write math equations.
2XelaP
For Kepler's third law I find it simpler to say the following (equivalent) blurb
Acceleration scales as distance/time^2. Gravity scales as 1/distance^2. These will be equal if the scalings satisfy time^2 = distance^3.
Shiboleths in AI discourse mask a lot of deep disagreement between people. We chatter a lot is about shibboleths. About timelines and P doom. But shibboleths are social markers. They exist in far mode. They are not what our inner simulate actually predicts, if you’d give it the chance. In other words, we’re faking most of our talk on AI.
Another factor masking disagreement is the vagueness of shibboleths. Timelines to what? Doomed by what? Two guys can have the same timelines and P doom but have radically different interpretations of what that means. One gu... (read more)
"So you make continuous simulations of systems using digital computers running on top of a continuous substrate that's ultimately made of discrete particles which are really just continuous fluctuations in a quantized field?"
"Yup."
"That's disgusting!"
"That's hurtful. And aren't you guys running digital machines made out of continuous parts, which are really just discrete at the bottom?"
"It's not the same! This is a beautiful instance of the divine principle 'as above, so below'. (Which I'm amazed your lot recognized.) Entirely unlike your ramshackle tower ... (read more)
Question: What's going on from a Bayesian perspective when you have two conflicting intuitions and don't know how to resolve them? Or learn some new info which rules out a theory, but you don't understand how precisely it rules it out?
Hypothesis: The correction flows down a different path than down the path which is generating the original theory/intuition. That is, we've failed to propagate info down our network and so you have a left-over circuit that believes in the theory which still has high weight.
People have trouble accepting conversational bids, even ones that seem obvious. I’ve seen cases where person A asks person B the same dang question 10 times throughout a conversation and B just can’t seem to give a response that’s in any way related to the question that was asked. Often, I feel like humans just confabulate vaguely plausible looking responses in a conversation to be used as a filler until they can opine on what they want to talk about. A less cynical explanation is that people aren’t making the conversational bids they appear to be making e... (read more)
Some interesting books from my reading list.
Analytical Mechanics - Nivaldo A Lems
In the tradition of V.I. Arnold, Goldstein, and Landau & Lifshitz. The author is really classical mechniacs pilled. Though the book is more focused on the mathematics of it all than the physics.
Fundamental Principles of Classical Mechanics
In the same tradition as Lems but more heuristic. It uses physical motivation to build up to formal ideas. Places a greater focus on a local co-ordinate approach. The author emphasises Cartan's Method of Moving Frames, for ped... (read more)
Kleinian view of geometry
About 1900, Klein discovered an interesting perspective on geometry. He found that you could view a Euclidean space as a real space plus a group of transformations that leave figures congruent to one another. This group is formed by combining rotations, translations and reflections. We call this the Euclidean group. A natural question to ask was whether other spaces could be characterized in terms of symmetry groups? Klein showed the answer is yes.
For an affine geometry, it is characterized by symmetry under the action ... (read more)
Why do leaves fall in autumn? The obvious answer is that there's a high upkeep cost to them, and during winter there is little sunlight, so tree leaves should fall down then. That is, they should be deciduous trees. So trees in Montreal or London be deciduous. Whereas in places like the Mediterranean, tree leaves shouldn't be deciduous, they should be evergreen. And it is so. Evergreen trees dominate in temperate, consistently sunny climates, whereas in places like the UK, only 10% of tree species are evergreen.
But that 10% figure is confusing. Doesn't it ... (read more)
I'm working on some articles why powerful AI may come soon, and why that may kill us all. The articles are for a typical smart person. And for knowledgable people to share to their family/friends. Which intros do you prefer, A or B.
A) "Companies are racing to build smarter-than-human AI. Experts think they may succeed in the next decade. But more than “building” it, they’re “growing” it — and nobody knows how the resulting systems work. Experts vehemently disagree on whether we’ll lose control and see them kill us all. And although serious people are... (read more)
5Lorec
[A], just 'cause I anticipate the 'More and more' will turn people off [it sounds like it's trying to call the direction of the winds rather than just where things are at].
[ Thanks for doing this work, by the way. ]
3Algon
Thanks for the feedback!
4Nathan Helm-Burger
A, since I think the point about growing vs constructing is good, but does need that explanation.
In "Proofs and Refutations", Imre Laktos[1] portrays a socratic discussion between teacher and students as they try to prove Euler's theorem V+F =E+2. The beauty of this essay is that the discussion mirrors the historical development of the subject, whilst also critiquing the formalist school of thought, the modern agenda of meta-mathematics, and how it doesn't fit the way mathematics is done in practice. Whilst I'm on board with us not knowing how to formalize mathematical practice, I think it is a solvable problem. Moreover, I am a staunch ultra-fin... (read more)
I have re-worked my best post so that it is easier to read. I recommend checking it out. https://www.lesswrong.com/posts/gtk2KqEtedMi7ehxN/do-one-new-thing-a-day-to-solve-your-problems
I was bored, so I decided to re-construct a max flow algorithm since I forgot how Ford Fulkerson works.
OK, you have a graph w/ positive integers denoting capacity for each edge, and start/end nodes. You want to get a flow going from input->output which maxes out the channel capacity.
How to do this? Well, I vaguely remember something about a blocking flow. Intuition: to figure out how much stuff you can push through a network, look at the bottlenecks.
What's a bottleneck? Well, it's a thingy which everything has to pass through that limits the performance... (read more)
When tracking an argument in a comment section, I like to skip to the end to see if either of the arguers winds up agreeing with the other. Which tells you something about how productive the argument is. But when using the "hide names" feature on LW, I can't do that, as there's nothing distinguishing a cluster of comments as all coming from the same author.
I'd like a solution to this problem. One idea that comes to mind is to hash all the usernames in a particular post and a particular session, so you can check if the author is debating someone in th... (read more)
4habryka
Yeah, I think the hide author feature should replace everyone with single letters or something, or give you the option to do that. If someone wants to make a PR with that, that would be welcome, we might also get around to it otherwise at some point (but it might take a while)
I've been thinking about exercises for alignment, and I think going through a list of lethalities and applying them to an alignment propsal would be a good one. Doing the same with Paul's list would be a bonus challenge. If I had some pre-written answer sheet for one proposal, I could try the exercise my self to see how useful it would be. This post, which I haven't read yet, looks like it would serve for the case of RLHF. I'll try it tomorrow and report back here.
Some random notes on harvesting energy at stellar scales. Again, from Grand Futures.
Harvesting dark energy:
Tying galaxies together: Anchor big rope to galaxies as they get pulled apart by dark matter. Build up elastic potential energy which can be harvested. Issue: inefficient. You get out < 10^{-39} times energy of rope. Needs negative energy density matter to give better efficiency. Not clear how you anchor rope to energy
Letting particles expand apart: again, very tiny amounts of energy compared to... (read more)
4habryka
We feed it back into the Recombee magical RL/ML algorithm which we use to generate those recommendations. I don't really know what they do with it.
2Algon
Fair enough. If you ever get round to figuring out how this all works, it would be nice to know.
6habryka
I mean, I think it's probably some kind of standard transformer architecture behind the scenes that predicts user behavior and giving it negative feedback is equivalent to a backropagation step, or maybe some kind of RL step. I don't have that much deep uncertainty about how it works, besides of course that we have no idea what's going on inside of deep neural nets.
4kave
They tout their transformer ("beeformer") in marketing copy, but I expect mostly its driven by collaborative filtering, like most recommendation engines
4habryka
My guess is most recommendation engines in use these days are ML/DL based. At least I can't think of any major platform that hasn't yet switched over, based on what I read.
2kave
I would definitely consider collaborative filtering ML, though I don't think people normally make deep models for it. You can see on Recombee's website that they use collaborative filtering, and use a bunch of weasel language that makes it unclear if they actually use anything else much at all
1) DATA I was thinking about whether all metrizable spaces are "paracompact", and tried to come up with a definition for paracompact which fit my memories and the claim. I stumbled on the right concept and dismissed it out of hand as being too weak a notion of refinement, based off an analogy to coarse/refined topologies. That was a mistake.
1a) Question How could I have fixed this?
1a1) Note down concepts you come up with and backtrack when you need to.
&nbs... (read more)
Rough thoughts on how to derive a neural scaling law. I haven't looked at any papers on this in years and only have vague memories of "data manifold dimension" playing an important role in the derivation Kaplan told me about in a talk.
How do you predict neural scaling laws? Maybe assume that reality is such that it outputs distributions which are intricately detailed and reward ever more sophisticated models.
Perhaps an example of such a distribution would be a good idea? Like, maybe some chaotic systems are like this.&nb... (read more)
You, yes you, could've discovered the importance of topological mixing for chaos by looking at the evolution of squash in water. By watching the mixture happening in front of your eyes before the max entropy state of juice is reached. Oh, perhaps you'd have to think of the relationship between chaos and entropy first. Which is not, in fact, trivial. But still. You could've done it.
Question: We can talk of translational friction, transactional friction etc. What other kinds of major friction are there?
Answers:
a) UI friction?
b) The o.g. friction due to motion.
c) The friction of translating your intuitions into precise, formal statements.
- Ideas for names for c: Implantation friction? Abstract->Concrete friction? Focusing friction! That's perhaps the best name for this.
- On second thought, perhaps that's an overloaded term. So maybe Gendlin's friction?
d) Focusing friction: the friction you experience when focusing.
Hypothesis: agency violating phenomena should be thought of as edge-cases which show that our abstractions of ourselves as agents are leaky.
For instance, look at addictive substances like heroin. These substances break down our Cartesian boundary (our intuitive seperation of the world into ourselves and the environment with a boundary) by chemically assaulting the reward mechanisms in our brain.
However, video games or ads don't obviously violate our Cartesian boundary, which may be one of many boundaries we assume exist. Which, if my hypothesis is tr... (read more)
My mind keeps returning to exercises which could clarify parts of alignment, both for me and others. Some of them are obvious: think about what kind of proof you'd need to solve alignment, what type of objects it would have to be talking about etc. and see whether that implies having a maths oracle would make the problem easier. Or try and come up with a list of human values to make a utility function and see how it breaks down under greater optimization pressure.
But what about new exercises? For skillsets I've never learnt? Well, there's the security mind... (read more)
One thing I'm confused about re: human brain efficiency is, if our brain's advantage over apes is just scaling and some minor software tweaks to support culture, what's that imply for Corvid brains? If you scaled Corvid brains up by the human-cortical-neuroun-count/chimp-cortical-neuoron-count, and gave them a couple of software tweaks, wouldn't you get a biological Pareto improvement over human brains?
Obvious thing I never thought of before:
Linear optimization where your model is of the form W1⋅W2....⋅Wn, the Wi being matrices, will likely result in an effective model of low rank if, you randomize the weights. Compared to just a single matrix -- to which the problem is naively mathematically identical, but not computationally -- this model won't be able to learn the identity function, or rotations or so on when n is large.
Note: Someone else said this on a gathertown meetup. The context was, that it is a bad idea to think abou... (read more)
1the gears to ascension
wow I'm not convinced that won't work. the only thing initializing with random weights should do is add a little noise. the naive mathematical interpretation should be the only possible interpretation up to your float errors, which, to be clear, will be real and cause the model to be invisibly slightly nonlinear. but as long as you're using float32, you shouldn't even notice.
[trying it, eta 120 min to be happy with results... oops I'm distractable...]
1Algon
EDIT: Sorry, I tried something different. I fed in dense layers, followed by batchnorm, then ReLU. I ended it with a sigmoid, because I guess I just wanted to constrain things to the unit interval. I tried up to six layers. The difference in loss was not that large, but it was there. Also, hidden layers were 30 dim.
I tried this, and the results were a monotonic decrease in performance after a single hidden layer. My dataset was 100k samples of a 20 dim tensor sampled randomly from [0,1] for X, with Y being a copy of X. Loss was MSE, optimizer was adam with weight decay 0.05, lr~0.001 , minibatch size was 32, trained for 100,000 steps.
Also, I am doubtful of the mechanism being a big thing (rank loss) for such small depths. But, I do think there's something to the idea. If you multiply a long sequence of matrices, then I expect them to get extremely large, extremely small, or tend towards some kind of equillibrium. And then you have numerical stability issues and so on, which I think will ultimately make your big old matrix just sort of worthless.
2the gears to ascension
oh by linear layer you meant nonlinear layer, oh my god, I hate terminology. I thought you literally just meant matrix multiplies
My body is failing me. I have been getting colds near weakly for a year and a half, after a particularly wretched cold. My soul is failing me. I have been worn down by a stressful environment, living with an increasingly deranged loved one. By my crippled minds inability to meet the challenge. Which causes body to further fail. Today, I grokked that I am in a doom spiral, headed down the same path as my kin's. I don't wish for so wretched an end, for an end it shall be.
But why my failing soul? Why does the algorithm which calls itself Algon fail when chall... (read more)
After thinking about how to learn to notice the feel of improving in pursuit of the meta, I settled on trying to reach the meta-game in a few video games.
After looking at some potential games to try, I didn't follow up on them and kept playing die in the dungeon. Nothing much happened, until my improvement decellarated. For whatever reason, I chose to look in the comments section of the game for advice. I rapidly found someone claiming they beat the game by removing every dice except 2 attack, 1 defence, 1 heal and one boost die and upgrading them to... (read more)
Sci-hub has a telegram bot which you can use with a desktop application. It is fast, and more importantly reliable. No more scrounging through proxy lists to find a link that works. Unfortunately, you need to install telegram on a phone first. Still, it has saved me some time and is probably necessary for an independant researcher.