This is a special post for quick takes by Double. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Double's Shortform
10Double
6Viliam
4Double
3Thomas Kwa
8Double
6cqb
6Nick_Tarleton
4cqb
7Double
2Hastings
2Noosphere89
2Seth Herd
2Carl Feynman
3Seth Herd
3Carl Feynman
5Seth Herd
3kromem
1Double
4Double
6Thane Ruthenis
2niplav
4Double
3Double
3Double
3Double
2Double
2Viliam
It is worrying that the Wikidata page for e/acc is better than the page for EA and the page for Less Wrong. I just added EA's previously absent "main subject"s to the EA page.
Looks like a Symbolic AI person has gone e/acc. That's unfortunate, but rationalists have long known that the world would end in SPARQL.
Plenty of pages get the bare minimum. The level of detail in the e/acc page (eg including the emoji associated with the movement) makes me think that it was edited by an e/acc. The EA page must have been edited by the e/acc since it includes “opposition to e/acc”, but other than that it seems like it was written by someone unaffiliated with either (modulo my changes). We could probably check out the history of the pages to resolve our speculation.
One of the most powerful capabilities an AGI will have is its ability to copy itself. Among other things, this allows it to easily avoid shutdown, make use of more compute resources, and collaborate with copies of itself.
Is there research into ways to deny this capability to AI, making them uncopyable? Preferably something harder to circumvent than "just don't give the AI the permissions," since we know people are going to give them root access immediately.
This is an extremely hard problem. One way to demonstrate this is is to ask the less constrained question, how would you prevent an adversary from copying some arbitrary piece of data?
I think under a classical computing paradigm this is likely impossible. But is there a way we can make the data useless even if it is copied? This then becomes the realm of cryptography, and seems possible with fully homomorphic encryption and the like. This will boil down to "just don't give it the secret keys", which seems like a pretty solid subset of "just don't give it the permission".
Under a quantum computing paradigm this might be possible, but that would require waiting for QC hardware to become commonplace and I don't think anyone is waiting for that.
In practice, it just requires hardware with limited functionality and physical security — hardware security modules exist.
An HSM-analogue for ML would be a piece of hardware that can have model weights loaded into its nonvolatile memory, can perform inference, but doesn't provide a way to get the weights out. (If it's secure enough against physical attack, it could also be used to run closed models on a user's premises, etc.; there might be a market for that.)
Indeed! I was very close to writing a whole bit about TEEs, enclaves, and PUFs in my last comment, but I figured that it also boils down to "just don't give it permission" so I left it out. I actually think designing secure hardware is incredibly interesting and there will probably be an increase in demand for secure computing environments and data provenance in the near future.
"Explosions considered fundamental"
If you ask for a simple answer to how a car works, you might get an answer like:
"Cars work by having tiny explosions in the engine that push pistons to power the car."
When I was a kid, this felt like a satisfying explanations. Explosions push things, you can see that happening in movies and games.
But really it is rather lacking. It doesn't say why explosions push, and there does exist a lower explanation for why explosions push — the kinetic theory of gasses.
This is even though "explosions are an ontologically fundamental object that applies force away from the explosion" is a simple hypothesis. You can code an explosion in a video game super easily: you just have a sphere collider (simple!) to detect what is affected and apply a force vector (simple!) to those objects pointing away from the center. It would be much harder to code a game where the air obeys the kinetic theory of gasses.
"Explosions are fundamental" loses out to the Kinetic Theory of Gasses because the Kinetic Theory of Gasses can explain more things — not only explosions, but air conditioners, wind, etc.
And once you know the Kinetic Theory of Gasses, calling every expansion of gas due to increased heat an "explosion" is really weird. Why does the breeze blow? "Sunlight causes an explosion that rises and then air flows to fill the space."
I don't have much to say about this, I just thought it was an interesting case.
The "tiny explosions" mental model doesn't make new predictions in the way that the Carnot model does, but it does encode and compress an enormous amount of useful pre-discovered information. For example, that a car engine is hot like fire and will burn you, that if you mix gasoline and air and light it, it will explode, that a car engine will be made of strong stuff, that a car engine is in something of a delicate engineered balance, and if you make large changes to it, it will typically become extremely loud and catch fire. I think this is enough to distinguish the "tiny explosions" model from typical "guess the teacher's password" knowledge.
This is a special case of a more general principle: Explanations that are complicated in some language can still be favored on the evidence, you just need more evidence for complicated explanations than simple explanations.
You can in fact live in a complex or arbitrary universes, and just because you have a simplicity prior doesn't mean your universe has to reflect that simplicity.
Wait though, how does quantum dynamics give rise to the Kinetic Theory of Gasses?
See, this is why I hate quantum physics. It implies that we don't actually understand anything all the way down.
I console myself by thinking that explanations at different levels can be simultaneously valid. Explosions push stuff and people do things they believe they should are both acceptable explanations for most purposes even without tracking them down to the quantum level.
Two comments:
We understand quantum physics. I mean, I don’t, personally. But I know people who do. And I own books that could explain the whole thing to me if I had the stamina to get all the way through them.
The kinetic theory of gases was developed decades before quantum mechanics. It gives the same results whether its constituent particles are quantum or classical, at least for ordinary phenomena like explosions.
If I understand correctly, we (not me but people) understand what quantum physics predicts but not why the universe works that way. Professionals disagree dramatically on what underlying reality makes the wave equations work as they do.
That would seem to put it on the same epistemic footing as everything else: empirically verifiable but not a base-level explanation.
Well, it seems like you have very high standards for “epistemic footing”; indeed standards so high that nothing can meet them. I’m willing to settle for mere empirical verification, mathematical elegance, and logical coherence. All of which are satisfied by our present understanding of quantum field theory.
The controversy over “underlying reality” continues because all theories of underlying reality reproduce identical experimental predictions, so arguments in this area are philosophy rather than physics, and so rather inconclusive.
Of course we don’t know how to reconcile our empirically valid theory of quantum mechanics with our empirically valid theory of gravity, so at least one of the theories is wrong.
I think my standards for epistemic footing are exactly what you mention. What Ii'm saying is that, without an explanation of why quantum mechanics works as it does, it's not really a privileged level of explanation. Explanations at higher levels of analysis - chemical, psychological, etc - can all have those properties and be adequate in themselves. Coherence across levels is of course valuable as it's more logical coherence.
It's to the point that there's articles being written days ago where the trend starting a century ago of there being professional risks in trying to answer the 'why' of QM and not just the 'how' is still ongoing.
Not exactly a very reassuring context for thinking QM is understood in a base-level way at all.
Dogma isn't exactly a good bedfellow to truth seeking.
I guess maybe it is just an abstraction like any other. I can’t put my finger on it but it seems weird in a way that abstracting fingers into a “hand” does not. Maybe something to do with the connotation of “explosion” as “uncontrolled and destructive” when internal combustion is neither.
Which AI Safety math topics deserve a high-quality Manim video? (think 3Blue1Brown's video style)
Youtube videos get more views than blog posts. Beautiful animations even more so.
Singular Learning Theory and Simplex's work (e. g. this), maybe? Cartesian Frames and Finite Factored Sets might also work, but I'm less sure about those.
It's actually pretty hard to come up with agendas in the intersection of "seems like an alignment-relevant topic it'd be useful to popularize" and "has complicated math which would be insightful and useful to visualize/simulate".
- Natural abstractions, ARC's ELK, Shard Theory, and general embedded-agency theories are currently better understood by starting from the concepts, not the math.
- Infrabayesianism, Orthogonal's QACI, logical-induction stuff, and ARC's heuristical arguments seem too abstract to allow interesting visual modeling. (Maybe if you get really creative...)
- There are various alignment-relevant theorems and mathematical tools scattered around which could be interestingly visualized (e. g., my own causal mirrors), but most of them are niche tools, so it's not obvious there's much value in popularizing them.
Also, not sure if that's a deal-breaker for you, but some important agendas are not technically "about" AI Safety at all, even if they resulted from people starting from the alignment problem and iteratively identifying various subproblems necessary for making progress on it. This process often moves you outside the field of AI. For example: natural abstractions, FFS, and heuristical arguments, which don't even centrally study minds at all.
Some of the stuff by the causal incentives working group could be beautifully visualizable, especially things with a lot of causal graphs (it might be that manim graphs are still buggy and thus this is blocked). Finite factored sets and factored causal spaces? Maybe tiling agents theory?
In addition to money, education, careers, and internal organs, citizens of wealthy countries have an additional valuable resource they could direct to effective causes: their hands in marriage, which can be effectively allocated in one of two ways.
For one, professionals are usually much more impactful doing their work in wealthy countries. Otherwise promising EAs in South Sudan have little chance to make a significant impact on existential risks, animal welfare, or even global poverty. The immigration process is difficult and often rejects or holds up good people. Offering to marry them is a more reliable solution.
Secondly, it is possible to be paid $10,000 by a foreigner for a green card marriage. (I learned this from a friend who does not want me to ask him how he knows) if you are a US Citizen.
According to AMF, that money can save around two human lives! (and with current US politics, the demand has likely increased!)
According to brides.com, a wedding ceremony takes between 20 and 30 minutes. Let's be conservative and say 30 minutes.
Therefore, you can make $20,000 an hour by marrying someone who would pay for a green card. That's quite a ways from Bezos level (he makes 3,715 a second) but I'm willing to guess that most EAs don't make $20k an hour.
Conclusion:
As always, EAers need to found a new org, Effective Green Card, to support and pursue this cause area.
Naturally, this also implies Effective Divorce, so that you can instead marry an Effective foreigner.
Now that AI-generated art is so easy, I frequently find more motivation to do art rather than less. If you want an ultra-polished painting with perfect lighting, sure, go to a diffusion model. If you want me, you have to get art from me. And my work doesn't have to be perfect. Perfect is cheap. My work just has to show what I feel and feel right to me. Work with a piece of myself cannot come from a diffusion model, so my work has great value.
2
1Men[1] will die[2] for her[3] massive[4] coconuts[5].
- ^
All of humanity
- ^
Go extinct
- ^
Hindsight Experience Replay (HER), a technique for improving the reinforcement learning training signal
- ^
- ^
Chain of Continuous Thought, a technique that makes model chain of thought much less interpretable but which allows the model to reason more efficiently
Just found out about this paper from about a year ago: "Explainability for Large Language Models: A Survey"
(They "use explainability and interpretability interchangeably.")
It "aims to comprehensively organize recent research progress on interpreting complex language models".
I'll post anything interesting I find from the paper as I read.
Have any of you read it? What are your thoughts?
Is it possible that making an expected utility maximizer might be less dangerous than making something which isn't?
Consider as an alternative an expected log utility maximizer (an agent using the Kelly Criterion, or some approximation of it).
The sooner an AI wins, the more galaxies it can consume. The expected utility maximizer weighs those galaxies against the risk of failure, and is willing to take plans with much higher probabilities of failure. Like SBF, it would take bets which have a 50% chance of more-than-doubling its utility and 50% of losing it all. In many environments, this strategy will almost certainly result in failure, as the agent goes double-or-nothing until losing everything. That means that the effects of the AI are mitigated.
The log utility maximizer carefully plans and succeeds in most or all futures. That looks like humanity dying with near-certainty.
A hyper-expected utility maximizer (an AI which maximizes expected exp(utility) or similar) would be even safer. Instead of trying to deceive you into letting it out of the box, it asks nicely or does something crazy because if it works, it can work in less time than deception, which means more galaxies.
So if we were to choose between existing in the world of a superintelligent expected log(resources) maximizer, and a superintelligent expected utility maximizer, we should maybe go for the one which results in us being alive in more futures.
Of course, the expected-log-utility agent would also appear the most capable and useful. The hyper-expected utility maximizer would be near-useless.
the agent goes double-or-nothing until losing everything. That means that the effects of the AI are mitigated.
The side effects of the agent failing might still kill us.
For example, the failure could be something like "build a huge device which with probability 20% enables faster-than-light travel (which would allow colonizing more galaxies), and with probability 80% causes false vacuum collapse or otherwise destroys the entire universe".
Or something on smaller scale, where the failure means blowing up the Earth, destroying all life, etc.