Currently there seems to be no good way to deal with this conundrum.
One wonders whether the AI alignment community should setup some sort of encrypted partial-sharing partial-transparancy protocol for these kinds of situations.
Shapely values are very cool. Let me mention some cool facts:
They arise in (cooperative) game theory but also in ML when doing credit allocation a combined prediction from mixing predictions from different modules of a system.
One piece of evidence of their fundamentalness is that they arise naturally from the Hodge theory on the hypercube of a coalition game: https://arxiv.org/abs/1709.08318
Another interesting fact I learned from Davidad: Shapley values are not compositional: a group of actors can increase their total Shapley value by forming a single caba...
Could you give a sneak peak on how Sylvia /Alain use these uniform distributions?
There is a third important aspect of functions-in-the-original-sense that distinguishes them from extensional functions (i.e. collection of input-output pairs): effects.
Describing these 'intensional' features is an active area of research in theoretical CS. One important thread here is game semantics; you might like to take a look:
https://link.springer.com/chapter/10.1007/978-3-642-58622-4_1
How can the uniform distribution on the natural numbers be used?
No problem!
Do you mean monoidal categories? I think that's what the central concept in the Abramsly-Coeke work & the Baez Rosetta stone paper is.
Category theory was developed in the 50s from considerations in algebraic topology. Algebraic topology was an extremely technically sophisticated field already in the 50s (and has by now reached literally incredible abstract heights).
I suppose one could imagine an alternate world where Galois invents category theory but it seems apparent to me that the amount of long-term development significantly divorced from direct applications (as calculus was) was needed for category theory to spring up and mature - indeed it is still in its teenage rebellious phase i...
Evidence Manipulation and Legal Admissible Evidence
[This was inspired by Kokotaljo's shortform on comparing strong with weak evidence]
In the real world the weight of many pieces of weak evidence is not always comparable to a single piece of strong evidence. The important variable here is not strong versus weak per se but the source of the evidence. Some sources of evidence are easier to manipulate in various ways. Evidence manipulation, either consciously or emergently, is common and a large obstactle to truth-finding.
Consider aggregating many ...
In the real world the weight of many pieces of weak evidence is not always comparable to a single piece of strong evidence. The important variable here is not strong versus weak per se but the source of the evidence. Some sources of evidence are easier to manipulate in various ways. Evidence manipulation, either consciously or emergently, is common and a large obstactle to truth-finding.
Consider aggregating many (potentially biased) sources of evidence versus direct observation. These are not directly comparable and in many cases we feel direct obser...
With all due respect with your brand as LessWrong's ornamental hermeneutic I'm afraid I'll need some clarification.
What is the monad of 1 exactly? A monad is a functor - what category are we talking about here?
In particular - what are the unit and multiplication maps?
(my guess: for the unit and but now I'm using nilsquare infinitesimals instead of invertible infinitesimals.)
I'm not sure what tangent space we are talking about - but I assume it's a Lie group (hyperfinite graph?) and we ...
Where can I learn more about hyperfinite Brownian motion?
Has this been developed deeply? (I am aware of Nelson's radically elementary probability book)
Where can I read more about this perspective?
I'm intrigued by the idea of linking the discrete and continuous Fourier transform through nonstandard analysis.
The idea is certainly beautifully elegant - has it been worked out in more detail somewhere?
Beautiful chaotic math energy here Alok, keep it up! =)
Roko's basilisk is a thought experiment which states that an otherwise benevolent artificial superintelligence (AI) in the future would be incentivized to create a virtual reality simulation to torture anyone who knew of its potential existence but did not directly contribute to its advancement or development.
Why Roko's basilisk probably doesn't work for simulation fidelity reasons:
Roko's basilisk threatens to simulate and torture you in the future if you don't comply. Simulation cycles cost resources. Instead of following through on torturing our wo...
And actually Barbara, Celarent, Darii, Ferio contain hints of phenomena not completely captured in first-order logic!
I like that this post can be read as in both jest and earnestness and in both readings it contains much Truth and Wisdom. =)
Imagine a data stream
assumed infinite in both directions for simplicity. Here represents the current state ( the "present") and while and represents the future
Predictible Information versus Predictive Information
Predictible information is the maximal information (in bits) that you can derive about the future given the access to the past. Predictive information is the amount of bits that you need from the past to make that optimal prediction.
Suppose you are...
"The links between logic and games go back a long way. If one thinks of a debate as a kind of game, then Aristotle already made the connection; his writings about syllogism are closely intertwined with his study of the aims and rules of debating. Aristotle’s viewpoint survived into the common medieval name for logic: dialectics. In the mid twentieth century Charles Hamblin revived the link between dialogue and the rules of sound reasoning, soon after Paul Lorenzen had connected dialogue to constructive foundations of logic." from the Stanford Encyclopedia ...
Yes - seems sensible. I believe ARC is doing some work tracing out various possible attack vectors of AI.
Yes. Please do.
This would be of interest to many people. The tractability of nanotech seems like a key parameter for forecasting AI x-risk timelines.
"I dreamed I was a butterfly, flitting around in the sky; then I awoke. Now I wonder: Am I a man who dreamt of being a butterfly, or am I a butterfly dreaming that I am a man?"- Zhuangzi
Questions I have that you might have too:
In this shortform I will try and...
Q: What is it like to understand advanced mathematics? Does it feel analogous to having mastery of another language like in programming or linguistics?
A: It's like being stranded on a tropical island where all your needs are met, the weather is always perfect, and life is wonderful.
Except nobody wants to hear about it at parties.
level 0: A state of ignorance. you live in a pre-formal mindset. You don't know how to formalize things. You don't even know what it would even mean 'to prove...
Agent Foundations Reading List [Living Document]
This is a stub for a living document on a reading list for Agent Foundations.
Causality
Book of Why, Causality - Pearl
Probability theory
Logic of Science - Jaynes
Modern type theory mostly solves this blemish of set theory and is highly economic conceptually to boot. Most of the adherence of set theory is historical inertia - though some aspects of coding & presentations is important. Future foundations will improve our understanding on this latter topic.
It seems that if the quadcopters are hovering close enough to friendly troops it shouldn't be too difficult to intercept a missile in theory. If you have a 10 sec lead time (~3 km at mach 1) and the drone can do 20 m/s that's 200 meters. With more comprehensive radar coverage you might be able to do much better.
I wonder how large the drone needs to be to deflect a missile however. Would it need to carry a small explosive to send it off course? A missile is a large metal rod - in space with super high velocity even a tiny drone would whack a missile off cou...
Thanks Daniel, that's good to know. Sam Altman's tweeting has been concerning lately. But it would seem that with a fixed size content window you won't be able to pass a true Turing test.
Ambiguous Counterfactuals
[Thanks to Matthias Georg Mayer for pointing me towards ambiguous counterfactuals]
Salary is a function of eXperience and Education
We have a candidate with given salary, experience and education .
Their current salary is given by
We 'd like to consider the counterfactual where they didn't have the education . How do we evaluate their salary in this counterfactual?
This is slightly ambiguous - there are two counterfactuals:
or
In the second c...
Hopfield Networks = Ising Models = Distributions over Causal models?
Given a joint probability distributions famously there might be many 'Markov' factorizations. Each corresponds with a different causal model.
Instead of choosing a particular one we might have a distribution of beliefs over these different causal models. This feels basically like a Hopfield Network/ Ising Model.
You have a distribution over nodes and an 'interaction' distribution over edges.
The distribution over nodes corresponds to the joint probability di...
Insights as Islands of Abductive Percolation?
I've been fascinated by this beautiful paper by Viteri & DeDeo.
What is a mathematical insight? We feel intuitively that proving a difficult theorem requires discovering one or more key insights. Before we get into what the Dedeo-Viteri paper has to say about (mathematical) insights let me recall some basic observations on the nature of insights:
(see also my previous shortform)
I agree that the mathematics of agency, goals and values must exist and mostly hasn't been found yet (though we have many parts, e.g. Von Neumann Utility theorem).
I am skeptical that this is purely an historical accident. It seems to me that this theory is sufficiently complex and requires breakthroughs in several different areas. I don't think it could have been solved in the fifties. Most relevant math was simply not developed enough.
That doesn't mean progress cannot be sped up. With the threat of AI x-risk this area of enquiry has seen renewed interest and we've seen several breakthroughs here in just the last few years. As an example, Critch's new work on Boundaries will probably play an integral part.
Woah, Andrew, this is fantastic work! I am seriously excited about this direction.! I liked your previous posts on boundaries very much too, but I had no idea your thoughts on boundaries were this technically refined - and that they tie in so beautifully with Markov blankets!
re: Friston.
Friston particular style that could justifiably be called obscurantist. His writing is extremely verbose, often fails to define key terms, and very nontrivial equations are often posited without derivation or citation. After spending considerable effort trying to understand...
Excellent post Adam!
I presume it is a canonical joke but I had to chuckle at the cat-gorizers =D
If you think about programmes as lambda terms then they have free variables in which you can plug values. Flipping the red and the green light or 0 with 1 becomes less mysterious from this point of view.
Taking a fully applied category/compositionality point of view we could be looking at "open systems" (so systems with input and output ports) and see how they compose.
Yeah follow-up posts will definitely get into that!
To be clear: (1) the initial posts won't be about Crutchfield work yet - just introducing some background material and overarching philosophy (2) The claim isn't that standard measures of information theory are bad. To the contrary! If anything we hope these posts will be somewhat of an ode to information theory as a tool for interpretability.
Adam wanted to add a lot of academic caveats - I was adamant that we streamline the presentation to make it short and snappy for a general audience but it...
Although good alignment research has been done that does not involve maths [e.g. here and here ] good math* remains the best high-level proxy of nontrivial, substantive, deep ideas that will actually add up to durable knowledge.
*what distinguishes good math from bad math? that's a tricky question that requires a strong inside view.
If I assign a probability to an event and my friend assign a probability to an event at what odds "should" we bet?
It seems that while there are a number of fairly natural suggestions there isn't '' one canonical answer to rule them all". I think the key observation here is that what bet gets made is underdetermined from just the probabilities.
We need to add more information to the beliefs of me and my friend to resolve this ambiguity. As mentioned above there is a duality between market order (order by number of ...
Another aspect of markets (and thermodynamic systems!) is that they may be open systems: they can have excess demand or supply of goods - and be open to the meddling of outside investors.
So an open market might have input./output nodes where we might have nonzero flows of goods (or particles). A formal mathematical model might make use of ideas from compositionality and applied category theory.
An inflow of a good will - all else equal - lower the price of that good. If we think of forecasting markets this would correspond to evidenc...
You are right.
Thanks for your comment Vladimir! This shortform got posted accidentally before it was done but this seems highly relevant. I will take a look!
There is a key difference between an abstract algorithm and instances of that algorithm running on a computer. To take just one difference: we might run several copies of the same algorithm on a computer/virtual environment. Indeed, even the phrasing: several copies of the same algorithms hints to their fundamental distinctness. A humorously inclined individual might perhaps like to baptise the abstract algorithm as the Soul, while the instances are the Material Body or Avatars. Things start to get interesting when we conside...
These estimates are questionable. You should be aware that historically the nuclear winter hypothesis has been the darling of Soviet propaganda.
https://apps.dtic.mil/sti/citations/ADA165794
anthropics cannot be used here.
The global stockpile of Nuclear weapons are not and never was sufficient to wipe out the human race. Nuclear war would be catastrophic but not close to the end of the world. In fact, even an all-out nuclear war would leave 50% of US population untouched.
Yeah apparently the nuclear winter story was actively promoted by Soviet spies
Unclear to me if this piece of Soviet propaganda was on net bad or good for preventing nuclear brinkmanship.
Hah no 'betray' in its less-used meaning as
unintentionally reveal; be evidence of.
"she drew a deep breath that betrayed her indignation"
Hah! Yes.
Also, a good definition does not betray all the definitions that one could try but that didn't make it. To truly appreciate why a definition is "mathematically righteous" is not so straightforward.
From personal anecdote it seems pretty clear to me people have idiosyncratic preferences orthogonal to general desirability that are real. Even spending a lot of time with people that don't fit those preferences does not make them attractive.
I do take your points about the state of the research.
(This shortform was inspired by the following question from Daniel Murfet: Can you elaborate on why I should care about Kelly betting? I guess I'm looking for an answer of the form "the market is a dynamical process that computes a probability distribution, perhaps the Bayesian posterior, and because of out of equilibrium effects or time lags or X, the information you derive from the market is not the Bayesian posterior and therefore you should bet somehow differently in a way that reflects that"?)
Timing the Market
Those with experience with financial marke...
(This was inspired by the following question by Daniel Murfet: "Can you elaborate on why I should care about Kelly betting? I guess I'm looking for an answer of the form "the market is a dynamical process that computes a probability distribution, perhaps the Bayesian posterior, and because of out of equilibrium effects or time lags or , the information you derive from the market is not the Bayesian posterior and therefore you should bet somehow differently in a way that reflects that"?")
[See also: Kelly bet or update and Superrational agents Kelly be...
Multi-Step Fidelity causes Rapid Capability Gain
tl; dr Many examples of Rapid Capability Gain can be explained by a sudden jump in fidelity of a multi-step error-prone process. As the single step error rate is gradually lowered there is a sudden transition from a low fidelity to a high fidelity regime for the corresponding multistep process. Examples abound in cultural transmission, development economics, planning & consciousness in agent, origin of life and more.
Consider a factory making a widget in N distinct steps. Each step has a probab...
Math research as Game Design
Math in high school is primarily about memorizing and applying set recipes for problems. Math at (a serious) college level has a large proof-theoretic component: prove theorems not solve problems. Math research still involves solving problems, and proving theorems but it has a novel dimension: stating conjectures & theorem, and most importantly the search for the 'right' definitions.
If math in high school is like playing a game according to a set of rules, math in college is like devising optimal strategies within the confin...
So happy to see this post appear! 🔥
The story about operons and the high interconnectedness of prokaryote genomes makes me wonder: bacteria kick out the antibiotic-coding gene after a few hours... but how does it know which gene to kick out?
Does it have a way to tell which genes are more 'alien' than others? (Or are we only talking about plasmids here?) I've heard it's hard to genomic manipulate some genomes because the cells keep kicking out new genes
One could speculate there is some sort of mechanism, perhaps epi-genetic, that is able to tell which genes are more alien / new than others somehow?
I'd love to hear your thoughts