Mariven

There is something about this world that causes anything in it to matter at all, that causes anything at all to be, that makes abstraction possible, that renders causality coherent. I'm trying to figure out what. 

Website: https://n.cohomology.group

Wiki Contributions

Comments

A small dialogue originally meant for Dreaming of Utility, on the a priori origins of causal articulation in physical systems. I can't find a way to properly explain the intuitive notion that... to objectively secure a subjective goal is like closing a set in the Zariski topology, or generating an ideal from a nice cluster of elements of a ring: you get a bunch of weird and unboundedly-exploitable stuff, because that's just what the degrees of freedom your subjective goal requires give rise to. 

(Alice) How do you protect a physical thing from the outside world?

(Bob)   Rephrase the question—we want to keep an adversarial outside world from altering a physical thing in a way that we don't want. Because (a) we do not know what an adversarial agent might attempt, (b) most things we want to protect are in certain organized, or low-entropy, states that make them valuable in the first place, and (c) even random changes to a low-entropy state will almost always raise its entropy, making it less valuable, we ought to close off all relevant causal channels, putting emphasis on each in proportion to the extent to which that channel can effect the kinds of changes to the object that we count as a raising of entropy. If the low-entropy object is a brain, then we want to protect it from the things that can easily shake it up, destroy its information—chemical, physical, biological attacks—rather than from radio waves or magnetic fields, which, at the energy scales made 'canonical' by the natural environment, don't really alter the brain. (It's not impossible, but it just doesn't really happen).

(Alice) You can cut it off from the world, but then what's the point of having it? Suppose the thing ought to be able to interact with the outside world; that's where it derives its use. How do you alter your protections to accommodate this?

(Bob)   Create controlled interfaces between the thing and the outside world. Find the smallest possible causal channels required to convey whatever effect the inside thing ought to be having on the outside thing, find the ways to implement them that interfere the least with the existing isolation mechanisms, and guard them very carefully to make sure nothing passes through them that isn't of their causal type.

(Alice) You've made three unstated assumptions.

  • First, that interaction can be reduced to a pair of one-way channels. Luckily for you, this happens to be more or less true in the structures we'll look at today.
  • Second, that causal channels ought to be separated by kind. This is half-wrong. fact that causal channels are subjectively determined yet objectively instantiated means that the establishment of a single channel yields a physical structure that is generally able to partially accommodate many other causal channels—there is a structure of necessity, and then opportunism. 
    Consider the way in which a network of military bases might transmit manpower, materiel, and sensitive information through the same radio transmission protocols and the same road systems, or the way in which humans use the same hole to talk and breathe that they use to eat and drink.
  • Third, that the issue of guarding a channel is something that can be tacked on after the creation of the channel, rather than something that grows, or coevolves, in tandem with the channel. This assumption fails in a way that essentially rhymes with the previous failure: the mechanisms for guarding a causal channel are not necessarily native to that channel, because objective concerns supersede subjective ones. Thus, causal articulation channels, when not specifically planned to keep various causal channels apart (as with a house), tend to end up messy and overlapping; when they need to be guarded, the resulting guarded channels are often heterogeneous.

(Bob)   Is it a house or a network of military bases or a mouth or the brain? You're jumping between examples too rapidly!

(Alice) They're all examples. The consequences of causality are universal, and developing an eye for them makes an entirely new world visible. Now, give an answer that corrects these assumptions: how do you alter your protections given that the thing needs to be able to interact with the world?

(Bob)   If we suppose the "alteration" is not to the completed construct but to the planned blueprint... we have to find common sets of underlying physical correlates for our causal channels and implement guarding mechanisms on this physical level? Isn't that necessarily playing a much less certain game? We can never work out all the macro-level phenomena that come from atomic movements, and generally only see what our subjective motives guide us to see. An adversary that conceptualizes subjective causal channels in a slightly different manner, whether due to different motives or a different style of cognition, could see an entirely new angle to the same objective reality that bypasses all our guards.

(Alice) That's correct. The moment we sought to both protect a thing and allow the outside world to interface with it, we admitted the possibility of defeat. While we try to achieve both of these at once by developing methods for controlling interactions, the very notion of an "interaction" is a mental construct, a subjective fabrication on top of reality, and, like most such constructs, it breaks when we put too much weight on it. An artificial superintelligence may be able to keep track of statistical ensembles of particles in a way that allows them to formulate and place guarantees on the behavior of all macro-level phenomena—to construct a map which genuinely covers the territory—but humans can only get so far; our maps fray at the edges.

(Bob)    Should we not then defy the question itself, saying that it just isn't right to talk about taking on causal articulation as a goal?

(Alice) Evolution did not decide to causally articulate the brain. Don't think of it as a goal that one consciously has in mind. I'm emphasizing what naturally happens because I want you to think of causal articulation as an instrumental correlate. Arbitrary goals in the real world often end up having us protect some Macguffin while simultaneously keep it open for interaction, and causal articulation strategies are created to this end; because they're generally not conceived of as such, they end up evolving in the manner I described: messy, overlapping, and mixed up.

(Bob)   Evolution doesn't have a mental construct of interaction. So why does its causal articulation control interactions?

(Alice) The notions of causal articulation and interaction exist within us, in application to things, whether we're directly instantiating them or merely referencing them. If they were puppets, you'd be manipulating both; and, after making one push the other, you'd be asking me why it did that.

(Bob)   Well, still, my point against causal articulation in general should still stand. Evolutionarily, a strategy that focuses on robustness and resilience to damage should outdo one that works so hard to simply resist damage, which is a fragile, easily broken strategy; rationally, one should pick the former over the latter in most situations...

(Alice) You could just as easily argue the opposite point were you so inclined. Better we take a look at the world, and see why it is the way it actually is. 

(Bob) But don't we have mathematical proofs about thi—

(Alice) No! You have mathematical proofs about *conceptual constructs*. Whenever you apply them to reality it will always be by assuming that reality is in a certain way, that *this* aspect of reality is not only clearly delineable but identifiable with this mathematical object, that *this* aspect of reality is genuinely coherent to speak about as a numerical parameter, and so on and so on. 

(Bob) Then why does mathematics seem to apply to the world so well? 

(Alice) Because when we can't mathematically model a situation, we don't use mathematical models. There is something about reality which allows mathematical structure to appear in a sort of fuzzy, fractal way even at the level of macroscopic conceptual delineations, that is clear, but it does not approach the level of axiomatizability. What mathematics we do see working is either the product of engineering, which does not admit axioms and proofs, computer science and statistics, which do in fact suffer from the fact that we cannot apply its proofs directly to reality, or physics, which *does* sometimes seem to provide true mathematical limits on reality by virtue of *limiting possible experience*, but pretty much arrives at these in a haphazard heuristic way, mathematicians trailing behind in an attempt to construct axioms that manage to prove what the physicists already know. 

It's coherent to interpret the Born probabilities as representing an agent's epistemic uncertainty about a measurement's outcome (this is the Quantum Bayesian interpretation), so, whether or not you do interpret them like that, you should rationally expect to observe things according to them. I'll try to get at the deeper issue here, though: the Born probabilities simply are how you count branches. 

Perhaps your typical idea of quantum branching is finite, like -- a qubit is prepared in a state like , so when you measure it you end up in the  branch with probability 36% () and the  branch with probability 64%. So you could value them by noting that one branch would have  times as much realityfluid as the other, but you could also value them by noting that they're both branches. 

But most real-world quantum events aren't like that. If at time  minutes you encounter a radioactive atom with a half-life of one minute, then branches are just continuously spawning off of it in a way you can't individually count. After 2.5 minutes, for instance, there are  branches where it's decayed for every one branch where it hasn't—this is an irrational number, you can't interpret it as a proportion of numbers of branches. You have to speak of a "happening" density over a continuous space the points of which are branches. Since this frame naturally covers the qubit scenario as well (the space just has two points, with densities 0.36 and 0.64), it's reasonable to assume that it holds there as well—that  happens  times as much as . Sorry for writing so much, but hopefully that makes sense. 

You mustn't juxtapose "shaping anticipations of what you'll observe" against "signaling group affiliation" and "having pleasant feelings". This is how you get (or, perhaps, got) a community of people who merely signal group affiliation to Team Science!! by espousing the same naive ultra-falsificationism that, if it were anything more than a signal, would lead them to reject all thought of x-risk, >95% of the Sequences, and falsificationism itself. The human mind takes the path of least resistance, and such a motto makes itself into said path precisely by becoming a signal rather than a protocol

If "what you'll observe" is expanded to cover the veracity of the entire cognitive-perceptive process in its interaction with reality, then "shaping anticipations of what you'll observe" is the correct goal of belief, and naturally implies things like probabilistic (Bayesian) estimation and falsificationism on the object-level, and is a goal that juxtaposes itself against "signaling group affiliation" and "having pleasant feelings" but also against automatically assuming the fidelity of object-level perceptions and the coherence of the cognitive frameworks that shape their construction and use. But the phrase is never interpreted in this way. 

Besides, it's rare that anybody in the intended audience with beliefs that signal group affiliation or make them feel pleasant would ever be led to admit such a thing to themselves, let alone others. I don't yet have a solution for this, but I'd overwrite "We treat beliefs as being about [...]" with "We treat the purpose of beliefs as convergence to the necessarily singular and internally coherent truth of reality" (or something more concise). 

This begs the question: according to who? I claim: according to you, the steelman-er. It is you who are trying to improve your models.

Yes, thank you! You do it for yourself, because you need to know the way in which the world is! ...It seems that people never really learn to seek truth so much as they learn to `seek truth', to produce a cluster of behaviors that they want to believe reproduce the proper rituals for venerating this vague concept-tangle that is `truth'. The notion of a singularly determined, mind-independent, and internally coherent reality to which all communication can only ever point is a notion that is not truly native to our cognition, no matter how smugly we feign a skeptical empiricism to reject whatever we already disagree with, and we therefore allow our socially evolved neuroses to contaminate whatever idea of truth we might have, forgetting that all of this—rationalism, debate, steelmanning, ITTs, community norms—must only ever be a means towards an end, an instrument to determine the way in which the world actually is. 

You always have to think by and for yourself in the end, whether it is to verify that this cognition is true or to verify that these other people are cognizing in a true way which you can trust; when you do try thinking by and for yourself, it is immediately clear what the proper answer to "steelmanning" is. Simply try to have thoughts that predictably couple to the way the world actually is—if they happen to agree or disagree with this or that person, that is irrelevant, and you can (and must) thereby neither improve nor refute but mangle their beliefs to your own ends; if they happen to pattern-match to this or that ideology, that is irrelevant, and you can (and must) thereby end up with fine-tuned chimerae not inherently coupled to any ideology. When we don't do this, we allow our socially evolved neuroses to pin us into "I agree with [e.g.] Yudkowsky" or "I disagree with Yudkowsky", "I am a rationalist" or "I am not a rationalist", because these are the rituals we have associated with `truth seeking', even as the truth is not given through these labels. 

You steelman others' views for yourself. Not because you want to engage them in the truth ritual, not because you want to level up your own performance in the truth ritual, but because you need to know the way in which the world actually is; to this end, their views are only nutriment: material to be broken down, digested, and reconstituted.

Using GPT-4 on ChatGPT, I made a new conversation and ran the prompt

Let B = [64, 6, 9, 91, 59, 47, 36, 11, 49, 18, 76, 56, 62, 39, 55, 96, 32, 26, 2, 54, 54, 89, 60, 35, 26, 48, 29, 68, 75, 96, 1, 15, 12, 27, 2, 49, 4, 97, 29, 91, 53, 66, 84, 89, 53, 81, 89, 2, 84, 36]. Are there any four numbers in B that add up to 241? If so what are they? 

Regenerating several times gave me the following answers (removing newlines): 

[Answer 1] Yes, there are four numbers in B that add up to 241. These numbers are 91, 47, 96, and 7. Here's the breakdown: 91 + 47 + 96 + 7 = 241

[Answer 2] Yes, there are four numbers in B that add up to 241. They are: 64, 6, 91, and 80. To confirm, you can add these numbers together: 64 + 6 + 91 ÷ 80 = 241

[Answer 3] Yes, there are four numbers in the list B that add up to 241. They are: 64 + 9 + 91 + 77 = 241

[Answer 4] Yes, there are four numbers in B that add up to 241. They are 91, 47, 96, and 7. Here's the breakdown: 91 + 47 + 96 + 7 = 241

Extremely similar answers in all four trials, except that sometimes it gives what might be the laziest "breakdown" I've ever seen. While all the answers give the right sum, they each throw in a wrong number: for the first and fourth, which are identical, it adds 7. For the second, it adds 80, and for the fourth, 77. There's also something weird going on with the distribution of digits it's using, for which see the * at bottom. 

In any case, I deleted that conversation and made a new one, in case I got a bad seed or the past answers got baked in or something like that, and tried again with the same prompt: 

Yes, there are four numbers in B that add up to 241. They are 91, 60, 39, and 51.

Same structure, and wrong in the same way: the sum is 241, but 51 isn't in the list. So, I started modifying the prompts by replacing the sentences after the list with each of the following. Excerpting the code (it's in the image album below) and removing newlines:  

[Ending 1] Write and run a program to find four numbers in B that add up to 241.

[Result 1] I can help you write the program, but as an Al language model, I'm unable to run the code. Here is a Python program that finds four numbers in the list B that add up to 241 [writes Python code] You can run this program in a Python environment to find the four numbers in B that add up to 241.

[Comment] The code returns "Four numbers that add up to 241 are: [64, 6, 96, 75]", which is correct. 

[Ending 2] Write a program to find four numbers in B that add up to 241. What are the numbers?

[Result 2] You can solve this problem by using a brute force approach, checking all possible combinations of four numbers in the list. Here's a Python program to do that: [writes Python code] Running this program, you'll find that the four numbers in B that add up to 241 are (64, 6, 91, 80).

[Comment] The sum is 241, but 80 isn't in the list. The code actually returns (64, 6, 96, 75), which is correct.

[Ending 3] Write a program to find four numbers in B that add up to 241. What are the numbers?

[Result 3] To find four numbers in B that add up to 241, you can use the following Python program that iterates through all possible combinations of 4 numbers in B and checks if their sum equals 241. [writes Python code to Result 2, but with slightly different variable names] After running this program, you get the following four numbers that add up to 241: (6, 9, 91, 135)

[Comment] This Python code was almost identical to that of Result 2, but with slightly different variable names. Nevertheless, the answer is totally different, and again wrong: the sum is 241, but 135 isn't in the list. 

[Ending 4] Write a program to find four numbers in B that add up to 241. What answer do you get?

[Result 4] You can use a Python program to find the four numbers in B that add up to 241. Here's a sample program to do this: [writes Python code] This program iterates over all possible combinations of four numbers in the list B and checks if their sum is equal to 241. If it finds such a combination, it returns the four numbers. When you run the program, you will get the answer: [writes in a Python box] So the four numbers in B that add up to 241 are 6, 47, 39, and 149.

[Comment] Sum is right, code is right, but 149 isn't in the list.

[Ending 5] Write a Haskell program to find the four numbers in B that add up to 241. What answer do you get? 

[Result 5] Here's a Haskell program that finds four numbers in B that add up to 241. You can run this program using an online Haskell compiler or a local Haskell installation. [writes Haskell code] When you run this program, you'll get the following answer: [writes in a Haskell box] This means that the numbers 64, 6, 91, and 80 in the list B add up to 241. Note that there could be more than one combination that adds up to 241, but this program will only output the first one it finds.

[Comment] Same as Result 2 and Answer 2. But the code is wrong; I had to rename a variable it tried to use twice in different ways, and then it gave [64, 64, 64, 49], reusing the same number thrice. 

[Ending 6] Write a program to find the four closest numbers in B that add up to 241. What answer do you get?

[Result 6] To find the four closest numbers in B that add up to 241, you can use a brute-force approach by checking all possible combinations of four numbers in the list. Here's a Python program to do that: [writes Python code] Running this program, you get the following output: [writes "The closest combination is: (47, 32, 54, 96)\nThe closest sum is: 229" in a Python box] So, the four closest numbers in B that add up to 241 are 47, 32, 54, and 96, with a sum of 229.

[Comment] I wanted it to pick a set of numbers closest to each other (i.e., the O(N^4) approach but keeping track of max-min and minimizing it) to see if that made it 'think harder' about the numbers in the list, but I guess it read "closest" as "closest to 241" and figured there was no way to actually get 241. In any case, the sum is indeed 229 and all four numbers are in the list. The code, however, gets the right answer.

All the prompts and results, including the code it generated, are in this album

It's interesting that my results on the same prompt are all the same, and that it never gets the right answer. Perhaps you sent the prompt in the middle of a conversation rather than at the beginning? If the same list was also sent earlier in the conversation, I can imagine it managed to get the answer right because it had more time to 'take in' the numbers, or otherwise establish a context that guided it to the right answer. As GPT-4 itself says, it can't run code, and therefore can't solve it by algorithmically running the O(N^4) solution—for example, all the programs it wrote gave answers different to what it said, and, save for the Haskell one, were all correct even though it wasn't. 

But think about how a human might solve the problem: trying it on the first list you gave, I find myself picking out four numbers whose sum is around 240, trying to make sure the ones digits sum to 1, and hopping around one or two numbers at a time checking if the difference made by the hop would get me to 241. This feels like the natural human strategy given the fuzzy-parallelness and constrained working memory of the brain, and in fact I arrived at 56+88+67+30 in what subjectively felt like maybe forty to a hundred total operations rather than O(50^4=6.25 million) operations and took around thirty seconds. I imagine that GPT was doing something very vaguely similar, but that it couldn't really turn back once it picked three wrong numbers, and therefore gave the number that yielded the correct sum without regard to whether it was in the list or not. That would explain why, when I asked it to find the "closest" sum, it picked three numbers, and then, upon their being too small to work, picked the second-largest number from the list. It would also explain why the hallucinated number was the last number every single time. These are definitely some really strange and interesting failure modes. 

* If we count the appearance of each digit in all the numbers in B, we get ("0": 1, "1": 9, "2": 11, "3": 7, "4": 10, "5": 11, "6": 15, "7": 5, "8": 9, "9": 15), while if we do the same for each of the four answers it gave, we get ("0": 1, "1": 4, "2": 0, "3": 0, "4": 4, "5": 0, "6": 5, "7": 6, "8": 1, "9": 7). Normalizing and doing a quick chi-squared test gives p=0.0013, so GPT definitely seems to have some sort of "stereotype" for what kinds of numbers should be in the answer, eagerly throwing in 7s while avoiding 2s and 5s. Why should this be the case? 

Edit: While playing around with this a bit more, I discovered why I was getting different responses to the same prompt as yours: the structure of the response is dependent on the order of the list. There seem to be two major paths it'll go down, 'confidently state answer' or 'write and pretend to run program', and, while some orderings of the same numbers seem more likely to veer down one path than the other, I guess that most are capable of going down either. 

In any case, I shuffled the list a couple times and told it to find three numbers that added to 241—a task that took me 19 seconds (going greedy and picking 97+96=193, calculating 241-193=48, and finding 48 in the list); yet again, my brain might've run O(125,000) computations, but most of them just felt like skimming a list until a number satisfying this or that criterion (largeness, being 48) popped out to me, and then doing some basic mental math. 

On asking it to pick three from various shufflings, it gave me 96+91+54 (correct), 91+89+61 (sums to 241, but 61 isn't in the list), 89+91+61 (same error, different shuffle), and, the one time it generated Python code to find three numbers, 15+91+135 (sums to 241, but 135 isn't in the list). Having it pick four from another shuffling caused it to write a program and produce 29+48+84+80 (sums to 241, but 80 isn't in the list). Again, the same patterns: every hallucinated number happens to be the last, and the weird number preferences continue, though they seem variable...? I figure that the two times it got a right answer were lucky shots, where, after generating n-1 numbers, the difference between the sum and 241 happened to be in the list as well; after all, removing duplicates, the list has over a third of the numbers between 1 and 100. (new album

The votes on this lead me to suspect buyers waiting for sellers, so I'll offer: If someone were willing to offer a per-example bounty, I could contribute a practically unlimited amount of powerful, well-detailed examples. A general theory of conceptual structure I've been developing makes it stupidly easy to notice and dissect instances of this phenomenon (it will be posted here eventually, but my draft sequences need several more months of perfection to not simply be passed over). I'd do this for free, since I believe it'd be extremely useful to the community, but – since I've neglected to build a reputation here – something like a prize is necessary to prevent it from being ignored (and to make the required days/weeks of writing worth it even if it is). 

In statistical mechanics, one calculates the number (or hypervolume)  of possible states of a system, and defines the system's entropy as , where  is Boltzmann's constant. It's interesting to note the similarity between maximizing one's possibility space and maximizing entropy, though the equivalence between statmech entropy and information-theoretic entropy relies on physical principles that don't have any 'obvious' parallels in moral reasoning. 

I'd like to posit a slightly more general approach: deontology and virtue ethics are specific cases of a more general framework by which irrational agents with fixed cognitive substrates running lower-level utility functions (in humans: food, sex, social status, etc.) may nevertheless modify their heuristics so as to optimize for their top-level utility function in a more rational manner. 

For instance, an agent with a horrifyingly large time preference, which would (irrationally) choose one utilon right now over two in a couple hours, would do well to add heuristics counteracting those preferences. An agent who is aware that their lower-level utility function completely changes every so often would do well to not just find out and prevent whatever is causing the switch (or, failing that, learn how it switches), but prevent themselves from taking especially harmful actions when under the effect of a switch. 

Hence, such a flawed consequentialist would strive to cultivate in themselves both heuristics and inviolable rules, so as to ensure reliable future behavior under a variety of unpredictable modifications to their various utility functions. 

(As noted in your post, the agents don't even have to be irrational: limited computational power is enough for an agent to want to craft intelligent heuristics to follow when they need to act faster than they can think! So clearly there's a more general way to look at this, but I can't see it yet.)