aysja

Wiki Contributions

Comments

This reminds me a lot of one of Kuhn's essays A Function for Thought Experiments. Where basically he's like "people often conflate variables together; thought experiments can tease apart those conflations." E.g., kids will usually start out conflating height with volume so that even though they watch the experimenter pour the "same" amount of water into a taller, thinner glass, they will end up saying that the left hand glass in (c) has more water than the one on the right. 

Sale > conservation of liquid piaget > in stock

 

Which is generally a good heuristic: height of water line and volume are usually pretty correlated.  Eventually, though, experience brings these two variables into tension and kids will update their models.  Kuhn argues that thought experiments are often playing this role, i.e., calling attention to and resolving conceptual tension between variables that were previously conflated.

In any case, I think the strategy "considering more possibilities" is really important for figuring out the "edges" of concepts... it feels sort of like "playing" with them until you have a "feel" for what they are.... which seems related to me to your ideas about "indexing," too.  Anyways, I thought a bunch of these examples were great. I now find myself confused about waves. 

How does the redundancy definition of abstractions account for numbers, e.g., the number three? It doesn’t seem like “threeness” is redundantly encoded in, for example, the three objects on the floor of my room (rug, sweater, bottle of water) as rotation is in the gear example, since you wouldn’t be able to uncover information about “three” from any one object in particular. 

I could imagine some definition based on redundancy capturing “threeness” by looking at a bunch of sets containing three things. But I think the reason the abstraction “three” feels a little strange on this account is that it is both highly natural (math!) but also can be highly “arbitrary,” e.g., “threeness” is wherever a mind can count three distinct objects (and those objects can be maximally unrelated!). 

Perhaps counting the three objects on the floor of my room is a non-natural use case of the abstraction “three,” but if so, why? And where is the natural abstraction “three” in the world? 

I really value "realness" although I too am not sure what it is, exactly. Some thoughts:

I cannot stand fake wood or brick or anything fake really, because it feels like it is trying to trick me. It's "lying," in sort of the same way I feel like people lie when they say they are doing something because it helps climate change or whatever, when really it seems clear that they are doing it for social approval or something of that nature. 

Moss feels very real to me, also, as do silky spider webs, or any slice of nature, really, when I'm in it. I think it's because the moss is not pretending to be something else, not to me anyways, it's just there

Homes can be real-seeming to me, like how warm, cozy fireplaces with the wind whipping past the window and redwood walls make spaces seem inviting and true. But I think they can also be very not real. Many household things feel kind of "fake" to me, in the sense of trickery, like my microwave. It's not really deceiving me in the sense that it is lying about itself—it will heat up my food if I press some buttons, but it's like... asking something of me? Trying to get me to use it on its terms. Food containers with words on them also feel kind of like this... trying to get me to read them, to consume them, and so on... 

"Trying for real" is an especially interesting one to me because it feels so important and I don't know quite what it is.  At least part of it seems related to trickery, like how "actually trying" to answer a question looks like not giving up until you have a satisfying-to-your-curiosity answer and "not really trying" looks more like getting a good-enough-to-pass-another-person's-test answer, or not really believing it'll work, or something like that. Where the "actually trying" bit seems much more fundamentally related to the thing the trying is about, hence "real," the not-trying bit seems more related to something else entirely and that disconnect feels "fake" to me. 

Thanks for writing this up! It seems very helpful to have open, thoughtful discussions about different strategies in this space. 

Here is my summary of Anthropic’s plan, given what you’ve described (let me know if it seems off): 

  1. It seems likely that deep learning is what gets us to AGI. 
  2. We don’t really understand deep learning systems, so we should probably try to, you know, do that. 
  3. In the absence of a deep understanding, the best way to get information (and hopefully eventually a theory) is to run experiments on these systems. 
  4.  We focus on current systems because we think that the behavior they exhibit will be a factor in future systems.

Leaving aside concerns about arms races and big models being scary in and of themselves, this seems like a pretty reasonable approach to me. In particular, I’m pretty on board with points 1, 2, and 3—i.e., if you don’t have theories, then getting your feet wet with the actual systems, observing them, experimenting, tinkering, and so on, seems like a pretty good way to eventually figure out what’s going on with the systems in a more formal/mechanistic way. 

I think the part I have trouble with (which might stem from me just not knowing the relevant stuff) is point 4. Why do you need to do all of this on current models? I can see arguments for this, for instance, perhaps certain behaviors emerge in large models that aren’t present in smaller ones. But I’ve never seen, e.g., a list of such things and why they are important or cruxy enough to justify the emphasis on large models given the risks involved. I would really like to see such an argument! (Perhaps it does exist and I am not aware). 

I also have a bit of trouble with the “top player” framing—at the moment I just don’t see why this is necessary. I understand that Anthropic works on large models, and that this is on par with what other “top players” in the field are doing. But why not just say that you want to work with large models? Why mention being competitive with Deepmind or OpenAI at all? The emphasis on “top player” makes me think that something is left unsaid about the motivation, aside from the emphasis on current systems. To the extent that this is true, I wish it were stated explicitly. (To be clear, "you" means Anthropic, not Miranda). 

Ah, thanks! Link fixed now. 

Yes, welp, I considered getting into this whole debate in the post but it seemed like too much of an aside. Basically, Lynch is like, “when you control for cell size, the amount of energy per genome is not predictive of whether it’s a prokaryote or a eukaryote.” In other words, on his account, the main determinant of bioenergetic availability appears to be the size of the cell, rather than anything energetically special about eukaryotes, such as mitochondria. 

There are some issues here. First, most of the large prokaryotes are outliers like Thiomargarita, in the sense that they have expanded their energy without expanding their functional volume. However, their genomes are still quite small, which means that their “energy/genome” will be large. Eukaryotic cells of the same size have way more energy and way longer genomes, making their “energy/genome” roughly equivalent to the large prokaryotes. 

Second, Lynch’s story is that strong selection keeps bacterial genomes short. The main reason that bacteria have strong selection is because there are so many of them, and there are so many of them because they’re so small. But why are they so small? It seems like an obvious contender is Lane’s story about them being energy bottlenecked by their surface area. So, in my opinion, these two hypotheses are synergistic and my best guess is that they’re both part of the story. 

Thanks!!

Yeah I think it’s a great question and I don’t know that I have a great answer. Plasmids (small rings of DNA that float around separately) are part of the story. My understanding here is pretty sketchy, but I think plasmids are way more likely to be deleted than the chromosomal DNA, and for some reason antibiotic resistant genes tend to be in plasmids (perhaps because they are shared so frequently through horizontal gene transfer)? So the “delete within a few hours” bit is probably overstating the average case of DNA deletion in bacteria. I would be surprised if it “knew” about the function of the gene, although I agree it seems possible that some epigenetic mechanism could explain it. I don’t know of any, though!

Good question! I don’t know, but I think that they don’t necessarily need to. Something I didn’t get into in the post but which is pretty important for understanding bacterial genomes is that they do horizontal gene transfer, which basically means that they trade genes between individuals rather than exclusively between parents and offspring. 

From what I understand, this means that although on average the bacteria shed the unhelpful DNA if given the opportunity, so long as a few individuals within the population still have the gene, it can get rapidly reacquired when needed. I don’t know exactly how the math works out, but I’d guess that in big enough populations, if antibiotic encounters are somewhat common, then probably they don’t need to do it de novo each time? 

This also means bacterial genomes are much more distributed than eukaryotic ones. So long as any individual bacteria has some gene, it’s “as if” the whole species has it. Which means their genomes are, in a sense, actually longer than they might naively seem. Being distributed has advantages: no single genome needs to be very long, yet the population can hold onto useful stuff. But it also has disadvantages: any adaptation that relies on genes being close together in a single genome is unlikely to develop (which includes e.g. all of the regulatory hierarchy stuff mentioned in the post). So I do still expect that the pressure towards short genomes meaningfully stunts bacterial complexity. 

This is an excellent post! Thank you for sharing your thoughts! I too am very curious about many of these questions, although I’m also at a half-baked stage with a lot of it (I’d also love to have a better footing here!). But in any case, here are some thoughts (in no particular order).

  1. I’ve been interested in the questions you pose around AlexNet for a while, in particular, how much computation is a function of observers assigning values versus an intrinsic property of the thing itself. And I agree this starts getting pretty weird and interesting when you consider that minds themselves are doing computations. Like, it seems pretty clear that if I write down a truth table on paper, it is not the paper or ink that “did” the computation, it was me. Likewise, if I take two atoms in a rock, call one 0, the other 1, then take an atom at a future state, call it 0, it seems clear that the computation “AND” happened entirely in my head and I projected it onto the rock (although I do think it’s pretty tricky to say why this is, exactly!). But what about if I rain marbles down on a circle circumscribed in a square (the ratio of which “calculates” pi)? In this case it feels a bit less arbitrary, the circle and the square chalked on the ground are “meaningfully” relating to the computation, although it is me who is doing the bulk of the work (taking the ratio)? This feels a bit more middle ground to me. In any case, I do think there is a spectrum between “completely intrinsic to the thing” and “agents projecting their own computation on the thing” and that this is largely ignored but incredibly interesting and instructive for how computation actually works.  
  2. Relatedly, people often roll their eyes at the Chinese Room thought experiment (and rightly so, because I think the conclusions people draw about it with respect to AI are often misguided). But I also think that it’s pointing to a deep confusion about computation that I also share. The standard take is that, okay maybe the person doesn’t understand Chinese but the “room does,” because all of the information is contained inside of it. I’m not really convinced by this. For the same reason that the truth table isn’t “doing” the computation of AND, I don’t think that the book that contains the translation is doing any meaningful computation, and I don’t think the human inside understands Chinese, either (in the colloquial sense we mean when we don’t understand a foreign language). There was certainly understanding when that book was generated, but all of that generative tech is absent in the room. So I think Searle is pointing at something interesting and informative here and I tentatively agree that the room does not understand Chinese (although I disagree with the conclusion that this means AI could never understand anything).
  3. I do agree that input/output mappings are not a good mechanistic understanding of computation, but I would also guess that it’s the right level of abstraction for grouping different physical systems. E.g., the main similarity between the mechanical and electrical adder is that, upon receiving 1 and 1, output 2, and so on. 
  4. I get confused about why minds have special status, e.g. “computation is a function of both the dynamics and an observer.” On the one hand, it feels intuitive that they are special, and I get what you mean.  And on the other hand, minds are also just physical systems. What is it about a mind that makes something a computation when it wasn’t otherwise? It’s something about how much of the computation stems from the mind versus the device? And how “entangled” the mind is with the computation, e.g., whether states in the non-mind system are correlated with states in the mind?  Which suggests that the thing is not exactly “mind-ness” but how “coupled” various physical states are to each other.  
  5. I also think that the adder systems are far less (or maybe zero) observer dependent computations, relative to the rock or truth table, in the sense that there are a series of physically coupled states (within the system itself) which reliably turn the same input into the same output. Like, there is this step of a person saying what the inputs “represent,” but the person’s mind, once the device is built, does not need to be entangled with the states in the machine in order for it to do the computation. The representation step seems important, but also not as much about the computation itself rather than “how that computation is used.” Like, I think that when we look at isolated cases of computation (like the adders), this part feels weird because computation (as it normally plays out) is part of an interconnected system which “uses” the outputs of various computations to “do” something (like in a standard computer, the output of addition might be the input to the forward-prop in a neural net or whatever). And a “naked” computation is strange, because usually the “sense making” aspect of a computation is in how it’s used, not the steps needed to produce it. To be clear, I think the representation step is interesting (and notably the thing lacking in the Chinese Room), and I do think that it’s part of how computation is used in real-world contexts, but I still want to say that the adder “adds” whether we are there to represent the inputs as numbers or not. Maybe similar to how I want to say that the Chinese room “translates Chinese” whether or not anyone is there to do the “semantic work” of understanding what that means (which, in my view, is not a spooky thing, but rather something-something “a set of interconnected computations”). 
  6. Maybe a good way to think of these things is to ask “how much mind entanglement do you need at various parts of this process in order for the computation to take place?”
  7. My guess is that computation is fundamentally something like “state reliably changes in response to other state.” Where both words (“state” and “reliably”) are a bit tricky to fully pin down and there are a bunch of thorny philosophical issues. For instance, “reliably” means something like “if I input the first state a bunch of times, the next state almost always follows”, but if-thens are hard to reconcile with deterministic world views. And “state” is typically referring to something abstract, e.g., we say “if the protein changes to this shape, then this gene is expressed,” but what exactly do we mean by “shape”? There is not a single, precise shape that works, there’s a whole class consisting of slight perturbations or different molecular constituents, etc. that will “get the job done,” i.e., express the gene. And without having a good foundation of what we mean by an abstraction, I think talking about natural computation can be philosophically difficult.
  8. “Is there anything purely inside of AlexNet that can tell us that 1 in the output node means cat and that 0 means not cat?” I’m not sure exactly what you’re gesturing at with this, but my guess is that there is. I’m thinking of interpretability tools that show that cat features activate when shown a picture of a cat, and that these states reliably produce a “1” rather than a “0.” But maybe you’re talking about something else or have more uncertainty about it than me?
  9. I agree that thinking is extremely wild! ‘Nough said.

I love this work! It’s really cool to see interpretability on toy models in such a clear way.

The trend from memorization to generalization reminds me of the information bottleneck idea. I don’t know that much about it (read this Quanta article a while ago), but they appear to be making a similar claim about phase transitions. I believe this is the paper one would want to read to get a deeper understanding of it.

I like this framework, but I think it's still a bit tricky about how to draw lines around agents/optimization processes.   

For instance, I can think of ways to make a rock interact with far away variables by e.g., coupling it to a human who presses various buttons based on the internal state or the rock. In this case, would you draw the boundary around both the rock and the human and say that that unit is "optimizing"? 

That seems a bit weird, given that the human is clearly the "optimizer" in this scenario.  And drawing a line around only the rock or only the human seems wrong too (human is clearly using the rock to do this strange optimization process and rock is relying on the human for this to occur). Curious about your thoughts. 

Also, I'm not sure that agents always optimize things far away from themselves. Bacteria follow chemical gradients (and this feels agent-y to me), but the chemicals are immediately present both temporally and spatially. There is some sense in which bacteria are "trying" to get somewhere far away (the maximum concentration), but they're also pretty locally achieving the goal, i.e., the actions they take in the present are very close in space and time to what they're trying to achieve (eat the chemicals). 

Load More