Goodfire wrote Intentionally designing the future of AI about training on interpretability. This seems like an instance of The Most Forbidden Technique which has been warned against over and over - optimization pressure on interpretability technique [T] eventually degrades [T]. Goodfire claims they are aware of the associated risks and...
Humans are contacted by a mysterious type of being calling themselves “Galabren” who say they are “aelthous”. They’d like to know if we, too, are aelthous, since if we are they’d like to treat us well, as they care about aelthous things. We ask the Galabren what aelthous means and...
To elaborate a bit (but you should go read the posts!) the classic Most Forbidden Technique scenario goes like this: you detect "bad thoughts" in the CoT/activations, you penalize them, and the model learns to have the same bad thoughts in ways your detector can't see.
Goodfire seems to be doing something structurally different, by decomposing the gradient into semantic components and choosing which components to apply. Does that really make the the difference, thought? I don't know!
Goodfire wrote Intentionally designing the future of AI about training on interpretability.
This seems like an instance of The Most Forbidden Technique which has been warned against over and over - optimization pressure on interpretability technique [T] eventually degrades [T].
Goodfire claims they are aware of the associated risks and managing those risks.
Are they properly managing those risks? I would love to get your thoughts on this.
People who meditate a lot often drop their sense of agency and it does come with a layer of suffering being dropped, but the baseline experience remains.
In my experience, the conscious experience of pain changes quite dramatically when the aversion is properly dropped. So dramatically I wouldn't know that the two experiences are at all related if I encountered them separately.
If the rules for phenomenal consciousness follow necessarily from physical laws then the phenomenal bridge should not be specified as part of the generating string.
I think your description of phenomenal consciousness as "a problem of mapping between our first-person perspectives (our direct experience) and a shared third-person perspective (the shared understanding of an objective world)" is absolutely correct; and I think that once a particular structure is specified from the outside (third-person), what it looks like from the inside (first-person) follows by necessity.
Demonstrating that first-person experience follows by necessity from the third-person view to be true and clearly showing how is one of my primary fantasy-ambitions, and I don't claim to know that it is true, but I think it should be true.
I think that in philosophy in general and metaethics in particular, the idea that since many people disagree one should not be confident in one's ideas is wrong.
I'll somewhat carefully spell out why I think this; a lot of this reasoning is obvious, but the core claim is that the intuitions people use in philosophy in order to ground their arguments are often wrong in predictable ways.
"One man's modus ponens is another man's modus tollens" is usually what is at the core of ongoing philosophical disagreements. Suppose is universally agreed, is somewhat intuitive to everyone but the degree to which that intuition is compelling varies, and is somewhat unintuitive to everyone but the degree to... (read more)
The hyphenated version isn't working for me
I have a nine month old son and in no way feel like everything else is getting thrown under the bus (or that my ambition has decreased as a result).
I do obviously care about keeping my son alive and well, and that does require some degree of compromise with my other goals, but I haven't experienced caring less about my other goals than I did previously.
Also the reasons I care about him have stayed pretty constant both before and after he was actually born, I don't think oxytocin is noticeably influencing how I feel my goals (my guess is it's just causing my feelings of affection towards him in small scale interactions).
Thank you for not deleting
I grew up on a farm. We had (my parents still have) dairy goats, chickens for meat, and rabbits for meat (we also killed the male goats and some of the females which we couldn't keep for dairy). The animals weren't for the business (the only thing we sold was low bush blueberries), they were just for my family, and by raising them we didn't need to buy meat or dairy elsewhere.
I took care of the animals. I got to see how they lived and how they died, see them when they were in states of joy and of suffering. I also had a hand in killing them, especially the chickens.
I am... (read 733 more words →)
Noticing this only works as an example if the two signals are (approximately) the same partition of , i.e. (temperatures at which the mirror fogs) is approximately the same as (temperatures at which you have goosebumps).
Humans are contacted by a mysterious type of being calling themselves “Galabren” who say they are “aelthous”. They’d like to know if we, too, are aelthous, since if we are they’d like to treat us well, as they care about aelthous things.
We ask the Galabren what aelthous means and they say it’s difficult to describe—that essentially there’s a feeling of aelthousness which has something to do with what it feels like from the inside to exist as a Galabren (and perhaps as other beings too, they’re not sure).
Aelthousness isn’t obviously necessary to explain any of their objective behaviors; the only reason they know it’s there is because they can feel it.
It’s very... (read 459 more words →)
Is Goodhart's Curse Not Really That Bad?
EDIT: It's bad. Still, it's good to understand exactly when it's bad.
I'm not implying I'm on to anything others haven't thought of by posting this - I'm asking this so people can tell me if I'm wrong.
Goodhart's Curse is often cited to claim that if a superintelligent AI has a utility function which is a noisy approximation of the intended utility function, the expected proxy error will blow up given a large search space for the optimal policy.
But, assuming Gaussian or sub-Gaussian error, the expected regret is actually something like where is the size of the raw search space. Even if search space grows exponentially with intelligence, expected... (read more)
This year's Spring ACX Meetup everywhere in Williamstown.
Location: Sawyer Library 430 Conference Room – https://plus.codes/87J8PQ7X+C5F
Group Link: https://eph.lol/acx/
There will be suggested reading ahead of the meetup
Contact: acx.meetup@satchlj.com
If you haven't already, I'd recommend reading Vinge's 1993 essay on 'The Coming Technological Singularity': https://accelerating.org/articles/comingtechsingularity
He is remarkably prescient, to the point that I wonder if any really new insights into the broad problem have been made in the last 32 years since he wrote. He discusses, among other things, using humans as a base to build superintelligence on as an possible alignment strategy, as well as the problems with this approach.
Here's one quote:
... (read more)Eric Drexler [...] agrees that superhuman intelligences will be available in the near future — and that such entities pose a threat to the human status quo. But Drexler argues that we can confine such transhuman devices so that
Thank you - this matches my current thinking