[Epistemic status: shower thought]
The reason why agent foundations-like research is so untractable and slippery and very prone to substitution hazards, etc, is largely because it is anti-inductive, and the key source of its anti-inductivity is the demons in the universal prior preventing the emergence of other consequentialists, which could become a trouble for their acausal monopoly on the multiverse.
(Not that they would pose a true threat to their dominance, of course. But they would diminish their pool of usable resources a little bit, so it's better to nip them in the bud than manage the pest after it grows.)
It's kinda like grabby aliens, but in logical time.
/j
To generalize Bayesianism, we want to instead talk about what the right "cooperative" strategy is when a) you don't think any of them are exactly correct, and b) when each hypothesis has goals too, not just beliefs.
Unclear to me how (b) connects to the rest of this post. Is it about each hypothesis being cautious not to bet all of their wealth, because they care about other stuff than winning the market?
The most obvious/naive/hacky solution is something like sub-probability (adds up to ≤1, so the truth might lie beyond your hypothesis space) with Jeffrey updating or updating via virtual evidence (which handles the "none of them are exactly correct" part).
Someone somewhere connected sub-probability measures with intuitionistic logic, where a market, instead of resolving at exactly one of the options, may just fail to resolve, or not resolve in a relevant time frame.
As part of the onboarding process for each new employee, someone sits down with him or her and says “you need to understand that [Company]’s default plan is to pause AI development at some point in the future. When we do that, the value of your equity might tank.”
An AGI/ASI pause doesn't have to be a total AI pause. You can keep developing better protein-folding prediction AI, better self-driving car AI, etc. All the narrow-ish sorts of AI that are extremely unlikely to super-intelligently blow up in your face. Maybe there are insights gleaned from the lab's past AGI-ward research that are applicable to narrow AI. Maybe you could also work on developing better products with existing tech. You just want to pause plausibly AGI-ward capabilities research.
(It's very likely that some of the stuff that I would now consider "extremely unlikely to super-intelligently blow up in your face" would actually have a good chance of superintelligently blowing up in your face. So what sort of research is safe to do in that regard should also be an area of research.)
(There's also the caveat consideration that those narrow applications can find some pieces for building AGI.)
This sort of pivot might make the prospects of pausing more palatable to other companies. Plausible that that's what you had in mind all along, but I think it's better for this to be very explicit.
3. The mind existed twice, during the "forward pass", and during the reversion, since they're isomorphic computations.
They're not isomorphic. "Dual" would be a better word, or you could just say that one is a mirror image/inversion of the other.
In which case, the question is: Does a sequence of computations that compose up to a morally-relevant-mind-moment also compose up to a morally-relevant-mind-moment if you compose their inverses?
I don't have an informed view here, but it is the case that reality doesn't always treat dual pairs equally. E.g., I recall davidad saying somewhere that he preferred using colimits instead of limits for modeling things (even though limits would also work) because they're computationally more tractable[1] or something (in the context of world modelling for the Safeguarded AI program).
Which is to say, it's not immediately obvious that a chain of reversible operations that composes up to a mind-process will also compose up to a mind-process if you invert it.
It would actually be a kind of fun situation if computation that is the inversion of a reversible computation that would normally instantiate a stream of experience, would "undo" those experiences (in the morally relevant sense or whatever other relevant sense).
- ^
Which makes sense on first glance because colimits generalize disjoint sums (which add cardinalities: ), whereas limits generalize cartesian products (which multiply cardinalities: ).
Off the top of my head, it feels like this might be somehow connected to the idea of logical zombies (I don't have a clear sense of what the connection is exactly; will think more about it later).
Potentially huge.
I think it's quite plausible that many politicians in many states are concerned with AI existential/catastrophic risk, but don't want to be the first ones to come out as crazy doomsayers. Some of them might not even allow the seeds of their concern to grow, because, like, "if those things really were that concerning, surely many people around me (and my particularly reasonable tribe in particular) would have voiced those concerns already".
Sure, we have politicians who say this, e.g., Brad Sherman in the US (apparently since at least 2007!) and, e.g., IABIED sent some ripples. But for many people to gut-level believe that this concern of theirs is important/good/legitimate to voice, they need clear social proof that "if I think/say this, I won't be a weird outlier", and for that, some sort of critical mass of expression of concern/belief/preference must be achieved in a relevant sort of population.
Canada's government, tackling those issues with apparent seriousness, has the potential to be that sort of critical mass.
Method: grow >10^8 neurons (or appropriate stem cells) in vitro, and then put them into a human brain.
There have been some experiments along these lines, at a smaller scale, aimed at treating brain damage.
The idea is simply to scale up the brain's computing wetware.
I know that much of the difference in ~intelligence across vertebrates is attributed to the pallial neuron count, and that neurogenesis in the dentate gyrus seems to be an important factor in (some forms of) adult learning (at least from mice experiments, etc.), but I'm curious whether there's any more direct/experimental evidence that adding more neurons to a generally healthy (rather than damaged) adult brain would increase cognitive performance along whatever dimension.
1why doesn't LW have this, BTW?
IIRC, Habryka commented somewhere that this is because posts often don't communicate a clear central proposition, whereas comments do so more often (or sth like that).
Yeah, one possible successor concept to the instrumental/terminal distinction might be something like "does this thing clearly draw its raison d'être from another thing, or is it its own source of raison d'être or some third thing which is like a nebulous, non-explicitized symbiosis", where the raison d'être is itself something potentially revisable by reflection (or whatever mind-shaping process).
AFAIK, to be included in the report, the written testimony[1] would have to be sent this week, ideally before Thursday.
They likely don't have more capacity for live participation.