I agree that I did not specify full, working implementations of "far coordination". There are details that I did not fill in to avoid prematurely reaching for rigour.
The kind of coordination I imagined is somewhat limited.
I guess this is an idea I may revisit and develop further sometime later. I do think there's something sensible/useful here, but maybe my exploration of it wasn't useful.
I had spoken with people that expected our descendants to diverge from us in ways we'd consider especially heinous, that were concerned about astronomical suffering and was persuaded by Hanson's argument that a desire to maintain civilisational unity may prevent expansion.
So I was in that frame of mind/responding to those arguments when I wrote this.
I'm sympathetic to this reasoning. But I don't know if it'll prevail. I'd rather we lock in some meta values and expand to the stars than not expand at all.
Well, I was arguing that we should discount in proportion to our uncertainty. And it seems you're pretty confident that it would kill future people (and more people, and we don't expect people in 10K years to want to die anymore than people in 100 years), so I think I would prefer to plant the 100 years landmine.
That said, expectations of technological progress, that the future would be more resilient, that they can better deal with the landmine, etc. means that in practice (outside the thought experiment), I'll probably plant the 10K landmine as I expect less people to actually die.
I don't buy this argument for a few reasons:
SBF didn't latch onto EA after he acquired some measure of power or when EA was a force to be reckoned with, but pretty early on. He was in a sense "homegrown" within EA.
The "SBF was a sociopath using EA to launder his reputation" is just motivated credulity IMO. There is little evidence in favour of it. It's just something that sounds good to be true and absolves us of responsibility.
Astrid's hypothesis is very uncredible when you consider that she doesn't seem to be aware of SBF's history within EA. Like what's the angle here? There's nothing suggesting SBF planned to enter finance as a college student before MacAskill sold him on earning to give.
Thanks for the answer!
Out of curiosity, what career did you transition to?
I think you should focus on who you want as a supervisor, rather than whether it will be in mathematics or in computer science...If you find the right supervisor, it is not important if they are in CS or maths or something else.
I think you should focus on who you want as a supervisor, rather than whether it will be in mathematics or in computer science
If you find the right supervisor, it is not important if they are in CS or maths or something else.
I'll be keeping this in mind.
That said computational/biological anchors are a good reason for longer timelines absent foundational breakthroughs in our understanding of intelligence.
Furthermore, I suspect that intelligence is hard, that incremental progress will become harder as systems become more capable, that returns to cumulative investment in cognitive capabilities are sublinear/marginal returns to cognitive investment decay at a superlinear rate, etc.)
I'd like to present a useful formalism for describing when a set is "self-similar".
Given arbitrary sets X,Y, an "equivalence-isomorphism" is a tuple (f:X→Y,f−1:Y→X,∼R⊂X∪Y×X∪Y), such that:
For a given equivalence relation ∼R, if there exist functions such that an equivalence-isomorphism can be constructed, then we say that the two sets are "isomorphic under ∼R".
The concept of "isomorphism under an equivalence relation" is meant to give us a more powerful mechanism for describing similarity/resemblance between two sets than ordinary isomorphisms afford.
Two sets are "similar" if they are isomorphic to each other under a suitable equivalence relation.
A set is "self-similar" if it's "similar" to a proper subset of itself.
This is deliberately quite bare, but I think it's nonetheless comprehensive enough (any notion of similarity we desire can be encapsulated in our choice of equivalence relation) and unambiguous given an explicit specification of the relevant equivalence relation.
I stuck to sets because I don't know any other mathematical abstractions well enough to play around with them in interesting ways.
Ordinarily, two sets are isomorphic to each other if a bijection exists between them (they have the same cardinality). This may be too liberal/permissive for "similarity".
By choosing a sufficiently restrictive equivalence relation (e.g., equality), we can be as strict as we wish.
Whatever notion of similarity we desire is encapsulated in our choice of equivalence relation.
For an AI that had no need to communicate with other agents, then the idea of truth serves as a succinct term for the map-territory/belief-reality correspondence.
It allows the AI to be more economical/efficient in how it stores information about its maps.
That's some value.
Saying that a proposition is true, is saying that it's an accurate description of the territory.
"The sentence 'X' is true iff X."
The territory may be physical reality ("'the sky is blue' is true"), a formal system ("'2 + 2 = 4' is true"), other maps, etc.
Even if truth judgments can only be made by comparing maps — even if we can never assess the territory directly — there is still a question of how the territory is.
Furthermore, there is value in distinguishing our model/expectations of the world, from our experiences within it.
This leads to two naive notions of truth: