Why work your way up at all? The lower you can keep your tolerance, the better, I'd guess?
I don't intend on ever switching away from my sencha/japanese green tea.
Given this as a foundation, I wonder if it'd be possible to make systems that report potentially dangerously high concentrations of compute, places where an abnormally large amount of hardware is running abnormally hot, in an abnormally densely connected network (where members are communicating with very low latency, suggesting that they're all in the same datacenter).
Could it be argued that potentially dangerous ML projects will usually have that characteristic, and that ordinary distributed computations (EG, multiplayer gaming) will not? If so, a system like this could expose unregistered ML projects without imposing any loss of privacy on ordinary users.
the less readable your posts become because the brain must make a decision with each link whether to click it for more information or keep reading. After several of these links, your brain starts to take on more cognitive load
I don't think it's reasonable to try to avoid the cognitive load of deciding whether to investigate subclaims or follow up on interesting ledes while reading. I think it's a crucial impulse for critical thinking and research and we have to have it well in hand.
Wondering if radical transparency about (approximate) wealth + legalizing discriminatory pricing would sort of steadily, organically reduce inequality to the extent that would satisfy anyone.
Price discrimination is already all over the place, people just end up doing it in crappy ways, often by artificially crippling the cheaper versions of their products. If they were allowed to just see and use estimates of each customer's wealth or interests, the incentives to cripple cheap versions would become negative, perhaps more people would get the complete featu... (read more)
Since everything can fit into the "agent with utility function" model given a sufficiently crumpled utility function, I guess I'd define "is an agent" as "goal-directed planning is useful for explaining a large enough part of its behavior." This includes humans while discluding bacteria. (Hmm unless, like me, one knows so little about bacteria that it's better to just model them as weak agents. Puzzling.)
Most of what people call morality is conflict mediation: techniques for taking the conflicting desires of various parties and producing better outcomes for them than war.
That's how I've always thought of the alignment problem. The creation of a very very good compromise that almost all of humanity will enjoy.
There's no obvious best solution to value aggregation/cooperative bargaining, but there are a couple of approaches that're obviously better than just having an arms race, rushing the work, and producing something awful that's nowhere near the average human preference.
Agreed. Humans are constantly optimizing a reward function, but it sort of 'changes' from moment to moment in a near-focal way, so it often looks irrational or self-defeating, but once you know what the reward function is, the goal-directedness is easy to see too.
Sune seems to think that humans are more intelligent than they are goal-directed, I'm not sure this is true, human truthseeking processes seems about as flawed and limited as their goal-pursuit. Maybe you can argue that humans are not generally intelligent or rational, but I don't think you can ju... (read more)
Do not use FAIR as a symbol of villainy. They're a group of real, smart, well-meaning people who we need to be capable of reaching, and who still have some lines of respect connecting them to the alignment community. Don't break them.
Seems useless if the first system pretends convincingly to be aligned (which I think is going to be the norm) so you never end up deploying the second system?
And "defeat the first AGI" seems almost as difficult to formalize correctly as alignment, to me:
sort of incoherent and not definable in the general case
Why? Solomonoff inducting, producing an estimate of the measure of my existence (the rate of the occurrence of the experience I'm currently having) across all possible universe-generators weighted inversely to their complexity seems totally coherent to me. (It's about 0.1^10^10^10^10)
infra-Bayesianism would (I think) tell you to act as if you're the brain whose future you believe to have the lowest expected utility
I haven't listened to that one yet, but ... wasn't it a bit hard to swallow as a decisio... (read more)
I don't really disagree with any of that, but yeah I think you might be missing the issue of curation, which is kind of most of the work that journals do.
A revolutionary publication, before being widely recognized, will look just like an error. Most of the time, it will turn out to be an error. Fully evaluating it takes time and energy, and if no one is paying reviewers to do that, it's generally totally unrewarding work and no one will do it and the diamonds in the rough wont be made discoverable.
If you understand why we must reward the boring work of rep... (read more)
I'm generally a fan of pursuing this sort of moral realism of the ideals, but I want to point out one very hazardous amoral hole in the world that I don't think it will ever be able to bridge over for us, lest anyone assume otherwise, and fall into the hole by being lax and building unaligned AGI because they think it will be kinder than it will.
(I don't say this lightly: Confidently assuming kindness that we wont get as a result of overextended faith in moral realism, and thus taking on catastrophically bad alignment strategies, is a pattern I see shockin... (read more)
In situations like that, I'd say, more.. you should process it with reduced energy, in correct proportion. I wouldn't say you should completely deafen yourself to anyone (unless it's literally a misaligned AIXI).
I think even this slackened phrasing is not applicable to the current situation, because the people I'm primarily listening to are mostly just ordinary navy staff who are pretty clearly not wired up to any grand disinformation apparatus about UAP.
and we are either completely left alone or have been put in a simulation, in which case occasional UFO sightings don't seem like an optimal feature of the outcome.
Agreed. A way of using our matter (the earth) for something else, without killing us.
So I've been thinking about that. For any simulator, there are things they do and don't care about capturing accurately in the simulation. I'd guess that the simulation has a lot to do with whether we hold to the reciprocal kind-colonization pacts that they're committed to themselves. For that, it's important tha... (read more)
Is there writing about that? Last time I thought deeply about reversible computing, it didn't seem like it was going to be useful for really anything that we care about.
I'll put it this way.. if you look at almost any subroutine in a real program, it consists of taking a large set of inputs and reducing them to a smaller output. In a reversible computer, iirc, the outputs have to be as big as the inputs, informationally (yeah that sounds about right). So you have to be throwing out a whole lot of useless outputs to keep the info balanced, that's what you h... (read more)
Even if stars only make up a small fraction of the matter in the universe, it's still matter, they'd still probably have something they'd prefer to do with it than this. I'm not really sure what kind of value system (that's also power-seeking enough to exert control over a broad chunk of the universe) could justify leaving it fallow.
I will politely decline to undergo epistemic learned helplessness as it seems transparently antithetical to the project of epistemic rationality
Even if it were true, how would they know it was a propulsion technology?
Uh, because there seemed to be a solid object (showed up in a kind of radar that we don't know how to spoof) that was moving around really fast in line with the visual. As stated, I still think it might not be a propulsion technology, but the witnesses don't tend to float any other possibility. I haven't seen them asked about the plasma image theory.
I wouldn't say I think that it's an alcubierre drive specifically, what I mean is I don't know what else to liken it to and it woul... (read more)
It's squarely relevant to the post, but it is mostly irrelevant to Eliezer's comment specifically, and I think the actual drives underlying the decision to make it a reply to Eliezer are probably not in good faith, like, you have to at least entertain the hypothesis that they pretty much realized it wasn't relevant and they just wanted eliezer's attention or they wanted the prominence of being a reply to his comment.
Personally I hope they receive eliezer's attention, but piggybacking messes up the reply structure and makes it harder to navigate discussions... (read more)
This is an inappropriate place to put this.
Compute is physically simpler than life. Where there is life, there is necessarily also compute. Where there is compute, there isn't necessarily also life.
Good, and cheap, is the thing. If we didn't have silicon computing, we would still have vacuum tubes, we'd still have computers. But as I understand it, vacuum tubes sucked, so I wouldn't expect that that machine learning would be moving so quickly at this point.
... (read more)If that were the case, there'd be more measure in the next year than in the next second, but you don't suddenly find yourself a year from now. (
Private information is evil. (Though I'm still on the fence as to whether it's a necessary evil to avoid world-sized preference falsification cascades.)
Clippy is not ideal, but better than humanity.
There's a weird genre of paranoia where people worry that the thing we value will turn out to be something we disvalue. But I guess you mean it's a case where the values of the average LWer disagree sharply from the values of the globe, right. (I don't see that, personally.)
I'm bullish on radical transparency at this point. Whoever is the most unrelentingly brash will seize the next moral aesthetics cycle.
Regarding moving beyond blame minimization, I think it's worth mentioning my Venture Granters, a system for protecting sane risk-takers in public funding institutions: https://www.lesswrong.com/posts/NY9nfKQwejaghEExh/venture-granters-the-vcs-of-public-goods-incentivizing-good
Research that makes the case for AGI x-risk clearer
I ended up going into detail on this, in the process of making an entry to the FLI's aspirational worldbuilding contest. So, it'll be posted in full about a month from now. But for now, I'll summarize:
Rationalists should be deeply interested in the Princeton-Nimitz encounters, regardless of whether it was confusion, aliens, or a secret human technology, because cases of confusion on this level teach us a lot about how epistemic networks operate, and if it were aliens or a secret human technology that would be strategically significant.
So, since those were pretty much the only possibilities, I was deeply interested.
I eventually settled loosely into the theory that the tictacs were probably a test of a long-range plasma volumetric display decoy/spoofing t... (read more)
The princeton-nimitz reports are unambiguously worth the oxygen it takes to contemplate them, given the consistency of the reports and the ramifications it would have even if it was "just" a human technology. So if you had the virtue of curiosity, you would contemplate it, and you would get led down the path that ends with the resolution that the "lie", "mistake", or "human technology" theories don't really make deep sense either, and a rationalist does indeed have to start considering the other theory, that some aliens end up being much stranger than we w... (read more)
Doesn't land a hit on the story as it's always been told: They're piloted by intelligent beings and that they only want to be seen occasionally. They'd notice that we're all carrying cameras and deliberately appear less frequently (while still acting aloof).
Btw, I'm open to the possibility that the answer is "yes, but it will accelerate alignment techniques more than capabilities, so it's still good to do."
(Note, though, not all acceleration of deployment is bad. Imagine that we manage to secure against the period of peril, where fully general capabilities have been pretty much found but aren't being deployed because the discoverers are too responsible to do it without a convincing alignment solution. That's a case where alignment work itself accelerates the deployment of AGI, but the acceleration is purely good.)
Ah, yeah link previews are good. I guess the problem with LW's ones that they're difficult to find out about on mobile, the user has to figure out to click and hold, then close the browser popup. I prefer gwern's way, where clicking a link on mobile will only open the preview, and you have to click again to traverse the link. Others have complained about that, though.
This post is relevant, and has more to say about the benefits of neighbors in approaching lightspeed travel https://www.lesswrong.com/posts/DWHkxqX4t79aThDkg/my-current-thoughts-on-the-risks-from-seti#Alien_expansion_and_contact
Apparently there's an armstrong - sandberg paper that found that getting 99% of lightspeed is totally feasible with coil guns. So the benefits are mild.
I notice that pay transparency seems to be a key subproblem here. If we just knew how salary was distributed in these organizations, then we preeety much know how power is distributed. It would simplify the auditing pretty drastically.
There are pros and cons to pay transparency (I'm mostly pro, but I do fear that envy is a bigger problem in the US than in Scandanavian countries where transparency is working well). But I'm not sure that's the key subproblem.
I'd expect it's cultural devaluation of women that's the key subproblem. Even where women aren't a small minority, there's an amazing double-standard about appearance, presentation, and discussion style throughout most US and UK (and I presume elsewhere, but I know less about that) businesses.
Will that cause more harmful project to succeed?
In reality I'm not sure the trap would remain effective for long enough for too many of those to start turning up. Humans aren't rigidly CDTish. They'll catch on. Perhaps many professional traders already have some principle against playing games of collective chicken.
I guess a good question here is... is the opportunity cost of assessing the failure rate, to the level of accuracy where the risk of project success is low enough that you can be sure that you'll get your refund bonus, actually lower than the re... (read more)
however there are professions that are more heavily concentrated by females and I would imagine that their dealings with sexism are more minimal.
Would it help at all to promote information about which finance firms are closest to having gender parity (cut through their PR), so that women who would strongly prefer not to be an extreme minority know which firms to give preference to, and apply to first?
For posterity, the original title was
Alex Tabarrok proposed improving crowdfunding mechanisms with Refund Bonuses. I think this might be a natural occurrence of a dutch book against Causal Decision Theory
I also removed these sections which I kind of left in by accident and had already decided at the time of posting that I couldn't really stand behind. Sorry about those.
Could be true, but I think I was probably understating the value of the credible signal that is sent by having refund bonuses, even for a LDT agent.
... (read more)As a Logical Decision Theory (LDT) s
I see how that can be misleading. I'll try to clarify it. The reason it ended up looking like that was that "kickstarter with refund bonuses added" is, as he acknowledges, a really good way of describing it, even though it was not a product of taking kickstarter and adding refund bonuses.
Mm, to add context, you're mentioning this because it's a very anti-inductive market, yes? And yet people keep participating. So why wouldn't they keep participating in the refund bonus extraction game of chicken.
What's do you think is wrong? I don't see any contradictions here.
Are you confused about who I'm saying is getting dutch booked? I'm saying pledgers dutchbook themselves, the project will be more than fine, it would be extremely good for the project. It seems like a very good mechanism from the project's perspective, and I approve of it.
I'd love to participate, but I'm a mostly single male without US residency (just NZ/Au/UK), which I realize is unlikely to be a bottlenecked resource, so I'll un-vote my comment here heh
For travel through neighboring grabby civs, mm, I guess you'd want to get to know them first. Are there ways they could prove that they're a certain kind of civ, with a certain trusted computing model, that lets them prove that they wont leak you?
For travel through neighboring primitive civs in the vulnerable stage... Maybe you'd send a warrior emissary who doesn't attribute negative utility to any of its own states of mind. If it's successful... Hmm... it establishes an encryption protocol with home, and only then do you start sending softer minds.
But tha... (read more)
It would be worth writing, yeah. It would be an update for me.
P(any civilization in its early computing stage will run any code that is sent to them) ≈ 1 for me, not sure about the other terms. Transmission would also require that a civilization within the broadcast radius enters its computer age, and notices the message, before they mature and stop being vulnerable to being hacked, all before that region of space is colonized by a grabby civ (Oh, note, though, this model of spread, if it is practical, we might be able to assume that grabby civs can't othe... (read more)
What are some of those components? We can put them on a list.
By the way, "myopic" means "pathologically short-term".
I like it. Less thoroughly descriptive, but it might generalize to more cases.
you are trying to build an incentive structure that will accelerate the development of AGI.
No, I'm not sure how you got that impression (was it "failing to coordinate"?), I'm asking for the opposite reason.
Okay, no, the Teilhardian laser-as-nanomanufacturer idea is probably not workable. I read an extremely basic article about laser attenuation and, bad news: lasers attenuate.
The best a laser could do to any of the planets about the nearest star seems to be making a pulse of somewhat bright light visible to all of them.
I still wonder about sending packets of resilient self-organizing material that could survive a landing, though.
On the other hand, even if we went extinct, the universe wouldn’t remain empty since some other civilization would be there to take our place.
Yes, but most of my existential risk comes from AGI Misalignment, which would not follow this law, because a Misaligned AGI is likely to spread up and fill our volume and be as immovable to alien civs as we would have been.
Moving quickly can allow humanity to gather a larger fraction of the universe for itself.
The incentives to move quickly were actually a lot greater before grabby aliens, due to accelerating cosmolo... (read more)
I guess they wouldn't need a firmament if they were doing a thing where.. they just let life-supporting planets see them until intelligent life emerges, because unintelligent life would be indifferent to them, and then once intelligent life starts to build telescopes they descend and scan everyone and move it into a simulation. This would get them a completely accurate biological history. The simulation, from then on, might not be completely accurate, but if so, I am not sensitive to what would be missing from it.
And you haven't been able to reset your tolerance with a break? Or would it not be worth it? (I can't provide any details about what the benefits would be sry)