All of Capybasilisk's Comments + Replies

O-risk, in deference to Orwell.

I do believe Huxley's Brave New World is a far more likely future dystopia than Orwell's. 1984 is too tied to its time of writing.

the project uses atomic weapons to do some of the engineering

Automatic non-starter.

Even if by some thermodynamic-tier miracle the Government permitted nuclear weapons for civilian use, I'd much rather they be used for Project Orion.

Isn't that what Eliezer referred to as opti-meh-zation?

Previously on Less Wrong:

Steve Byrnes wrote a couple of posts exploring this idea of AGI via self-supervised, predictive models minimizing loss over giant, human-generated datasets:

Self-Supervised Learning and AGI Safety

Self-supervised learning & manipulative predictions

I'd especially like to hear your thoughts on the above proposal of loss-minimizing a language model all the way to AGI.

I hope you won't mind me quoting your earlier self as I strongly agree with your previous take on the matter:

If you train GPT-3 on a bunch of medical textbooks and prompt it to tell you a cure for Alzheimer's, it won't tell you a cure, it will tell you what humans have said about curing Alzheimer's ... It would just tell you a plausible story about a situation related to the prompt about curing Alzheimer's, based on its training data. Ra

... (read more)
6Charlie Steiner3mo
Ah, the good old days post-GPT-2 when "GPT-3" was the future example :P I think back then I still thoroughly understimated how useful natural-language "simulation" of human reasoning would be. I agree with janus that we have plenty of information telling us that yes, you can ride this same training procedure to very general problem solving (though I think including more modalities, active leaning, etc. will be incorporated before anyone really pushes brute force "GPT-N go brrr" to the extreme). This is somewhat of a concern for alignment. I more or less stand by that comment you linked and its children; in particular, I said Simulating a reasoner who quickly finds a cure for Alzheimer's is not by default safe (even though simulating a human writing in their diary is safe). Optimization processes that quickly find cures for Alzheimer's are not humans, they must be doing some inhuman reasoning, and they're capable of having lots of clever ideas with tight coupling to the real world. I want to have confidence in the alignment properties of any powerful optimizers we unleash, and I imagine we can gain that confidence by knowing how they're constructed, and trying them out in toy problems while inspecting their inner workings, and having them ask humans for feedback about how they should weigh moral options, etc. These are all things it's hard to do for emergent simulands inside predictive simulators. I'm not saying it's impossible for things to go well, I'm about evenly split on how much I think this is actually harder, versus how much I think this is just a new paradigm for thinking about alignment that doesn't have much work in it yet.

Charlie's quote is an excellent description of an important crux/challenge of getting useful difficult intellectual work out of GPTs.

Despite this, I think it's possible in principle to train a GPT-like model to AGI or to solve problems at least as hard as humans can solve, for a combination of reasons:

  1. I think it's likely that GPTs implicitly perform search internally, to some extent, and will be able to perform more sophisticated search with scale.
  2. It seems possible that a sufficiently powerful GPT trained on a massive corpus of human (medical + other) k
... (read more)
I think talking of "loss minimizing" is conflating two different things here. Minimizing training loss is alignment of the model with the alignment target given by the training dataset. But the Alzheimer's example is not about that, it's about some sort of reflective equilibrium loss, harmony between the model and hypothetical queries [] it could in principle encounter but didn't on the trainings dataset. The latter is also a measure of robustness. Prompt-conditioned behaviors of a model (in particular, behaviors conditioned by presence of a word, or name of a character) could themselves be thought of as models, represented in the outer unconditioned model. These specialized models (trying to channel particular concepts [] ) are not necessarily adequately trained, especially if they specialize in phenomena that were not explored in the episodes of the training dataset. The implied loss for an individual concept (specialized prompt-conditioned model) compares the episodes generated in its scope by all the other concepts of the outer model, to the sensibilities of the concept. Reflection reduces this internal alignment loss by rectifying the episodes (bargaining [] with the other concepts), changing the concept to anticipate the episodes' persisting deformities, or by shifting the concept's scope to pay attention to different episodes. With enough reflection, a concept is only invoked in contexts to which it's robust, where its intuitive model-channeled guidance is coherent across the episodes of its reflectively settled scope, providing acausal coordination among these episodes in its role as an adjudicator [https://www.less

"Story of our species. Everyone knows it's coming, but not so soon."

-Ian Malcolm, Jurassic Park by Michael Crichton.

LaMDA hasn’t been around for long

Yes, in time as perceived by humans.

LaMDA (baring some major change since [] ) is a transformer model, and so only runs when being trained or being interacted with, so time would be measured in number of inputs the neural net saw. Each input would be a tick of the mental clock.

why has no one corporation taken over the entire economy/business-world

Anti-trust laws?

Without them, this could very well happen.

I've got uBlock Origin. The hover preview works in private/incognito mode, but not regular, even with uBlock turned off/uninstalled. For what it's worth, uBlock doesn't affect hover preview on Less Wrong, just Greater Wrong.

I'm positive issue is with Firefox, so I'll continue fiddling with the settings to see if anything helps.

There is an icon in the lower right that looks like this which toggles previews on or off. Do they come back if you click on it?

Preview on hover has stopped working for me. Has the feature been removed?

I'm on Firefox/Linux, and I use the Greater Wrong version of the site.

I still see it working on Greater Wrong. Do you have any extensions that might be blocking it?

It's also an interesting example of where consequentialist and Kantian ethics would diverge.

The consequentialist would argue that it's perfectly reasonable to lie (according to your understanding of reality) if it reduces the numbers of infants dying and suffering. Kant, as far as I understand, would argue that lying is unacceptable, even in such clear-cut circumstances.

Perhaps a Kantian would say that the consequentialist is actually increasing suffering by playing along with and encouraging a system of belief they know to be false. They may reduce infant... (read more)

I think we’ll encounter civilization-ending biological weapons well before we have to worry about superintelligent AGI:

My assumption is that, for people with ASD, modelling human minds that are as far from their own as possible is playing the game on hard-mode. Manage that, and modelling average humans becomes relatively simple.   

Interesting. Though I think extremes represent fewer degrees of freedom; where certain traits/characteristics dominate, and heuristics can better model behaviour. The "typical" person has all the different traits pushing/pulling, and so fewer variables you can ignore. i.e. the typical person might be more representative of hard-mode.

Williams Syndrome seems to me to just be the opposite of paranoia, rather than autism, where the individual creates a fictional account of another human's mental state that's positive rather than negative. 

That's to say, their ability to infer the mental states of other humans is worse than that of the typical human. 

That’s the problem with Kolmogorov complexity: it is the shortest program given unlimited compute. And it spends any amount of compute for a shorter program

I don't see why it's assumed that we'd necessarily be searching for the most concise models rather than, say, optimizing for CPU cycles or memory consumption. I'm thinking of something like Charles Bennett's Logical Depth.

These types of approaches also take it for granted that we're conducting an exhaustive search of model-space, which yes, is ludicrous. Of course we'd burn through our limited comput... (read more)

As I already pointed out, we already do. And turns out that you need to optimize more for CPU/memory, past the kilobytes of samples which are already flabby and unnecessary from the point of view of KC. And more. And more. Go right past 'megabyte' without even stopping. Still way too small, way too compute/memory-hungry. And a whole bunch more beyond that. And then you hit the Hutter Prize size, and that's still too optimized for sample-efficiency, and we need to keep going. Yes, blow through 'gigabyte', and then more, more, and some more - and eventually, a few orders of magnitude sample-inefficiency later, you begin to hit projects like GPT-3 which are finally getting somewhere, having traded off enough sample-inefficiency (hundreds of gigabytes) to bring the compute requirements down into the merely mortal realm. You can locate the Earth in relatively few bits of information. Off the top of my head: the observable universe [] is only 45 billion lightyears radius; how many bits could an index into that possibly take? 24 bits to encode distance from origin in lightyears out of 45b, maybe another 24 bits to encode angle? <50 bits for such a crude encoding, giving an upper bound. You need to locate the Earth in time as well? Another <20 bits or so to pin down which year out of ~4.5b years. If you can do KC at all, another <60 bits or so shouldn't be a big deal...

Going forward, I think discovery in the natural sciences will entirely be about automated searches in equation-space for models that fit datasets generated by real-world systems.

Why does one model work and not the other? Hopefully we'll know, most likely we won't. At any rate, the era of a human genius working these things out with pen and paper is pretty much over (Just consider the amount of combined intellectual power now needed to make incremental improvements. Major scientific papers these days will usually have a dozen+ names from several institution... (read more)

9Ben Pace10mo
Wow! Sounds like you should be able to exploit this knowledge for a lot of prestige and scientific discovery :)

“no, we swear there’s going to be a Higgs boson, we just need to build a more powerful particle accelerator”

Particle physicists also made other confident predictions about the LHC that are not working out, and they're now asking for a bigger accelerator.

Survivorship bias might be at play, wherein we forget all the confident pronouncements that ended being just plain wrong.

5Charlie Steiner10mo
I mean, the other main things to look for were WIMPs and supersymmetry, but almost everyone was cautious about chances of finding those. []

"How Much Are Games Like Factorio And EVE Online Sapping Away The Intellectual Potential Of Humanity?"

For example, as a consumer of Hollywood movies, you see an aged businessman in a slick suit and automatically associate him with “evil capitalists”, even if the man is Chuck Feeney or Elon Musk. The reason is simple: you have learned to have such associations, by consuming the fiction written by people of certain political views, from Simpsons to Star Trek to pretty much every movie that depicts businessmen.

Many people have had enough interaction with businessmen in slick suits to independently form negative associations. No fiction needed.

I thought the meme was that physicists think they can ride into town and make sweeping contributions with a mere glance at the problem, but reality doesn't pan out that way.

Relevant XKCD.

That is indeed a meme. Though if the physicists' attempts consistently failed, then biologists would not joke about physicists being like gunslingers.

Perhaps a followup study can investigate if trying to sneak culture war topics into ostensibly non-political spaces also maps to dark triad.

5Ben Smith1y
I know it's a touchy topic. In my defense, the research is solid, published in social psychology's top journal. I suppose the study deals with rhetoric in a political context. This community has a long history of drawing on social and cognitive psychological research to understand fallacies of thought and rhetoric (HPMOR), and I posted in that tradition. Apologies if I have strayed a little too far into a politicized area. One needn't see this study as a shot at any particular political side--I can imagine people engaging 'virtuous-victimhood-signalling' within a wide range of different politicized narratives, as well as in completely apolitical contexts. It also shouldn't be read to delegitimize victims from speaking out about their perspective! But perhaps it does provide evidence that sympathy can be weaponized in rhetorical conflict. We can all recognize this in political opponents and be blind to it amongst political allies.
Every society will have some non-zero number of topics associated with the “culture war” of the day. To cater to the stigma and ignore those topics would mean subjecting discussions and experimentation to the whims of the worst political actors on any given day. However, the callout is probably warranted. NOT discussing these to avoid culture war issues is the wrong solution, but pretending they’re not politically fraught is also incorrect. Doing so would likely lead to forgoing proper precautions for discussion of these topics, and result in worse outcomes. It does sometimes feel like the best conversations come out of topics that no one strongly identifies with in the moment.

I guess you could posit natural selection as being objective reality's value system, but I have the feeling that's not the kind of thing moral realists have in mind.

Indeed. A certain coronavirus has recently achieved remarkable gains in Darwinist terms, but this is not generally considered a moral triumph. Quite the opposite, as a dislike for disease is a near-universal human value. It is often tempting to use near-universal human values as a substitute for objective values, and sometimes it works. However, such values are not always internally consistent because humanity isn't. Values such as disease prevention came into conflict with other values such as prosperity during the pandemic, with some people supporting strict lockdowns and others supporting a return to business as usual. And there are words such as "justice" which refer to ostensibly near-universal human values except people don't always agree on what that value is or what it demands in any specific case.
Thank you. Also Sloman'sThe structure of the space of possible minds [] . All these are worthwhile articles, but I'd like to highlight Goertzel's discussion of kinds of embodiment.

From the point of view of the alien scientists, I'd argue that it's rational to ignore claims of unprecedented insight into reality unless the claimant can demonstrate impressive and unexplained feats using said insight.

I would think that gaining an entirely new sensory modality would lead to unparalleled advantages over fellow members of your species. At the very least, it would let you do things that would confound them.

New colors.

I don't mean discerning ever finer gradations in existing colors, but entirely new color qualia.

What would it be like? Would our brains be able to integrate this new phenomenal experience?

And most importantly, if color is a property of brains rather than something in the external world, does that imply the number of "possible colors" is infinite? I.e. seeing as biological brains "choose" how to internally represent a particular wavelength of EM radiation, is the seemingly Platonic realm from which these colors are plucked from inexhaustible?

We c... (read more)

Experiments with squirrel monkeys [] suggests that the monkeys can effectively learn new color qualia. It requires gene therapy that's a bit more risky then brewing your own vaccine together but it's technology that works. I don't see how that changes anything about that. If you use real numbers to model color you already have infinitive shades of colors. Adding the number of possible cone configurations in the eye adds a lot of additional one's but there's a physical limit of ways those can be configured.
Finer gradations are boring, but there are (1) infrared and ultraviolet colors, and (2) colors that our current system of three types of cone cells [] perceives as the same, but could be perceived as different by having more types of cones, or even different types of cones. We could use the new qualia for these.
Interview with David Pearce with the H+ magazine [], 2009 Or, in other words: I hear your "new colors" and raise you new qualia varieties that are as different from sight and taste as sight and taste are from each other.

The Arbital entry on Unforeseen Maximums [0] says:

"Juergen Schmidhuber of IDSIA, during the 2009 Singularity Summit, gave a talk proposing that the best and most moral utility function for an AI was the gain in compression of sensory data over time. Schmidhuber gave examples of valuable behaviors he thought this would motivate, like doing science and understanding the universe, or the construction of art and highly aesthetic objects.

Yudkowsky in Q&A suggested that this utility function would instead motivate the construction of external objects that wo... (read more)

Thanks for sharing this.

Would there be any advantages to substituting brute force search with metaheuristic algorithms like Ant Colony Optimization?

I do think that something could be done. I'm not very familiar with Ant Colony Optimization in particular, and it's not immediately obvious to me how it would be applied here. Specifically, how would it take advantage of the algorithmic complexity if that's all we know beforehand? Often, we're working with infinite parameter spaces where even reasonable subsets will grow exponentially large as we expand them. Is there something like a lazy version of Ant Colony Optimization which can be applied here? I do think some kind of randomized, learning search would be helpful. I mentioned genetic programming and I think stochastic supercompilation like searches (a la Stoke) would likely work as well.

Of possible interest, Roman Yampolskiy's paper, "The Universe Of Minds".

The paper attempts to describe the space of possible mind designs by first equating all minds to software. Next it proves some interesting properties of the mind design space such as infinitude of minds, size and representation complexity of minds. A survey of mind design taxonomies is followed by a proposal for a new field of investigation devoted to study of minds, intellectology. A list of open problems for this new field is presented.

Are random trading strategies more successful than technical ones?

In this paper we explore the specific role of randomness in financial markets, inspired by the beneficial role of noise in many physical systems and in previous applications to complex socioeconomic systems. After a short introduction, we study the performance of some of the most used trading strategies in predicting the dynamics of financial markets for different international stock exchange indexes, with the goal of comparing them with the performance of a completely random strategy. In

... (read more)

But a lot of that feeling depends on which animal's insides you're looking at.

A closely related mammal's internal structure is a lot more intuitive to us than, say, an oyster or a jellyfish.

4Ben Pace2y
Is that also proportional to its complexity? Like, I'm guessing it's harder to understand the internal structure of the things that don't actually have many features.

For example, take the idea that an AI should maximise “complexity”. This comes, I believe, from the fact that, in our current world, the category of “is complex” and “is valuable to humans” match up a lot.

The Arbital entry on Unforeseen Maximums elaborates on this:

Juergen Schmidhuber of IDSIA, during the 2009 Singularity Summit, gave a talk proposing that the best and most moral utility function for an AI was the gain in compression of sensory data over time. Schmidhuber gave examples of valuable behaviors he thought this would motivate, like doing sc

... (read more)

Robin Hanson posits that the reason why there isn’t wider adoption of prediction markets is because they are a threat to the authority of existing executives.

Before we reach for conspiracies, maybe we should investigate just how effective prediction markets actually are. I'm generally skeptical of arguments in the mold of "My pet project x isn't being implemented due to the influence of shadowy interest group y."

As someone unfamiliar with the field, are there any good studies on the effectiveness of PM?

There's nothing shadowy about the claim that CEO's like to be able to decide on the strategy of their company and don't like the idea of giving up that power by delegating it to a prediction market. To measure how effective it is for companies to let their strategy decision be guided by prediciton markets you would need some companies to do that. We don't live in a world where that's the case.
I'm not sure "conspiracy" is appropriate here. The existing Powers That Be (both political and corporate) have individual and collective interests in maintaining their current conditions. That they might each and all act to preserve the status quo (where they are powerful) probably does not actually require coordination of any kind, nor the secrecy that usually accompanies the term "conspiracy". I expect that no matter how effective prediction markets are, they will generally lack the necessary slack [] to dominate the existing systems.

This would just greatly increase the amount of credentialism in academia.

I.e., unless you're affiliated with some highly elite institution or renowned scholar, no one's even gonna look at your paper.

I agree this is a likely outcome, though I also think there's at least a 30% chance that the blackhats could find ways around it. Journals can't just lock it down to people the editors know personally without losing the large majority of their contributors.

Look on the bright side. If it turns out to be a disaster of Black Death proportions, the survivors will be in a much stronger bargaining position in a post-plague labour market.

The demand for products (and indirectly for labor to produce these products) will fall too, won't it?
I don't see why there would be more demand and less supply of labour after such a catastrophe. Why do you think that? (Or was it a joke? I cannot tell.) I would rather say that the bright side is that a societal collapse brought on by a pandemic might at least delay the climate change collapse a bit.

Consider the trilobites. If there had been a trilobite-Friendly AI using CEV, invincible articulated shells would comb carpets of wet muck with the highest nutrient density possible within the laws of physics, across worlds orbiting every star in the sky. If there had been a trilobite-engineered AI going by 100% satisfaction of all historical trilobites, then trilobites would live long, healthy lives in a safe environment of adequate size, and the cambrian explosion (or something like it) would have proceeded without them.

... (read more)

Altman's responsiveness test might be more useful if it measured a founder's reply time to an underling rather than to the guy holding the purse strings/ contact book.

An alternative hypothesis is that Graham has a bunch of tests/metrics that he uses to evaluate people on, and those tests/metrics work much better when people do not try to optimize for doing well on them

Isn't it a bit odd that PG's secret filters have the exact same output as those of staid, old, non-disruptive industrialists?

I.e., strongly optimizing for passing the tests of <Whitebread Funding Group> doesn't seem to hurt you on YC's metrics.

I'm not sure whether that's true. One way to optimize for the tests of <Whitebread Funding Group> might be to go to a convention where you can get a change to pitch yourself in person to them. There are also other ways you can spend a lot of time to someone to endorse you to <Whitebread Funding Group>. Spending that means that's time not spent getting clear about vision, building product or talking to users and that will be negative for applying to VC.

Relatedly, is there any work being done to come up with distance measures for agent states (including any of momentary experiences, life-sums, or values)? We talk a lot about marginal and absolute value of lives and make comparisons of experiences (motes vs torture) - what, if anything, has been done to start quantifying?

It turns out that measuring the distance between possible worlds is...complicated.

What about when the poor men come barging through the rich man’s gate? I take it that too is factored into the divine plan?

Only insofar as even man's sinning is part of the divine plan; but though part of the divine plan, it is still sin. The social order is the divine order, each is born into the position that God has ordained, and the King rules by the grace of God. So it has been believed in some former times and places. There is still a trace of that in our (British) coins, which have a Latin inscription meaning "[name of monarch] by the grace of God King/Queen, defender of the faith."

And what interesting insights into reality does it give us?

Risky question to ask when dealing with pure mathematics

I don’t think people should use this site to promote their personal blogs. Sure, you can add a link to your blog at the bottom of your post, but this teaser excerpt BS is really irritating. I don’t click through just out of principle.

If you have something to say, post the whole thing here. If I like what you have to say, I might check out your other stuff, but I’m not going to be forced into it.

Fwiw, while I much prefer full cross-posts, in practice we can't actually force people to do stuff. Insofar as people are making the choice between "not post on LW, while posting elsewhere" and "link post on LW" I still prefer the latter. But note that we generally don't curate (or at least have a higher bar for curating) posts that are not fully crossposted, so there's at least some incentive to post the full text on LW.
4Said Achmiz4y
I agree with this sentiment. I am in favor of link-posts that are also cross-posts; this is the best of both worlds: readers don’t have to leave Less Wrong to read the post (and thus get to take advantage of LW’s or GW’s far superior features to those of most blog software), and there’s a clear link to the originating blog if a reader wants to check out more of the author’s stuff. However, there is a caveat: sometimes copying over a post is quite impractical, at best. (Such was the case with one of my posts [] .)
I do also think link-posts are fine. I personally prefer short-excerpts over no excerpt, but a lot of people seem to be annoyed by it, so I would generally recommend copying over the whole thing (and I am happy to set up crossposting for people so that that happens automatically when you apply a tag on your blog).