This is a special post for quick takes by Nate Showell. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I've come to believe (~65%) that Twitter is anti-informative: that it makes its users' predictive calibration worse on average. On Manifold, I frequently adopt a strategy of betting against Twitter hype (e.g., on the LK-99 market), and this strategy has been profitable for me.
I was comparing it to base-rate forecasting. Twitter leads people to over-update on evidence that isn't actually very strong, making their predictions worse by moving their probabilities too far from the base rates.
Metroid Prime would work well as a difficult video-game-based test for AI generality.
- It has a mixture of puzzles, exploration, and action.
- It takes place in a 3D environment.
- It frequently involves backtracking across large portions of the map, so it requires planning ahead.
- There are various pieces of text you come across during the game. Some of them are descriptions of enemies' weaknesses or clues on how to solve puzzles, but most of them are flavor text with no mechanical significance.
- The player occasionally unlocks new abilities they have to learn how to use.
- It requires the player to manage resources (health, missiles, power bombs)
- It's on the difficult side for human players, but not to an extreme level.
There are no current AI systems that are anywhere close to being able to autonomously complete Metroid Prime. Such a system would probably have to be at or near the point where it could automate large portions of human labor.
I find myself betting "no" on Manifold a lot more than I bet "yes," and it's tended to be a profitable strategy. It's common for questions on Manifold to have the form "Will [sensational event] happen by [date]." These markets have a systematic tendency to be too high. I'm not sure how much of this bias is due to Manifold users overestimating the probabilities of sensational, low-probability events, and how much of it is an artifact of markets being initialized at 50%.
Is trade ever fully causal? Ordinary trade can be modeled as acausal trade with the "no communication" condition relaxed. Even in a scenario as seemingly causal as using a vending machine, trade only occurs if the buyer believes that the vending machine will actually dispense its goods and not just take the buyer's money. Similarly, the vending machine owner's decision to set up the machine was informed by predictions about whether or not people would buy from it. The only kind of trade that seems like it might be fully causal is a self-executing contract that's tied to an external trigger, and for which both parties have seen the source code and verified that the other party have enough resources to make the agreed-upon trade. Would a contract like that still have some acausal element anyway?
Physical causality is naturally occuring acausal dependence (between physically interacting things), similarly to how a physical calculator captures something about abstract arithmetic. So the word "acausal" is unfortunate, it's a more general thing that shouldn't be defined by exclusion of the less general special case of immense practical importance, acausal dependence is something like logical/computational causality. And acausal trade is trade that happens in situations within the fabric of acausal dependencies, how an agent existing within acausal ontology might think about regular trade. But since a clearer formulation remains elusive, fixing the terminology seems premature.
Bug report: when I'm writing an in-line comment on a quoted block of a post, and then select text within my comment to add formatting, the formatting menu is displayed underneath the box where I'm writing the comment. For example, this prevents me from inserting links into in-line comments.
An edgy writing style is an epistemic red flag. A writing style designed to provoke a strong, usually negative, emotional response from the reader can be used to disguise the thinness of the substance behind the author's arguments. Instead of carefully considering and evaluating the author's arguments, the reader gets distracted by the disruption to their emotional state and reacts to the text in a way that more closely resembles a trauma response, with all the negative effects on their reasoning capabilities that such a response entails. Some examples of authors who do this: Friedrich Nietzsche, Grant Morrison, and The Last Psychiatrist.
Allow me to quote from Lem’s novel “Golem XIV”, which is about a superhuman AI named Golem:
Being devoid of the affective centers fundamentally characteristic of man, and therefore having no proper emotional life, Golem is incapable of displaying feelings spontaneously. It can, to be sure, imitate any emotional states it chooses— not for the sake of histrionics but, as it says itself, because simulations of feelings facilitate the formation of utterances that are understood with maximum accuracy, Golem uses this device, putting it on an "anthropocentric level," as it were, to make the best contact with us.
May not this method also be employed by human writers?
It's a natural tendency to taunting, which is meant to motivate the reader to attack the author, who is frustrated at the lack of engagement. The more sure you are of yourself, the more provocative you tend to be, especially if you're eager to put your ideas to the test.
A thing which often follows edginess/confidence, and the two may even be a cause of eachother, is mania. Even hypomanic moods has a strong effect on ones behaviour. I believe this is what happened to Kanye West. If you read Nietzsche's Zarathustra, you might find that it seems to contain a lot of mood-swings, and it was written in just 10 days as far as I know (and periods of high productivity are indeed a characteristic of mania)
I think it makes for great reading, and while such people have a higher risk of being wrong, I also think they have more interesting ideas. But I will admit that I'm a little biased on this topic as I've made myself a little edgy (confidence has a positive effect on mood)
What do other people here think of quantum Bayesianism as an interpretation of quantum mechanics? I've only just started reading about it, but it seems promising to me. It lets you treat probabilities in quantum mechanics and probabilities in Bayesian statistics as having the same ontological status: both are properties of beliefs, whereas in some other interpretations of quantum mechanics, probabilities are properties of an external system. This match allows quantum mechanics and Bayesian statistics to be unified into one overarching approach, without requiring you to postulate additional entities like unobserved Everett branches.
My probability that quantum Bayesianism is onto something is .05. It went down a lot when I read Sean Carroll's book Something Deeply Hidden. .05 is about as extreme as my probabilities get for the parts of quantum physics that are not settled science since I'm not an expert.
Could you summarize what Carroll says that made you update so strongly against it?
My memory is not that good. I do recall that it is in the chapter "Other ways: alternatives to many-worlds".
Simulacrum level 4 is more honest than level 3. Someone who speaks at level 4 explicitly asks himself "what statement will win me social approval?" Someone who speaks at level 3 asks herself the same question, but hides from herself the fact that she asked it.
Simulacra levels aren't a particularly good model for some interactions/topics, because they blend together in idiosyncratic ways. It's unspecified in the model whether it's intentional or not, or whether a speaker is hiding anything from themselves, vs cynically understanding the levels and using the one that they think is most effective at the time.
I don't think so. simulacrum 4 has trouble making coherent reference to physical causal trajectory. simulacrum 3 and 1 are compatible in fact, in some circumstances. not so with 4
The orthogonality question is an engineering question
People usually think about the orthogonality question ("Is the orthogonality thesis true?") as a philosophical question. The usual way of approaching the orthogonality question is by taking a starting point of "assume an AGI exists" and then reasoning about what goals the AGI would have. But one can flip the usual starting point around and ask, for a specific goal, "is it realistically achievable to create a general intelligence that has this goal?" This reframing turns the orthogonality question into an engineering question that has more direct practical relevance than the philosophical version. The engineering version is a question about the types of results an AI developer can expect from different engineering decisions, rather than speculation about an idealized AGI; it's grounded in what's realistically achievable instead of what might be theoretically possible.
Instances of the engineering version of the orthogonality question also open the broader orthogonality question up to empirical testing. And so far, the empirical evidence we've received has pointed toward the answer "no." Ever since the early days of reinforcement learning, researchers have been creating models with narrow goals, and so far, none of those systems has shown full generalization in the type of intelligence it's developed. Protein-folding models only fold proteins; chess engines don't model their environments outside the confines of the 64 squares. Language prediction has generalized further than most other training objectives, but language models still perform poorly at non-linguistic tasks (understanding images, acting within physical environments) and have jagged capabilities even within the set of language-based problems. Each new failure to create general intelligence from a narrow training objective is (usually weak) empirical evidence that narrow training signals are too impoverished to let a model develop highly general capabilities. Maybe general intelligence from a narrow goal would be possible with truly gargantuan amounts of compute, but recall that the engineering version of the orthogonality question is about what's practically achievable.
I'm done with LessWrong. I'm leaving. Seeing one of the administrators advocate violence against AI lab employees, and seeing the userbase upvote and agree-vote his comment on net, has made me no longer want to spend time on the site or be associated with it. That comment is a particularly strong example of a pattern I've been observing on LessWrong recently, and it's tipped me over into deciding to stop visiting the site.
In the past couple years I've noticed a trend of increasing dogmatism on LessWrong, where some users have become intolerant of disagreement with the cluster of beliefs according to which AI extinction is highly likely, timelines are short, and pause/stop advocacy is necessary to survive. In the past couple months this trend has gotten noticeably worse. It's turned into a purity spiral that's now starting to turn against prosaic AI alignment researchers: I've noticed a more posts and comments lately that discuss, often in with-us-or-against-us social terms, whether prosaic alignment research and the people who do it are acceptable. From the vote counts on most of those posts, the "no" side appears to be winning.
Some of you are probably thinking something along the lines of, "But AI really will kill us all, and soon. Isn't radicalism justified in circumstances like these?" If something like that comes to mind for you, I'd like to request that you take a moment to consider the perspective of someone with a significantly different worldview. How would you sound to them? Set aside, for a moment, your cached objection to the term "outside view" and try taking it. Learn from history what groups tend to do when their members have orthodoxic commitments to beliefs in apocalyptic stakes.
(And how did that objection to "outside view" end up in your cache in the first place? What purpose might the objection serve, to you... or someone else?)
I've gotten practical value out of LessWrong in the past. After leaving, I'll end up missing some posts I would've liked to read. LessWrong used to succeed as a place for people to improve their rationality. But now I've come to think that it's having the opposite effect now that people here are using arguments as soldiers so frequently. Most of the times I visit LessWrong nowadays it leaves my mind unsettled and my thinking less clear. I feel the need to defend, to myself, my disagreements with the emerging doctrines and feel as though I'm about to be exposed as an impostor. My local rationalist and EA group doesn't have the same problems I mentioned, and I'll continue to engage with it. But continuing to use LessWrong has become counterproductive to me. To improve my truth-seeking abilities and learn to act more wisely, I've decided to leave this site behind.
I'm sorry it turned out this way : / I know what you mean about the shifting vibe - I am in it.
Given the huge number of clarifications Habryka issued later, he did not mean to compare lab employees to Nazis to be hung. He did want serious, large-scale, unprecedented legal inquiries that take AI risk into account. I suppose you probably see that as a motte, though. The upvoters won't be able to offer an explanation.
I wish I could say something better. Often I wonder if I'm in a bad bubble, or if everything I believe is wrong. I constantly worry about maintaining the good levels of chill, because I want to be able to talk to everyone. I'm glad there in person stuff is there for you. I hope everything goes well for you.