I guess orgs need to be more careful about who they hire as forecasting/evals researchers.

Sometimes things happen, but three people at the same org...

This is also a massive burning of the commons. It is valuable for forecasting/evals orgs to be able to hire people with a diversity of viewpoints in order to counter bias. It is valuable for folks to be able to share information freely with folks at such orgs without having to worry about them going off and doing something like this.

But this only works if those less worried about AI risks who join such a collaboration don't use the knowledge they gain to cash in on the AI boom in an acceleratory way. Doing so undermines the very point of such a project, namely, to try to make AI go well. Doing so is incredibly damaging to trust within the community.

Now let's suppose you're an x-risk funder considering whether to fund their previous org. This org does really high-quality work, but the argument for them being net-positive is now significantly weaker. This is quite likely to make finding future funding harder for them.

This is less about attacking those three folks and more just noting that we need to strive to avoid situations where things like this happen in the first place. This requires us to be more careful in terms of who gets hired.

Hegel - A Very Short Introduction by Peter Singer - Book Review Part 1: Freedom

Hegel is a philosopher who is notorious for being incomprehensible. In fact, for one of his books he signed a contract that assigned a massive financial penalty for missing the publishing deadline, so the book ended up being a little rushed. While there was a time when he was dominant in German philosophy, he now seems to be held in relatively poor regard and his main importance is seen to be historical. So he's not a philosopher that I was really planning to spend much time on.

Given this, I was quite pleased to discover this book promising to give me A Very Short Introduction, especially since it is written by Peter Singer, a philosopher who write and thinks rather clearly. After reading this book, I still believe that most of what Hegel wrote was pretentious nonsense, but the one idea that struck me as the most interesting was his conception of freedom.

A rough definition of freedom might be ensuring that people are able to pursue whatever it is that they prefer. Hegel is not a fan abstract definitions of freedom which treat all preferences the same and don't enquire where they come from.

In hi... (read more)

Book Review: Communist Manifesto

“The history of all hitherto existing society is the history of class struggles. Freeman and slave, patrician and plebeian, lord and serf, guild-master and journeyman, in a word, oppressor and oppressed, stood in constant opposition to one another, carried on an uninterrupted, now hidden, now open fight, that each time ended, either in the revolutionary reconstitution of society at large, or in the common ruin of the contending classes”

Overall summary: Given the rise of socialism in recent years, now seemed like an appropriate time to review the Communist Manifesto. At times I felt that Marx’s writing was keenly insightful, at other times I felt he was in ignorance of basic facts and at other times I felt that he held views that were reasonable at the time, but for which the flaws are now obvious. In particular, I found the first-half much more engaging than I expected because, say what you like about Marx, he’s an engaged and poetic writer. Towards the end, the focused shifted into particular time-bounded political disputes for which I neither had the knowledge to understand nor the interest to acquire. At the start, I fe... (read more)

Book Review: So Good They Can't Ignore You by Cal Newport:

This book makes an interesting contrast to The 4 Hour Workweek. Tim Ferris seems to believe that the purpose of work should be to make as much money as possible in the least amount of time and that meaning can then be pursued during your newly available free time. Tim gives you some productivity tips in the hope that it will make you valuable enough to negotiate flexibility in terms of how, when and where you complete your work, plus some dirty tricks as well.

Cal Newport's book is similar in that it focuses on becoming valuable enough to negotiate a job that you'll love and downplays the importance of pursuing your passions in your career. However, while Tim extolls the virtues of being a digital nomad, Cal Newport emphasises self-determination theory and autonomy, competence and relatedness. That is, the freedom to decide how you pursue your work, the satisfaction of doing a good job and the pleasure of working with people who you feel connected to. He argues that these traits are rare and valuable and so that if you want such a job you'll need skills that rare and valuable to offer in return.

That's... (read more)

Someone should see how good Deep Research is at generating reports on AI Safety content.

There is a world that needs to be saved. Saving the world is a team sport.  All we can do is to contribute our part of the puzzle, whatever that may be and no matter how small, and trust in our companions to handle the rest. There is honor in that, no matter how things turn out in the end.

If this were a story, there'd be some kind of academy taking in humanity's top talent and skilling them up in alignment.

Most of the summer fellowships seem focused on finding talent that is immediately useful. And I can see how this is tempting given the vast numbers of experienced and talented folks seeking to enter the space. I'd even go so far as to suggest that the majority of our efforts should probably be focused on finding people who will be useful fairly quickly.

Nonetheless, it does seem as though there should be at least one program that aims to find the best talent (even if they aren't immediately useful) and which provides them with the freedom to explore and the intellectual environment in which to do so.

I wish I could articulate my intuition behind this clearer, but the best I can say for now is that my intuition is that continuing to scale existing fellowships would likely provide decreasing marginal returns and such an academy wouldn't be subject to this because it would be providing a different kind of talent.

I think that there's good reasons why the discussion on Less Wrong has turned increasingly towards AI Alignment, but I am also somewhat disappointed that there's no longer a space focusing on rationality per se.

Just as the Alignment forum exists as a separate space that automatically cross-posts to LW, I'm starting to wonder if we need a rationality forum that exists as a separate space that cross-posts to LW, as if I were just interested in improving my rationality I don't know if I'd come to Less Wrong.

(To clarify, unlike the Alignment Forum, I'd expect such a forum to be open-invite b/c the challenge would be gaining any content at all).

One thing I'm finding quite surprising about shortform is how long some of these posts are. It seems that many people are using this feature to indicate that they've just written up these ideas quickly in the hope that the feedback is less harsh. This seems valuable; the feedback here can be incredibly harsh at times and I don't doubt that this has discouraged many people from posting.

Less Wrong might want to consider looking for VC funding for their forum software in order to deal with the funding crunch. It's great software. It wouldn't surprise me if there were businesses who would pay for it and it could allow an increase in the rate of development. There's several ways this could go wrong, but it at least seems worth considering.

I don't want to comment on the whole Leverage Controversy as I'm far away enough from the action that other people are probably better positioned to sensemake here.

On the other hand, I have been watching some of Geoff Anders' streams does seem pretty good at theorising by virtue of being able to live-stream this. I expect this to be a lot harder than it looks, when I'm trying to figure out my position on an issue, I often find myself going over the same ground again and again and again, until eventually I figure out a way of putting what I want to express into words.

That said, I've occasionally debated with some high-level debaters and given almost any topic they're able to pretty much effortlessly generate a case and how the debate is likely to play out. I guess it seems on par with this.

So I think his ability to livestream demonstrates a certain level of skill, but I almost view it as speed-chess vs. chess, in that there's only so much you can tell about a person's ability in normal chess from how good they are at speed chess.

I think I've improved my own ability to theorise by watching the streams, but I wouldn't be surprised if I improved similarly from watching Eliezer,  A... (read more)

Why the focus on wise AI advisors? (Metapost^[1] with FAQ 📚🙋🏻‍♂️, Informal^[2] Edition 🕺, Working Draft 🛠️)

About this Post  - 🔗:  🕺wiseaiadvisors.com ,  🕴️Formal Edition (coming soon) 

This post is still in draft, so any feedback would be greatly appreciated 🙏. It'll be posted as a full, proper Less Wrong/EA Forum/Alignment forum post, as opposed to just a short-form, when it's ready 🌱🌿🌳.

 ✉️ PM via LW profile   

📲 QR Code

This post is a collaboration between Chris Leong (primary author) and Christopher C

I really dislike the fiction that we're all rational beings. We really need to accept that sometimes people can't share things with us. Stronger: not just accept but appreciate people who make this choice for their wisdom and tact. ALL of us have ideas that will strongly trigger us and if we're honest and open-minded, we'll be able recall situations when we unfairly judged someone because of a view that they held. I certainty can, way too many times to list.

I say this as someone who has a really strong sense of curiosity, knowing that I... (read more)

I've recently been reading about ordinary language philosophy and I noticed that some of their views align quite significantly with LW. They believed that many traditional philosophical question only seemed troubling because of the philosophical tendency to assume words like "time" or "free will" necessarily referred to some kind of abstract entity when this wasn't necessary at all. Instead they argued that by paying attention to how we used these words in ordinary, everyday situations we could see that the way people used the... (read more)

Book Review: Waking Up by Sam Harris

This book aims to convince everyone, even skeptics and athiests, that there is value in some spiritual practises, particularly those related to meditation. Sam Harris argues that mediation doesn't just help with concentration, but can also help us reach transcendental states that reveal the dissolution of the self. It mostly does a good job of what it sets out to do, but unfortunately I didn't gain very much benefit from this book because it focused almost exclusively on persuading you that there is value here,... (read more)

Anti-induction and Self-Reinforcement

Induction is the belief that the more often a pattern happens the more likely it is to continue. Anti-induction is the opposite claim: the more likely a pattern happens the less likely future events are to follow it.

Somehow I seem to have gotten the idea in my head that anti-induction is self-reinforcing. The argument for it is as follows: Suppose we have a game where at each step a screen flashes an A or a B and we try to predict what it will show. Suppose that the screen always flashes A, but the agent initially think... (read more)

Here's one way of explaining this: it's a contradiction to have a provable statement that is unprovable, but it's not a contradiction for it to be provable that a statement is unprovable. Similarly, we can't have a scenario that is simultaneously imagined and not imagined, but we can coherently imagine a scenario where things exist without being imagined by beings within that scenario.

Rob Besinger:

If I can imagine a tree that exists outside of any mind, then I can imagine a tree that is not being imagined. But "an imagined X that i

The sad thing about philosophy is that as your answers become clearer, the questions become less mysterious and awe-inspiring. It's easy to assume that an imposing question must have an impressive answer, but sometimes the truth is just simple and unimpressive and we miss this because we didn't evolve for this kind of abstract reasoning.

I was talking with Rupert McCallum about the simulation hypothesis yesterday. Rupert suggested that this argument is self-defeating; that is it pulls the rug from under its own feet. It assumes the universe has particular properties, then it tries to estimate the probability of being in a simulation from these properties and if the probability is sufficiently high, then we conclude that we are in a simulation. But if we are likely to be in a simulation, then our initial assumptions about the universe are likely to be false, so we've disproved the assu... (read more)

Three levels of forgiveness - emotions, drives and obligations. The emotional level consists of your instinctual anger, rage, disappointment, betrayal, confusion or fear. This is about raw raws. The drives consists of your "need" for them to say sorry, make amends, regret their actions, have a conversation or emphasise with you. In other words, it's about needing the situation to turn out a particular way. The obligations are very similar to the drives, except it is about their duty to perform these actions rather than your desire to make it... (read more)

Book Review: The Rosie Project:

Plot summary: After a disastrous series of dates, autistic genetics professor Don Tilman decides that it’d be easier to just create a survey to eliminate all of the women who would be unsuitable for him. Soon after, he meets a barmaid called Rosie who is looking for help with finding out who her father is. Don agrees to help her, but over the course of the project Don finds himself increasingly attracted to her, even though the survey suggests that he is completely unsuitable. The story is narrated in Don’s voice. He tells us... (read more)

There's been a lot of discussion about how Less Wrong is mostly just AI these days.

If that's something that folk want to address, I suspect that the best way to do this would be to run something like the Roots of Progress Blog-Building Intensive. My admittedly vague impression is that it seems to have been fairly successful.

Between Less Wrong for distribution, Lighthaven for a writing retreat and Less Online for networking, a lot of the key infrastructure is already there to run a really strong program if the Lightcone team ever decided to pursue this.

Ther... (read more)

Was thinking about entropy and the Waluigi effect (in a very broad, metaphorical sense).

The universe trends towards increasing entropy, in such an environment it is evolutionarily advantageous to have the ability to resist it. Notice though that life seems to have overshot and resulted in far more complex ordered systems (both biological or manmade) than what exists elsewhere.
 
It's not entirely clear to me, but it seems at least somewhat plausible that if entropy were weaker, the evolutionary pressure would be weaker and the resulting life and systems produce by such life would ultimately be less complex than they are in our world. 

On free will: I don't endorse the claim that "we could have acted differently" as an unqualified statement.

However, I do believe that in order to talk about decisions, we do need to grant validity to a counterfactual view where we could have acted differently as a pragmatically useful fiction.

What's the difference? Well, you can't use the second to claim determinism is false.

I've been thinking about Rousseau and his conception of freedom again because I'm not sure I hit the nail on the head last time. The most typical definition of freedom and that championed by libertarians focuses on an individual's ability to make choices in their daily life. On the more libertarian end, the government is seen as an oppressor and a force of external compulsion.

On the other hand, Rousseau's view focuses on "the people" and their freedom to choose the kind of society that they want to live in. Instead of being se... (read more)

As I wrote before, evidential decision theory can be critiqued for failing to deal properly with situations where hidden state is correlated with decisions. EDT includes differences in hidden state as part of the impact of the decision, when in the case of the smoking lesion, we typically want to say that it is not.

However, Newcomb's problem also has hidden state is correlated with your decision. And if we don't want to count this when evaluating decisions in the case of the Smoking Lesion, perhaps we shouldn't count this in the case of Newc... (read more)

Parent comment for: Why the focus on wise AI advisors?

I'm beginning to warm to the idea that the reason why we have evolved to think in terms of counterfactuals and probabilities is rooted in these are fundamental at the quantum-level. Normally I'm suspicious at rooting macro level claims in quantum level effects because at such a high level of abstraction it would be very easy for these effects to wash out, but the multi-world hypothesis is something that wouldn't wash out. Otherwise it would seem to be all a bit too much of a coincidence.

("Oh, so you believe that counterfactuals and probability are at least... (read more)