Holy cow, you've walked the exact path I'm walking right now! I used to be super into Bayesian epistemics, AI risk, etc. Then my world model kept seeing ... large prediction errors. I quit AI risk and quant trading, and now see most of our institutions (science, press) as religious/fallible. Now I'm super into vipassana, storytelling, Kegan levels, NVC, IFS, form is emptiness. I'm even considering training at a monastery.
I have a couple questions for you:
Another thing I'm interested in is making System 1 models more palatable to System 2 minded people. Because man, I was ignorant for a year or two, and my orbit still think I'm crazy for going on about vipassana and Kegan levels. This is more a word salad for other people with similar interests, but it may interest you too:
I feel like we're the blind men with the elephant more often than we'd like to admit. A lot of the time when two people make conflicting claims about society, really they're both right about their substrate of society and the world is just twice as big as either thought.
Another shocker for most people: 20 million people in the US live in trailer parks. People with similar life circumstances tend to accumulate in similar places, only see those places, and thus vastly underestimate the diversity of life experience. (This is also true of everything in https://www.lesswrong.com/posts/KpMNqA5BiCRozCwM3/social-dark-matter)
I've played Figgie a fair bit and don't think it's a good tool for teaching epistemics outside of a Jane Street trading internship.
To actually learn with it, you first need a number of motivated playing partners. They need to be quite skilled for their actions to be informative, and the game sucks before they reach this point. In my experience people don't reach this skill level within their first couple hours of play. It also requires N custom-made decks for N rounds of play, or for everyone to install a specific app. So it's really only practical for a trading firm internship program.
I have what I would call unusually good epistemics, and I mostly got there by obsessively reading the financial news (Bloomberg, Reuters) and making concrete predictions about how various stories would develop, and seeing where I went wrong. I'd ask myself questions like "how, concretely, did this article come into being?" I did this for about 9 months, and by the end I was rarely surprised by news items, could spot the signs of failure modes like circular reporting (super common), and had working mental models of many important institutions.
The other valuable thing I did was putting probabilities on everyday occurrences (65% confident I get home within 15 mins, etc.). I recorded these, and reflected on what went wrong when I deviated too far from perfect calibration.
Love this idea, and +1 on Jon Gjengset and Neel Nanda.
For finance though, I'm skeptical of your recommendations (DeepFuckingValue and Martin Shkreli). The former made his money pumping-and-dumping meme stocks, and I get the impression the latter has been selected for fame (like recommending Neil deGrasse Tyson to learn physics).
In general, I think finding good resources in finance requires a much stronger epistemic immune system than nearly any other field! There's so much adverse selection, and charlatans can hide behind noisy returns and flashy slide decks for a very long time. The KL-divergence between what competent looking YouTubers say and what actually works is extreme.
My recommendations would be:
Other recommendations:
For a mix of legal and reputational reasons, I will have to be a little vague in describing the experiences I had.
Part 1: social forces (outside view)
There were a bunch of social forces that led me to form an outside view that the AI risk community doesn't converge to truth. I'll describe one, storytelling.
One thing I've learned is that the stories we tell ourselves are models of reality. Life experiences are too numerous; every narrative is a simplification, and there are many narratives (models) that fit the experiences (data). And the default process that humans use to construct narratives is very unscientific - lots of rationalising, forgetting, selecting based on what frame generates the best emotional response. Our narratives and ontologies end up contouring our revealed preferences, while we perceive ourselves as virtuous. (This is where Kegan levels, equanimity meditation and narrative therapy come in.)
So often, when I see people explain their choices, their narrative seems better explained by "trying to justify their job" or "trying to feel good about themselves" than by "genuine truth-seeking and abyss staring". It's a subconscious process, and the truth hurts.
There were a bunch of things I saw / heard about in the AI risk community that didn't seem right. For example, I was abused in college, their friends enabled them, I self-ostracised, they pursued people 2 more years. It's hard when personality disorder is involved; we were all kids so I've forgiven. But when I heard rumours about similar things in AI risk, like the TIME article, and saw a (still-)high-status man inappropriately touch a young woman, you have to wonder - if the supposed adults in the room don't have the EQ to avoid bungling abuse allegations or taking money from crypto moguls, and don't take responsibility, do they have the self-awareness to pivot when their research agendas are failing? Or are they telling themselves stories there too? I truly love these people, but I don't want to be one.
I saw similar things in an overlapping community where, if I could legally talk about it, it would front the New York Times. Around this time I started finding Eric Jang's haikus and Le Guin's Omelas very relatable.
I don't know if you can logic someone into my perspective here. The bottleneck is abyss-staring, and some of the abysses I stared into were enabled by the deep equanimity I gained from my meditation practice. If an unpleasant hypothesis generates such strong emotional / ruminatory reactions that it can't be stably held in your mind, you will stay in denial in the world where it is true. Whether this manifests as getting uncomfortable, abruptly changing the topic, rationalising it away, or nit-picking an irrelevant detail; doesn't matter.
The door to the path I've taken started with meditating a ton, and then pouring equanimity into the thoughts that made me most uncomfortable.
Part 2: my model of AIs (inside view)
I've been thinking about ML since 2018, and my high level take is "you get exactly what you wished for". Models are models of the data: they are shaped by the loss function, regularities in the data set, the inductive biases their functional form encodes, and that's ~it. Like, the "it" in AI models is the dataset, or Hutter's "compression is intelligence".
If you take Anthropic's circuit model of LLMs, circuits need to pay rent in the form of decreasing loss. There's no deception unless you trained it on deceptive data, or created an inductive bias towards deception. Any model capacity allocated towards deception would be optimised out otherwise. Every time we do mech interp, we see the models are just doing the most algorithmically simple/efficient way to encode grammar or Othello board states (before you mention Neel's modular addition circuit, trigonometry soup was the simplest solution for that particular functional form and input/output encoding :P). Point is, there's no room for anything suspicious to hide.
To believe the practice is different from math, you'd have to believe something weird about the differences in real training runs. Adam's loss geometry (because compression=intelligence assumes KL-divergence loss), floating point math, RLHF, etc. I'm glad people are thinking about this. But I'm personally unconvinced it's super high value; it feels better explained by "generating a laundry list of hypotheticals" than by "genuine truth-seeking".
The MIRI ontology has never felt adaptive to modern ML to me, but then I never really understood it.
Maybe we do active inference agents (fixed priors induce action), or get good at RL, or prompt engineer models to be more agentic; it feels wrong to me to think these are x-risky but I can't put into words why. Runaway growth models in particular feel very wrong to me, the world is so (so!) much more complicated and less digitised than people in SV tend to think.
The research agendas I've seen up close all contradict these personal views in some way.
I know there are many other schools of thought on AI risk, and I haven't thought about them all, and I don't doubt my model here has many problems. My only goal here is to convey that I have thought hard about the object-level too, it is not just social ick.