Practical Decision-Making was the first course of the Guild of the Rose Beta. The course was an introduction to conscious decision making and spanned five weeks from 09/01/2021 to 10/06/2021. It followed our standard flipped-classroom model and was taught by Matt Freeman, Guild Council member. Topics addressed included:
- Calibration; or the ability to make probabilistic predictions with an accurate confidence level
- Utility; an economics concept that can be used to quantify outcomes
- Decision trees; a technique for breaking complex decision down into concrete outcomes and paths of action
Fundamental decision making is an underrated skill. Few people are aware of the concept of a decision tree, and even fewer are able to build one. Even among rationalists, there is too much emphasis on theoretical minutiae. If one cannot use the basics of decision making in their daily life, it matters little whether they can write one thousand words due Friday on functional vs causal decision theory.
The Art must have a purpose other than itself, or it collapses into infinite recursion.
--Eliezer Yudkowsky, The Twelve Virtues of Rationality
Successes
As an introduction to decision trees, Practical Decision-Making was a solid success. Multiple students reported using the techniques to make difficult decisions. Topics ranged from the mundane (Should I get a haircut?) to the life-altering (Should I stay in academia or get a job?), and several people ultimately chose courses of action that they had initially leaned against.
Metaculus predictions
As part of an ongoing homework, students were asked to vote on one Metaculus question per day. Metaculus is a website that lets users vote on how likely they think various events are to happen. Questions cover a broad range of topics, from sports to politics to technology. The intent of the daily task was that making predictions would sharpen two skills:
Identifying nonspecific claims. Metaculus predictions are quantified and precise, in direct contrast to common everyday predictions. For example, consider the difference between saying "There is an 85% chance it will be raining tomorrow at 09:00 AM" and "It's probably going to rain". Most people would say the second one, but it's very vague. While it's certainly possible to check whether it rained, it's hard to say much beyond that. However, if a Metaculus predictor says there's a 99% chance of rain and it doesn't rain (and they keep doing this), then that tells you a lot more about their predictive capabilities.
An example of a quantified, precise Metaculus prediction.
Improve probabilistic thinking. Probabilistic thinking is the skill of estimating the likelihood of a specific outcome happening. It can be done consciously (via tools such as Fermi estimates) or intuitively (via the "gut impression" or "vibe" of a situation). Don't make the mistake of thinking that intuitive reasoning is always inferior to explicit reasoning. In a world where each moment is determined by many factors, an intuitive grasp of probability allows one to make quick, precise, and effective decisions.
We did not create an objective test to measure skill in these two areas. However, students who stuck with the daily prediction tasks did report improvements.
Videos
For Practical Decision-Making, the Guild experimented with creating weekly videos. None of the Council had experience with scripted video content, so filming the ten lectures for the course posed a steep learning curve. While the initial videos were quite rough, quality improved steadily across the course's five weeks.
Given that the Guild is still in Beta, we consider this to be a success. However, the experience has shown the infeasibility of producing high-quality video content ourselves. We can do better by hiring a third-party video editor. This approach will also have the added benefit of allowing instructors to concentrate on the script and content of their classes.
Mistakes
Miscommunication
The intent was for each class's homework to build upon the last. To do so, students were expected to tackle the same major life decision each week.
Unfortunately, this expectation was not communicated and most students worked on different decisions each week. This led to them feeling blindsided and overwhelmed when the fourth week's assignment was to build a decision tree. Instead of merely assembling pieces worked on over the past three weeks, they ended up having to do the entire task from scratch.
Future iterations of Practical Decision-Making will inform students that they should work on the same decision each week. We also intend to outline the structure of the course, so it's clearer that each week is meant to build on the last.
Choices vs outcomes
The first class of Practical Decision-Making was called Thinking in Buckets. This is a technique that simplifies the chaos of reality by placing similar outcomes into "buckets". For example, when considering whether to play a game of roulette, one might choose the following buckets: "jackpot", "break even", and "lost money".
Unfortunately, about half the students came away from the first video lecture with the wrong interpretation. Many people thought that the buckets were supposed to represent choices, rather than outcomes. To use the simplified roulette example above, the outcomes are determined by which slice of the wheel the ball falls upon. The choices, however, are quite simple: whether to play the game and, if so, where to place one's chips.
Powerball
One of the homework assignments involved creating an expected value table for the Powerball lottery. An expected value table is a way to calculate how much a Powerball ticket is "worth", on average. Unfortunately, Powerball was a poor choice for an introductory assignment due to its complexity. Many students got hung up on irrelevant rules minutiae such as Power Plays or Powerballs vs regular balls. In retrospect, we should have used a lottery with simpler prize mechanics.
The Credence Game
The Credence Game is an app that measure's one's calibration; or ability to make precise probabilistic predictions. Unfortunately, the Credence Game is not cross-platform and several students couldn't run it. Future version of Practical Decision-Making will instead rely on Metaculus's web-based Calibration Practice.
Metaculus time horizons
Most Metaculus questions don't resolve for months or years. Loose feedback loops make it very hard to train one's intuitive calibration. Calibration is a valuable skill, and ideally it would be possible to practice it on real-world questions. However, the Metaculus prediction task was still effective at training general probabilistic thinking.
Lessons Learned
- Hire a professional video editor
- Script video content in advance; don't try to improvise on the fly
- Have a second set of eyes review homework assignments for clarity prior to posting them
Wrap-Up
Overall, Practical Decision-Making was a solid class and a great start to the Guild Beta. We consider the course a success and will likely run an improved version of it in the future.