- China has fewer than 100,000 COVID cases this year (official estimate): 30% → Hold.
This is graded as not happening. I think it’s pretty ambiguous. Yes the official statistics simply started ignoring reality, but there were various officials giving out ‘estimates’ that were several orders of magnitude higher than the threshold.
The reality could also have gone either way.
I think if you interpret the scenario that happened as resolving to no, then 30% is too high, because there is too big a risk of the official statistics simply ignoring reality, so now you need to win quite the parlay. If you don’t need that, 30% is too low, the dam broke rather late if anything.
I'm having trouble parsing this, but it sounds like you might have the inequalities reversed? As in, if it's graded as not happening, then it's graded as "China had more than 100,000 cases" but it sounds like you might think it's graded as "China had fewer than 100,000 cases".
New U.S. sanctions on Russia (70%): Scott holds, I sell to 60%.
This seems like a better sale than the sale on Russia going to war, by a substantial amount. So if I was being consistent I should have sold more here. Given that I was wrong about the chances of the war, the sale would have been bad, but I didn’t know that at the time. Therefore this still counts as a mistake not to sell more.
This seems like a conjunctive fallacy. "US sanctions Russia" is very possible outside "Russia goes to war", even if "Russia goes to war" implies "US sanctions Russia". You had 30% on "major flare up in Russia-Ukraine". Perhaps you are anchoring your relative sells or something?
I obviously agree that you know these things, and am only noting a self-flagellation that seemed unearned. Thanks for writing Zvi!
Previously: 2022 ACX Predictions Buy/Sell/Hold, Evaluating 2021 ACX Predictions
As is the yearly tradition, let’s see how we did. Scott noted in his results post that on pure numbers I did pretty well, coming in 54th out of 508 entries.
My own evaluation, by my own standards, is I did a mediocre job.
2022 was the first low-effort year, no looking at prediction markets and five minute time limits. For 2023, I put even less effort into things – I didn’t spend the five minutes even once, and mostly put down numbers on instinct. I modestly regret not trying harder, but am confused as to whether or not that regret makes sense.
Pure numbers is not how I evaluate predictions. They are one input, but it is more important to do deliberate practice. Look back, and ask whether your prediction made sense, whether your logic was good, what would have been the right odds. A Briar score is quite a crude measure, as is looking at calibration. They can tell you vaguely where you are at in some sense, but it won’t much help you improve otherwise.
Then one can look at the big picture, and see what patterns and errors emerge.
There are very clear patterns of error in my predictions from 2022.
How should we think about prediction markets outscoring all but one person’s five minute efforts? I think we should be disappointed in prediction markets here, given the asymmetries. I agree that the one person who beat them outright in five minutes got lucky. It still seems clear that anyone thinking well would be able to defeat the prediction markets ‘playing offense’ rather soundly, and that weighing smarter opinions more improves them a lot.
So we go through each question in turn, and ask if good decisions got made. Keep in mind we weren’t allowed to look at prediction markets (both Scott and myself).
Did not happen or come close.
Predictor average was 36%. On reflection I didn’t sell quite enough here. Biden had a bunch of good and bad news and none of it mattered much at all. Market was 14%, so if I’d known that I’d have sold down to where it was.
Did not happen or come close.
This was despite Roe vs. Wade being overturned, and not a peep. My logic was that $250 million was not very much in damages, and the bae rate of this wasn’t so low. Hard to know what the base rate truly is in a case like this. I don’t hate my buy here, especially with market recorded as 36%, but I do think 36% is insanely high when you look at the historical record. If Trump was president, maybe, but with Biden?
Happened and was never close to not happening.
It seems like Biden would have to credibly have said he wasn’t running for this to happen, or a combination of a clear alternative with a truly horrendous midterm. He even got a war against Russia by proxy and nothing much moved. So this was damn unlikely and if anything should have been higher.
Did not happen, could have gone either way, him and DeSantis flip every so often.
Donald Trump had a maximally bad midterm and generally things went quite poorly for him, while they went extremely well for DeSantis, and this still almost came in. There was a lot of resilience here along the baseline scenario, such that this seems like it starts out at 80%+, as we essentially needed a trifecta of (Trump failure, DeSantis success, and a coin flip). So the question is how often we get a different Trump failure mode, of failing health or severe legal trouble. The buy here seems good to me on reflection, even though it lost.
Not on the compilation page but I’d say it happened. Presumably ambiguity was the issue here. Given the level of precautions, I’ll take a 2% shot at a hitch here every time on any reasonable definition.
Very much happened.
I was a big skeptic on this and I was wrong big, with only my anchoring keeping me from making a very bad prediction on likely the most important question. As with many others, I assumed that something this stupid and pointless was highly unlikely, and wow was that logic wrong. There are many lessons here for sure, including ‘authoritarian information environments and incentives are even more warped and SNAFU than I realized’ and ‘no really, a person can’t escape their nature and will do crazy things’ and also some things about Russian culture.
This is also one case where being right about one big thing (that Ukraine could win and the Russian attack was really dumb) and then letting that make me very wrong about an even more important thing, because I thought others would understand that. Whoops.
Did not happen.
On reflection this was probably a little low given how such distributions work. I like the superforecaster average of 7%.
Did not happen.
I said ‘even if there is a shooting war in Ukraine this won’t happen.’ There were some scary moments given how bad this would be if it happened, and we will never know for sure how close Xi came to trying something deeply stupid. Given that Ukraine happened, I’m going to say this sale was pretty bad, both in terms of the Ukraine war making the situation more volatile than other years (as did the party congress) and also reflecting a world model where this is more likely. So I don’t mind the sale in 2021 but in 2022 I think you couldn’t do that. Which is super scary. Going into 2023, knowing what we know now and being passed the windows, I’m back to being more of a seller than a buyer.
Did not happen. I suppose ‘no reasonable person’ is a high bar, but they did repeal the law and are clearly hostile. Superforecasters had this at 16%, average predictor was at 13%, and I think I didn’t buy enough here, you have to go at least to 10% even with my lack of information and the anchoring. Which, in turn, means that no reasonable person should have invested before, as I noted. If you have a 15% yearly chance of things becoming non-viable, that’s not a good investment, even if it doesn’t tank everything automatically on the spot. Even 5% is quite a lot.
It is often said that all progress depends on the unreasonable man.
Did not happen, and was never going to happen given they outright repealed the law. Likely this should have been sold down a lot if one understood the situation. Seems reasonable to defer when totally in the dark. Would have been a good place to actually spend five minutes doing research, if I’d been going all out to win the competition.
Did not happen, was way down instead.
Started year at 112. I think the market is getting short end of the stick being called 50 here because there is a long tail of super crazy and also who exactly is going to be shorting this for a year? Hard to know what one is supposed to think about GameStop levels of irrational behavior. On reflection I think this should have been more like 20%, and yes that makes GameStop a clear sale at the time but… that seems fine, no?
Neither happened, both were down instead.
At the time Ether was ~2.5K as my notes indicate, still seems like these are reasonable market-neutral positions to take. One could argue that there were signs one should have been bearish, but it seems fine to pass on such questions. Market is marked as being more skeptical than this and I think that is the market pricing options badly – if I wanted to be long crypto, didn’t have to worry about taxes and was happy to do complex weird things and not worried about counterparty risk, I think buying far-out-of-the-money call options on crypto is a better buy than crypto – party exactly because of all those reasons not to do it.
Happened, it’s 0.07 today.
Crypto was indeed down and this held, which suggests that holding BTC instead of ETH was an error. I do think the 90% is expressing a rather strong opinion on major financial instruments here.
It is now 107K and was clearly lower.
Wasn’t graded due to several issues.
The other problem is that the question says ‘below’ rather than ‘above.’ It is very bad to misread markets like that. You can lose quite a lot. What I clearly meant here was that I wanted to buy this, and I don’t think I would have cared too much about Scott’s opinion here so likely to have bought to 65% and maybe the full 70%. Could the NFT craze have sustained itself for the whole year, at least for the ‘top stuff’? Yeah, maybe it had a 25% or so chance, you never know. And I wouldn’t rule these out as potentially super valuable in 20 years, consider what people pay for basically anything from our childhoods that’s rare.
Did not happen.
90% was clearly very nuts. I said ‘check the market’ and that was marked as 54% here, whereas superforecasters were still at 75% and average predictor was 80%. I still can’t see how it is as low as 54% if you assume market is efficient, I’m willing to accept 70% was too high and reflected me actually being bullish. Hey, I was and still am long.
Did not happen.
Market is marked at 30%. A 24% chance of landing in between these numbers seems fine, I decided 30% was reasonable given my decision not to look at option prices and that all seems fine. Given the 70% was too high, this was too high as well, but it reflects a copy of the same mistake.
Did not happen, finished 6.5%.
I fell pretty stupid here for letting Scott’s number trick me into staying that high. There is simply no way this should have been anywhere near 80%. Market here is marked at 45%. Part of the difference is that I didn’t expect the Ukraine War, and with that I think 45% is too low, but anything above ~65% is still crazy talk. In general, it seems like I anchored far too much.
Rate was about 3.7%, well below 5%. Market is listed at 73%, superforecaster average 68%. Given it is clear I was overinfluenced by Scott’s numbers, I’m going to say I was modestly too low, and 75% or even 80% would have been better.
Was graded as not happening.
Market is listed as 14%, but superforecasters said 52%. Not sure where market comes from here and the ambiguity in what counts as what is large. I noted that the 50% here threw me, which is likely another case of me anchoring too much on not-so-considered numbers. Didn’t sell enough.
Did not happen.
It seems most everyone did expect this to happen, market is listed as 94%. It does feel like an awful lot of confidence for something like this where any number of things can go wrong. Market being that high gives me pause but I’d like to have sold this more, given I did not know that. And on reflection if I’d known market was 94% I should have sold a tiny bit there too.
Did not happen or get anywhere close.
I was right that a lot of people would stop testing and reporting and caring about Covid. There were still a lot of Covid cases. If Omicron hadn’t kept evolving would this have had a chance? I think yeah, but December is a rough month and 10k is a low number. The buy here is at best aggressive.
Quote is ‘yeah, not going to happen.’ Seems fine to not try and sell sub-1% on principle, but here is a place one could have thought about it.
Happened, despite the vaccination campaign in 2022 being widely seen as disastrous.
Whenever things go pretty badly and you win anyway like this, in a place where the variance seems reasonably low, it means the uncertainty was in your thinking. I said I wasn’t excited, and presumably that means the calculation was off a bit.
Did not come anywhere near happening.
I interpreted this as cumulative, still not sure if that was intended, it doesn’t happen either way. It still seems like a mystery why this very much did not happen, and I am fine with having bought a bit. Superforecaster average was also 10%.
Did not happen.
I still think that I was pretty crazy to only buy this to 2% – only this past week I saw at least one doctor reverse course on Vitamin D and say it was unethical not to recommend it. And if I was worried about Covid, I would be damn sure to take my Vitamin D. Was a reversal here unlikely? Sure, but 5%-style not 2%-style. The other two were sub-1% shots.
Sadly did not happen, in a situation where it obviously should have happened and also on reflection obviously was a long shot to have happened. The FDA is very much about not approving things and I feel silly still being as high as 50% even if others were mostly higher.
Did not happen, there were new variants that displaced old ones but they were classified as sub-variants of Omicron. Given all the near misses in that sense, 60% still seems fine.
I was too anchored here, and too worried (well, too worried in this particular context) that San Francisco can stay irrational longer than you can stay solvent. This was never going to happen and I should have been more confident.
This does seem like exactly the type of place we could plausibly have stayed irrational, but one needs to justify 35% as being high rather than as being low. Market was still at 51%, so this was likely appropriately low.
Graded as happening due to the Omicron boosters.
This is more too low than too high, but it seems mostly inevitable that there would be an Omicron booster and that they’d tell everyone to get it. So if that alone counts, as I think it would have, then yeah, probably a bit low.
This is graded as not happening. I think it’s pretty ambiguous. Yes the official statistics simply started ignoring reality, but there were various officials giving out ‘estimates’ that were several orders of magnitude higher than the threshold.
The reality could also have gone either way.
I think if you interpret the scenario that happened as resolving to no, then 30% is too high, because there is too big a risk of the official statistics simply ignoring reality, so now you need to win quite the parlay. If you don’t need that, 30% is too low, the dam broke rather late if anything.
Skipping the personal predictions because I don’t have grades for them, and I mostly didn’t trade them anyway.
88. No new real-money prediction market becomes bigger than Polymarket: 70% → Buy to 75%.
Happened.
Prediction markets are a hard business. There are a lot of people trying. There is a lot of value to the world in making them succeed. The problem is it is highly unclear how you the prediction market website makes real money. People want to bet on sports and politics, where there are clearly defined high profile outcomes without too much insider information, quick resolution, good drama. That doesn’t mean there is great appetite for weirdness that mostly happens over long period and also yes everything is probably a security (not investment advice). And then there’s all the Hansonian issues that people don’t want accurate information that is visible to all because it disrupts elite power and decision making. I am very happy to keep working with prediction markets to try and make them work, but the problem increasingly seems super hard to commercialize successfully where it would count.
In any case, yeah, if anything I didn’t buy enough here, Polymarket filled the niche of ‘crypto prediction market company actually trying’ and there was no reason to expect a new entrant that would be bigger.
Happened. Market is listed here as 91%, I am guessing that was a Manifold Market, which for reasonably obvious reasons is kind of cheating? Still, I do think that 30% chance for a company with no huge costs or reason to fold up shop keeping their doors open a year was on reflection super low and I feel dumb here not going to 60% or higher. Wow I anchored too much last year.
Market is listed as 12% and the superforecasters averaged 6% but I am happy with my call here. It indeed seems super difficult to get these things approved and off the ground quickly, market seems saturated, even if something does arrive it’s not going to get to half Kalshi size within a year of getting on our radar.
Yep, I doubt anything new is on the horizon, and if it is it’s going to take a while to ramp up most of the time. Still, could have happened, who knows, seems OK to stop at 15%. The market 42% here seems crazy.
January 23, and I’m guessing what happened counts. My theory was Scott might choose to slightly miss, which did not happen. On reflection no, Scott mostly wants to meet his obligations, and I shouldn’t have sold.
Now the Vox predictions, so numbers reset.
Yeah, 95% was completely insane, should have known the market would never be anything like that high, 90% still insanely high. I did bet on Republicans and on net lost a bit at the market odds, which I mostly stand by given where there was uncertainty. It took a lot to have this not happen.
2. Inflation in the US will average under three percent: 80% → Scott holds, I sell to 60%.
Yeah, once again even my level of confidence in Team Transitory here was crazy and this was dumb.
Seems like this already happened and I was confused so I ignored it. Then in the spreadsheet this gets listed as under 3%. Super weird, in any case essentially a repeat where I make the same mistakes.
This did famously happen.
It’s hard to know how inevitable it was. Feels like it could have gone either way, and the leak reflected that this was up in the air no matter who or which side did it. Everyone was in the 50%-65% range which is actually reasonably narrow. I’m pretty much fine with it.
Yeah, that escalated quickly, whoops.
Seems I was on the wrong side of this, it was hard for him not to make the finals and hard for him to lose a heads-up against whoever it was because they were likely to either be hard right or hard left, despite his Covid policies. I didn’t think enough about how the dynamics work in France. It felt touch and go but not coin-flip level touch and go.
Did not happen, was close, market was highly skeptical so presumably I was missing things here. I haven’t looked into details enough to know what to think.
Happened. Both sources here are clearly not making good predictions so it’s not clear who to favor if I have no idea. If I had to guess I’d say re-elections are more likely than they are giving them credit for in general, but seems fine to pass.
10. China will not reopen its borders in the first half of 2022 (80%): Scott buys to 90%, I held at 80%.
Happened. Market was listed at 98% and yeah, that was insane confidence. I can see the argument for 90%, but given they did reopen in the second half, my guess is 80% was closer to the right answer than 90%.
11. Chinese GDP will continue to grow for the first 3/4 of the year (95%): Scott sells to 90%, as do I.
Happened.
It was clear at the time, and seems even more clear now, that the calibration on the Vox predictions was quite bad, confidence was way too high. You need a good reason not to sell in this spot.
My logic was wrong here. I thought there was approval uncertainty, and instead we couldn’t find 20% of parents willing to do this despite approval, so that’s bad predicting no matter the resulting numbers. I was definitely surprised that the early child vaccine was rejected so resoundingly – sure I expected that in some areas, but 20% is not a lot.
Did not happen.
Everyone was in the 70s, and on reflection this should have been more like 60% for the same reason as other logic above. Too much anchoring once more.
This happened. Still haven’t looked into it.
This happened. I interpreted Scott’s buy as ‘there is at least one obvious country that will count’ and I’m guessing that it didn’t come to anything like that and it was a gimme and I chickened out.
Happened, as part of the continuing trend. Doesn’t feel like it was inevitable, so still ‘seems high’ without rising to the threshold of action.
This was graded as having happened, I stand by this being too ambiguous.
Happened, alas. What I said here: “Buy to 75%, we’re really stupid. I mean really, really stupid and terrible at this.” Yep. If anything should have been higher.
19. The Biden administration will set the social cost of carbon at $100/ton or more (70%): Scott Holds, I buy to 80%.
There is a resolution dispute, FP/Vox claims probably true, I agree with Scott this was false, ‘will happen real soon now’ does not in any way count, doesn’t matter what they do in 2023. My mistake here was assuming that it almost certainly wouldn’t take them into 2023 to set a social cost of carbon. Somehow I overestimated government efficiency. I’d like to say it won’t happen again.
Happened. As I noted this is a math question, not clear I did enough of the relevant math, but coming off a down year this was presumably a sizable favorite.
As I said, uninformed sale, I was right that Dune had no chance. I had too much respect for Vox’s predictions and calibration, could have easily sold this down to 25% despite being uninformed – if Belfast had been that likely to win other sources would have mentioned it, although it’s not obvious I’d have seen them.
Not on spreadsheet but this happened and they had 37 versus next highest being 27. So the right answer was much higher, this was never going to go another way. Doesn’t mean one could find that out in five minutes. There’s ‘this should be knowable in advance’ pulling in favor of going higher given that Vox said 60%, there’s ‘Vox is overconfident in things’ pulling it down. Not sure which should win that fight.
Yglesias Predictions
1. Democrats lose both houses of Congress (90%): Scott holds – as above, hold here.
2. Democrats lose at least two Senate seats (80%): Scott holds
3. Democrats lose fewer than six Senate seats (80%): Scott holds
I also went with ‘he’s thought about this a lot, might as well go with it.’ Distribution continues to seem vaguely reasonable.
I sold to 60% as it seemed uncertain. Scott resolved this as ambiguous since it’s not clear if it means leadership or her house seat. I’m inclined to say Matt meant the leadership even if it’s technically ambiguous and I’m pretty sure that’s what I was predicting. I mostly like being modestly more uncertain.
Of course, noticing and avoiding ambiguous language is an important skill too.
The BBB question wasn’t graded, presumably because of the news in the interim being quite bad for its chances. I am pretty sure the IRA does not count and this did not happen, and if that is considered clear even 40% was too high.
Market says 91% but this is one of those classic Free Money situations where the people on the other side are being dumb. This was a good buy.
Happened. As I noted there are a lot of them. I don’t even know who it was. I like being high here, 75% still seems a bit low.
As I noted in my post, this is a lower threshold than Scott’s which is why probability is lower here. I do think 90% was a bit low here even with everything we learned, but 95% was likely a bit high. The chance of a minor flareup does not seem all that high, China either goes for it or it doesn’t and Taiwan isn’t starting anything.
This seems like a better sale than the sale on Russia going to war, by a substantial amount. So if I was being consistent I should have sold more here. Given that I was wrong about the chances of the war, the sale would have been bad, but I didn’t know that at the time. Therefore this still counts as a mistake not to sell more.
This did not happen. I notice I would have been actively surprised if it did happen, which means being as high as 50% was wrong. Needed to be down to 40% or lower.
Happened. Should we have been this confident? Barring a new variant this was essentially a Can’t Happen, so it would have required a new variant that spread everywhere while also being either deadlier or able to ignore previous immunity with respect to death not only with respect to infection. Was that more of a 5% or a 10% shot at the time? I love the buy to 90%, less certain about the further buy to 95%. The market price here says 96%, which is surprising to me, I would have expected them to express more uncertainty. Given that, the 95% is obviously fine.
This isn’t listed in the spreadsheet. I looked up what this was and then looked at Wikipedia and it looks like they did get around the debt brake but did it by getting a two-thirds majority to vote for it. That’s not a loophole, so I’m grading this false, and my sale directionally correct for the right reasons.
For a second I thought I was being super aggressive for no reason then I realized this says 2021 not 2022. Free money. Past is easier to predict than the future. Didn’t make it onto the spreadsheet. If 2021 was a typo, of course, 90% would have been absurdly overconfident.
16. Liz Cheney loses primary (80%): Scott holds, I hold
She lost in a landslide, and never had a chance. I wrote I wanted to sell, but thought better of it. We know this was in fact almost impossible, the question is how confident we could have been in this. The 80% number still seems reasonable, perhaps a bit low because Cheney had already made it clear she would rather lose with honor and dignity than give herself a better chance. I approve, but didn’t help her chances.
17. Some version of USICA passes Congress (70%): Scott holds, I laugh and sell to 55%.
It did happen on a bipartisan basis, and market was even higher at 80%. My sale was based on the prior that congress does not pass things, especially bipartisan things. Biden did quite a good job of passing bipartisan things, whether or not you think they were good things. Market, it seems, was right to anticipate this type of thing as likely.
Same thoughts apply here, except this is an even more explicit reminder that while we had to choose between Hillary Clinton and Donald Trump, we can take comfort in the fact that it really can be strictly worse on both counts. What a vote to have to cast.
19. China officially abandons Covid Zero (70%): Scott holds, I sell to 60%.
This did happen right at the end. My comment hints at the final result: Abandon Covid zero, have infinite Covid, simply decide not to count any of it. It’s a bold choice. Given how hard China fought to stall this decision, it seems right to have been lower than 70%.
20. Fewer U.S. Covid-19 deaths in 2022 than in 2020 (80%): Scott buys to 90%, I buy to 95% as per above.
21. Additional booster shots of mRNA vaccines authorized for seniors (80%): Scott holds, I buy to 85%.
Happened, as I expected. Seems hard to imagine a scenario where this didn’t happen, if anything 85% feels a little low.
This came in at 7.1%, so it didn’t happen. Same mistake as the other inflation markets, so not much to say.
23. November 2022 year-on-year CPI growth is above 4% (70%): Scott sells to 50%, I sold to 40%.
My justification mentions the possibility of hedging in TIPS, so perhaps I wasn’t being as crazy as this now looks? Either way, more bad inflation take department.
24. The Fed ends up doing more than its currently forecast three interest rate hikes (60%): Scott holds, I buy to 70% saying it’s a dumb forecast.
Turned out to be a very dumb forecast. So this was a good prediction on its own, but the real question is, am I contradicting my inflation predictions here? I think no, this was a really dumb forecast and even with 4% inflation they would have hiked four times anyway, just less, and it was only my bad inflation takes keeping me as low as 70%. The market’s 84% seems much better and 70% was unacceptably low (despite the superforecaster average being 57%).
Once again, my mistake, and a rather important one.
26. Viktor Orbán loses power in Hungary (60%): Scott holds, I sell to 50%.
Market is listed as 26%, and, well, yeah. No credit for selling only to 50%, that’s terrible, my logic of ‘people predict these things every year and mostly they don’t happen’ was exactly right and I needed to be much lower, at most 40% and ideally less.
Market was 80% so turns out this was more likely. In the absence of time or willingness to do more research it’s not obvious which direction I should have moved.
I notice that given what I know now about the world, I would have been more inclined to sell even without doing research, saying that without knowing whether agreement was reached – Biden seems highly willing to fail to reach mutually beneficial agreements on trade and such things are generally a mess, and I should have known this at the time. Then when I looked at Wikipedia I realized this is a long standing dispute. So if I’d given this a few minutes and thought about it cleanly I would have sold down to at most 50% or so. And no, as far as I can tell we didn’t solve this.
This did not happen, Democrats had a good night overall and no one cared much about Covid so my logic was wrong. We all moved on. Was pretty silly to be more skeptical of Democratic chances than Matt was being, given his other probabilities.
I found broad support for the idea that I should do a 2023 predictions post, but I have less to work off this year in terms of source posts and also the events to predict mostly seem less interesting than usual. Commenters are invited to link to sources of others’ predictions to work off and of relevant prediction markets, or use other arguments or incentives, to try and convince me to do this. Currently I am leaning weakly against doing so.
The Substack version of this post has two polls at the end if you are interested.