This wouldn't really solve much of the problem though, since ETFs are still pretty expressive. For example, when they have a sense for whether an important clean-energy bill will pass or fail, they could buy/sell a clean-energy-tracking ETF.
Some ETFs are pretty high-weight Nvidia, so it would be pretty easy to still trade it indirectly, albeit a little bit less efficiently.
And honestly even the S&P500 will still move a lot based on various policy outcomes.
Just so you know, this is still missing on your personal site.
Also the image here doesn't exist on your personal site's post.
Thanks for writing all these wonderful resources Neel!
You probably also want to do some kind of normalization here based on how many total posts the user has upvoted. (So you can't just i.e. upvote everything.) (You probably actually care about something a little different from the accuracy of their upvoted-as-predictions on average though...)
Here's a good/accessible blog post that does a pretty good job discussing this topic. https://ericneyman.wordpress.com/2019/09/17/least-squares-regression-isnt-arbitrary/
I think that this is true of the original version of alphastar, but they have since trained a new version on camera inputs and with stronger limitations on apm (22 actions/5s) (Maybe you'd want some kind of noise applied to the inputs still, but I think the current state is much closer to human-like playing conditions.) See: https://www.deepmind.com/blog/alphastar-grandmaster-level-in-starcraft-ii-using-multi-agent-reinforcement-learning
In other words, we should be telling children 'be careful of roads/cars' (including on Halloween) Not 'be careful of Halloween'
I agree with the post, but I will point out that you really do need to emphasize the utility per micromort here. If you keep your utility constant, it is the total risk that matters. Just like if you were going to go on a long car ride tomorrow (on safer-than-usual roads, but not enough to outweigh the total driving) and someone points out you're much more likely to die than usual - sure, you can point out 'ah yes, but the chance I die per-mile is lower than usual!' but that's not the right reference point if your utility isn't a function of the driving-amount.
All that said, the total number of deaths is only ~double on Halloween? That feels so insane, roads must be SO much safer than usual.
Here are some objections I have to your post:
How are you going to specify the amount of optimization pressure the AI exerts on answering a question/solving a problem? Are you hoping to start out training a weaker AI that you later augment?
If so, I'd be concerned about any distributional shifts in its optimization process that occur during that transition
If not, it's not clear to me how you have the AI 'be safe' through this training process.
At the point where you, the human, is labeling data to train the AI to identify concepts with measurements/feature - you now have a loss function that's dependent on human feedback, and which, once again, you can't specify in terms of the concepts you want the AI to identify. It seems like the AI is pretty incentivized to be deceptive here (or really at any point in the process).
I.e. if i's superintelligent and you accidentally gave it the loss function 'maximize paperclips', but it models humans as potentially not realizing they gave it this loss function, then I think it would act indistinguishably from an AI with the loss function you intended (at least during this stage of training you outline).
Even if, say, it does do things at first that look like things a paperclip maximizer would try to do, instead of whatever you actually want it to do (label things appropriately) - say, it tries to get a human user to upload it to the internet or something, but your safe-guards are sufficiently strong to prevent things like this - then I think as you train away actions like this, you're not just training it to have better utility functions or whatever, but you're training it to be more effectively deceptive.
Unfortunately, comparing the returns isn't a great way of evaluating the portfolio compared to the S&P 500. You should really be comparing their Sharpe ratios (or just annualized tstat). If you have, for example, 5% annualized returns on $x in excess of the risk-free rate, you can just double your pnl to 10% by borrowing $x more money and investing it (assuming you can borrow at a competitive rate). Why not do that? Well, you'll also have more variance in your portfolio. Probably what you really care about is risk-adjusted returns.
The most common way to evaluate this is to compare the (daily mean returns)/(daily stdev returns), where maybe you adjust the first thing by the rate you can borrow money at.
(Eyeballing it, SPY was probably like 1.5-2x as good as the other portfolio by this metric.)
Happy to explain more if this is confusing or you're curious and have other questions.
Edit: I see your other post/comment now that has a Sharpe ratio and portfolio that looked like it outperformed this one; maybe this isn't new/interesting or useful, but I'll leave it up in case someone else finds it useful.