TL;DR We apply mechinterp techniques on VPT, OpenAI's Minecraft agent. We also find a new case of goal misgeneralization - VPT kills a villager when we force one to stand under some tree leaves. Abstract > Understanding the mechanisms behind decisions taken by large foundation models in sequential decision making...
TL;DR Different misgeneralized goals can appear based on training random seed alone, including outliers on a scale of 1 in 500. Abstract > We explore colour versus shape goal misgeneralization originally demonstrated > by Di Langosco et al. (2022) in the Procgen Maze environment, where, given > an ambiguous choice,...
We apply mechinterp techniques on VPT, OpenAI's Minecraft agent. We also find a new case of goal misgeneralization - VPT kills a villager when we force one to stand under some tree leaves.
... (read more)Understanding the mechanisms behind decisions taken by large foundation models in sequential decision making tasks is critical to ensuring that such systems operate transparently and safely. In this work, we perform exploratory analysis on the Video PreTraining (VPT) Minecraft playing agent, one of the largest open-source vision-based agents. We aim to illuminate its reasoning mechanisms by applying various interpretability techniques. First, we analyze the attention mechanism while the agent solves its training task - crafting a diamond pickaxe. The agent
Different misgeneralized goals can appear based on training random seed alone, including outliers on a scale of 1 in 500.
... (read 1083 more words →)We explore colour versus shape goal misgeneralization originally demonstrated
by Di Langosco et al. (2022) in the Procgen Maze environment, where, given
an ambiguous choice, the agents seem to prefer generalization based on colour
rather than shape. After training over 1,000 agents in a simplified version of the
environment and evaluating them on over 10 million episodes, we conclude that the
behaviour can be attributed to the agents learning to detect the goal object through
a specific colour channel. This choice is arbitrary. Additionally, we show how,
due to underspecification, the preferences can change when retraining the agents
using exactly the
Would "delta hedging" be useful here? It helps hedge long option exposure by shorting some amount of a stock.
For example, at the money calls generally have a delta of 0.5, so holding 100 at the money calls and shorting 50 shares makes you roughly neutral for small moves in the underlying asset.
Would probably require monthly rebalancing based on how many options you effectively hold and market moves. It also wouldn't work well if AGI happens at GDM and Google stock goes exponential ("volatility smile" problem).
non pdf arxiv link: https://arxiv.org/abs/2305.15324
For the four examples of
24-16=12, 53-25=25, 34-16=13, 63-17=16
is this the pattern?
ab-cd=ca
DreamerV3 is not a great example, as they use so many hacks to make the task easier that it barely counts as getting a diamond or Minecraft anymore. Action shaping, macro actions, instant block breaking, fake "bug fixing", all to get a diamond in 0.4% of episodes.
More info here: https://x.com/Karolis_Ram/status/1785750372394348632