Reading into VLM hallucinations using the Jacobian lens
Reading and editing the visual workspace of a vision-language model. Vision-language models hallucinate: ask one whether some object is in a picture and it will happily say yes whether it's there or not. Why is this? Using Anthropic's recent J-lens method I found that LLaVA-1.5-7B's internal state seems to register...
Jul 108