Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models

TL;DR: We discovered the possibility of using steering vectors from text-only models to enhance visual reasoning in multimodal LLMs (MLLMs). The technique is simple: extract textual representations for concepts like "spatial relationships" and "counting" from the LLM backbone and then apply them to the vision-language model's hidden states. This steering not only changes how the model understands visual content but also leads to meaningful improvements - 15.8% on spatial reasoning tasks and 34.2% on counting tasks for example, which suggests that these models maintain some unified representations that bridge text and vision.

The Core Insight: Text Steers Vision

Here's something that surprised us: you can make a vision-language model better at visual understanding by steering it with vectors derived purely from text.

Many multimodal large language models (MLLMs) like PaliGemma^[1] and Idefics^[2] are built by taking a pre-trained text-only LLM backbone and teaching it to process images. We wondered: do these models preserve enough of their original text representations that we could use text-only steering techniques to improve their visual reasoning?

The answer appears to be a resounding yes.

The Mechanics of Steering Vectors

Let's clarify exactly what we mean by "steering vectors" and how they work:

What Is a Steering Vector?

A steering vector is a direction in the model's high-dimensional activation space that corresponds to some concept or behavior^[3]^[4]. Think of it as a "conceptual compass" - it points toward the neural representation of ideas like "spatial relationships" or "counting."

How Are Steering Vectors Applied?

The application is surprisingly straightforward. During inference, at a specific layer $ℓ$ , we modify the model's hidden states:

$h^{'} (ℓ) = h (ℓ) + α \times v (ℓ)$

Where:

$h (ℓ)$ = original hidden states at layer $ℓ$
$v (ℓ)$ = our steering vector for that layer
$α$ = steering strength (how much to intervene)
$h^{'} (ℓ)$ = modified hidden states

We can also choose to where to apply this steering:

Image tokens only: Steer only on hidden states of image tokens
Text tokens only: Steer only hidden states of text tokens
Both: Steer on hidden states of all tokens except special tokens like <bos>

An important notice here is that we never steer the output token i...

Posts

Wikitag Contributions

Comments