My instinctive meaning of "impression" is more what we mean when we say "first impression," meaning the System 1 reaction. I believe that Katja's intention here is that impression means your own considered System 2 reaction before taking into account the opinions of others, to avoid anchoring and separate the beliefs you came to on your own from those you got by factoring in the beliefs of others.
I can see value in keeping track of impressions in both senses, First impressions are a potentially valuable thing people put a lot of weight on, that has highly uncertain accuracy that likely varies a lot based on situation and user. System 2 reactions ("second impressions?") also seem valuable, if they can be successfully isolated. In some cases they can and I'll even be explicit about this and say something like "my own fair value was 20%, but given your fair value, I now think 40%" and tracking that 20% value's accuracy seems good.
One worry I instinctively have is that we're still incorporating a lot of other people's beliefs and models when we do our initial estimate. There's no alternative to that. So, when I give my 20% estimate that includes all sorts of estimates and beliefs I've gotten from other people about similar or logically related questions in the past. Often in some sense I have a truly unanchored belief that's different from that, too. Another thing that happens is I think something like "20%, but my model of you says you'll think 60%, so my current belief is 40%," and now things are quite muddled.
The more of these "anticipation of the wisdom of others" steps you go through, the more accurate your impression record likely becomes, but at some point it also becomes less useful. This seems like a big problem?
It seems to me that to a large extent impressions can be framed in terms of vague predictive statements with no explicit probabilistic or causal content, influenced much more by external factors than reasoned beliefs.
"He seemed nice" corresponds to "X% chance that if I met him more often, we would continue getting along well".
"That sounds crazy" corresponds to "I can't really tell you why but I find that rather improbable".
If I am right about this, the first and main step would be lto earn how to turn impressions into explicit probabilistic statements which are easy to test. Keeping track of their status should not be any different from anything else then.
In order to offer suggestions I'd like to understand your view better.
Would you agree that, in your terminology, the relation between an impression and a belief is analogous to that between a crux and an crucial argument? The former relates to the actual causal reason you think something is true, the latter relates to your socially-adjusted preferred excuse for thinking it is true.
In my terminology, 'impression' is your own sense of what seems true before taking into account other people's views (unless another person's view actually changes your own sense) and 'belief' is what you would actually bet on, given that you are not vastly more reliable than everyone with different impressions.
For example, perhaps my friend is starting a project, and based on talking to her about it a bit I feel like it is stupid and will never work. But several other friends who work on similar projects are really excited about it. So I might decide that it is probably going to be successful after all, though it doesn't look exciting to me. Then my impression of the project was that it was unpromising, but my belief is that it is promising.
It is good to separate impressions from beliefs.
It is good to keep track records.
Is it good to keep separate impression and belief track records?
My default guess would be ‘a bit, but probably too much effort, since we hardly manage to keep any track records.’
But it seems maybe more than a bit good, for these reasons:
Having good first impressions, and being good at turning everyone’s impressions into a good overall judgment might be fairly different skills, so that some people are good at one and some are good at the other, and you get a clearer signal if you separate them.
We probably by default mostly learn about beliefs and not impressions, because by assumption if I have both and they are different, I suspect the impression is wrong, and so will make me look worse if I advertise that I hold it.
Impressions are probably better than beliefs to have track records for, because the point of the track records is to know how much to weight to give different sources when constructing beliefs, and it is more straightforward to know directly which sources are good than to know which aggregations of sources are good (especially if they are mostly bad, because nobody has track records).
As in, perhaps we mostly keep belief track records when we keep track records, but would do better with impression track records. What would we do if we wanted to keep impression track records instead? (Do we already?)