I think even if the model wasn't specifically trained for geolocation, it's a reasonable assumption that metadata from photos (which often includes geo data) somehow gets passed to the models and this created a huge annotated dataset of (geo, photo) pairs during training for stuff like searching Google images.

I mean this is a summary of a talk I didn't see so I want to reserve some judgement. But at the same time I just can't imagine being in a situation where I don't want to use the word signal because the other person might think I'm talking about cell signal, so instead I bust out "kodo"

In general, I think we should be pretty wary of taking basic ideas and dressing them up in fancy words. It serves no purpose other than in-group signaling.

That makes sense. However, I do see this as being meaningfully different from evals or mech interp in that to make progress you really kind of want access to frontier models and lots of compute/tooling, so for individuals who want to prioritize this approach it might make sense to try to join a safety team at a lab first.

How are these ideas different from "signal" and "noise"?

Do you think building a new organization around this work would be more effective than implementing these ideas at a major lab?

I think this might be a result of o-series being trained in a non-chat setup for most of the CoT RL phase and then being hamfistedly finetuned right at the end so it can go into ChatGPT, which just makes them kind of bad at chat and so o3 gets confused when the conversation has a lot of turns. Retraining it to be good at multi-turn chat with separate reasoning traces would probably just be super expensive and not worth the squeeze. (this is just a guess)

No, that is not the definition of a prior. There are priors which imply an expected number of buses, and priors that don't. If you select a prior that doesn't, you can still get a meaningful posterior distribution even if that posterior distribution doesn't have a real-valued mean.

No, that's not the posterior distribution--clearly, the number of buses cannot be lower than 1546, but that distribution has material probability mass on low integers. I'm not quite sure how you got that equation.

But regardless, I think this shows where we disagree. That prior has mean 2... that's a pretty strong assumption about the distribution of n. If you want to avoid that kind of assumption, you can get posterior distributions but not a posterior expectation.

I'm not disagreeing with that categorically--for many priors the posterior distribution is well defined. But all of those priors carry information (in the information theoretical sense) about the number of buses. If you have an uninformative reference prior, your posterior distribution does not have a mean.

You can see the sketch of this proof if you consider the likelihoods of seeing the bus for any given n. If there are 1546 buses, there was a 1/1546 chance you saw this one. If there were 1547, there was a 1/1547 chance you saw this one. This is the harmonic series, which diverges. That divergence is the fundamental issue that's going to cause the mean to be undefined.

You can't make claims about the posterior without setting at least some conditions on what your prior is--obviously, for some priors the posterior expectation is well-defined. (Trivially, if I already think n=2000 with probability 1, I will still think that after seeing the bus.) But I claim that all such priors make assumptions about the distribution of the possible number of buses. In the uninformative case, your posterior distribution is well-defined (as I said, it's a truncated zeta distribution) but it does not have a finite mean.

The intuition is that if we both saw bus 1546, and you guessed that there were 1546 buses and I guessed that there were 1547, you would be a little more likely to be correct but I would almost certainly be closer to the real number.

The Bayesian update isn't generally well-defined because you get a divergent mean. Your implicit prior is  1/n which is an improper prior. This is fine for deriving a posterior median, which in this case happens to be about 3,100 buses, and a posterior distribution, which in this case is a truncated zeta distribution with s=2 and k=1546. But the posterior mean does not exist.