I’ve been looking at TypeSafe’s Jev model and I still don’t really get what it’s doing. Could somebody explain?

At first I thought this was basically old-school ML with better marketing. If the output is just “yes/no” or a probability, then we already have XGBoost, classifiers, etc. But the important difference is that the input can still be arbitrary language/code, and the question itself can also be arbitrary. So it’s not “train a churn classifier”; it’s more like “give me this giant messy state, then answer this new question about it.”

That does seem meaningfully different.

I can sort of see how removing autoregressive output could plausibly make it much faster. If you only need a few probabilities, having an LLM generate a whole chain of tokens is obviously wasteful. You could imagine a model doing a lot of internal computation and then directly emitting a few scores. I don’t really believe this, but I can see it being plausible at least.

But I’m still very skeptical of the idea that this gets you around the hard part.

My intuitive model of LLMs is that pretraining is where most of the raw, general intelligence comes from. Posttraining makes it useful, safe, better at specific tasks, etc., but the base semantic/world model comes from massive-scale pretraining. So if Jev can understand arbitrary code and arbitrary natural-language questions well enough to match strong LLM monitors on some tasks, where did that ability come from? The tasks it can do are pretty primitive, but still they seem like they should require some sort of higher level reasoning.

I don’t see how you avoid scaling. Even if the architecture is much better suited to this task than an autoregressive transformer, you still need to learn English, code, abstractions, weird semantic relationships, etc. That seems expensive in a very fundamental way. Maybe TypeSafe found a much better architecture, but I have a strong prior against “small startup discovers way to get frontier-level primitive intelligence at 1/100 the compute.”

The backdoor-detection result is the thing that made me update. One yes/no question gets AUROC 0.976 and catches 90% of backdoors at 2% FPR, apparently comparable to o3-mini with CoT in the cited control paper, while costing basically nothing. That is not what I’d expect from a dumb classifier.

So clearly something real is happening, but I just have no idea what. Or does nobody know, and the company is just keeping this a secret? Okay, they have an API and they say they “can’t prove it isn’t subsidized”. That’s a pretty funny joke. For example a hypothesis is that this is some subsidized scam? Or maybe this has some real progress but also subsidized heavily? Or another hypothesis is that this is all real, and the sort of problems it can do well on, what they call System 1, is actually a whole lot more like chess than general reasoning on a fundamental level. But I’m so confused, I don’t have high probabilities assigned to anything.

Okay, just for the sake of it, let’s say this really works. The black-box aspect bothers me.

With chain of thought, the reasoning trace is not perfectly faithful, but sometimes it is informative. You can often see what the model noticed, what intermediate claim it made, where it went wrong, etc. If Jev just outputs 0.93, then that entire internal process is hidden. Like, LLMs seem like blessings in comparison?

So, I’m genuinely confused what this is doing and how I should update, and if this really really works, I think this is a very bad sign. Like we should 100% stop this right now? More than likely I’m just confused.