williawa

List of Excerpts from Mythos model card. Tried to include interesting things, but also included some boring to be expected things. I omitted some things that were too long. Also wanna note, 1. that this list of excerpts highlights "concerning" things above the rate at which they occur in the document. 2. I frequently say "Anthropic seems to think ..." or "their theory appears to be that ...", and this doesn't mean I think the opinion is unsubstantiated or that they are wrong, its just a natural way to phrase things for me. Capability Stuff Anthropic Staff Opinion About whether Mythos is a drop-in replacement for entry Research Eng/Scientist > We did an n=18 survey on Claude Mythos Preview’s strengths and limitations. 1/18 participants thought we already had a drop-in replacement for an entry-level Research Scientist or Engineer, and 4 thought Claude Mythos Preview had a 50% chance of qualifying as such with 3 months of scaffolding iteration. We suspect those numbers would go down with a clarifying dialogue, as they did in the last model release, but we didn’t engage in such a dialogue this time. Model hallucinates much less and also gets dramatically better scores on memory-loaded benchmarks like simple-qa. Not surprising considering this is a new big base model. Mythos displays dramatic above trend acceleration on ECI I don't have direct quotes to point to, but they do stress that this is sensitive to which benchmarks are included, and that many benchmarks you'd wanna use are saturated, so there's fairly few datapoints to base the fit on, which makes it higher variance. But they still seem pretty confident in the qualitative takeaway. Anthropic Seems to think this is due to human-research and not due to AI-based internal acceleration > The gains we can identify are confidently attributable to human research, not AI assistance. We interviewed the people involved to confirm that the advances were made with