Evaluating LLMs by their writing abilities is hard for several reasons. one that comes to mind is that LLM writing abilities seem dependent on post-training rather than pre-training; that is, the base models are probably a lot better at writing than the models we actually see, but their good writing tendencies are trained out of them in the effort to make them friendly, helpful assistants. That being said, I do take writing abilities somewhat seriously as a gut-check eval and haven’t gotten around to testing the new ones, so this is interesting to know!