Like many people, I appreciate the information on Twitter/X (despite all of the waves of exodus), but I don’t necessarily like the toxicity or the time sink.
So I (and my buddy Claude Fable) made a digest app that gives you the day’s news and science discussions.
The “science” section is based on links to journal articles, ranked by engagement and classified by field.
The “news” section is based on keywords related to “straight” world-affairs news topics, like “war” or “election”, clustered by story and ranked by engagement. The idea is to cover the sorts of things that would be on the front page of a traditional newspaper, as opposed to entertainment or opinion. Keywords are translated into the top non-English languages on Twitter/X (Japanese, Spanish, Portuguese, Arabic, and Indonesian) and posts in any language are auto-translated into English.1
Summaries of tweets and their associated articles use Sonnet 5; classification uses Haiku 4.5. Links to original tweets and associated articles are included.
Both Science and News sections are based on advanced search queries using the API. There are no cherrypicked accounts being followed except some wire services like AP and Reuters.
News stories link to newspapers, news sites, and wire services where relevant, but they’re explicitly not limited to topics discussed by the “mainstream media”; social-media-only “citizen reporting” is included, though flagged as “unsubstantiated” when there are no links to professional news sources.
The repo is here if you want to take a look; please do comment with any bugs or feature requests if you’re interested in trying it out.
Twitter/X is very popular in Japan, so if Japanese stories are overrepresented, that’s why. Likewise, Chinese stories will be underrepresented because Twitter/X is blocked in China.
The digest is lovely, but is it worth 60$/month? I am thinking of creating something like this for newsletters in my inbox, but I would consider the cost, especially in the long-term.
How do you think about this? Or is it just the price to pay for learning/experimenting?
use the Anthropic Batch API (flat 50% discounting, and this is exactly the right application for it)
use prompt caching
Haiku for classification is probably overkill. consider GPT Luna. OpenAI has positioned it as the budget option. (maybe this new Jev product for filtering?)
use deterministic python to de-dup and perhaps require a link to a known good repository ... but this requires actual human work to build the allow/deny list
keep an eye out for free LLM API keys - opportunities come and go - e.g., right now Google is handing out free Gemini Flash keys that could replace GPT Luna as a classifier (the only one thing Gemini is good at)
twitter API costs are irreducible, but you should be able to otherwise cut LLM costs to yield a total somewhere in $25-35/month
This morning I was looking for hardware/models to run locally (for a different purpose). Given that this curation is not a low latency task, it could be done by some small local model.
OMG I love this idea! A couple wishes: 1: Important plots should show up in the body, without me having to look through the source. 2: Maybe have claude rank them? 3: I think I really want a weekly or biweekly digest - a day is just not that long. 4: Automated factchecking, in case anything's obvious enough that an LLM can find blatant evidence? If it works for community notes... (they recently got some automated notes, instructed to be conservative with the suggested notes, and I think I remember seeing some stats that suggested it's been doing a pretty good job) 5: Ability to see the twitter discussion, but with an automated filter for constructive comments
Like many people, I appreciate the information on Twitter/X (despite all of the waves of exodus), but I don’t necessarily like the toxicity or the time sink.
So I (and my buddy Claude Fable) made a digest app that gives you the day’s news and science discussions.
The “science” section is based on links to journal articles, ranked by engagement and classified by field.
The “news” section is based on keywords related to “straight” world-affairs news topics, like “war” or “election”, clustered by story and ranked by engagement. The idea is to cover the sorts of things that would be on the front page of a traditional newspaper, as opposed to entertainment or opinion. Keywords are translated into the top non-English languages on Twitter/X (Japanese, Spanish, Portuguese, Arabic, and Indonesian) and posts in any language are auto-translated into English.1
Summaries of tweets and their associated articles use Sonnet 5; classification uses Haiku 4.5. Links to original tweets and associated articles are included.
Both Science and News sections are based on advanced search queries using the API. There are no cherrypicked accounts being followed except some wire services like AP and Reuters.
News stories link to newspapers, news sites, and wire services where relevant, but they’re explicitly not limited to topics discussed by the “mainstream media”; social-media-only “citizen reporting” is included, though flagged as “unsubstantiated” when there are no links to professional news sources.
The repo is here if you want to take a look; please do comment with any bugs or feature requests if you’re interested in trying it out.
Twitter/X is very popular in Japan, so if Japanese stories are overrepresented, that’s why. Likewise, Chinese stories will be underrepresented because Twitter/X is blocked in China.