Twitter Twitches

[-]Simon Berens2y140

Bloomberg reported 2 weeks ago that Twitter resumed paying Google Cloud: https://www.bloomberg.com/news/articles/2023-06-21/twitter-resumes-paying-google-cloud-patching-up-relationship

[-]GeneSmith2y117

Twitter owner Elon Musk continues to be surprised by how Twitter works. Last week he learned that their code ‘shadowbanned’ any account with low reputation score, preventing them from trending, and the calculation was based on ‘how many times were you reported’ so every big account got shadowbanned.

I roll to disbelieve. If this were the case, virtually zero tweets by large accounts would ever trend. There must either be some additional code that overrides this shadowban or some list that nullifies its effects.

It's still hilarious that it was in the codebase at all.

I don't know what to think about Musk not paying the GCP bill. He obviously has the money. Does he really not want to sell more Tesla stock that badly? Why would you risk a 44 billion dollar investment involving a ton of your own money (not to mention that of many of your friends) over a 1 billion dollar bill?

[-]Ben Pace2y110

After a ~year of not reading Twitter, I coincidentally returned to reading Twitter the exact day that the rate limits were added. I'll say, this externally imposed daily limit is precisely the feature I've wanted in social media, and each day I've been glad of it.

[-]Edmund Nelson2y20

The number is way too high for that. I use twitter almost an hour a day (way way too much time) and I don't hit the rate limit

[-]Ben Pace2y20

I was initially using a brand new account and hit the rate limits in under an hour. Perhaps the rate limits were different for that account. I notice I didn't hit it yesterday (after I managed to login to my old account).

[-]jimrandomh2y91

We know that Twitter was in multiple datacenters, including their own datacenter, plus Google Cloud, plus AWS. We know that they were trying to get out of these contracts, possibly using default as a negotiating tactic. We know that their technical-debt level was extraordinarily bad. After joking that they probably had a history of their engineers being spies who obfuscating things in order to make it easier to hide their sponsors' backdoors, I thought about it a bit more and decided that was probably literally true. They were (and probably still are) using orders of magnitude more computing resources than running a service like Twitter ought to take, if it was well engineered. And we know that they started having capacity problems, with timing that seems to line up suspicioously with what we might infer is a monthly billing cycle.

But there are a bunch of very different interpretations of this, which we can't easily distinguish:

They defaulted on their bill with GCP or AWS, had a bunch of servers shut off, and discovered they no longer had sufficient capacity;
They declined to renew or scaled back their expenditure with GCP or AWS, thinking that their remaining compute resources were adequate, but they weren't;
They declined to renew or scaled back their expenditure with GCP or AWS, and the remaining capacity was adequate, but they had problems with the migration that would be easier to fix given extra capacity-headroom;
They're using the same servers as before, and had problems with some crawler/scraper changing its activity patterns

One thing I can say, from running LW, is that from a capacity perspective crawlers are a much bigger issue for websites than you'd naively expect. (LessWrong does have rate limits per-IP-address, they're just high enough that you won't ever hit them under normal usage.) So even if there was a capacity-reduction related to their supply of hardware, it may still be the case that most of their capacity was going to scrapers, and to try to limit the scrapers as a way of regaining capacity. It seems fairly likely that the rate-limiting option was set up in advance as a quick-response option for any sort of capacity issue (including capacity issues created by things like a developer accidentally deploying slow code, or surges in usage).

The main problem with crawlers is that their usage patterns don't match those of regular users, and most optimization effort is focused on the usage patterns of real users, so bots sometimes wind up using the site in ways that consume orders of magnitude more compute per request than a regular user would. And some of these bots have been through many iterations of detection and counter-detection, and are routing their requests through residential-IP botnets, with fake user-agent strings trying to approximate real web browsers.

As for the shadowbanning thing--the real bug was probably a bit more subtle than the tweet-length description, but the bug itself is not surprising, and given the high-technical-debt codebase, probably not nearly as stupid as it sounds. Or rather: the effect may have been that stupid, but the code itself probably didn't look on cursory inspection like it was that bad. Ie, I would assign pretty high probability to that code containing an attempt to normalize for visibility that didn't work correctly, or an uncompleted todo-item to make a visibility correction score. A codebase like Twitter is going to have bugs like this, they can ony be discovered by skilled programmers doing forensic investigations, and executives will only know about them during the narrow time window between when they're discovered and when they're fixed.

[-]siclabomines2y*10

The main problem with crawlers is that their usage patterns don't match those of regular users, and most optimization effort is focused on the usage patterns of real users, so bots sometimes wind up using the site in ways that consume orders of magnitude more compute per request than a regular user would.

And Twitter has recently destroyed his API, I think? Which perhaps has the effect of de-optimizing the usage patterns of bots.

[-]RomanS2y10

And some of these bots have been through many iterations of detection and counter-detection, and are routing their requests through residential-IP botnets, with fake user-agent strings trying to approximate real web browsers.

As someone who has done scraping a few times, I can confirm that it's trivial to circumvent protections against it, even for a novice programmer. In most cases, it's literally less than 10 minutes of googling and trial & error.

And for a major AI / web-search company, it could be a routine task, with teams of dedicated professionals working on it.

[-]RomanS2y40

I think the both explanations can be true at the same time:

Twitter is refusing to pay a bill to Google
Twitter is severely abused by data scrapers.

One likely scenario is where Google itself is a main culprit.

E.g. Elon learned that Google is scraping twitter data on industrial scale to train its AIs, without paying anything to Twitter. This results in massive infrastructure expenses for Twitter, to be paid to... Google. Outraged Elon stormed into the Alphabet headquarters, but was politely asked to get lost. Hilarity ensues.

LESSWRONG
LW

LESSWRONG
LW

34

34

34

Well You See What Happened Was…

The Plan

Oh No, That’s Not the Reason, Except It Is

People Dealing With It

Other Twitter News

General Twitter Outlook Going Forward