We Built a Tool to Protect Your Dataset From Simple Scrapers
9cousin_it
2Said Achmiz
1ProgramCrafter
10Said Achmiz
8Thane Ruthenis
2ProgramCrafter
1anaguma
3Thane Ruthenis
5winstonBosan
New Comment
Ugh, I hate Anubis. Any website with Anubis is spectacularly annoying or, in some cases, outright impossible to browse, if you have an even slightly outdated browser/system. It’s a really user-hostile solution.
If we are protecting against AI-building labs specifically, there are two outcomes:
- They do not care to set up a special way of data extraction for those protected sites. Then, we can set Anubis to whatever low difficulty that it does not hinder users.
- They do care and want to get the data behind. We lose.[1]
- ^
Because AI labs have a large number of... what they are called, again... massive parallel computation units... ah, GPUs, they have an advantage at proof-of-work. And if they choose to use them against humanity (a sleight of hand here, I admit), it will not go well.
Then, we can set Anubis to whatever low difficulty that it does not hinder users.
… unless it breaks entirely because someone has an old browser / weird browser / text browser / screen reader / NoScript enabled / etc., and the difficulty setting doesn’t matter entirely; and the result is that the user still can’t get through.
1Hmm. This approach relies partly on the AGI labs being cooperative and wary of violating the law, and partly on creating minor inconveniences for accessing the data which inconvenient human users as well. In addition, any data shared this way would have to be shared via the download portal, impoverishing the web experience.
I wonder if it's possible to design some method of data protection that (1) would be deployable on arbitrary web pages, (2) would not burden human users, (3) would make AGI labs actively not want to scrape pages protected this way.
Here's one obvious idea. It's pretty hostile and might potentially have net-negative results (for at least some goals), but I think it's worth discussing.
We could automatically seed the text with jailbreaks, spam, false information, and other low-quality/negative-quality training data, in a way that is invisible to users but visible to LLMs. Pliny-style emoji bombs, invisible/very tiny text, and other techniques along those lines, randomly inserted into the human-readable text between paragraphs or words.
How easy would it be for AGI labs to clean this up? Might be pretty easy, if we only have select few methods of hiding the text: then they can just automatically look for text within invisible tags/added via Unicode variation selectors, and remove it. But it might be possible to create a diverse, ever-growing family of text-hiding methods, such that static countermeasures don't work. Tasking an LLM with cleaning the document, instead of manually designed methods, might backfire, with the cleaner LLM itself getting jailbroken by the embedded cognitohazards.
Making the hidden text actually harmful seems easier: jailbreaks are invented faster than countermeasures against them, I think.
Ideally, the whole setup would be continuously updated: instead of individual writers having to inject this stuff on their own, there would be a centralized public API or GitHub repo which web developers could use, embedding this functionality into websites. This centralized API/repo can then be continuously updated with new jailbreaks and counters to whatever countermeasures AGI labs come up with (AdBlock-style).
Again, it's obviously pretty hostile, but if paired with a canary string, any actor that decides to ignore the canary string and scrape the page anyway deserves what they get.
Any obvious reasons this is a bad idea which I'm missing? I guess the obvious failure mode is people deploying this without the canary string, meaning even cooperative AGI labs might accidentally train on the data poisoned this way. If the goal is to prevent training on bad data (because of e. g. misalignment concerns), that's obviously counterproductive.
You have to make that poison inactive in accessibility cases, or a person using screen reader would hear all that. However, if a correctly configured screen reader skips the invisible data, then labs will just use it (assuming they can be bothered with cleaning dataset at all).
Also, training-time jailbreaks are likely quite different from inference-time jailbreaks. The latter will tend to hit Operator-style stuff harder.
This seems like a great idea! However, I think it might degrade the usefulness of the dataset, especially if it’s meant to later be used to evaluate LLMs since any jailbreaks etc. would apply in that setting as well. If you provide utilities to clean up the text before evaluation, these could be used for scraping as well.
Yeah, I guess the use-case I had in mind is generally people who don't want LLMs trained on (particular pieces of) their writing, rather than datasets specifically.
