LESSWRONG
LW

World Optimization
Personal Blog

15

Scraping websites currently free due to coronavirus

by emmab
14th Apr 2020
1 min read
3

15

World Optimization
Personal Blog

15

Scraping websites currently free due to coronavirus
9Kenny
3philh
3Derek M. Jones
New Comment
3 comments, sorted by
top scoring
Click to highlight new comments since: Today at 8:36 PM
[-]Kenny5y90

Why is this a good idea?

I understand that this is great opportunity to scrape otherwise-inaccessible sites, but whom is this intended to benefit? Is it just the people scraping?

Is this not malicious abuse of charitable giving?

Reply
[-]philh5y30

I agree with the question. I don't know if we should avoid this. But I think we should consider whether, for example, these sites are opening up with the implicit expectation that people will take advantage of it for covid-related purposes; and whether, if we just start scraping everything, that'll make them or similar sites less likely to open up in future.

Reply
[-]Derek M. Jones5y30

The ACM is offering free download of their articles: https://dl.acm.org/

Reply
Moderation Log
More from emmab
View more
Curated and popular this week
3Comments

Some sites are free due to the coronavirus and we should scrape them. Please list any temporarily-free websites you know of, such as https://www.uptodate.com/

To download sites I recommend https://github.com/ArchiveTeam/grab-site

Consider whether you want to use --no-offsite-links. Otherwise it will pull every linked page and its resources (and sometimes even embedded videos).