Comments on: Scraping Google To See What Happens

By: Judy Salter

Judy Salter — Fri, 11 May 2012 23:16:00 +0000

Google exists because it scraped everything and everyone without caring if they allow it or not.

Now they are large and don’t want to be scraped, that just sucks imho.
But anyway, it is possible to scrape Google. http://google-rank-checker.squabbel.com is an open source project that is able to scrape millions of hits without issues.Might be worth to add to the blog.

By: MetaSearch

MetaSearch — Wed, 22 Nov 2006 20:18:29 +0000

How do metasearch engines survive? Don’t they just scrape results?

By: Vitaliy

Vitaliy — Wed, 25 Oct 2006 15:25:46 +0000

I like Google and have the toolbar, but please scraping Google. I can’t find spyware.

By: Steve Crcker

Steve Crcker — Fri, 15 Sep 2006 05:27:09 +0000

Perhaps in a legalistic sense Google has not "stolen" anything. But I understand Daniel to be saying, essentially, that Google has profited hugely from the free. volunteer and idealistic efforts of others. And has given nothing back to those whose efforts have made Google possible. Now this last point is certainly debatable. One could argue that by providing search capability, access is facilitated to many small sites which would otherwise go unnoticed. But wait, it's not that simple. The growing comercialization of the search industry, led by Google, has created a situation where noise increasingly drowns out information in search results - at least if you are a user searching for information for its own sake - and not for some buying opportunity. Commercialization of the search function virtually insures that paid results will tend to dominate and crowd out material put on the Web as a volunteer "labor of love" - the very kind of material which caused many of us to gravitate toward the Internet in the first place.

Just a thought,
-Steve

P.S. No relation to the Steve Crocker who helped invent the Internet.

By: pb

pb — Wed, 12 Jan 2005 17:25:24 +0000

Looks like we'll need to go to Microsoft to scrape search results:
http://blogs.msdn.com/msnsearch/archive/2005/01/11/351064.aspx

Google used to offer this simple, RESTful API method but ditched it for its current, cumbersome SOAP-based APIs.

By: Miles Barr

Miles Barr — Wed, 12 Jan 2005 11:10:29 +0000

In the UK in addition to regular copyright law we also have the Database Act:

http://www.legislation.hmso.gov.uk/si/si1997/1973032.htm

This allows (among other things) someone to compile of a database of public domain works and have IP rights on that database. This would make what Scroogle does illegal in the UK. Is there something similar in the US?

By: Adam

Adam — Wed, 12 Jan 2005 04:54:10 +0000

Mr. Brandt is an idiot, and a rude idiot at that.

1) Don't want your sites on the Web spidered by Google? Put up a simple robots.txt file. Voila.

2) Don't like Google? Don't use it. Mr. Brandt's effort to use Google's bandwidth, R&D, and processing power while stripping their ads is akin to the selfishness of BugMeNot's childishly gleeful facilitation of helping people access registration-required news sites while violating the terms of those sites. Same thing: Don't want to register at the New York Times? Don't read their articles.

3) There are many things I don't like about Google. Their blog is typically fluffy and uninformative, their hiring practices are tiresome and inefficient, and they've done a surprisingly poor job of integrating their various services. With that said, though, Google's done a LOT of stuff right... and most of it quite unselfishly and without being evil. If only the same could be said of Andrew O at The Register (another annoying twit) and Daniel Brandt.

By: brian

brian — Wed, 12 Jan 2005 00:10:23 +0000

But, but…well Google and other search engines by the strictest definition of copyright law (an affirmative right) could be construed to have cached web pages and images of others without the explict permission of the owners.