Google thinks I'm a bot
s part of some assistance we’re giving to a research project in another department, a colleague and I have spent the past few days building a bibliography of earthquake research. After scraping all the usual academic databases, we turned to Google Scholar, which often picks up papers that the other databases don’t, particularly newly-published material that the others haven’t added yet, and the so-called “grey literature” – research papers written by government departments and other non-academic institutions. The search we ran turned up about 2,000 results, so I started downloading the citations a page at time into Zotero (a bibliographic system similar to Endnote) so we could compare them to the references we already had. Except I only got 40-odd pages in before Google popped up with a message telling me I’d been blocked because it looked like I was running a script (a bit ironic, really, that Google doesn’t like people running scripts over their data, when that’s what they do to everyone else’s websites…). I wasn’t, but apparently it’s not normal human behaviour to want to download thousands of references. So that’s it – I’m officially not human any more – Google says so, so it must be true 🙂