We just ran 23 million queries of the World Bank's website. Technically, a piece of computer code did the work, occupying a PC in an empty cubicle in our office for about 9 weeks, gradually sweeping up nearly every bit of information available in the World Bank's global database on poverty and inequality, known as PovcalNet.
Why did we go through all this trouble? The parochial answer is that we wanted to use the data for our own research and got frustrated with the World Bank website designed to dole out the data in bite-size chunks, rather than the large swaths researchers might want. After a somewhat, erm, delicate negotiation with colleagues at the World Bank, we've just posted the resulting paper, data set, and code online, so data-oriented readers can now download the full income ... [view whole blog post ]