G2 Crawler News

You may have noticed that the number of “Unknown”/”??” countries has been increasing since the graphs came back in June. This is because the country is determined by using MaxMind’s GeoLite Country database which maps IP addresses to countries. The version the crawler was using hadn’t been updated since September 2005 and many new IP blocks have been added since then. So the database is now current and hopefully the country stats will more accurately reflect where people really are. Some of the countries which should see increases are Australia, Costa Rica, India, Japan, Korea, Mexico, New Zealand, and Thailand.

Today a new best hub uptime was established beating the old one of 209h 10m 45s. This is because for some reason unknown to me the crawler has decided to run for over a week continuously without dying. I did however check system logs recently and it does look like it may be crashing because of a memory leak. With any luck magic fixed it and it will now run smoothly forever.