G2 Crawler News

Anyone who has read through this blog knows that the crawler has tended to crash fairly often. In recent times it was crashing to much to even continue running it. But rather than give up entirely I decided to write my own crawler. Five weeks later and g2paranha has emerged. To go along with the new crawler is a redesigned website written by kevogod. g2paranha has been designed to be distributed. Currently I run one crawler and Datz kindly volunteered to run another.

Over the next few weeks expect there to be a few bumps as any bugs get ironed out. I’ll also be rounding out the feature set provided by the original crawler. Right now that means adding country flags and keeping track of unique nodes. This last feature is the best estimate of the network size so until it is implemented the network counter won’t be put back on the front page. The leaf count on the history page gives an over count of the network size because each leaf can connect to multiple hubs.

I’ll try to keep the news updated as I progress.