G2 Crawler News

So it’s been nearly a year since the last post. For the most part the crawler website has stayed the same with only the occasional bug fix as necessary.

However I have continued to experiment with new ideas and I am taking advantage of the more powerful computer to open up new possibilities. One such experiment is the G2 Network Map. Two years ago I created a small time lapse video showing the hubs on the network. I continued with that theme by creating a much higher resolution map of the network. Rather than doing a time lapse video it is instead and overlay of all hubs and leaves that have been seen in the last 2.5 years.

Take a look at the G2 Network Map.

Double click to zoom into an area of the map. Click and drag to pan around. In the upper right you can choose which image to look at. Right now the choice is to see all hubs or all leaves for the last 2.5 years. The color shows the number of IPs seen in a particular block. The scale goes from black and blue being the least number of IPs to orange and red being the most IPs. Yellow, green and the other colors fall somewhere in the middle. IP blocks are labeled based on who owns those IPs. RIPE (Europe), ARIN (North America), APNIC (Asia), LACNIC (Latin and South America), and AfriNIC (Africa) are known as Regional Registrars. They have been allocated large blocks of IPs which they then distribute to companies in their geographic regions. If you zoom in further you can see blocks allocated to individual companies. I did all of the individual company blocks by hand and I mostly focused on areas where there was a lot of activity on the G2 network. So the main area I focused on was Europe in the lower left. But also some of LACNIC in the lower right and a few ARIN blocks in the upper left. If you see a white box around an area but the name doesn’t show up then zoom in a few more levels and the name should appear. In addition to showing the name the IP block is also shown.

To make the network map interactive I took advantage of a piece of software called OpenLayers. Normally it is used for showing geographic maps. That is why on the initial view half of the page is missing. Geographic maps are done in a 2:1 ratio and the IP map is 1:1.

To create the images I used the same ipv4-heatmap that was used to create the images for the video. But instead I rendered them at extremely high resolution. Because web browsers don’t make it easy to view images that are 16000×16000 I needed an alternative method. That’s when I decided to use OpenLayers, the same technology that is used for showing maps online. The way OpenLayers works is that it loads small images called tiles, which are pieces of the bigger image. I used a set of perl scripts and ImageMagick to chop up the high resolution images into tiles that can then be loaded by your browser as necessary when you pan around and zoom in.

I hope you like it. Let me know if you have any suggestions.