The crawler website has moved to a new host and a new domain name. crawler.trillinux.org will now redirect to crawler.doxu.org.
The crawler itself will be hosted on the same machine as the website and the database. This will allow more interactive use of the data that wasn’t possible before.
Previously the crawler and database were being hosted by Datz on one of his personal machines. The website was being generated on my personal machine every 5 minutes by querying the database hosted on Datz’s machine and a static copy was then uploaded to the web host. Because of the relatively slow upload speed of residential internet connections this process was slower than it needed to be.
Post a Comment
You must be logged in to post a comment.