Commit Graph

58 Commits (573d03c7d79f617b2a0bc919e933a539098d89e3)

Author SHA1 Message Date
orbiter 28d5703f8a - fixed a bug in Robots.txt loader which could have caused that robots.txt files had been loaded from the same domain more than once
17 years ago
orbiter a6719dfd2b - refactoring of robots parser
17 years ago
orbiter 474e29ce4a added options to configure the 'corporate identity'-icons, the home page link and the greeting line from
17 years ago
orbiter 474659a71f - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order
17 years ago
orbiter b928ae492a some code-cleanup and possible speed enhancements in different core methods
17 years ago
danielr 7feae906aa - organize imports
17 years ago
orbiter dd75b3cabc - patch for bad profiles
17 years ago
orbiter 1689030ee8 refactoring: moved all crawler classes into their own package
17 years ago