9 Commits (ac9540dfb6713b46df543907337a125151716a4a)

Author	SHA1	Message	Date
Michael Peter Christen	0833937c1c	better balancing and duetime-cumputation also for no-delay intranet ... hosts	13 years ago
Michael Peter Christen	c326aa8f67	disabled writing new entries to crawl stacks to prevent that a domain ... with many documents block refreshing of the crawl queue	13 years ago
Michael Peter Christen	c25d7bcb80	- added concurrency for robots.txt loading ... - changed data model for domain counter	13 years ago
Michael Peter Christen	2d9e577ad0	replaced the custom robots.txt loader by the standard http loader	13 years ago
Michael Peter Christen	a33e2742cb	- removed unnecessary synchronized and deadlock in crawler ... - removed problem with monitoring object on Balancer.wait - added missing user agent settings	13 years ago
orbiter	8952153ecf	update to Balancer algorithm: ... - create a load list from the current list of known hosts - do not create this list for each Balancer.pop access - create the list from those hosts which have a zero-waiting time - select 1/3 from that list which have the most urls waiting - get hosts from the wainting list in random order - fixes for some delta-time computations - always load all urls from hosts which have never been loaded before	13 years ago
Michael Peter Christen	1533bfd63b	refactoring	13 years ago
Michael Peter Christen	8219a445f3	refactoring	13 years ago
Michael Peter Christen	00c1c777fa	refactoring	13 years ago