0fe8be7981
enhaced data structures for balancer and latency computation which
...
should produce a bit better prognosis about forced waiting times.
13 years ago
0833937c1c
better balancing and duetime-cumputation also for no-delay intranet
...
hosts
13 years ago
c25d7bcb80
- added concurrency for robots.txt loading
...
- changed data model for domain counter
13 years ago
2d9e577ad0
replaced the custom robots.txt loader by the standard http loader
13 years ago
a33e2742cb
- removed unnecessary synchronized and deadlock in crawler
...
- removed problem with monitoring object on Balancer.wait
- added missing user agent settings
13 years ago
5f0ab25382
removed the option to prevent removal of & parts inside of the
...
MultiProtocolURI during normalform computation because that should
always be done and also be done during initialization of the
MultiProtocolURI Object. The new normalform method takes only one
argument which should be 'true' unless you know exactly what you are
doing.
13 years ago
00c1c777fa
refactoring
13 years ago
Michael Peter Christen