6 Commits (bcc623a84320564dcacf2830288713620ebae230)

Author	SHA1	Message	Date
Michael Peter Christen	bcc623a843	refactoring of load_delay: this is a matter of client identification	12 years ago
Michael Peter Christen	5878c1d599	- refactoring of log to ConcurrentLog: ... jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger	12 years ago
Michael Peter Christen	8f2d3ce2f9	reduced locking situation in crawler: shifted synchronized location and ... reduced time-out of robots.txt load limit	12 years ago
Michael Peter Christen	2d9e577ad0	replaced the custom robots.txt loader by the standard http loader	13 years ago
Michael Peter Christen	5f0ab25382	removed the option to prevent removal of & parts inside of the ... MultiProtocolURI during normalform computation because that should always be done and also be done during initialization of the MultiProtocolURI Object. The new normalform method takes only one argument which should be 'true' unless you know exactly what you are doing.	13 years ago
Michael Peter Christen	abebb3b124	added a crawl start checker which makes a simple analysis on the list of ... all given urls: shows if the url can be loaded and if there is a robots and/or a sitemap.	13 years ago