yacy_search_server

Commit Graph

Author	SHA1	Message	Date
Michael Peter Christen	2d9e577ad0	replaced the custom robots.txt loader by the standard http loader	13 years ago
Michael Peter Christen	5f0ab25382	removed the option to prevent removal of & parts inside of the MultiProtocolURI during normalform computation because that should always be done and also be done during initialization of the MultiProtocolURI Object. The new normalform method takes only one argument which should be 'true' unless you know exactly what you are doing.	13 years ago
Michael Peter Christen	abebb3b124	added a crawl start checker which makes a simple analysis on the list of all given urls: shows if the url can be loaded and if there is a robots and/or a sitemap.	13 years ago

Author

SHA1

Message

Date

Michael Peter Christen

2d9e577ad0

replaced the custom robots.txt loader by the standard http loader

Michael Peter Christen

5f0ab25382

removed the option to prevent removal of & parts inside of the

MultiProtocolURI during normalform computation because that should
always be done and also be done during initialization of the
MultiProtocolURI Object. The new normalform method takes only one
argument which should be 'true' unless you know exactly what you are
doing.

Michael Peter Christen

abebb3b124

added a crawl start checker which makes a simple analysis on the list of

all given urls: shows if the url can be loaded and if there is a robots
and/or a sitemap.

3 Commits (ee9d7fd93d3859a668f93abb013eb847977567c9)