theli
|
f8ad65eae1
|
*) First trial implementation of robots.txt support
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@674 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
allo
|
9300689dde
|
bugfix *gr*
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@662 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
allo
|
ebc39a7b9a
|
minor fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@659 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
allo
|
f90f699ab1
|
missing package line.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@655 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
|
allo
|
06a451768f
|
a simple robotsParser.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@652 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
20 years ago |
