Unnamed repository; edit this file 'description' to name the repository.
git clone git://
Log | Files | Refs | README | LICENSE

commit 3df47bfd42a8ea5142333e0cf5fa652752a04363
parent a38b003bd5b90529d98350f162e7403f25353b7e
Author: René Wagner <>
Date:   Wed, 27 Jan 2021 13:35:54 +0100

add "/robots.txt" route to

It's a hard coded approach to serve a robots.txt to other crawlers.
No crawler may access /add-seed & /threads and all relevant virtual agents
may not access /search and /backlinks

Mserve/ | 4++++
1 file changed, 4 insertions(+), 0 deletions(-)

diff --git a/serve/ b/serve/ @@ -57,6 +57,10 @@ gus = GUS() def status(request): return Response(Status.SUCCESS, "text/plain", "ok") +@app.route("/robots.txt", strict_trailing_slash=False) +def status(request): + return Response(Status.SUCCESS, "text/plain", + "User-agent: researcher\nUser-agent: indexer\nUser-agent: archiver\nDisallow: /search\nDisallow: /backlinks\n\nUser-agent: *\nDisallow: /add-seed\nDisallow: /threads") @app.route("/favicon.txt", strict_trailing_slash=False) def favicon(request):