indymedia
/
epicyon
mirror of https://gitlab.com/bashrc2/epicyon
7
4
Fork 1
Code Issues 14 Releases Wiki Activity

Don't block crawlers on news instances

merge-requests/30/head
Bob Mottram 2021-06-28 18:28:14 +01:00
parent 97ec53143c
commit 784aca35f3
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
daemon.py
View File

@ -466,6 +466,8 @@ class PubServer(BaseHTTPRequestHandler):
# is this a web crawler? If so the block it # is this a web crawler? If so the block it
agentStrLower = agentStr.lower() agentStrLower = agentStr.lower()
if 'bot/' in agentStrLower or 'bot-' in agentStrLower: if 'bot/' in agentStrLower or 'bot-' in agentStrLower:
if if self.server.newsInstance:
return False
print('Blocked Crawler: ' + agentStr) print('Blocked Crawler: ' + agentStr)
return True return True
# get domain name from User-Agent # get domain name from User-Agent