indymedia
/
epicyon
mirror of https://gitlab.com/bashrc2/epicyon
7
4
Fork 1
Code Issues 14 Releases Wiki Activity

Extra check for search bots

merge-requests/30/head
Bob Mottram 2022-03-08 20:03:41 +00:00
parent 7ce38ceef8
commit 118571ae3f
1 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
crawlers.py
View File

@ -126,8 +126,10 @@ def blocked_user_agent(calling_domain: str, agent_str: str,
contains_bot_string = False
for bot_str in bot_strings:
if bot_str in agent_str_lower:
contains_bot_string = True
break
if '://bot' not in agent_str_lower and \
'://robot' not in agent_str_lower:
contains_bot_string = True
break
if contains_bot_string:
if agent_str_lower not in known_bots:
known_bots.append(agent_str_lower)