indymedia
/
epicyon
mirror of https://gitlab.com/bashrc2/epicyon
7
4
Fork 1
Code Issues 14 Releases Wiki Activity

Merge branch 'main' of gitlab.com:bashrc2/epicyon

merge-requests/30/head
Bob Mottram 2022-03-09 22:56:19 +00:00
parent 130e4ef5e5 118571ae3f
commit 14e86227af
1 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
crawlers.py
View File

@ -126,8 +126,10 @@ def blocked_user_agent(calling_domain: str, agent_str: str,
contains_bot_string = False
for bot_str in bot_strings:
if bot_str in agent_str_lower:
contains_bot_string = True
break
if '://bot' not in agent_str_lower and \
'://robot' not in agent_str_lower:
contains_bot_string = True
break
if contains_bot_string:
if agent_str_lower not in known_bots:
known_bots.append(agent_str_lower)