indymedia
/
epicyon
mirror of https://gitlab.com/bashrc2/epicyon
7
4
Fork 1
Code Issues 14 Releases Wiki Activity

Extra llm crawlers

main
bashrc 2026-04-14 20:20:27 +01:00
parent 3897a637a2
commit 259d51d00f
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
crawlers.py
View File

@ -167,7 +167,7 @@ def blocked_user_agent(calling_domain: str, agent_str: str,
'openai', 'panscient', 'phindbot', 'qualifiedbot', 'quillbot',
'sbIntuitionsbot', 'semrushbot', 'shapbot', 'terracotta',
'velenpublicwebcrawler', 'wpbot', 'yak', 'amzn-', 'azureai',
'kagi-fetcher'
'kagi-fetcher', 'aranet-', 'apifybot', 'apifyweb', 'exabot'
)
for bot_str in llm_bot_strings:
if bot_str in agent_str_lower: