TheCloud/anubis
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2025-08-03 09:48:08 -04:00
Code Issues Packages Projects Releases Wiki Activity

chore(docs/manifest): allow common crawl to test with the team

Browse Source 
Signed-off-by: Xe Iaso <me@xeiaso.net>
This commit is contained in:
Xe Iaso 2025-07-04 00:22:43 +00:00
parent 7c0996448a
commit 2f1e78cc6c
No known key found for this signature in database
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/manifest/cfg/anubis/botPolicies.yaml
View File

@ -11,6 +11,7 @@
## /usr/share/docs/anubis/data or in the tarball you extracted Anubis from.
bots:
- import: (data)/crawlers/commoncrawl.yaml
# Pathological bots to deny
- # This correlates to data/bots/deny-pathological.yaml in the source tree
# https://github.com/TecharoHQ/anubis/blob/main/data/bots/deny-pathological.yaml