7 Commits

Author SHA1 Message Date
Xe Iaso
845095c3f6
chore(robots.txt): don't block CCBot
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-07-04 00:23:21 +00:00
Dryusdan
1d5fa49eb0
Bump ai.robots.txt to v1.37 (#689)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
2025-06-18 13:30:53 -04:00
Dryusdan
281b6c5c00
Bump ai.robots.txt to v1.34 (#632) 2025-06-08 14:54:47 -04:00
Dryusdan
11081aac08
Bump AI-robots.txt rules to version 1.31 (#538)
* Bump AI-robots.txt rules to version 1.31

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-05-23 16:15:12 +00:00
Dryusdan
961320540b
Bump AI-robots.txt rules to version 1.30 (#509)
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-05-16 16:40:25 +00:00
Dryusdan
76514f9f32
Bump AI-robots.txt rules to version 1.29 (#383) 2025-04-27 20:52:08 -04:00
Yulian Kuncheff
6156d3d729
Refactor and split out things into cmd and lib (#77)
* Refactor anubis to split business logic into a lib, and cmd to just be direct usage.

* Post-rebase fixes.

* Update changelog, remove unnecessary one.

* lib: refactor this

This is mostly based on my personal preferences for how Go code should
be laid out. I'm not sold on the package name "lib" (I'd call it anubis
but that would stutter), but people are probably gonna import it as
libanubis so it's likely fine.

Packages have been "flattened" to centralize implementation with area of
concern. This goes against the Java-esque style that many people like,
but I think this helps make things simple.

Most notably: the dnsbl client (which is a hack) is an internal package
until it's made more generic. Then it can be made external.

I also fixed the logic such that `go generate` works and rebased on
main.

* internal/test: run tests iff npx exists and DONT_USE_NETWORK is not set

Signed-off-by: Xe Iaso <me@xeiaso.net>

* internal/test: install deps

Signed-off-by: Xe Iaso <me@xeiaso.net>

* .github/workflows: verbose go tests?

Signed-off-by: Xe Iaso <me@xeiaso.net>

* internal/test: sleep 2

Signed-off-by: Xe Iaso <me@xeiaso.net>

* internal/test: nix this test so CI works

Signed-off-by: Xe Iaso <me@xeiaso.net>

* internal/test: warmup per browser?

Signed-off-by: Xe Iaso <me@xeiaso.net>

* internal/test: disable for now :(

Signed-off-by: Xe Iaso <me@xeiaso.net>

* lib/anubis: do not apply bot rules if address check fails

Closes #83

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-03-22 18:44:49 -04:00