Gives us many nice things like:
* Windows support for yeet (modulo TecharoHQ/yeet#29)
* Removes the dependency on /bin/sh or /bin/bash thanks to
mvdan.cc/sh/v3
* Checksum-compliant reproducible builds by default
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Split up AI filtering files
Create aggressive/moderate/permissive policies to allow administrators to choose their AI/LLM stance.
Aggressive policy matches existing default in Anubis.
Removes `Google-Extended` flag from `ai-robots-txt.yaml` as it doesn't exist in requests.
Rename `ai-robots-txt.yaml` to `ai-catchall.yaml` as the file is no longer a copy of the source repo/file.
* chore: spelling
* chore: fix embeds
* chore: fix data includes
* chore: fix file name typo
* chore: Ignore READMEs in configs
* chore(lib/policy/config): go tool goimports -w
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* Define OpenAI bot ALLOW policies
Allows OpenAI bots to be allowlisted at the choice of the Anubis administrator. None are enabled by default.
* Define MistralAI bot ALLOW policy
* chore: spelling
* Add Applebot definition
Adds Apple's search indexing bot, and allowlists it by default.
Allowlisted by default because it is equivalent to Googlebot/Bingbot. Remove Applebot from `ai-robots-txt.yaml` for the same reasons.
Remove `Applebot-Extended` from `ai-robots-txt.yaml` as it has no effect.
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* feat(lib): annotate cookies with what rule was passed
Anubis JWTs now contain a policyRule claim with the cryptographic hash
of the rule that it passed. This is intended to help with a future move
away from proof of work being the default.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix cookie storage logic
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(bench): await benchmark loop and adjust outline styles in templates
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor: remove unused showContinueBar function and clean up video error handling
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: format code for consistency and readability using prettier
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
For some reason, Google Chrome will randomly send a "full"
Accept-Language header, and other times it will send a "partial"
Accept-Language header. This makes the challenge construction
inconsistent.
This commit fixes this issue by only considering up to the first five
characters of the Accept-Language header when making a challenge string.
Signed-off-by: Xe Iaso <me@xeiaso.net>
Closes#565
The page already had the version number embedded into it, but that was
not printed to the page. This prints the version number set at compile
time to the page.
Signed-off-by: Xe Iaso <me@xeiaso.net>
This seems counter-intuitive at first glance, but let me cook.
One of the problems with Anubis is that the rule matching is super
deterministic. This means that attackers can figure out what patterns
they are hitting and change things to bypass them.
The randInt function lets you have rulesets behave nondeterministically.
This is a very easy way to hang yourself, but can be great to
psychologically mess with scraper operators. Consider this rule:
```yaml
- name: deny-lightpanda-sometimes
action: DENY
expression:
all:
- userAgent.matches("LightPanda")
- randInt(16) >= 4
```
It would match about 75% of the time.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(expression): add validation for empty ExpressionOrList
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(imports): block empty file imports with improved error checking logic
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* docs(expression): improve validation to error on empty CEL expressions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Closes#531
This changes `anubis_challenges_issued` to be a vector counter that
records the challenge issuance method.
Signed-off-by: Xe Iaso <me@xeiaso.net>
For websites hosted on non-standard ports (`https://example:com:8443`,
`http://www.example.net:8080`, etc.), the domains listed in
`REDIRECT_DOMAINS` must contain the port number. This commit documents
this requirement on the Installation page.
Fixes#517.
Signed-off-by: Max Chernoff <git@maxchernoff.ca>
Closes#520
For some reason, Chrome and Firefox are very picky over what they use to
match cookies that need to be deleted. Listen to me for my tale of woe:
The basic problem here is that cookies were an early hack added on the
side of the HTTP spec and they're basically impossible to upgrade or
change because who knows what relies on the exact behavior cookies use.
As a result, cookies don't just match by name, but by every setting that
exists on them. You can also have two cookies with the same name but
different values. This spec is a nightmare lol.
Even more fun: browsers will make up values for cookies if they aren't
set, meaning that getting a challenge token at `/docs` is semantically
different than a challenge token you got from `/`.
This PR fixes this issue by explicitly setting the "make sure cookie
support is working" cookie's path to `/`, meaning that it will always be
sent. Additionally, cookies are expired by setting the expiry time to
one minute in the past.
Hopefully this will fix it. I'm testing this locally and it seems to
work fine.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib): ensure that clients store cookies
If a client is misconfigured and does not store cookies, then they can
get into a proof of work death spiral with Anubis. This fixes the
problem by setting a test cookie whenever the user gets hit with a
challenge page. If the test cookie is not there at challenge pass time,
then they are blocked. Administrators will also get a log message
explaining that the user intentionally broke cookie support and that this
behavior is not an Anubis bug.
Additionally, this ensures that clients being shown a challenge support
gzip-compressed responses by showing the challenge page at gzip level 1.
This level is intentionally chosen in order to minimize system impacts.
The ClearCookie function is made more generic to account for cookie
names as an argument. A correlating SetCookie function was also added to
make it easier to set cookies.
* chore(lib): clean up test code
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Previously this made ClearCookie always clear cookies by name even when
CookieDomain was set. This change fixes this and adds tests to make sure
that this doesn't happen again.
Signed-off-by: Xe Iaso <me@xeiaso.net>
