* feat(ogtags): enhance target URL handling for OGTagCache, support Unix sockets
Closes: #323#319
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* docs: update CHANGELOG.md to include Opengraph passthrough support for Unix sockets
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* docs: update CHANGELOG.md to include Opengraph passthrough support for Unix sockets
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(ogtags): add option to consider host in Open Graph tag cache key
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(ogtags): add option to consider host in OG tag cache key
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* test(ogtags): enhance tests for OGTagCache with host consideration scenarios
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(ogtags): extract constants for HTTP timeout and max content length
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(ogtags): restore fetchHTMLDocument method for cache key generation
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(ogtags): replace maxContentLength field with constant and ensure HTTP scheme is set correctly
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(fetch): add proxy headers
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Closes#372
Fun YAML fact of the day:
What is the difference between how these two expressions are parsed?
```yaml
foo: >
bar
```
```yaml
foo: >-
bar
```
They are invisible in yaml, but when you evaluate them to JSON the
difference is obvious:
```json
{
"foo": "bar\n"
}
```
```json
{
"foo": "bar"
}
```
User-Agent strings, URL path values, and HTTP headers _do_ end in
newlines in HTTP/1.1 wire form, but that newline is usually stripped
before the server actually handles it. Also HTTP/2 is a thing and does
not terminate header values with newlines.
This change makes Anubis more aggressively detect mistaken uses of the
yaml `>` operator and nudges the user into using the yaml `>-` operator
which does not append the trailing newline.
I had honestly forgotten about this YAML behavior because it wasn't
relevant for so long. Oops! Glad I released a beta.
Whenever you get into this state, Anubis will throw a config parsing
error and then give you a message hinting at the folly of your ways.
```
config.Bot: regular expression ends with newline (try >- instead of > in yaml)
```
Big thanks to https://yaml-multiline.info, this helped me realize my
folly instantly.
@aiverson, this is official permission to say "told you so".
Signed-off-by: Xe Iaso <me@xeiaso.net>
Previously Anubis would aggressively make sure that the client cookie
matched exactly what it should. This has turned out to be too paranoid
in practice and has caused problems with Happy Eyeballs et. al.
This is a potential fix to #303 and #289.
* Add check endpoint which can be used with nginx' auth_request function
* feat(cmd): allow configuring redirect domains
* test: add test environment for the nginx_auth PR
This is a full local setup of the nginx_auth PR including HTTPS so that
it's easier to validate in isolation.
This requires an install of k3s (https://k3s.io) with traefik set to
listen on localhost. This will be amended in the future but for now this
works enough to ship it.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(cmd|lib): allow empty redirect domains variable
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(test): add space to target variable in anubis container
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin): rewrite subrequest auth docs, make generic
* docs(install): document REDIRECT_DOMAINS flag
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib): clamp redirects to the same HTTP host
Only if REDIRECT_DOMAINS is not set.
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
It would seem the file was originally edited for 2-space indentation but accidentally used tabs instead of actual spaces.
Signed-off-by: Diego E <diegoe@gnome.org>
* feat(config): support importing bot policy snippets
This changes the grammar of the Anubis bot policy config to allow
importing from internal shared rules or external rules on the
filesystem.
This lets you create a file at `/data/policies/block-evilbot.yaml` and
then import it with:
```yaml
bots:
- import: /data/policies/block-evilbot.yaml
```
This also explodes the default policy file into a bunch of composable
snippets.
Thank you @Aibrew for your example gitea Atom / RSS feed rules!
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(data): update botPolicies.json to use imports
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(cmd/anubis): extract bot policies with --extract-resources
This allows a user that doesn't have anything but the Anubis binary to
figure out what the default configuration does.
* docs(data/botPolices.yaml): document import syntax in-line
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib/policy): better test importing from JSON snippets
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin): Add import syntax documentation
This documents the import syntax and is based on the block comment at
the top of the default bot policy file.
* docs(changelog): add note about importing snippets
Signed-off-by: Xe Iaso <me@xeiaso.net>
* style(lib/policy/config): use an error value instead of an inline error
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(cmd/anubis): compute full XFF header
this one is pretty important to not pass
through blindly, as many applications and
frameworks will trust them
* feat(cmd/anubis): skip XFF compute if remote address is loopback
* docs: update CHANGELOG
Closes#333
I'm very bad at design so I just picked colors that looked reasonable
enough to me. Hopefully this will be enough to get us to the next stage!
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix: improve error handling for resource closing and JSON encoding in MakeChallenge
* chore: update CHANGELOG with recent changes and improvements
* refactor: simplify RenderIndex function and improve error handling
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
This makes each check into its own type that has encapsulated check
logic, meaning that it's easier to add new checker implementations in
the future.
Signed-off-by: Xe Iaso <me@xeiaso.net>
After deploying Anubis bot traffic is drastically reduced but I still
see a lot of requests from User-Agents that claim to be 'Opera' like so:
"Opera/9.90.(Windows NT 6.0; mt-MT) Presto/2.9.173 Version/10.00"
"Opera/8.46.(X11; Linux i686; fo-FO) Presto/2.9.161 Version/11.00"
Add 'Opera' to the generic-browser rule to also challenge them.
Signed-off-by: Michael Jeanson <mjeanson@efficios.com>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* Add documentation example for a NGINX configuration that demonstrates how to insert Anubis in the middle of a normal configuration.
Signed-off-by: Jeroen Massar <jeroen@massar.ch>
* Add changelog entry
Signed-off-by: Jeroen Massar <jeroen@massar.ch>
* docs/admin/installation: rephrasing and diagrams
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs/admin/installation: flatten down the nginx config
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs/admin/installation: other fixups and note the assumptions at play
Thanks @SuperSandro2000!
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Jeroen Massar <jeroen@massar.ch>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* Create anubis.freebsd
add freebsd rc.d script so can be run as a freebsd daemon
Signed-off-by: Paul Wilde <31094984+pswilde@users.noreply.github.com>
* Update CHANGELOG.md
Signed-off-by: Paul Wilde <31094984+pswilde@users.noreply.github.com>
---------
Signed-off-by: Paul Wilde <31094984+pswilde@users.noreply.github.com>
* fix(fetch): improve error handling for Content-Type parsing
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(fetch): rename OgHandledError to ErrOgHandled for statichcheck to like me
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
