510 Commits

Author SHA1 Message Date
dependabot[bot]
1ca531b930
build(deps): bump the gomod group with 4 updates (#709)
Bumps the gomod group with 4 updates: [github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus](https://github.com/grpc-ecosystem/go-grpc-middleware), [github.com/grpc-ecosystem/go-grpc-middleware/v2](https://github.com/grpc-ecosystem/go-grpc-middleware), [google.golang.org/grpc](https://github.com/grpc/grpc-go) and [k8s.io/apimachinery](https://github.com/kubernetes/apimachinery).


Updates `github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus` from 1.0.1 to 1.1.0
- [Release notes](https://github.com/grpc-ecosystem/go-grpc-middleware/releases)
- [Commits](https://github.com/grpc-ecosystem/go-grpc-middleware/compare/providers/prometheus/v1.0.1...v1.1.0)

Updates `github.com/grpc-ecosystem/go-grpc-middleware/v2` from 2.1.0 to 2.3.2
- [Release notes](https://github.com/grpc-ecosystem/go-grpc-middleware/releases)
- [Commits](https://github.com/grpc-ecosystem/go-grpc-middleware/compare/v2.1.0...v2.3.2)

Updates `google.golang.org/grpc` from 1.72.2 to 1.73.0
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.72.2...v1.73.0)

Updates `k8s.io/apimachinery` from 0.33.1 to 0.33.2
- [Commits](https://github.com/kubernetes/apimachinery/compare/v0.33.1...v0.33.2)

---
updated-dependencies:
- dependency-name: github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus
  dependency-version: 1.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: gomod
- dependency-name: github.com/grpc-ecosystem/go-grpc-middleware/v2
  dependency-version: 2.3.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: gomod
- dependency-name: google.golang.org/grpc
  dependency-version: 1.73.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: gomod
- dependency-name: k8s.io/apimachinery
  dependency-version: 0.33.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: gomod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-23 15:59:08 -04:00
Xe Iaso
f9259299b9
chore: release v1.20.0-pre2
Signed-off-by: Xe Iaso <me@xeiaso.net>
v1.20.0-pre2
2025-06-23 15:17:13 -04:00
Xe Iaso
16a4e04027
fix(lib): fix invalid response after success in Chrome (#711)
Closes #564

This one is really dumb. Take a seat and listen to my tale of woe.

While @victorvalenca was working on #693 we ran into a strange issue.
The tests would consistently pass on Firefox but instantly failed on
Chrome. After adding increasingly desperate debugging logs to the mix,
we found out that somehow Chrome was randomizing the contents of its
Accept-Language header. This was making the challenge string get
calculated differently, thus making things spuriously fail. I cannot
figure out what causes Chrome to do this other than you being in an
environment where you have more than one "system language" set.

Either way, this should finally fix this issue and bring peace to the
land forever*.

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-23 15:11:56 -04:00
dependabot[bot]
8c79870edb
build(deps): bump the github-actions group with 3 updates (#708)
Bumps the github-actions group with 3 updates: [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action), [actions-hub/kubectl](https://github.com/actions-hub/kubectl) and [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv).


Updates `docker/setup-buildx-action` from 3.10.0 to 3.11.1
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](b5ca514318...e468171a9d)

Updates `actions-hub/kubectl` from 1.33.1 to 1.33.2
- [Release notes](https://github.com/actions-hub/kubectl/releases)
- [Commits](f632a31512...d50394b7d7)

Updates `astral-sh/setup-uv` from 6.1.0 to 6.3.0
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](f0ec1fc3b3...445689ea25)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: 3.11.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: actions-hub/kubectl
  dependency-version: 1.33.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
- dependency-name: astral-sh/setup-uv
  dependency-version: 6.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-23 08:41:04 -04:00
Eric T. Johnson
060b10ea2d
fix(web/js): broken progress bar with slow algo (#673)
This was revealed by the reformat in #546.

Signed-off-by: Eric T. Johnson <yut23@users.noreply.github.com>
2025-06-22 20:05:37 -04:00
Xe Iaso
4c74934e9f
fix(default-config): Techaro -> Zombocom
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-22 20:04:40 -04:00
Xe Iaso
5870f7072c
feat: implement imprint/impressum support (#706)
* feat: implement imprint/impressum support

Closes #362

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(docs/anubis): enable an imprint

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: fix the end of the sentence, comment out a default impressum

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: link back to impressum page

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-22 18:09:37 -04:00
Xe Iaso
3c1d95d61e
fix(default-config): off-by-one error in the default thresholds (#701)
I don't know how I missed this in testing.
2025-06-20 11:47:34 -04:00
Jan Alexander Steffens
ab801a3597
Makefile: Build robots2policy (#699)
* Makefile: Build robots2policy

* Update metadata

check-spelling run (pull_request) for build-robots2policy

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

---------

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Co-authored-by: Jason Cameron <git@jasoncameron.dev>
2025-06-20 11:08:56 -04:00
Xe Iaso
ecc716940e
chore: release v1.20.0-pre1
Signed-off-by: Xe Iaso <me@xeiaso.net>
v1.20.0-pre1
2025-06-19 19:32:49 -04:00
Xe Iaso
4948036f39
feat: add default OpenGraph tags to configuration file (#694)
* feat(config): opengraph passthrough configuration

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(ogtags): use config.OpenGraph for configuration

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: wire up ogtags config in most of the app

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(ogtags): return default tags if they are supplied

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: make OpenGraph legal so we have some sanity in reviewing

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): use OpenGraph.Enabled

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib): load default config file if one is not specified in spawnAnubis

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(config): fix ST1005

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: document open graph defaults and its new home in the policy file

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(installation): point to weight threshold new home

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: rename default to override

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(default-config): add off-by-default opengraph settings to bot policy file

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(anubis): make build

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib): fix build

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-19 18:00:44 -04:00
Xe Iaso
7aa732c700
fix(config): actually load threshold config (#696)
* fix(config): actually load threshold config

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib): fix test failures

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-19 17:13:01 -04:00
Xe Iaso
226cf36bf7
feat(config): custom weight thresholds via CEL (#688)
* feat(config): add Thresholds to the top level config file

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(config): make String() on ExpressionOrList join the component expressions

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(config): ensure unparseable json fails

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(config): if no thresholds are set, use the default thresholds

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(policy): half implement thresholds

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(policy): continue wiring things up

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib): wire up thresholds

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(lib): handle behavior from legacy configurations

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: document thresholds

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: update CHANGELOG, refer to threshold configuration

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): fix build

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(lib): fix U1000

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Co-authored-by: Jason Cameron <git@jasoncameron.dev>
2025-06-18 16:58:31 -04:00
Dryusdan
1d5fa49eb0
Bump ai.robots.txt to v1.37 (#689)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
2025-06-18 13:30:53 -04:00
Lothar Serra Mari
97c1d4f353
docs(known-instances): add extensions.typo3.org (#691)
Signed-off-by: Lothar Serra Mari <mail@serra.me>
2025-06-18 08:06:23 -04:00
hydrargyrum
244f1c505a
fix(geo): correct typo "counties" to "countries" (#678) 2025-06-17 23:50:42 -04:00
Jason Cameron
ae4d3b0ce5
chore: remove duplicate CHANGELOG entry (#684)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
2025-06-17 22:49:30 +00:00
prettysunflower
e60c43cdd2
docs(known-instances): add wiki.koha-community.org (#683)
Signed-off-by: prettysunflower <me@prettysunflower.moe>
2025-06-17 12:14:15 -04:00
Jason Cameron
b2b2679bae
perf: replace cidranger with bart for significant performance improvements (#675)
* feat: replace cidranger with bart improving performance by 3-20x

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* perf: replace cidranger with bart for IP range checking

- Replace cidranger.Ranger with bart.Lite in RemoteAddrChecker
- Use netip.ParsePrefix instead of net.ParseCIDR for modern IP handling
- Improve performance: 3-20x faster lookups with zero heap allocations
- Update imports to use github.com/gaissmai/bart and net/netip
- Remove cidranger dependency from go.mod

Benchmark results:
- IPv4 lookups: 4x faster (15.58ns vs 63.25ns, 0 vs 2 allocs)
- IPv6 lookups: 3x faster (26.51ns vs 76.96ns, 0 vs 2 allocs)
- Insertions: 20x faster (976ns vs 19,191ns)
- Large tables: 14x faster (5.2ns vs 74.85ns)

* docs: clarify CHANGELOG to not give false impressions

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* perf: optimize string concatenation in RemoteAddrChecker hash generation

Replace fmt.Fprintln with strings.Join for 7x faster performance:
- Before: 935.1 ns/op, 784 B/op, 22 allocs/op
- After: 133.2 ns/op, 192 B/op, 1 alloc/op

The hash is used for JWT cookie validation and error code generation.
Comma separation provides the same deterministic uniqueness as newlines
but with significantly better performance during policy initialization.

* chore: remove accidentally commited string benchmark

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* style: apply Copilot suggestions

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix: reference the right var name

i cannot write a merge commit

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

---------

Signed-off-by: Jason Cameron <git@jasoncameron.dev>
2025-06-17 11:57:55 -04:00
Jason Cameron
e2b46fc5e7
perf: Replace internal SHA256 hashing with xxhash for 4-6x performance improvement (#676)
* perf(internal): Use FastHash for internal hashing
docs: Add xxhash performance improvement to changelog entry
feat(hash): Add fast non-cryptographic hash function

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* test(hash): add xxhash benchmarks and collision tests

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Update metadata

check-spelling run (pull_request) for json/hash

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

---------

Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
2025-06-16 22:53:53 -04:00
hyperdefined
3437e575d4
chore(sponsors): update canine.tools logo (#672) 2025-06-16 14:09:35 -04:00
Xe Iaso
ae064be710
chore(docs/manifest): it helps if you terminate strings properly
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-16 12:11:04 -04:00
Xe Iaso
e3826df3ab
feat: implement a client for Thoth, the IP reputation database for Anubis (#637)
* feat(internal): add Thoth client and simple ASN checker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): cached ip to asn checker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: go mod tidy

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(thoth): minor testing fixups, ensure ASNChecker is Checker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): make ASNChecker instances

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): add GeoIP checker

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): store a thoth client in a context

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: refactor Checker type to its own package

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(thoth): add thoth mocking package, ignore context deadline exceeded errors

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thoth): pre-cache private ranges

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(lib/policy/config): enable thoth ASNs and GeoIP checker parsing

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(thoth): refactor to move checker creation to the checker files

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(policy): enable thoth checks

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(thothmock): test helper function for loading a mock thoth instance

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat: wire up Thoth, make thoth checks part of the default config

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(thoth): mend staticcheck errors

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(admin): add Thoth docs

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(policy): update Thoth links in error messages

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: update CHANGELOG

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(docs/manifest): enable Thoth

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: add THOTH_INSECURE for contacting Thoth over plain TCP in extreme circumstances

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(thoth): use mock thoth when credentials aren't detected in the environment

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(cmd/anubis): better warnings for half-configured Thoth setups

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(botpolicies): link to Thoth geoip docs

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-16 11:57:32 -04:00
Xe Iaso
823d1be5d1
chore(docs/manifest): explicitly allow blog RSS feed
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-16 11:50:53 -04:00
Xe Iaso
0c6a820372
chore(docs/manifest): enable OG_PASSTHROUGH
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-16 09:31:56 -04:00
Xe Iaso
81f6380dd4
Add the blog section back (#670)
* Revert "docs/blog: remove (#273)"

This reverts commit df3509ec998d002e8bcef6285c7c0bc16b54d513.

* chore: intro to the blog post

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-16 09:28:21 -04:00
dependabot[bot]
e5455c02d8
build(deps): bump the github-actions group with 3 updates (#666)
Bumps the github-actions group with 3 updates: [docker/login-action](https://github.com/docker/login-action), [actions/attest-build-provenance](https://github.com/actions/attest-build-provenance) and [github/codeql-action](https://github.com/github/codeql-action).


Updates `docker/login-action` from 3.0.0 to 3.4.0
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...74a5d142397b4f367a81961eba4e8cd7edddf772)

Updates `actions/attest-build-provenance` from 2.3.0 to 2.4.0
- [Release notes](https://github.com/actions/attest-build-provenance/releases)
- [Changelog](https://github.com/actions/attest-build-provenance/blob/main/RELEASE.md)
- [Commits](db473fddc0...e8998f9491)

Updates `github/codeql-action` from 3.28.19 to 3.29.0
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](fca7ace96b...ce28f5bb42)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: 3.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: actions/attest-build-provenance
  dependency-version: 2.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
- dependency-name: github/codeql-action
  dependency-version: 3.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Jason Cameron <git@jasoncameron.dev>
2025-06-15 21:13:56 -04:00
Colin Finck
1d8033d69e
Add ReactOS to known-instances.md (#664)
Signed-off-by: Colin Finck <colin@reactos.org>
2025-06-15 15:30:54 +00:00
Jason Cameron
e0781e4560
feat: add robots2policy CLI to convert robots.txt to Anubis CEL (#657)
* feat: add robots2policy CLI utility to convert robots.txt to Anubis challenge policies

* feat: add documentation for robots2policy CLI tool

* feat: implement crawl delay handling as weight adjustment in Anubis rules

* feat: add various robots.txt and YAML configurations for user agent handling and crawl delays

* test: add comprehensive tests for robots2policy conversion and parsing

* fix: update example URL in usage instructions for robots2policy CLI

* Update metadata

check-spelling run (pull_request) for json/robots2policycli

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

* docs: add crawl delay weight adjustment and deny user agents option to robots2policy CLI

* Update cmd/robots2policy/main.go

Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* Update cmd/robots2policy/main.go

Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* fix(robots2policy): use sigs.k8s.io/yaml

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(config): properly marshal bot policy rules

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore(yeetfile): expose robots2policy in libexec

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(yeetfile): put robots2policy in $PATH

Signed-off-by: Xe Iaso <me@xeiaso.net>

* Update metadata

check-spelling run (pull_request) for json/robots2policycli

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

* style: reorder imports

* refactor: use preexisting structs in config

* fix: correct flag check in main function

* fix: reorder fields in AnubisRule struct for better alignment

* style: improve alignment of struct fields in AnubisRule and OGTagCache

* Update metadata

check-spelling run (pull_request) for json/robots2policycli

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

* fix: add validation for generated Anubis rules from robots.txt

* feat: add batch processing for robots.txt files to generate Anubis CEL policies

* fix: improve usage message and error handling for input file requirement

* refactor: update AnubisRule structure to use ExpressionOrList for improved expression handling

* refactor: reorganize policy definitions in YAML files for consistency and clarity

* fix: correct indentation in blacklist and complex YAML files for consistency

* test: enhance output comparison in robots2policy tests for YAML and JSON formats

* Revert "fix: improve usage message and error handling for input file requirement"

This reverts commit ddcde1f2a326545d3ef2ec32e5e03f55f4f931a8.

* fix: improve usage message and error handling in robots2policy

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

---------

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-14 23:41:00 -04:00
Lothar Serra Mari
7a195f1595
docs(known-instances): add bugs.scummvm.org and gitlab.postmarketos.org (#661)
* docs(known-instances): add bugs.scummvm.org and gitlab.postmarketos.org

Signed-off-by: Lothar Serra Mari <mail@serra.me>

* chore: clean uri

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

---------

Signed-off-by: Lothar Serra Mari <mail@serra.me>
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Co-authored-by: Jason Cameron <git@jasoncameron.dev>
2025-06-14 13:55:55 +00:00
Jason Cameron
2904ff974b
refactor(ogtags): optimize URL construction and memory allocations (#647)
* refactor(ogtags): optimize URL construction and memory allocations

* test(ogtags): add benchmarks and memory usage tests for OGTagCache

* refactor(ogtags): optimize OGTags subsystem to reduce allocations and improve request runtime by up to 66%

* Update docs/docs/CHANGELOG.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* refactor(ogtags): optimize URL string construction to reduce allocations

* Update internal/ogtags/ogtags.go

Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>

* test(ogtags): add fuzz tests for getTarget and extractOGTags functions

* fix(ogtags): update memory calculation logic

Prev it would say that we had allocated 18pb

=== RUN   TestMemoryUsage
    mem_test.go:107: Memory allocated for 10k getTarget calls: 18014398509481904.00 KB
    mem_test.go:135: Memory allocated for 1k extractOGTags calls: 18014398509481978.00

    Now it's fixed with

    === RUN   TestMemoryUsage
    mem_test.go:109: Memory allocated for 10k getTarget calls:
    mem_test.go:110:   Total: 630.56 KB (0.62 MB)
    mem_test.go:111:   Per operation: 64.57 bytes
    mem_test.go:140: Memory allocated for 1k extractOGTags calls:
    mem_test.go:141:   Total: 328.17 KB (0.32 MB)
    mem_test.go:142:   Per operation: 336.05 bytes

* refactor(ogtags): optimize meta tag extraction for improved performance

* Update metadata

check-spelling run (pull_request) for json/ogmem

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

* chore: update CHANGELOG for recent optimizations and version bump

* refactor: improve URL construction and meta tag extraction logic

* style:  cleanup fuzz tests

---------

Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-13 09:53:10 -04:00
Jason Cameron
3b3080d497
feat: add a strip-base-prefix option (#655)
* style: fix formatting in .air.toml and installation.mdx

* feat: add --strip-base-prefix flag to modify request paths when forwarding

Closes: #638

* refactor: apply structpacking (betteralign)

* fix: add validation for strip-base-prefix and base-prefix configuration

* fix: improve request path handling by cloning request and modifying URL path

* chore: remove integration tests as they are too annoying to debug on my system
2025-06-12 17:46:08 -04:00
Jason Cameron
60ba8e9557
fix(ci): conditionally run SSH jobs for TecharoHQ/anubis repository (#654)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
2025-06-11 21:18:43 +00:00
Jason Cameron
14c80483a9
fix(gitattributes): update pattern for generated files (#652)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
2025-06-11 21:00:37 +00:00
Xe Iaso
d1452b6d39
test(ssh-ci): re-enable GOARCH=ppc64le (#651)
This reverts commit 5e95da6b6c820bd12b96e06c732dc9000dbaa81e.
2025-06-11 14:01:48 -04:00
Xe Iaso
5e95da6b6c
test(ssh-ci): disable GOARCH=ppc64le for now
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-11 12:58:32 -04:00
dependabot[bot]
988fc0941b
build(deps): bump github.com/cloudflare/circl from 1.6.0 to 1.6.1 (#650)
Bumps [github.com/cloudflare/circl](https://github.com/cloudflare/circl) from 1.6.0 to 1.6.1.
- [Release notes](https://github.com/cloudflare/circl/releases)
- [Commits](https://github.com/cloudflare/circl/compare/v1.6.0...v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/cloudflare/circl
  dependency-version: 1.6.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-11 16:52:10 +00:00
Xe Iaso
f5140ae57b
test: introduce SSH based CI for non-native test hosts (#644)
* feat: ssh based CI

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test: implement SSH ci with caches and github actions

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): fix known hosts secret

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): clone the repo, that's important

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): speed up ci by prebaking the SSH CI image

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): set -euo

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): enable pull_request_target so things work

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): oh goody it's broken

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): add cronjob to rebuild ci runner image

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): also run yeet

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): force git version for yeet

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): run set -x in the container

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): fix yeet?

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): remove yeet for now

Signed-off-by: Xe Iaso <me@xeiaso.net>

* test(ssh-ci): disable for PRs for now

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-11 12:50:01 -04:00
Jason Cameron
bbdee34f37
fix(anubis): improve challenge handling and error reporting (#645) 2025-06-11 12:47:06 -04:00
Stephanie Gawroriski
6e2eeb9e65
Update known-instances.md to include SquirrelJME (#643)
Include SquirrelJME, which is an emulator for Java ME 8.

Signed-off-by: Stephanie Gawroriski <xer@multiphasicapps.net>
2025-06-10 23:20:50 +00:00
Xe Iaso
c638653172
feat(lib): implement request weight (#621)
* feat(lib): implement request weight

Replaces #608

This is a big one and will be what makes Anubis a generic web
application firewall. This introduces the WEIGH option, allowing
administrators to have facets of request metadata add or remove
"weight", or the level of suspicion. This really makes Anubis weigh
the soul of requests.

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): maintain legacy challenge behavior

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(lib): make weight have dedicated checkers for the hashes

Signed-off-by: Xe Iaso <me@xeiaso.net>

* feat(data): convert some rules over to weight points

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: document request weight

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(CHANGELOG): spelling error

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs: fix links to challenge information

Signed-off-by: Xe Iaso <me@xeiaso.net>

* docs(policies): fix formatting

Signed-off-by: Xe Iaso <me@xeiaso.net>

* fix(config): make default weight adjustment 5

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-09 15:25:04 -04:00
Fierelier
0fe46b48cf
Make progress bar styling more compatible (UXP, etc) (#636)
* Make progress bar styling more compatible (UXP, etc)

* Add 'Make progress bar styling more compatible (UXP, etc)'

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Fierelier <fier@airmail.cc>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-09 12:19:38 -04:00
David Chandek-Stark
d6e5561768
Adds ability to toggle off stripping of private addrs from XFF (#619)
* Adds ability to toggle off stripping of private addrs from XFF

* chore: spelling

Signed-off-by: Xe Iaso <me@xeiaso.net>

* chore: refactor to flow better

Signed-off-by: Xe Iaso <me@xeiaso.net>

---------

Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
2025-06-09 13:33:19 +00:00
dependabot[bot]
6594ae0eef
build(deps): bump github/codeql-action in the github-actions group (#635)
Bumps the github-actions group with 1 update: [github/codeql-action](https://github.com/github/codeql-action).


Updates `github/codeql-action` from 3.28.18 to 3.28.19
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](ff0a06e83c...fca7ace96b)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: 3.28.19
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-09 08:56:41 -04:00
Lothar Serra Mari
ad09f82c3c
docs(admin/environments): Prefer IPv6 over IPv4 for apache2 listener directive (#628)
Signed-off-by: Lothar Serra Mari <mail@serra.me>
2025-06-09 08:56:30 -04:00
Xe Iaso
372b797f64
chore: go generate
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-08 20:52:22 -04:00
dependabot[bot]
6eaf0e13a2
build(deps): bump the gomod group with 2 updates (#634)
Bumps the gomod group with 2 updates: [github.com/a-h/templ](https://github.com/a-h/templ) and [golang.org/x/net](https://github.com/golang/net).


Updates `github.com/a-h/templ` from 0.3.887 to 0.3.898
- [Release notes](https://github.com/a-h/templ/releases)
- [Changelog](https://github.com/a-h/templ/blob/main/.goreleaser.yaml)
- [Commits](https://github.com/a-h/templ/compare/v0.3.887...v0.3.898)

Updates `golang.org/x/net` from 0.40.0 to 0.41.0
- [Commits](https://github.com/golang/net/compare/v0.40.0...v0.41.0)

---
updated-dependencies:
- dependency-name: github.com/a-h/templ
  dependency-version: 0.3.898
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: gomod
- dependency-name: golang.org/x/net
  dependency-version: 0.41.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: gomod
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-08 20:44:25 -04:00
Dryusdan
281b6c5c00
Bump ai.robots.txt to v1.34 (#632) 2025-06-08 14:54:47 -04:00
Jason Cameron
9539668049
style: Some minor fixes (#548)
* chore(deps): update dependencies in go.mod and go.sum

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* refactor: rename variables for clarity in anubis.go and main.go

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix(checker): handle error when inserting IP range in ranger

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* fix(tests): simplify boolean checks in header and URL value tests

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* refactor(api): remove unused /test-error endpoint and restrict /make-challenge to development

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* build(deps): update golang-set to v2.8.0 in go.sum

Signed-off-by: Jason Cameron <git@jasoncameron.dev>

* Update metadata

check-spelling run (pull_request) for json/stuff

Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>

---------

Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
2025-06-07 18:21:22 +00:00
Xe Iaso
8eff57fcb6
chore(docs/manifest): try no-js challenge to see how it impacts false positive rate
Signed-off-by: Xe Iaso <me@xeiaso.net>
2025-06-06 21:40:28 -04:00