I'm gonna be totally honest here, I'm still not sure why #564 is still
an issue. This is really confusing and I'm going to totally throw out
how Anubis issues challenges and redo it with Valkey (#201, #622).
The problem seems to be that I assume that the makeChallenge function in
package lib is idempotent for the same client. I have no idea why this
would be inconsistent, but for some reason it is and I'm just at a loss
for words as to why this is happening.
This stops the bleeding by improving the UX as a stopgap.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* lib/localization: implement localization system
Locale files are placed in lib/localization/locales/. If you add a
locale, update manifest.json with available locales.
* Exclude locales from check spelling
* tests(lib/localization): add comprehensive translations test
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(challenge/metarefresh): enable localization
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix: use simple syntax for localization in templ
Also localize CELPHASE into French according to the wishes of the
artist.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore:(js): fix forbidden patterns
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: add goi18n to tools
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/localization): dynamically determine the list of supported languages
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* fix(bench): await benchmark loop and adjust outline styles in templates
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor: remove unused showContinueBar function and clean up video error handling
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: format code for consistency and readability using prettier
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Closes#565
The page already had the version number embedded into it, but that was
not printed to the page. This prints the version number set at compile
time to the page.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor: reorder import statements in fetch.go and fetch_test.go
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix: optimize struct field alignment to reduce memory usage
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(js): use pure JS SHA256 library, refactor
Closes#458
Additionally, I made a horrifying discovery: Firefox seems to actively
hinder performance if you are using more than one Worker per page. It
does not spread the load out across cores like I expected. Instead it
seems to make that one Worker thrash and have to constantly context
switch, which caused a lot of slowdown.
The benchmarks in #155 continue to be the best contribution ever made to
Anubis. What clued me into there being a problem here was the fact that
the "slow" algorithm was faster than the "fast" algorithm on my laptop.
This made no intuitive sense to me so I dug further.
Either way I think this is a Firefox bug at its core, but for now we
have to work around it by doing the hacky terrible thing that I hate.
I also swapped the SHA256 operations to @aws-crypto/sha256-js on the
advice of a trusted cryptography expert. I don't know what performance
differences this makes, but I'm getting 150-225 kilohashes per second,
which is pretty dang good.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(js): apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(js): use fast algo for fast worker
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add check endpoint which can be used with nginx' auth_request function
* feat(cmd): allow configuring redirect domains
* test: add test environment for the nginx_auth PR
This is a full local setup of the nginx_auth PR including HTTPS so that
it's easier to validate in isolation.
This requires an install of k3s (https://k3s.io) with traefik set to
listen on localhost. This will be amended in the future but for now this
works enough to ship it.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(cmd|lib): allow empty redirect domains variable
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(test): add space to target variable in anubis container
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin): rewrite subrequest auth docs, make generic
* docs(install): document REDIRECT_DOMAINS flag
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib): clamp redirects to the same HTTP host
Only if REDIRECT_DOMAINS is not set.
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* fix(xess): suppress Go inspection warning for boolean expressions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: use outline shorthand
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Add Open Graph tag support (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Prevent nil pointer dereference in test (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat!: Implement Open Graph tag caching and passthrough functionality (WIP)
I'm going to sleep. currently tags are passed to renderIndex.
see https://github.com/TecharoHQ/anubis/issues/131
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Add configuration for air tool with build and logger settings
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Move OG tags to base template (og-tags)
Moves the Open Graph (OG) tags from the index template to
the base template. This allows OG tags to be set on any
page, not just the index. Also adds a
BaseWithOGTags function to the web package to allow
passing OG tags to the base template. Removes the
ogTags parameter from the Index function and template.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Delete CHANGELOG.md
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Add language attribute to HTML tag in template
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(tests): Fix nil pointer ref
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Add timeout to http client (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: fix line endings & indentation
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: add inspection comment for GoBoolExpressions in UnchangingCache
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Implement Open Graph tag fetching and caching
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(og-tags): Simplify Open Graph tag extraction logic
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(og-tags): Add nil check in isOGMetaTag and enhance test cases
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Add approved tags and prefixes for Open Graph extraction
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* test(og-tags): Update tests with approved tags and improve clarity
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: Add changelog notes
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix: Improve stability of the target fetcher?
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix: Update template error handling and improve Open Graph tag integration
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: format files and remove deubg logs
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Credit CELPHASE for mascot design (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Credit CELPHASE for mascot design (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: Allow twitter prefixed OG tags by default
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: replace /tmp with /var
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Update docs/docs/CHANGELOG.md
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* Update docs/docs/admin/configuration/open-graph.mdx
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* chore: add fediverse to default prefixes (#og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Remove og-query-distinct flag
This commit removes the `og-query-distinct` flag and
associated logic. URLs with different query parameters
will now always be treated as the same cache key for Open
Graph tags. This simplifies the caching logic and
improves performance.
Additionally, the http client used for fetching OG tags
is now a member of the OGTagCache struct, rather than a
global variable. This improves testability and allows
for more flexible configuration in the future.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Update docs/docs/admin/configuration/open-graph.mdx
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* docs: remove og tags references
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor: rename url > u to not overlap package name
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Update internal/ogtags/cache.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* Update internal/ogtags/cache.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* fix(tests): Don't use network when network access is disabled
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Handle nil URL in GetOGTags (og-tags)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: sort installation docs alphabetically
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(tests): validate that no duplicate requests are made
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style(tests): remove unused ok var
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* docs: convert to table fmt
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): Enhance OG tag fetching and caching
Adds additional approved OG tags (`keywords`, `author`), improves
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update generated templ's after format
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(tests): update integration_test.go to reflect the new behavior of fetchHTMLDocument
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Revert "data/botPolicies: allow iMessage scraper by default (#178)"
This reverts commit 21a9d777
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Simplify ogTags access in cache test.
Didn't know this was possible! wow!
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Handle request timeouts when fetching OG tags (#og-tags)
Cache a nil result for half the TTL to avoid repeatedly
requesting a timed-out URL.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: make OG tags passthrough option function.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Fix: Handle timeouts and non-200 responses when fetching OG tags (og-tags)
- Cache empty results for timeouts and non-200 status codes
to avoid spamming the server.
- Use a non-nil empty map to represent empty results in the
cache, as nil would be a cache miss.
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(og-tags): switch to http.MaxBytesReader
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore(og-tags): add noindex, nofollow meta tag and update error line numbers
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* web/js: update page to allow users to read the "Why am I seeing this?", complete with a button to send them through after challenge completed, and a 30s timeout that does the same.
* .gitignore: added .DS_store.
* docs/docs/CHANGELOG: added to the Unreleased section as requested in code quality guidelines
* web: pushing index_templ.go alongside this update.
* package.json: added postcss to dependencies list.
* package-lock: added postcss to dependencies
* Revert "package-lock: added postcss to dependencies"
This reverts commit bf02e7ba56e8bf8705821d4f4864c66b1ef614bf.
* Revert "package.json: added postcss to dependencies list."
This reverts commit 1a38c63049dc75099dc652ed725c7862eef4b3e4.
* web/js: OG comments are important
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* web/js: add project license in the JavaScript used by Anubis
This will allow LibreJS users to pass the captcha without problems
without having to whitelist anubis manually.
* Update docs/docs/CHANGELOG.md
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Fijxu <fijxu@nadeko.net>
---------
Signed-off-by: Fijxu <fijxu@nadeko.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* cmd/anubis: add a debug option for benchmarking hashrate
Having the ability to benchmark different proof-of-work implementations
is useful for extending Anubis. This adds a flag `--debug-benchmark-js`
(and its associated environment variable `DEBUG_BENCHMARK_JS`) for
serving a tool to do so.
Internally, a there is a new policy action, "DEBUG_BENCHMARK", which
serves the benchmarking tool instead of a challenge. The flag then
replaces all bot rules with a special rule matching every request
to that action. The benchmark page makes heavy use of inline styles,
because currently all global styles are shared across all pages. This
could be fixed, but I wanted to avoid major changes to the templates.
* web/js: add signal for aborting an active proof-of-work algorithm
Both proof-of-work algorithms now take an optional `AbortSignal`, which
immediately terminates all workers and returns `false` if aborted before
the challenge is complete.
* web/js: add algorithm comparison to the benchmark page
"Compare:" is added to the benchmark page for testing the relative
performance between two algorithms. Since benchmark runs generally have
high variance, it may take a while for the averages to converge on a
stable difference.
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
Since the challenge is done off of the main thread, there is no simple
way to report the progress done towards completing it. This change
adds a callback parameter, `progressCallback`, which is called with
the most recently attempted nonce every ~1024 iterations (should this
be configurable?). For the single-threaded "slow" algorithm, this is
exactly every 1024 iterations. For the multi-threaded "fast" algorithm,
threads take turns reporting in a round-robin as then notice they
have passed a multiple of 1024. This complexity is to avoid individual
threads falling behind their siblings due to the overhead of messaging
the main thread. To minimize this overhead as much as possible, a
regular number is sent instead of an object.
With the new information provided by the callback, a hash rate display
is added to the challenge page. This display is updated at most once
per second and set with tabular numbers to avoid the constantly changing
value being too visually distracting.
* web: show a progress bar based on completion probability
To provide more feedback to the user, the spinner is replaced with a
progress bar of the probability the challenge is complete. Since it
looks a little weird that a progress bar would fill up a quarter of the
way and then jump to the end (even though the probability would make
that happen 1 in 4 times), the bar is mapped with a quadratic easing
function to move faster at the beginning and then slow down as the
probability of redirection increases. If the probability exceeds 90%,
a message appears letting the user know things are taking longer than
expected and to continue being patient.
Signed-off-by: Xe Iaso <me@xeiaso.net>
Closes#125Closes#40
Among other things, this moves all of the asset generation to run within
the context of an npm script. Developer documentation stubs have been
added so that people can get started more easily.
The top-level Dockerfile (which is no longer used in production) has
been removed as its presence has been causing confusion. This changeset
will break it anyways.
These changes will make for less "repo churn" as the static assets are
built and rebuilt, at the cost of making the build step more complicated
for downstream packagers. If this becomes a burden, we can explore
making a "release tarball" that contains pre-massaged outputs.
* Refactor anubis to split business logic into a lib, and cmd to just be direct usage.
* Post-rebase fixes.
* Update changelog, remove unnecessary one.
* lib: refactor this
This is mostly based on my personal preferences for how Go code should
be laid out. I'm not sold on the package name "lib" (I'd call it anubis
but that would stutter), but people are probably gonna import it as
libanubis so it's likely fine.
Packages have been "flattened" to centralize implementation with area of
concern. This goes against the Java-esque style that many people like,
but I think this helps make things simple.
Most notably: the dnsbl client (which is a hack) is an internal package
until it's made more generic. Then it can be made external.
I also fixed the logic such that `go generate` works and rebased on
main.
* internal/test: run tests iff npx exists and DONT_USE_NETWORK is not set
Signed-off-by: Xe Iaso <me@xeiaso.net>
* internal/test: install deps
Signed-off-by: Xe Iaso <me@xeiaso.net>
* .github/workflows: verbose go tests?
Signed-off-by: Xe Iaso <me@xeiaso.net>
* internal/test: sleep 2
Signed-off-by: Xe Iaso <me@xeiaso.net>
* internal/test: nix this test so CI works
Signed-off-by: Xe Iaso <me@xeiaso.net>
* internal/test: warmup per browser?
Signed-off-by: Xe Iaso <me@xeiaso.net>
* internal/test: disable for now :(
Signed-off-by: Xe Iaso <me@xeiaso.net>
* lib/anubis: do not apply bot rules if address check fails
Closes#83
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
