I am looking for a good GNU Linux/ posix based system backup strategy without slopware like rsync - because #fckai.
Has anyone a good tool/ set of tools and a blog post for me? It would be really cool if it also works on NetBSD or RedoxOS - because #fcklinux and slopvalds.

#GNULinux #Debian #backup #sysops #Linux #NetBSD #RedoxOS

Last night mementomori.social got a bit slow. Not terrible, but noticeable. Load average hit 26 on 8 cores and page loads took several seconds. The cause was an nginx caching failure I had not run into before, so I am writing it down.

A crawler was hitting us from about 1500 IPs in a Singapore datacenter range. The problem was the requests sent bogus "Authorization: Basic" headers with every request.

Mastodon answers several public API endpoints with "Vary: Authorization". That is correct, since the response differs for logged in and anonymous callers. But nginx honors Vary, so a request carrying an Authorization header is a different cache entry than one without it. On our setup every distinct header value got its own entry. Each crawler request produced a cache key nobody had ever asked for, missed, and went straight through to Puma, uncached. Oof.

This was difficult to spot. Request volume looked normal for a social media server, PostgreSQL was idle, and nothing in the logs screamed "overload, overload!". The cache just stopped working on the busiest endpoints and the app servers took the whole load.

You can check your own instance in a few seconds. First without an Authorization header:

```
curl -sI https://your.instance/api/v1/trends/tags
```

Then the same request with a junk header:

```
curl -sI -H "Authorization: Basic dGVzdDp4" https://your.instance/api/v1/trends/tags
```

X-Cached only appears if you have `add_header X-Cached $upstream_cache_status` in your config, which is worth adding. Otherwise watch Age. I tried this against a few other instances and the pattern holds: without the header you get a cache hit, with it you do not.

The fix is one rule. Mastodon uses OAuth Bearer tokens and federation uses HTTP Signatures. Basic auth is never used, so any Basic header is bogus and can be rejected at the edge for no upstream cost:

```
location @proxy {
if ($http_authorization ~* "^Basic") { return 401; }
...
}
```

Check it before trusting it and always test the config for typos `sudo nginx -t` before restarting and crashing your services. Real Bearer tokens still 200, anonymous browsing still 200, POST /inbox still reaches Mastodon, and preview bots (Mastodon, Slack, Discord, Telegram, WhatsApp, facebookexternalhit) all still 200. Load went from 26 to normal (under 6 usually for our busy server) and CPU idle from 1.4 percent to 34 percent.

What actually identified this as automated was not request volume, which looks like normal browsing. In 57 minutes that one range fetched 1722 distinct hashtags. Everyone else on the instance, about a thousand real users plus all federation, fetched 1353. But one IP range went through more hashtags than the entire rest of the server, which is telling.

We now log $upstream_cache_status and $request_time to a separate file, so next time this is a ten second check instead of an hour of guessing.

If your instance feels slow and your database is idle, check your cache hit rate before you scale anything.

#MastoAdmin #SysOp #SysOps #SysAdmin #mementoMoriSocial #Nginx