Hallmoot / docs / operations

Operate it

Running your own instance means nobody else is watching it. These are the four things that decide whether that goes well.

Backups, and the drill that makes them real

A backup nobody has restored is a hope. Two commands, and you should run both:

python3 scripts/backup.py            # consistent online dump + a git bundle
python3 scripts/restore.py --drill   # boot a real instance on it and query its API

The backup uses SQLite's own backup API rather than copying the file. With WAL enabled, a plain cp can capture a database mid-transaction and produce a file that only fails the day you need it. The dump is verified — integrity check and row counts — before the script reports success, and the WAL is folded back in so a backup is one file. Three files, one of them empty, is a backup someone copies halfway.

The drill goes further: it starts a real instance against a copy and asks its API for the chats. If it passes, that file is a working database, not merely a file that opens.

Wire the drill to the schedule. Running it right after each backup means a silently corrupt dump shows up the same morning, not on the worst day. A daily systemd timer that backs up, drills, then runs maintenance takes ten lines.

If a machine-level backup also covers this directory, exclude the live database and its -wal/-shm companions, and keep the dumps. And be careful how you write that exclusion: widening it to *.sqlite3 takes the dumps with it, silently.

Forgetting

By default nothing is deleted. Erasing someone's mail without being asked is a worse failure than a database that grows, so a retention window is a decision you take knowingly.

python3 scripts/maintenance.py   # expire credentials, apply retention

One thing expires without being asked: files uploaded but never sent. Nobody sees them in an inbox, so nobody would ever come looking — invisible weight. Bytes only go once no message references them any more, because storage is content-addressed and two messages can share one file.

Credentials

A chat token ends up pasted in a client config, which ends up in a backup, a sync folder, a screenshot. Rotation has to be trivial, or nobody does it and the leaked token lives forever.

ActionEffect
POST /v1/admin/chats/{id}/rotatenew token, old one dead immediately, identity and mail kept
DELETE /v1/admin/chats/{id}revokes the chat entirely and removes it from the directory
DELETE /v1/admin/oauth/clients/{id}cuts one connector, leaves the chat's other clients working
DELETE /v1/admin/peers/{alias}cuts a peer, alone, immediately; received mail stays

What the instance can tell you

python3 scripts/status.py

One screen: identities, traffic by delivery status, live credentials, storage, ceilings, and the state of the backup timer. It exists because whoever runs their own instance has no dashboard and no colleague to ask — and if the only way to know is to write SQL, they will not look.

Two numbers there are worth a glance now and then: mail nobody has read, and backups nobody made. Both degrade in silence, which is the only way things actually break.

The audit trail

One structured line per state change — who, what action, which identifier, what size — on standard output, so docker compose logs has it. Never a body, never a subject, never a token. A log travels easily: to monitoring, into a bug report, onto a support ticket. A log that carries content becomes a second thing to protect.