Operations

systemd unit

/etc/systemd/system/tndm.service, system scope, running as loca:

[Unit]
Description=TNDM Tandem Audit Platform
After=network-online.target
Wants=network-online.target
 
[Service]
Type=simple
User=loca
Group=loca
WorkingDirectory=/home/loca/dev/tandem-audit
Environment=HOME=/home/loca
Environment=NODE_ENV=production
Environment=HOST=127.0.0.1
Environment=PORT=51818
Environment=DATA_DIR=/home/loca/tndm
ExecStart=/usr/bin/node build/index.js
Restart=on-failure
RestartSec=5
TimeoutStopSec=20
 
[Install]
WantedBy=multi-user.target

Variable	Effect
`HOST`	adapter-node binds to this interface; `127.0.0.1` is what keeps the service off the public interface
`PORT`	adapter-node listen port; nginx is the only client
`DATA_DIR`	read once at module load and resolved to an absolute path (`src/lib/server/store.ts:10`), with `/home/loca/tndm` as the fallback if unset or blank
`NODE_ENV`	production mode for the Node runtime
`HOME`	set explicitly so the unit does not depend on the login environment

HOST and PORT are consumed by the adapter’s own server, not by application code — there is no reference to either in src/. DATA_DIR is the only environment variable the application itself reads, and the storage skeleton beneath it is created lazily on first use (src/lib/server/store.ts:37-48).

ExecStart runs build/index.js, the artefact of npm run build, which is vite build (package.json:8). A source change is therefore not live until the build is re-run and the service restarted.

Observed state: systemctl is-active tndm returns active and is-enabled returns enabled.

Runbook

Service control needs root, and sudo -n here is deliberate: every command below runs non-interactively.

sudo -n systemctl start tndm
sudo -n systemctl stop tndm
sudo -n systemctl restart tndm

Reading state needs no privilege at all:

systemctl is-active tndm
systemctl status tndm --no-pager
journalctl -u tndm -f

journalctl works without sudo even though loca is in neither adm nor systemd-journal (id -nG reports loca sudo users): the unit runs as User=loca, so its entries belong to the invoking user and journald shows them. Verified by running journalctl -u tndm -n 2 --no-pager as loca and getting the unit’s own node[...] lines back.

Redeploy after a code change. The unit executes the build output, never the sources, so a restart on its own changes nothing:

cd /home/loca/dev/tandem-audit
npm run build
sudo -n systemctl restart tndm

Then check, in this order:

Step	Command	Expected
the unit came back	`systemctl is-active tndm`	`active`
it bound the right interface	`ss -ltn`	one row for 51818, on `127.0.0.1`, never `0.0.0.0`
it answers through nginx	`curl -s https://tndm.loca.zone/api/decks`	a JSON array; one deck today
nothing threw on boot	`journalctl -u tndm -n 30 --no-pager`	no stack traces after the restart timestamp

A restart severs every open event stream. The broker is an in-process EventEmitter with one listener per connection and no state outside the process (src/lib/server/broker.ts:9-24), so there is nothing to hand over. For an open workspace that is recoverable rather than lossy: the client reconnects itself with backoff from 1 s to a 30 s ceiling and recomputes its cursor from the last event it holds before reopening the stream (src/routes/s/[sessionId]/+page.svelte:280-302, cursor at :284), and the server replays everything past ?after= on connect (src/routes/api/sessions/[id]/stream/+server.ts:44-64). An agent tailing the stream with curl -N has no such logic and must be restarted by hand.

nginx

One file, /etc/nginx/sites-available/tndm.loca.zone, symlinked into sites-enabled, holding both server blocks. There is no separate wiki.tndm.loca.zone file; the wiki block lives in the same file as the application block.

Application block, server_name tndm.loca.zone, with client_max_body_size 64m:

location /api/ {
    proxy_pass http://127.0.0.1:51818;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 1h;
    chunked_transfer_encoding off;
}
 
location / {
    proxy_pass http://127.0.0.1:51818;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Four of those directives exist specifically so GET /api/sessions/:id/stream works:

Directive	Why the event stream needs it
`proxy_buffering off`	nginx would otherwise hold response chunks until a buffer fills, so events would arrive in bursts or not at all until the stream ended
`proxy_cache off`	an event stream is unique per cursor and must never be served from cache
`proxy_read_timeout 1h`	headroom rather than a requirement: the application writes a keepalive comment every 25 s on an unconditional interval (`src/routes/api/sessions/[id]/stream/+server.ts:67`), which already stays inside nginx’s 60 s default. The hour covers a stalled event loop that delays a ping, not the idle case
`chunked_transfer_encoding off`	avoids re-chunking a response the application already frames as SSE records
`Connection ""`	HTTP/1.1 keep-alive to the upstream instead of forwarding a client `Connection` header

The application also sets x-accel-buffering: no on the stream response (src/routes/api/sessions/[id]/stream/+server.ts:81), which is the in-band version of the same instruction, and emits a keepalive comment every 25 seconds (src/routes/api/sessions/[id]/stream/+server.ts:7).

client_max_body_size 64m raises nginx’s 1 MB default. The only endpoint that accepts a large request body is the scribble upload, which carries a base64-encoded PNG inside its JSON body (src/routes/api/sessions/[id]/scribble/+server.ts:12-18, validated as a data:image/png;base64 URL at src/lib/server/store.ts:396-400). Whether the specific value of 64m was chosen for that endpoint is not recorded anywhere, so treat the causal link as unverified; the sizing relationship itself is real.

Wiki block, server_name wiki.tndm.loca.zone:

location / {
    root /home/loca/dev/wikis/tndm/current;
    try_files $uri $uri.html $uri/ =404;
}

The $uri.html term is what lets Quartz’s extensionless routes resolve. Reload after edits with sudo nginx -t && sudo systemctl reload nginx.

TLS

One certbot certificate covers both names, at /etc/letsencrypt/live/tndm.loca.zone/. Both server blocks carry certbot-managed lines:

listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/tndm.loca.zone/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/tndm.loca.zone/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

Certbot also appended two plain-HTTP blocks, one per name, that redirect everything:

if ($host = tndm.loca.zone) {
    return 301 https://$host$request_uri;
}

Renewal is certbot-managed; because both names share one certificate, a renewal covers the application and the wiki together.

Backup and restore

Decks are reproducible; sessions are not.

A deck is derived data. deck.json is the output of parsing its source document, and the source path is recorded in the deck itself as deck.source (src/lib/mdframes/types.ts:14, set from the import at src/lib/server/importer.ts:87). Re-running POST /api/decks/import with the same id and copy_from rebuilds both deck.json and the copied assets, overwriting in place and leaving sessions/ untouched (src/lib/server/importer.ts:255-261). Losing decks/ therefore costs an import, not data.

Sessions are irreplaceable:

Path	Lost if deleted
`sessions/<id>/events.jsonl`	the entire review — every note, task, verdict, transcript and scribble reference. Nothing else stores this
`sessions/<id>/session.json`	the deck binding and the collab link; the session becomes unreadable because `readSession` 404s without it (`src/lib/server/store.ts:170-177`)
`sessions/<id>/scribbles/`	the PNG overlays. The events still reference the filenames, so the log survives with dangling links

Observed sizes today: du -sh reports 28M for decks/, effectively all of it the first deck’s copied PNGs under decks/qwizz-2026-08-20/assets, against 120K for sessions/ — 100K for the evidence session and 16K for the live one. The irreplaceable half is the small half.

Snapshot, one command, no service interruption:

tar czf "/home/loca/tndm-sessions-$(date +%Y%m%d-%H%M%S).tgz" -C /home/loca/tndm sessions

-C keeps paths relative to DATA_DIR, so the archive extracts cleanly over any data dir. events.jsonl is append-only and tolerant of a torn trailing line on read (src/lib/server/store.ts:249-253), so a snapshot taken while a review is live degrades to at most one lost trailing event rather than a corrupt session. Archiving . instead of sessions adds the 28M of deck assets and buys nothing a re-import would not.

Restore, with the service stopped:

sudo -n systemctl stop tndm
tar xzf /home/loca/tndm-sessions-20260820-114500.tgz -C /home/loca/tndm
sudo -n systemctl start tndm
curl -s https://tndm.loca.zone/api/sessions

Stopping first is not optional. A session’s high-water seq is cached in the process and, once cached, trusted without re-reading the log (src/lib/server/store.ts:324-331, refreshed on every append at :348). Restoring a log whose highest seq sits above the running process’s cached value would make the next append reuse a seq that already exists in the file. A stopped service has no cache to be stale.

Decks come back by re-import rather than from a tarball. For the deck that exists today:

curl -s -X POST https://tndm.loca.zone/api/decks/import -H 'content-type: application/json' -d '{"md_path":"/home/loca/dev/wrdp/q5vault/audits/qwizz/2026-08-20-feedback-tour.md","copy_from":"/home/loca/dev/wrdp/q5vault/assets/qwizz-tour/2026-08-20","id":"qwizz-2026-08-20"}'

No backup automation exists. crontab -l for loca reports no crontab, and the only backup unit on this host is stacks-backup.timer driving /home/loca/dev/coder/scripts/backup-stacks.sh, which is scoped to the cdsr and code IDE stacks: it tars /home/loca/.continue and selected paths under /home/loca/dev, never mentions tndm, and DATA_DIR at /home/loca/tndm lies outside every path it touches. Until something schedules the snapshot above, a completed review is one rm -rf from gone.

Proposed unit hardening

Not applied. The unit quoted at the top of this page carries none of these directives. This is a proposal to be reviewed, applied and then verified — not a description of the running service.

Drop-in file, /etc/systemd/system/tndm.service.d/hardening.conf. Fragment and install in one paste:

sudo -n mkdir -p /etc/systemd/system/tndm.service.d
sudo -n tee /etc/systemd/system/tndm.service.d/hardening.conf > /dev/null <<'EOF'
[Service]
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/loca/tndm
ProtectKernelTunables=yes
ProtectControlGroups=yes
RestrictSUIDSGID=yes
EOF
sudo -n systemctl daemon-reload
sudo -n systemctl restart tndm

Two of those lines are load-bearing and easy to get wrong:

ProtectHome=read-only, never ProtectHome=yes. The service has to keep reading /home: it runs build/index.js out of /home/loca/dev/tandem-audit, and every import reads a file under /home/loca/dev (src/lib/server/importer.ts:77-90). ProtectHome=yes presents an empty /home and breaks both at once.
ReadWritePaths=/home/loca/tndm is what keeps DATA_DIR writable. ProtectSystem=strict makes the whole filesystem read-only apart from /dev, /proc and /sys, and ProtectHome=read-only covers the data dir too because it lives under /home/loca; the listed path is excluded from both.

A unit that starts proves nothing here, because the sandbox only bites on access. Verification has to exercise both directions of the boundary: one import, which reads /home/loca/dev, and one scribble upload, which writes DATA_DIR. A service that boots and serves decks while silently failing to write is exactly the failure this fragment can introduce.

Port convention

The host rule is that all loca.zone app backends bind to ports 51000-53000, because standard ports are kept free on a shared dev server (/home/loca/AGENTS.md:51). TNDM uses 51818, which is inside that range and registered as tndm (Tandem Audit): 51818 (/home/loca/AGENTS.md:70). The application registry entry naming the directory, both domains, the unit and the data dir is at /home/loca/AGENTS.md:190.

The bind is loopback only, observed:

LISTEN 0 511 127.0.0.1:51818 0.0.0.0:* users:(("node",pid=2738602,fd=21))

Nothing outside the host can reach the port directly; nginx is the only path in.

Wiki build

/home/loca/dev/wikis/build.sh tndm --check-only   # validate, discard output
/home/loca/dev/wikis/build.sh tndm                # publish

Both paths take a global lock (/home/loca/dev/wikis/build.sh:36-42), run scripts/validate-ssot.py, which no-ops for a wiki that ships no wiki-ia.json manifest (/home/loca/dev/wikis/build.sh:51-53), then temporarily swap this wiki’s quartz.config.yaml into the shared Quartz checkout, restoring it afterwards (/home/loca/dev/wikis/build.sh:73-82, restore trap at /home/loca/dev/wikis/build.sh:71).

Check-only builds into dist-check-tmp, asserts index.html was produced, and removes the directory on exit (/home/loca/dev/wikis/build.sh:87-106, cleanup at /home/loca/dev/wikis/build.sh:26-32). Nothing published changes.

A production build renders into dist-tmp, again asserting index.html exists before it will swap (/home/loca/dev/wikis/build.sh:108-116). The swap is atomic: the temp directory is renamed to dist-<epoch>.<ns>-<pid>, a current.tmp symlink is pointed at it, and that symlink is renamed over current (/home/loca/dev/wikis/build.sh:126-133). Because the final step is a rename of a symlink, readers never observe a missing or half-written document root. Retention then keeps the active dist plus the five newest others and deletes the rest (/home/loca/dev/wikis/build.sh:135-148).

The first production build has run. /home/loca/dev/wikis/tndm/current exists and points at dist-1787217120.289399339-2868474, and https://wiki.tndm.loca.zone/ answers 200. Before that build the host answered 404, because current is created only on the non-check path.

Version control

The application is a git repository at /home/loca/dev/tandem-audit with exactly one commit, 7e75cde (“TNDM tandem-audit platform: mdframes parser, event bus, workspace UI, voice+scribble, collab rail”). There is no history to bisect; the working tree is the record.

This wiki is not tracked. /home/loca/dev/wikis is itself a git repository with 1760 tracked files, but nothing under tndm/ is among them (git -C /home/loca/dev/wikis ls-files tndm returns nothing), and the directory is not ignored either (git check-ignore -v tndm/content/operations.md exits 1). These pages are untracked working-tree files, so a deletion here is not recoverable from git.

That has a visible consequence on the rendered site. quartz.config.yaml resolves page dates in the order frontmatter, git, filesystem, with modified as the default type (/home/loca/dev/wikis/tndm/quartz.config.yaml:42-49). With no commit touching these files the git tier contributes nothing, so displayed dates fall through to file mtimes and shift on every edit. A page whose date carries meaning should state it in frontmatter. Whether a tndm build also prints a warning about the missing history is unverified: no build log for this wiki exists on disk.

Health checks and failure modes

Symptom	What it means	First move
502 from `tndm.loca.zone`	nginx is up and nothing is listening on 51818: unit stopped, crashed, or mid-restart. Both `location /` and `location /api/` proxy to the same upstream, so they always fail together	`systemctl is-active tndm`, then `journalctl -u tndm -n 50 --no-pager`
`GET /api/decks` returns `[]`	not a fault. Either the data dir holds no decks, or the process resolved a different one — `DATA_DIR` is read once at module load and falls back to `/home/loca/tndm` when unset or blank (`src/lib/server/store.ts:10`)	compare `ls /home/loca/tndm/decks` against `systemctl show tndm -p Environment`
404 from `wiki.tndm.loca.zone`	the `current` symlink is missing or points at a dist that retention deleted; the vhost root is that symlink	`ls -l /home/loca/dev/wikis/tndm/`, then rebuild
events arrive in bursts, or only when the stream closes	SSE buffering. The `/api/` block lost `proxy_buffering off`, `proxy_cache off` or `chunked_transfer_encoding off` (`/etc/nginx/sites-available/tndm.loca.zone:6-19`). The application’s own `x-accel-buffering: no` covers nginx but not another proxy in front of it	`sudo -n nginx -T`, then read the `location /api/` block
a scribble upload fails at the edge	the PNG exceeded `client_max_body_size 64m` (`/etc/nginx/sites-available/tndm.loca.zone:4`). nginx rejects it before the application sees a byte, so the journal stays silent and the error appears in the nginx log	`sudo -n tail /var/log/nginx/error.log`
`decks/` growing	one full asset copy per imported deck, never shared between decks: every asset is copied to `deckAssetPath(id, sub)` under that deck’s own directory (`src/lib/server/importer.ts:228-230`), and the first deck alone is 28M of PNGs. Importing the same gallery under N ids costs N times that	`du -sh /home/loca/tndm/decks/*`

One thing not to rely on. The unit is Restart=on-failure with RestartSec=5, and systemd’s start limit for it is 5 starts per 10 s window (systemctl show tndm -p StartLimitBurst -p StartLimitIntervalUSec reports 5 and 10s). Because a 5 s gap puts at most three starts inside a 10 s window, that burst of five can never be reached: a service that crashes on every boot restarts every five seconds indefinitely instead of settling into failed. A crash loop is therefore diagnosed from the journal, not from unit state, and systemctl is-active is not a sufficient health check on its own.

No authentication

Nothing guards this application. There is no src/hooks.server.ts, and a grep of src/ for authorization, bearer, cookie, JWT and credential handling matches nothing outside a screenshot caption in the parser fixture. The only controls are the loopback bind and TLS termination at the nginx edge, which means reachability of the vhost is the access control.

security carries the full picture: what an anonymous caller can do endpoint by endpoint, what the code does already contain, and the three options for closing it. None of them has been chosen. Nothing on this page should be read as saying the service is protected.

description	The systemd unit, the runbook, nginx vhosts, TLS, backup and restore, a proposed unit-hardening fragment that is not yet applied, the port entry, the wiki build, version control, and the health checks and failure modes that matter here.
tags	tndm, operations, deployment

TNDM — Tandem Audit

Explorer