Installation repair

Fresh single-VM Debian 13 installations now record a private recovery baseline and install /usr/local/sbin/nodara-repair. The tool uses the system Python standard library; it remains available when the panel or its virtual environment is broken. It does not run through a privileged web endpoint.

sudo nodara-repair check
sudo nodara-repair plan
sudo nodara-repair apply
# Only after an interrupted/incomplete repair:
sudo nodara-repair recover

check and plan inspect the installation without changing managed files or services. apply shows the current file paths, service states and interruptions, then requires REPAIR. recover requires RECOVER. Reports never display environment values or private keys. A nonzero exit means inspection or recovery could not finish; retain the baseline and journal.

Repair restores the exact recorded application source, three Nodara service definitions, environment files, and Nodara proxy site/enable links. This also replaces intentional changes to those files: review the plan. Nginx, Apache and Caddy are supported; Caddy's Nodara site is separate from its shared Caddyfile so other site files can remain untouched. The whole proxy configuration must validate before activation.

Secrets and agent credentials are restored unchanged from protected copies. The command does not upgrade code, download packages, regenerate credentials, change firewall rules, renew certificates, accept licences, migrate schemas, restore databases, or restart Docker. Dependencies, Docker availability, certificate/key consistency, database integrity and pending jobs are checked first. Missing dependencies and damaged data need manual recovery.

Maintenance and rollback

Run from the VM console or an independent SSH connection. Stop submitting work during maintenance. Pending or running jobs block repair; resolve them before proceeding. For changes affecting the shared application, the tool drains the panel, worker and agent together. A short database write lock prevents a new worker claim between the idle check and stopping services. Game containers are not stopped by the repair command. It refuses to continue if a graceful stop cannot be established.

Only affected Nodara services and the selected proxy are reconciled. Stopped services appear explicitly in the plan. Success requires authenticated agent health, trusted HTTPS login and stable service processes. A login-page health check is not a guarantee that every user workflow works.

Before mutation, the tool persists originals and prior service states in a root-only journal. Files are replaced atomically. A failed validation or health check triggers file rollback and attempts to restore prior service states. The attempt still fails; a restart or completed operation cannot be undone. If an unrelated proxy error prevents rollback from restarting the proxy, the journal remains unfinished. Correct that unrelated error, then use recover.

After a crash, recovery accepts only the recorded original or replacement file states. Later edits block automatic recovery. Do not delete the journal or baseline to bypass this protection. Recovery restores the pre-repair state, which may still be broken; run check afterward.

Baseline and boundaries

Back up /var/lib/nodara-repair as credentials: directory mode 0700, protected files 0600, owned by root. It contains source/configuration recovery copies, including the original environment secrets. Keep separate database, node-state and Docker-volume backups. Renewing certificates remain outside the frozen baseline.

Existing/manual installations without a baseline cannot be automatically adopted. The installation updater validates and publishes a replacement baseline during a successful upgrade; ordinary repair never refreshes it. Both tools share a maintenance lock and reject unresolved recovery state. Unknown systemd overrides, unsafe path substitutions and incompatible dependency state block automatic changes.

This version checks registered game-volume ownership/existence and refuses unresolved or unrecognized node recovery state. It cannot determine whether arbitrary game files or world contents are semantically intact. Use the existing runtime/world/add-on recovery workflows for those domains; installation repair never rewrites their journals or volumes. Public CA issuance/renewal, real Minecraft gameplay, SMTP delivery and SFTP transfers remain separate pilot checks.

Automated acceptance

Current release ea2d761 passed the Linux/four-VM installation-update matrix, including all three fresh-install proxies and legacy Ubuntu commissioning. The earlier repair-specific evidence below is retained for its original revision.

Verified on 2026-09-29: all three Debian 13 proxy jobs passed at code commit 273400a. Each VM ran 16 repair checks, 198 panel tests (2 skipped), and 245 agent tests (14 skipped), followed by the installation/fault/reboot checks below. The skipped opt-in integrations remain outside this evidence.

.github/workflows/installation-repair.yml boots a fresh Debian 13 QEMU VM on a GitHub Linux runner for each proxy. It uses a disposable private CA, a fixture administrator and a synthetic console container. The tests exercise the real installer, source/unit/environment/proxy faults, file and credential restoration, unrelated-site validation failure, rollback/recovery, database/world sentinel/volume preservation, administrator login, worker-to-agent console execution, and reboot. No production credentials or baseline artifacts are uploaded.

Run the filesystem transaction checks as root in disposable Linux:

python3 -m unittest scripts.test_repair_onprem

These checks exercise file ownership/modes, secret preservation, symlinks/hardlinks, locks, stale plans, corrupted recovery material, insufficient space, failed validation, crash recovery and interrupted rollback. Consult the actual workflow result for VM acceptance; adding the workflow alone is not evidence of a successful installation.