Magent/docs/archive/production-cutover-2026-09-07.md

Historical production cutover notes — superseded

These notes describe the temporary AMS-DEV01 setup, not the current production deployment. Do not run these cutover or rollback instructions against the live service. See current production instructions.

Production uses main, /home/zak/magent-production on AMS-DEV01 and docker-compose.production.yml. The legacy prod deployment and beta are not overwritten. Main runs CI verification; production activation is deliberately manual during the initial cutover.

Only API connection URLs/credentials and SMTP configuration are exported by scripts/prepare_production_settings.py. It reads the source's effective settings, uses an explicit allowlist, refuses existing output directories, and creates private files. It never copies a database, users, invite codes, issues, history, tokens, sessions, branding or notification templates. A new bootstrap admin and JWT secret are generated. Retrieve the bootstrap credentials from the protected bootstrap-admin.json on the server; never commit them.

The initial production .env enables MAGENT_COMING_SOON=true and disables BACKGROUND_TASKS_ENABLED. This presents the cover at / and pauses automatic imports and repair emails. The cover is not an authentication/security boundary; normal API authentication remains in force. Administrators can use /login.

Run docker compose -f docker-compose.production.yml up -d --build from the production directory. Caddy should proxy this hostname to 10.30.1.32:3200; Next forwards /api internally. The backend health port is localhost-only at 8200. Do not alter beta's route or other Caddy sites.

Before public activation, validate Caddy config, save its existing configuration, verify HTTPS, admin login, connection diagnostics and the empty-client-data state. Do not send SMTP tests without approval. Keep the old upstream for rollback.

At launch, set MAGENT_COMING_SOON=false and BACKGROUND_TASKS_ENABLED=true, then recreate the container. External service records can then be imported through normal synchronization; no beta client data is migrated. Review quality profiles, root folders, invite policy and notification rules in admin settings before use.

Initial cutover — 7 September 2026

• Public HTTPS cover and /api/health verified after cutover.
• Caddy: AMS-CAD01, /etc/caddy/Caddyfile, systemd caddy.service.
• SSH worked via 10.30.40.254 using HostKeyAlias=10.30.41.254.
• Only the magent.grizzlyflix.co.nz upstream changed, from 10.30.1.81:3002 to 10.30.1.32:3200. Both beta blocks were unchanged.
• Rollback configuration: /etc/caddy/Caddyfile.bak-magent-prod-20260907T0130. Restore it, run sudo caddy validate --config /etc/caddy/Caddyfile, then sudo systemctl reload caddy. Review subsequent edits before restoring the whole file; the old application was not stopped or deleted.
• Initial database: one newly generated bootstrap admin; zero invites, issues, cached requests, actions or snapshots. Login smoke-testing subsequently creates normal admin login activity only.
• Retrieve /home/zak/magent-production/bootstrap-admin.json securely on AMS-DEV01. Sign in at /login, then open /admin while the cover is active.
• No SMTP message was sent as part of validation. Background jobs remain paused.

Cover resilience update

The application host subsequently became unreachable over TCP from Caddy (both 3100 and 3200 timed out, despite responding to ping). The cover is now served directly by Caddy from /var/lib/caddy/magent-cover/index.html, sourced from docker/coming-soon.html, for /, /coming-soon and /coming-soon/. It needs no application server, JavaScript, API or external assets. Other paths retain the production reverse proxy. Full launch now also requires removing the @landing/static handle block from the production Caddy site once upstream connectivity is stable; the environment switch alone is insufficient. Pre-static configuration backup: /etc/caddy/Caddyfile.bak-magent-static-20260907T0145.