# Go-Live Runbook — HQ ops console (`apps/hq`)

How to put the internal console on your AWS box for real use. The store product is
unaffected (it stays local, D14); this is the vendor-side cloud app doc 11 always
prescribed. Engine: SQLite for dev, **Postgres in production via `DATABASE_URL`**
(D19 — the dual-engine switch is delivered; see §7).

## 0. The decided target (D20)

| Piece | Decision |
|---|---|
| URL | **`https://hq.simssoftware.com`** (subdomain of the company domain) |
| DNS | **GoDaddy** — simssoftware.com stays at the registrar; one A record added |
| Server | The existing company **AWS EC2** — HQ gets its own section on the box (`/opt/sims-hq`), beside whatever else runs there |
| TLS | Caddy reverse proxy (auto Let's Encrypt) — or a server block in the box's existing nginx if one already fronts other sites |
| Process | pm2, started **from the repo root** |
| Prod data | **Born in RDS Postgres** — provision RDS first, set `DATABASE_URL`, THEN run the APEX cutover import straight into it. Sequencing: database first, app section second, DNS/TLS last. |

**GoDaddy DNS (5 minutes):** My Products → simssoftware.com → DNS → Add record:
Type `A`, Name `hq`, Value `<EC2 Elastic IP>`, TTL 600 (raise to 1h once stable).
Nothing else on the domain changes; existing www/mail records are untouched.

## What you provide (the only blockers)
1. **A server** — a small AWS EC2 instance (t3.small is plenty for 5 users), Ubuntu 22.04+,
   with a domain or Elastic IP and HTTPS (via a reverse proxy — step 4).
2. **A GitHub repo** — so the box can pull the code (or we copy a build artifact).
3. Later, per capability: a Google Cloud OAuth app (Gmail), a read-only AWS IAM user
   (cost data), and your **APEX CSV export** (the 300 clients).

## 1. Get the code on the box
```bash
sudo mkdir -p /opt/sims-hq && sudo chown $USER /opt/sims-hq
git clone <your-repo-url> /opt/sims-hq        # after `gh repo create sims-hq` + push
cd /opt/sims-hq
# Node 20+ (nvm or NodeSource). Then:
npm install
```

## 2. Configure
```bash
cp apps/hq/.env.example apps/hq/.env
# Fill at minimum SIMS_PIN_PEPPER and HQ_SECRET_KEY (generators are in the file).
# Set HQ_DATA_DIR to a path on a backed-up volume, e.g. /var/lib/sims-hq/data.
```
`SIMS_PIN_PEPPER` **must be set before the first real login** and never changed after
(changing it invalidates every password). `HQ_SECRET_KEY` must be set before connecting
Gmail.

## 3. Build + run under a process manager
```bash
cd apps/hq-web && npm run build && cd ../..     # builds the web UI the server serves
# First boot prints a one-time owner password — capture it from the logs.
env $(grep -v '^#' apps/hq/.env | xargs) node apps/hq/dist/server.cjs   # smoke test, Ctrl-C

# Keep it running with pm2 (or a systemd unit). Start it FROM THE REPO ROOT:
# the DB path resolves to ./data/hq.db against the process CWD, so a pm2 app
# started from apps/hq would boot an empty database in apps/hq/data instead.
npm i -g pm2
env $(grep -v '^#' apps/hq/.env | xargs) pm2 start apps/hq/dist/server.cjs --name sims-hq
pm2 save && pm2 startup
```
Owner login on first boot is `admin@tecnostac.com` + the printed password (change it
immediately). The Gmail-disconnected banner is expected until step 6.

## 4. Put HTTPS in front (never expose Node directly)
Reverse-proxy 443 → `localhost:5182` with Caddy (simplest) or nginx + certbot:
```
# /etc/caddy/Caddyfile
hq.simssoftware.com {
  reverse_proxy localhost:5182
}
# If the box already runs nginx for other sites, use a server block instead:
#   server {
#     listen 443 ssl; server_name hq.simssoftware.com;
#     # certbot --nginx -d hq.simssoftware.com issues the cert
#     location / { proxy_pass http://127.0.0.1:5182; proxy_set_header Host $host;
#                  proxy_set_header X-Forwarded-For $remote_addr; }
#   }
```
Lock the EC2 security group so only 443 is public; 5182 stays localhost-only. The public
`/share/:token` links are served through this same proxy — that's intended (they're the
one unauthenticated route, token-guarded, rate-limited, revocable).

## 5. Backups (do this on day one)
```bash
chmod +x apps/hq/scripts/backup-hq.sh
# Nightly to S3 + 14-day local rotation:
crontab -e
0 2 * * * HQ_DATA_DIR=/var/lib/sims-hq/data S3_BUCKET=s3://your-bucket/hq-backups /opt/sims-hq/apps/hq/scripts/backup-hq.sh >> /var/log/sims-hq-backup.log 2>&1
```
Do a **restore drill** once: copy a backup to a scratch box and boot the app against it.

## 6. Connect the capabilities (each optional, each unlocks a feature)
- **Gmail** (sending): create a Google Cloud OAuth client, publish the app to Production,
  set `GOOGLE_CLIENT_ID/SECRET` + `HQ_SECRET_KEY`, then `npx tsx apps/hq/scripts/gmail-connect.ts`
  and grant consent as the company mailbox. The banner clears; auto-reminders start sending.
- **AWS cost per client**: create a read-only IAM user with Cost Explorer access, tag each
  client's cloud resources with a `client` cost-allocation tag, set `AWS_*`. The scheduler
  pulls monthly.
- **Your 300 clients**: export `clients.csv` + `invoices.csv` from APEX into `import-drop/`,
  then run the importer (verification report first, `--commit` after you approve).

## 6b. Funnel slice knobs (D16 — set these at go-live)
- **`share.base_url` setting** (required before quote follow-ups can email): set it to the
  public origin, i.e. `https://hq.simssoftware.com`, via the settings API or a one-line SQL
  insert into `setting`. The escalating quote follow-up **refuses to send** without it —
  a client email carrying a dead relative link is worse than no email.
- **`quote.followup.policy` setting**: `manual` (default — nudges queue on the dashboard
  and a person clicks Send) or `auto` (client emails go out unattended after each daily
  scan). Start on `manual`; flip to `auto` once the letter text has been seen in anger.
- **Reminder cadences are dated rows**, not env vars: `reminder_schedule` ships seeded with
  `quote_followup` at 3/7/14 days and `invoice_overdue` at 7/15/30 days past invoice date.
  To change a cadence (or the follow-up email wording), insert a **new dated row** —
  effective from a business date, no release, no restart.
- **Employee accounts**: after first boot, open **Employees** (owner-only nav) and create
  the real manager/staff logins. Staff see only their own pipeline and reminder queue;
  owner/manager see everything. Deactivating someone kills their session immediately.
- **Account owners**: assign an owner on each client (Client 360 → Account owner) so
  leads route to the right person's queue; unassigned rows stay visible to owner/manager.

## 7. Production database — RDS Postgres (do this FIRST; D19 delivered the switch)
The dual-engine work is **done** (D19): the app selects its engine from `DATABASE_URL` —
unset → SQLite (dev), set → Postgres. Migrations apply automatically from
`migrations-pg.ts` on boot. Go-live order (so data is born in Postgres, never migrated):

1. Provision RDS PostgreSQL 17 (db.t4g.micro is plenty), same VPC as the EC2, **not**
   publicly accessible; security group allows 5432 from the EC2's group only.
2. Create database `hq` + a scoped `hq` role; put
   `DATABASE_URL=postgres://hq:<pw>@<rds-endpoint>:5432/hq` in `apps/hq/.env`.
3. Boot once — migrations run, first-boot seed prints the owner password.
4. **Then** run the APEX cutover UI (stage → verify → commit): the 300 clients import
   directly into Postgres.

Known nuance (D19): client search is case-sensitive on Postgres until repos move to
`LOWER(...) LIKE LOWER(?)`. SQLite + `data/hq.db` remains the dev/test path; the §5
sqlite backup script is superseded in prod by **RDS automated snapshots** (enable 7-day
retention) — keep the S3 dump cron as belt-and-braces via `pg_dump` if desired.

## Health & operations
- `GET /api/health` → `{ ok: true }` for your uptime monitor.
- Logs: `pm2 logs sims-hq`. The scheduler runs on boot and every 6h (reminders, bounces,
  AWS pull when configured).
- Every mutation is audit-logged in the DB; the public share route and previews write nothing.