# ZUCE Group — deployment & server access

Reference doc for how this project gets from a local commit to running on
the production server. Deliberately holds *no* secret values (private keys,
tokens, passwords) — only what each credential is, where it actually lives,
and the commands that use it.

## Resource discipline (read this before touching production)

This hosting account has a tight process ceiling and has repeatedly locked
itself out (SSH and even cPanel itself unreachable) from accumulated
duplicate worker processes - most recently four stacked `lsnode:` workers
found by Truehost support on 2026-10-21. A full audit (October 2026) found
no leak in the app's own code (no child_process usage, no file watchers, no
unclosed DB connections, no runaway timers) - the cause has consistently
been **the deploy/restart step itself**, not application behavior. Three
rules, no exceptions:

1. **Never run a bare `cloudlinux-selector restart`/`start`/`stop`, in a
   shell or cPanel's UI.** Always `bash deploy/safe-restart.sh` - see
   "Known issue: restart doesn't always restart" below for why.
2. **`.htaccess` at the app root is tracked in this repo** (it previously
   wasn't, which meant the `NODE_OPTIONS`/`UV_THREADPOOL_SIZE` caps it
   carries existed only as an unversioned file on the server with no way to
   confirm from a repo checkout whether they were ever silently wiped by
   cPanel's Node.js Selector UI). If `git pull` on the server ever
   complains that `.htaccess` is untracked/would be overwritten, that means
   it drifted from what's committed - compare the two before resolving,
   don't just force-overwrite blindly.
3. **Any new server-side scheduled script must close every connection it
   opens and call `process.exit()` explicitly, in every code path** - see
   "Reducing how often this happens" below. The prospect-mining automation
   and the Outlook-bridge scripts are a separate, local-machine concern;
   see their own notes further down.

## Server access

- **Host:** `51.89.254.255` (cPanel account `wemqjdto`, server `lon106`,
  Truehost shared hosting)
- **SSH port:** `1624` (non-standard — confirmed via cPanel > SSH Access)
- **App root on server:** `/home/wemqjdto/app.zucegroup.com`
- **SSH alias:** `zucegroup-prod`, configured in `~/.ssh/config` on the dev
  machine (Majid's). Connect with `ssh zucegroup-prod`.
- **SSH key:** dedicated deploy keypair at `~/.ssh/zucegroup_deploy_ed25519`
  (private, stays on the dev machine only) / `~/.ssh/zucegroup_deploy_ed25519.pub`
  (imported into cPanel > SSH Access > Manage SSH Keys > Import Key, then
  authorized there). Follows the same per-project deploy-key convention as
  this machine's other server projects.
- **cPanel login / control panel:** not captured here — see cPanel's own
  login, or the account welcome email.

## GitHub access

- **Remote:** `https://github.com/graomogo/zucegroup.git`
- **Auth:** Windows Git Credential Manager, using a **fine-grained personal
  access token** under the `graomogo` GitHub account
  (github.com/settings/personal-access-tokens), scoped to cover every active
  `graomogo` project on this dev machine (zucegroup, akodo, anoadvocates,
  teenseed, theroyalpets) rather than one token per repo. New repos still
  need adding to the token's repository list — if a push fails with
  `403 Write access to repository not granted`, that's the first thing to
  check, followed by whether the token itself has expired.
- **Credential isolation:** `credential.https://github.com.useHttpPath` is
  set to `true` globally (`git config --global ...`) — the same fix Git for
  Windows already ships by default for `dev.azure.com`. Without this, every
  repo under `github.com` shared **one** cached credential slot, so
  re-authenticating any *other* `graomogo` project on this machine would
  silently overwrite zucegroup's cached token and break its push until
  re-entered — this was the actual cause of several recurring "PAT expired"
  pushes failures this project hit, not real expiry each time. With it on,
  each repo gets its own slot (`git:https://github.com/graomogo/zucegroup.git`,
  not the old shared `git:https://github.com`), so this can't happen again.
- Cached credential can be inspected (not read) via Windows Credential
  Manager / `cmdkey /list`, entry
  `git:https://github.com/graomogo/zucegroup.git`.

## Database

- MySQL, created via cPanel's MySQL Databases tool.
- Local dev `.env` currently has placeholder credentials (`DB_USER=change-me`)
  — real credentials only exist in the server's `.env` (not committed;
  entered via cPanel's Node.js App environment variables UI) and in
  whatever password manager/notes the account owner keeps them in.

## Standard deploy procedure

1. **Local:** commit, then `git push origin main`.
2. **SSH in:** `ssh zucegroup-prod`
3. `cd ~/app.zucegroup.com`
4. `git pull origin main`
5. `npm install` (only needed if `package.json`/`package-lock.json` changed)
6. `npm run db:sync` — **only** if a model/schema changed this round (adds/
   alters tables, doesn't drop anything; see `scripts/sync-db.js`)
7. Run any one-time seed/fix script this round needs (see `package.json`
   `scripts` for the full list — `db:seed-prospects`, `db:seed-diplomatic-directory`,
   `db:fix-entities`, etc. Each is safe to re-run; they upsert rather than
   duplicate.)
8. **Restart the app — but see "Known issue: restart doesn't always
   restart" below first.** Trusting a plain Restart click/command has
   repeatedly left this account with a stuck old worker still serving
   requests (or worse, no worker at all).

`DEPLOY_NOTES.txt` in the repo root carries the specific steps/changelog for
whatever the *current* pending deploy is — this file is the general,
lasting reference for how the pipeline works.

## Known issue: restart doesn't always restart

Confirmed repeatedly on this account: `cloudlinux-selector restart` (and
cPanel's Restart button, which calls the same thing) can report
`{"result": "success"}` while the actual `lsnode:` worker process for this
app is untouched — same PID, unchanged uptime. Old workers have been
observed surviving multiple restart cycles, each one still counting against
the account's process-count ceiling (a hard limit on this hosting plan,
confirmed via cPanel's Resource Usage panel at 50/50 more than once) until
the account runs out of headroom entirely and even basic `git`/`npm`/`ssh`
commands start failing with `cannot fork()`.

**The safe restart sequence is now a script - always use it, never a bare
`cloudlinux-selector restart` or the cPanel Restart button:**

```bash
cd ~/app.zucegroup.com
bash deploy/safe-restart.sh
```

`deploy/safe-restart.sh` (committed in the repo, pulled down with everything
else) does exactly what used to be a manual four-step sequence, and exits
non-zero if any step fails rather than silently reporting success:

1. Finds every `lsnode:` worker matching this app's exact path.
2. Kills each one explicitly by PID (never trusts that Restart already did
   this).
3. Confirms zero workers remain before proceeding - aborts rather than
   starting a second worker alongside a stuck one if the kill didn't fully
   take.
4. Starts fresh via `cloudlinux-selector restart`.
5. Hits the *real* domain (`zucegroup.com`, not `app.zucegroup.com` - the
   app root directory is named `app.zucegroup.com` but that is not the
   domain the app is actually configured to serve) to force the lazy worker
   to spawn, and checks it returns HTTP 200.
6. Verifies exactly one fresh worker exists at the end.

It was built and tested directly on this account in October 2026, after
discovering that a prior version using `mapfile < <(...)` (process
substitution) silently failed when run as a non-interactive SSH command on
this host - every check it did would report "clean" regardless of actual
state. The committed version avoids that - see the comments at the top of
the script if modifying it, and re-test any change the same way: run it,
then independently check `ps` yourself rather than trusting its own output
alone, at least once.

**Never deploy with a bare `cloudlinux-selector restart`/`start`/`stop`
call, in cPanel's UI or a shell.** It has repeatedly reported success while
leaving the previous worker running untouched - that is how multiple
stacked `lsnode:` workers have accumulated more than once on this account
(Truehost support found four of them on 2026-10-21), each still counting
against the account's process ceiling until the whole account locks up.

**If the account is already maxed out** (SSH itself starts failing to
fork, `cannot fork()` on basic commands) and you can't even get a shell to
run the above: the one thing that has reliably freed it every time is
**cPanel's Setup Node.js App page directly clicking Stop, then Start**
(not Restart) - going through cPanel's own management layer rather than a
shell seems to bypass whatever is stuck. Confirm the process count dropped
on the Resource Usage panel before trying SSH again.

**Reducing how often this happens:**
- `.htaccess` in the app root already sets `NODE_OPTIONS=--v8-pool-size=2`
  and `UV_THREADPOOL_SIZE=2` for the main app, to cap its own thread
  footprint. Any new script that runs continuously or on a tight schedule
  should set the same env vars.
- Any one-off script (anything under `scripts/`) must close every
  connection/client it opens (DB, IMAP, etc.) and call `process.exit()`
  explicitly at the end, in every code path including catch/finally -
  rather than trusting Node to drain the event loop naturally. A lingering
  timer or open handle is exactly how a "quick" script turns into a
  long-running ghost process that eats into the same process ceiling.
  (The old `check-inbox.js` cron script used to be the canonical example of
  this pattern - it was removed in September 2026 when inbox polling moved
  to the local Outlook bridge in `outlook-sync/`, see
  CRM_AUTOMATION_ROADMAP.md. There is currently no server-side cron script
  running at all - `crontab -l` on the account should show nothing beyond
  `MAILTO`/`SHELL` env lines; if it ever shows more than that, something
  was added outside this documented process and needs checking before
  trusting it.)
- If a scheduled script is ever reintroduced, keep its interval as
  infrequent as the task actually needs - a human-review workflow doesn't
  need sub-hourly freshness, and halving the frequency halves that script's
  contribution to process churn.
- **The prospect-mining/outreach automation (`scripts/seed-beyta-ngo-prospects-batchNN.js`,
  scheduled via `scripts/ngo-mining-progress.md`) runs in a separate cloud
  environment with no SSH or production-server access at all** - it only
  researches, writes seed scripts, and pushes to GitHub. It cannot be a
  source of server-side resource exhaustion by construction. Loading a
  batch into the live CRM is always a separate, manual step taken from an
  interactive session with real server access - see "Standard deploy
  procedure" above.