Paradaux
IssuesPAR-157Done
0

Root-cause the remaining slow pages (indexes / static docs / query rewrites — not shared cache)

Context

PAR-156 put a shared-Redis SWR cache over the tenant-global aggregate pages. Some slow pages can't (or shouldn't) be solved that way and need real fixes. From the Faro p75-LCP-by-route analysis (7d), the remaining offenders:

  • /me (p75 ~1740ms) — per-user, so not shared-cacheable. Needs query tuning / indexes on the player-scoped aggregates (findAccountsForPlayer, getPlayerTrajectory, countPlayerTransactions, getPlayerCounterparties).
  • /docs (p75 ~1630ms) — static Markdown content has no reason to be this slow. Verify it's genuinely statically rendered (no force-dynamic / per-request work leaking in), trim client bundle/hydration weight, ensure cache headers.
  • /chestshop/items/[key] (~1.27M pain) and /firms/[name] (~0.38M) — per-entity, view-gated; EXPLAIN their queries and index rather than cache.
  • Underlying heavy queries flagged by EXPLAIN: the money-flow account-type self-join, the account 365-day activity calendar + transaction scans, etc.

Approach (mechanisms other than the shared cache)

  1. EXPLAIN each slow query against prod; add missing indexes via a new economy-flyway V15 migration (additive; mind lock/space on ledger_postings ~3.5M / ledger_txns ~1.8M — create off-peak).
  2. Make /docs genuinely static (SSG + static cache headers; no dynamic bail-out).
  3. Rewrite queries where an index won't fix the shape (e.g. self-joins, function-wrapped predicates that defeat indexes).

Note

The 7d Faro window overlaps the probe/restart incident, so absolute LCPs are inflated; use relative ranking + EXPLAIN as the source of truth. Follow-up to PAR-156.

Comments

tesks · Jun 14, 2026, 1:22 PM

Investigation (EXPLAIN against prod) + what's done

Table sizes: accounts 114k, account_members 3.2k, firm_players 111k, ledger_txns 1.78M, ledger_postings 3.49M. Busiest single account = 1.26M postings; account 103850 (the 15s LCP in Faro) has only 40 — so its 15s was the incident window, not its query.

✅ Done — money-flow correlated subquery rewrite

EXPLAIN showed (SELECT COUNT(*) FROM ledger_postings WHERE p.txn_id = lt.txn_id) = 2 as a DEPENDENT SUBQUERY run per txn over the 892k-row window — the reason the page was "completely inaccessible." Pre-aggregating the 2-posting txn set once in a derived table (EXPLAIN: DERIVED, materialised once) removes it. Verified row-for-row identical to the old query at a fixed settlement_time boundary (the only diffs in a NOW()-based comparison were boundary drift between executions). Shipped to develop.

Findings — no index migration needed

The hot join columns are already indexed: firm_players PK = player_uuid_bin (name joins are PK lookups), accounts.idx_accounts_owner (owner_uuid_bin), account_members.idx_member_uuid + idx_member_active, ledger_postings has (account_id) and (account_id, txn_id, amount). The /me queries are already window-bounded and pass the account-id set in. So /me and account-detail are not index-starved — I'm deliberately not adding speculative indexes to 3.5M-row tables.

Remaining (need a decision — riskier / refactors)

  1. account/[id] for mega-accountsWHERE lp.account_id=? ORDER BY lt.settlement_time DESC LIMIT 20 can't use one index (settlement_time is on ledger_txns), so a high-posting account (up to 1.26M) does a full sort for a LIMIT 20. Fixes: (a) order by lp.txn_id DESC (monotonic proxy, uses the existing composite index — tiny semantic change for out-of-order settlements), or (b) denormalise settlement_time into ledger_postings (ledger write-path change, owned by Treasury). Only affects a handful of system/gov whales; PAR-156 caching already covers repeat views.
  2. /docs genuinely static — pages are SSG-prerendered, but the root layout calls getViewer() (reads the session cookie), forcing dynamic SSR app-wide. Real fix = move the auth-dependent header into a client component so the server layout is static and docs serve as static HTML. App-wide header refactor — wants its own change.
  3. /me auth overhead — the data queries are fine; the per-request cost is getViewer() doing ~3 remote-DB round-trips in the layout for authenticated users. Could cache identity/role per session (short TTL) to cut it.

Much of the Faro p75 inflation was the probe/restart incident (the 7d window overlaps it); recommend re-pulling p75 after a clean week before chasing 1/2/3.

tesks · Jun 14, 2026, 1:46 PM

Done — all three landed on develop

  1. money-flow correlated subquery → derived table (verified row-for-row equal at a fixed window). c46e5d5
  2. account/[id] orderingORDER BY lp.txn_id DESC instead of lt.settlement_time: EXPLAIN on the busiest account (#4, 1.26M postings) went from temporary+filesort over 1.75M rows to a ~40-row indexed read. 6f16224
  3. /docs genuinely static + auth off the render path — root layout is now cookie-free (theme/mode from env + client-side pre-paint script; viewer-dependent header → client islands fed by one /api/viewer fetch in the persistent layout). /docs/[[...slug]] builds as SSG; authenticated pages SSR without an auth round-trip on the critical path. 7e57753

#3 note: deliberately did not cache the role/capability resolution — a TTL there would let a just-revoked admin keep write access to the admin server-actions. The layout decoupling already takes auth off every page's render path, and the client /api/viewer fetch runs once per load (provider lives in the persistent layout), so caching wasn't needed for the win.

Validation: build green (correct static/dynamic split), lint clean, unit tests 65 passing, no stale API usages. Full Playwright E2E (auth/public) needs the seeded MariaDB shim — recommend running in CI before the release merges. Server-side page gating was not touched.

Left as known/inherent (caching covers them): aggregations over a single mega-account's full posting history (e.g. getCounterparties for account #4, ~1.26M postings) are genuinely O(postings) and rely on the PAR-156 cache for repeat views; a denormalised settlement_time on ledger_postings would be the only structural fix and that's a Treasury ledger write-path change.

Activity

  • ParadauxIO linked a pull request — PR #15 open — Release: money-flow streaming fix + persistent last-known-good cache (PAR-156, PAR-157)Jun 14, 2026, 2:44 PM
  • ParadauxIO linked a commit — Commit 5730e2e — Merge pull request #15 from MCCitiesNetwork/developJun 14, 2026, 2:39 PM
  • ParadauxIO linked a commit — Commit 4e45ca4 — money-flow: stream the 2-leg pivot by txn_id so it stays under the statement timeout (PAR-157)Jun 14, 2026, 2:39 PM
  • ParadauxIO linked a pull request — PR #15 merged — Release: fix /money-flow never caching (streaming txn_id pivot) (PAR-157)Jun 14, 2026, 2:39 PM
  • ParadauxIO linked a pull request — PR #15 open — Release: fix /money-flow never caching (streaming txn_id pivot) (PAR-157)Jun 14, 2026, 2:37 PM
  • ParadauxIO linked a commit — Commit 4e45ca4 — money-flow: stream the 2-leg pivot by txn_id so it stays under the statement timeout (PAR-157)Jun 14, 2026, 2:37 PM
  • ParadauxIO linked a commit — Commit a2e839d — Merge pull request #14 from MCCitiesNetwork/developJun 14, 2026, 1:48 PM
  • ParadauxIO linked a commit — Commit 7e57753 — Make the root layout cookie-free so /docs is static + auth is off the render path (PAR-157)Jun 14, 2026, 1:48 PM
  • ParadauxIO linked a commit — Commit 6f16224 — Order account transactions by txn_id, not settlement_time (PAR-157)Jun 14, 2026, 1:48 PM
  • ParadauxIO linked a commit — Commit c46e5d5 — Rewrite money-flow to drop the per-txn correlated subquery (PAR-157)Jun 14, 2026, 1:48 PM
  • ParadauxIO changed status to Status → DoneJun 14, 2026, 1:48 PM
  • ParadauxIO linked a pull request — PR #14 merged — Release: explorer performance — shared-Redis cache + query/static fixes (PAR-156, PAR-157)Jun 14, 2026, 1:48 PM
  • ParadauxIO linked a pull request — PR #14 open — Release: explorer performance — shared-Redis cache + query/static fixes (PAR-156, PAR-157)Jun 14, 2026, 1:47 PM
  • tesks changed status to Status → Pending ReleaseJun 14, 2026, 1:46 PM
  • tesks commentedJun 14, 2026, 1:46 PM
  • ParadauxIO linked a commit — Commit 7e57753 — Make the root layout cookie-free so /docs is static + auth is off the render path (PAR-157)Jun 14, 2026, 1:45 PM
  • ParadauxIO linked a commit — Commit 6f16224 — Order account transactions by txn_id, not settlement_time (PAR-157)Jun 14, 2026, 1:36 PM
  • tesks commentedJun 14, 2026, 1:22 PM
  • ParadauxIO linked a commit — Commit c46e5d5 — Rewrite money-flow to drop the per-txn correlated subquery (PAR-157)Jun 14, 2026, 1:21 PM
  • tesks created the issueJun 14, 2026, 1:14 PM