PAR-157
Root-cause the remaining slow pages (indexes / static docs / query rewrites — not shared cache)
Context
PAR-156 put a shared-Redis SWR cache over the tenant-global aggregate pages. Some slow pages can't (or shouldn't) be solved that way and need real fixes. From the Faro p75-LCP-by-route analysis (7d), the remaining offenders:
- /me (p75 ~1740ms) — per-user, so not shared-cacheable. Needs query tuning / indexes on the player-scoped aggregates (findAccountsForPlayer, getPlayerTrajectory, countPlayerTransactions, getPlayerCounterparties).
- /docs (p75 ~1630ms) — static Markdown content has no reason to be this slow. Verify it's genuinely statically rendered (no force-dynamic / per-request work leaking in), trim client bundle/hydration weight, ensure cache headers.
- /chestshop/items/[key] (~1.27M pain) and /firms/[name] (~0.38M) — per-entity, view-gated; EXPLAIN their queries and index rather than cache.
- Underlying heavy queries flagged by EXPLAIN: the money-flow account-type self-join, the account 365-day activity calendar + transaction scans, etc.
Approach (mechanisms other than the shared cache)
- EXPLAIN each slow query against prod; add missing indexes via a new economy-flyway V15 migration (additive; mind lock/space on ledger_postings ~3.5M / ledger_txns ~1.8M — create off-peak).
- Make /docs genuinely static (SSG + static cache headers; no dynamic bail-out).
- Rewrite queries where an index won't fix the shape (e.g. self-joins, function-wrapped predicates that defeat indexes).
Note
The 7d Faro window overlaps the probe/restart incident, so absolute LCPs are inflated; use relative ranking + EXPLAIN as the source of truth. Follow-up to PAR-156.
Resources
- PR#15 Release: money-flow streaming fix + persistent last-known-good cache (PAR-156, PAR-157)MCCitiesNetwork/economy-explorer
- commit5730e2e Merge pull request #15 from MCCitiesNetwork/developMCCitiesNetwork/economy-explorer
- commit4e45ca4 money-flow: stream the 2-leg pivot by txn_id so it stays under the statement timeout (PAR-157)MCCitiesNetwork/economy-explorer
- commita2e839d Merge pull request #14 from MCCitiesNetwork/developMCCitiesNetwork/economy-explorer
- commit7e57753 Make the root layout cookie-free so /docs is static + auth is off the render path (PAR-157)MCCitiesNetwork/economy-explorer
- commit6f16224 Order account transactions by txn_id, not settlement_time (PAR-157)MCCitiesNetwork/economy-explorer
- commitc46e5d5 Rewrite money-flow to drop the per-txn correlated subquery (PAR-157)MCCitiesNetwork/economy-explorer
- PR · merged#14 Release: explorer performance — shared-Redis cache + query/static fixes (PAR-156, PAR-157)MCCitiesNetwork/economy-explorer
Comments
tesks · Jun 14, 2026, 1:22 PM
Investigation (EXPLAIN against prod) + what's done
Table sizes: accounts 114k, account_members 3.2k, firm_players 111k, ledger_txns 1.78M, ledger_postings 3.49M. Busiest single account = 1.26M postings; account 103850 (the 15s LCP in Faro) has only 40 — so its 15s was the incident window, not its query.
✅ Done — money-flow correlated subquery rewrite
EXPLAIN showed (SELECT COUNT(*) FROM ledger_postings WHERE p.txn_id = lt.txn_id) = 2 as a DEPENDENT SUBQUERY run per txn over the 892k-row window — the reason the page was "completely inaccessible." Pre-aggregating the 2-posting txn set once in a derived table (EXPLAIN: DERIVED, materialised once) removes it. Verified row-for-row identical to the old query at a fixed settlement_time boundary (the only diffs in a NOW()-based comparison were boundary drift between executions). Shipped to develop.
Findings — no index migration needed
The hot join columns are already indexed: firm_players PK = player_uuid_bin (name joins are PK lookups), accounts.idx_accounts_owner (owner_uuid_bin), account_members.idx_member_uuid + idx_member_active, ledger_postings has (account_id) and (account_id, txn_id, amount). The /me queries are already window-bounded and pass the account-id set in. So /me and account-detail are not index-starved — I'm deliberately not adding speculative indexes to 3.5M-row tables.
Remaining (need a decision — riskier / refactors)
- account/[id] for mega-accounts —
WHERE lp.account_id=? ORDER BY lt.settlement_time DESC LIMIT 20can't use one index (settlement_time is on ledger_txns), so a high-posting account (up to 1.26M) does a full sort for a LIMIT 20. Fixes: (a) order bylp.txn_id DESC(monotonic proxy, uses the existing composite index — tiny semantic change for out-of-order settlements), or (b) denormalise settlement_time into ledger_postings (ledger write-path change, owned by Treasury). Only affects a handful of system/gov whales; PAR-156 caching already covers repeat views. - /docs genuinely static — pages are SSG-prerendered, but the root layout calls
getViewer()(reads the session cookie), forcing dynamic SSR app-wide. Real fix = move the auth-dependent header into a client component so the server layout is static and docs serve as static HTML. App-wide header refactor — wants its own change. - /me auth overhead — the data queries are fine; the per-request cost is
getViewer()doing ~3 remote-DB round-trips in the layout for authenticated users. Could cache identity/role per session (short TTL) to cut it.
Much of the Faro p75 inflation was the probe/restart incident (the 7d window overlaps it); recommend re-pulling p75 after a clean week before chasing 1/2/3.
tesks · Jun 14, 2026, 1:46 PM
Done — all three landed on develop
- money-flow correlated subquery → derived table (verified row-for-row equal at a fixed window).
c46e5d5 - account/[id] ordering —
ORDER BY lp.txn_id DESCinstead oflt.settlement_time: EXPLAIN on the busiest account (#4, 1.26M postings) went from temporary+filesort over 1.75M rows to a ~40-row indexed read.6f16224 - /docs genuinely static + auth off the render path — root layout is now cookie-free (theme/mode from env + client-side pre-paint script; viewer-dependent header → client islands fed by one
/api/viewerfetch in the persistent layout)./docs/[[...slug]]builds as●SSG; authenticated pages SSR without an auth round-trip on the critical path.7e57753
#3 note: deliberately did not cache the role/capability resolution — a TTL there would let a just-revoked admin keep write access to the admin server-actions. The layout decoupling already takes auth off every page's render path, and the client /api/viewer fetch runs once per load (provider lives in the persistent layout), so caching wasn't needed for the win.
Validation: build green (correct static/dynamic split), lint clean, unit tests 65 passing, no stale API usages. Full Playwright E2E (auth/public) needs the seeded MariaDB shim — recommend running in CI before the release merges. Server-side page gating was not touched.
Left as known/inherent (caching covers them): aggregations over a single mega-account's full posting history (e.g. getCounterparties for account #4, ~1.26M postings) are genuinely O(postings) and rely on the PAR-156 cache for repeat views; a denormalised settlement_time on ledger_postings would be the only structural fix and that's a Treasury ledger write-path change.
Activity
- ParadauxIO linked a pull request — PR #15 open — Release: money-flow streaming fix + persistent last-known-good cache (PAR-156, PAR-157)
- ParadauxIO linked a commit — Commit 5730e2e — Merge pull request #15 from MCCitiesNetwork/develop
- ParadauxIO linked a commit — Commit 4e45ca4 — money-flow: stream the 2-leg pivot by txn_id so it stays under the statement timeout (PAR-157)
- ParadauxIO linked a pull request — PR #15 merged — Release: fix /money-flow never caching (streaming txn_id pivot) (PAR-157)
- ParadauxIO linked a pull request — PR #15 open — Release: fix /money-flow never caching (streaming txn_id pivot) (PAR-157)
- ParadauxIO linked a commit — Commit 4e45ca4 — money-flow: stream the 2-leg pivot by txn_id so it stays under the statement timeout (PAR-157)
- ParadauxIO linked a commit — Commit a2e839d — Merge pull request #14 from MCCitiesNetwork/develop
- ParadauxIO linked a commit — Commit 7e57753 — Make the root layout cookie-free so /docs is static + auth is off the render path (PAR-157)
- ParadauxIO linked a commit — Commit 6f16224 — Order account transactions by txn_id, not settlement_time (PAR-157)
- ParadauxIO linked a commit — Commit c46e5d5 — Rewrite money-flow to drop the per-txn correlated subquery (PAR-157)
- ParadauxIO changed status to Status → Done
- ParadauxIO linked a pull request — PR #14 merged — Release: explorer performance — shared-Redis cache + query/static fixes (PAR-156, PAR-157)
- ParadauxIO linked a pull request — PR #14 open — Release: explorer performance — shared-Redis cache + query/static fixes (PAR-156, PAR-157)
- tesks changed status to Status → Pending Release
- tesks commented
- ParadauxIO linked a commit — Commit 7e57753 — Make the root layout cookie-free so /docs is static + auth is off the render path (PAR-157)
- ParadauxIO linked a commit — Commit 6f16224 — Order account transactions by txn_id, not settlement_time (PAR-157)
- tesks commented
- ParadauxIO linked a commit — Commit c46e5d5 — Rewrite money-flow to drop the per-txn correlated subquery (PAR-157)
- tesks created the issue