PAR-156
0
Shared-Redis SWR cache for explorer aggregates + client refresh in sync with cache reset
DoneHighUnassignedEconomy Explorer
Incident
Explorer pages 500'd in prod with max_statement_time exceeded. Two-part root cause: (1) probes hit the heavy / → restart cascade (fixed in gitops, probes → /healthz); (2) the dashboard's unstable_cache had no request-coalescing, so a cold pod under a burst ran the two ~890k-row 30-day ledger aggregations once per concurrent miss; /economy/health was force-dynamic and entirely uncached.
Fix (this issue)
Shared-Redis cache replacing per-pod memoisation, so the heavy aggregates recompute at most once per window for the whole fleet:
lib/redis.ts— lazy ioredis singleton fromREDIS_*, TENANT-prefixed keys (DC/SC share one prod Redis), graceful in-memory fallback whenREDIS_HOSTunset.lib/cache.tsmemo(key, windowMs, producer)— epoch-aligned windows (bucket = floor(now/windowMs)), stale-while-revalidate (serve prior window instantly, never block a request), cross-pod single-flight via a Redis lock (one recompute per window for the fleet), L1 map to absorb the synchronized client burst.- Applied to dashboard / market / chestshop / health. Dashboard derives gini/top-share inside the producer so the cached blob stays small.
AutoRefreshaligned to wall-clock window boundaries (+lag) so the client refresh lands in lockstep with the cache reset and hits the warm cache.- gitops:
REDIS_*env wired (prod → redis.production + app-secrets/REDIS_PASSWORD; dev → redis.development).
Follow-up to the probe fix. SQL unchanged — only how often / how concurrently the aggregates run, and where the result is cached.
Resources
- commit5bac3a8 Merge pull request #16 from MCCitiesNetwork/developMCCitiesNetwork/economy-explorer
- PR · merged#16 Release: persist last-known-good cache value (no cold runs on low-traffic pages) (PAR-156)MCCitiesNetwork/economy-explorer
- commit073ca3f cache: persist last-known-good instead of expiring on a short TTL (PAR-156)MCCitiesNetwork/economy-explorer
- PR#15 Release: money-flow streaming fix + persistent last-known-good cache (PAR-156, PAR-157)MCCitiesNetwork/economy-explorer
- commita2e839d Merge pull request #14 from MCCitiesNetwork/developMCCitiesNetwork/economy-explorer
- commit68b86a5 Cache the slowest remaining pages found via Faro RUM (PAR-156)MCCitiesNetwork/economy-explorer
- commit6ef31ea Extend the Redis aggregate cache to the remaining slow pages (PAR-156)MCCitiesNetwork/economy-explorer
- PR · merged#14 Release: explorer performance — shared-Redis cache + query/static fixes (PAR-156, PAR-157)MCCitiesNetwork/economy-explorer
- commitca10496 Back the aggregate cache with shared Redis + sync client refresh (PAR-156)MCCitiesNetwork/economy-explorer
- commite2ce4f6 Single-flight memo cache for heavy aggregate pages (PAR-156)MCCitiesNetwork/economy-explorer
Comments
No comments yet.
Activity
- ParadauxIO linked a commit — Commit 5bac3a8 — Merge pull request #16 from MCCitiesNetwork/develop
- ParadauxIO linked a pull request — PR #16 merged — Release: persist last-known-good cache value (no cold runs on low-traffic pages) (PAR-156)
- ParadauxIO linked a commit — Commit 073ca3f — cache: persist last-known-good instead of expiring on a short TTL (PAR-156)
- ParadauxIO linked a pull request — PR #16 open — Release: persist last-known-good cache value (no cold runs on low-traffic pages) (PAR-156)
- ParadauxIO linked a pull request — PR #15 open — Release: money-flow streaming fix + persistent last-known-good cache (PAR-156, PAR-157)
- ParadauxIO linked a commit — Commit 073ca3f — cache: persist last-known-good instead of expiring on a short TTL (PAR-156)
- ParadauxIO linked a commit — Commit a2e839d — Merge pull request #14 from MCCitiesNetwork/develop
- ParadauxIO linked a commit — Commit 68b86a5 — Cache the slowest remaining pages found via Faro RUM (PAR-156)
- ParadauxIO changed status to Status → Done
- ParadauxIO linked a commit — Commit 6ef31ea — Extend the Redis aggregate cache to the remaining slow pages (PAR-156)
- ParadauxIO linked a pull request — PR #14 merged — Release: explorer performance — shared-Redis cache + query/static fixes (PAR-156, PAR-157)
- ParadauxIO linked a commit — Commit ca10496 — Back the aggregate cache with shared Redis + sync client refresh (PAR-156)
- ParadauxIO linked a commit — Commit e2ce4f6 — Single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a pull request — PR #14 open — Release: explorer performance — shared-Redis cache + query/static fixes (PAR-156, PAR-157)
- ParadauxIO linked a pull request — PR #14 open — Release: single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a pull request — PR #14 open — Release: single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a pull request — PR #14 open — Release: single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a pull request — PR #14 open — Release: single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a commit — Commit 68b86a5 — Cache the slowest remaining pages found via Faro RUM (PAR-156)
- ParadauxIO linked a pull request — PR #14 open — Release: single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a pull request — PR #14 open — Release: single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a commit — Commit 6ef31ea — Extend the Redis aggregate cache to the remaining slow pages (PAR-156)
- tesks renamed the issue
- tesks description: Description updated
- ParadauxIO linked a pull request — PR #14 open — Release: single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a pull request — PR #14 open — Release: single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a commit — Commit ca10496 — Back the aggregate cache with shared Redis + sync client refresh (PAR-156)
- tesks changed status to Status → Pending Release
- ParadauxIO linked a pull request — PR #14 open — Release: single-flight memo cache for heavy aggregate pages (PAR-156)
- ParadauxIO linked a commit — Commit e2ce4f6 — Single-flight memo cache for heavy aggregate pages (PAR-156)
- tesks created the issue