Hardening & observability: limits, metrics, logs, processes

Distillation of hardening & merge session. Source:
the GLAV-PATTERNS document, blocks M (4-22, 25-29, 33-43, 47-59),
G (51-55), F (45-50).

1. Limits and caps

• SAFE DEFAULT ESCALATION — default limit is restrictive: REFERRAL_MONTHLY_CAP=20, not 0. Admin raises via env.
• ENV-CONFIGURABLE LIMITS WITH DEFAULTS — each limit: _env_int("NAME", default) with meaningful non-zero default. 0 = unlimited — explicit semantics.
• CAP ZERO = UNLIMITED — if cap and invited >= cap — cap=0 skips the check. if invited >= cap without check blocks everyone at 0.
• PRE-CAP SIDE-EFFECT GUARD — check cap BEFORE _ensure()/INSERT; rejected leaves no garbage in DB.
• MONTHLY ROTATION KEY — monthly limits keyed by YYYY-MM; UNIQUE(referrer_id, month); no cron resets.
• CAP WARNING LOG — on reaching cap — log.warning("... cap reached referrer=%s month=%s cap=%s") — admin knows the limit is choking growth.
• RATE LIMIT CHEAPLY AT THE EDGE — simple per-user window BEFORE expensive logic: in-memory dict[uid] = last_call; N per minute; 0 = off; message "please wait." Don't bring Redis for one handler.
• RATE LIMIT ALERT TEXT — reject contains a user-friendly message with wait time: "You can check once every 10 seconds." Not a silent return.
• monotonic() FOR RATE LIMITS — intervals — time.monotonic() (time.time() can go backwards); absolute timestamps — time().

2. Metrics

• METRIC ON ACTUAL STATE CHANGE — mutator function returns newly_credited=True; metric_inc ONLY on first change. Otherwise the metric lies N times.
• INVOICE IDEMPOTENCY MARKER — out["newly_credited"] = True/False; caller checks the flag.
• METRIC TABLE: UPSERT PATTERN — name TEXT PRIMARY KEY, value INTEGER, updated REAL + INSERT ... ON CONFLICT DO UPDATE SET value=value+excluded.value — atomic, no race.
• METRICS SNAPSHOT FUNCTION — metrics_snapshot() -> dict[str, int] in one query — for /admin and debugging.
• METRIC AT POINT OF ACTION — metric_inc right in the handler, at the event point; not in a separate reporting function that gets forgotten.
• CACHE HIT METADATA — cache result logs cost=0; isCached=true — otherwise metrics lie.

3. Logging

• THREE-TIER LOGGING — INFO = audit (charge/invoice/referral), DEBUG = diagnostics (provider bookkeeping), ERROR = alerts (cap reached, conflict). Everything at one level = no gradation.
• CHARGE AUDIT TRAIL — log.info("charge uid=%s seconds=%.3f from_free=%.3f from_bonus=%.3f from_paid=%.3f") — grep by uid restores history.
• INVOICE AUDIT LOG — invoice_created invoice=%s uid=%s + invoice_paid invoice=%s uid=%s added=%s.
• REFERRAL AUDIT LOG — referral new=%s referrer=%s bonus=%s month=%s.
• PROVIDER EXCEPTION NARROWING — except: pass → except Exception as exc: debug-log with provider, key_index, error. Not silent pass, but not log.error (floods prod with harmless failures).
• PREMIUM CHARGE SHORT CIRCUIT — premium doesn't go through cart math: early return with separate log charge ... premium=%s.
• LOG LEVEL PER HANDLER — each handler with explicit .setLevel(): stdout=INFO, bot_err.log=ERROR.
• TWO-PHASE ERROR LOG SETUP — RotatingFileHandler (INFO+) + separate RotatingFileHandler (ERROR only), different maxBytes/backupCount.
• LAUNCHER STDERR SEPARATION — launcher writes to launcher_out.log/launcher_err.log, NOT to code's RotatingFileHandler files (>> bot.log 2>&1 with own RotatingFileHandler = PermissionError on Windows).