Product Antifraud -- Log-Based Fraud Detection

Aspect	Detail
Purpose	Rule-based fraud detection from application logs (registration + auth flows)
Approach	Pure Python + pandas -- counting, grouping, threshold problems
Not for	ML classification -- use at moderate volumes (~50K entries/day)
Outputs	Markdown report + CSV alerts for security teams

---

When to Use This Skill

Task	This Skill Applies
Building fraud detection scripts for registration or auth logs	Yes
Analyzing K8s application logs for suspicious behavioral patterns	Yes
Detecting bots, credential stuffing, or velocity abuse from structured logs	Yes
Auditing logs for GDPR PII exposure (unmasked emails, phones, names)	Yes
Designing tunable threshold-based rule engines with JSON config	Yes
Reviewing or extending existing antifraud detection rules	Yes
Building fraud alerting reports (Markdown + CSV) for security teams	Yes
ML-based fraud scoring (real-time model inference)	No -- use ai-ml-data-science
Application security hardening (OWASP, auth implementation)	No -- use software-security-appsec
Infrastructure log analysis (access logs, firewall, WAF)	No -- use ops-devops-platform
Real-time streaming fraud detection	No -- use data-lake-platform

---

Quick-Start Checklist

Step	Action	Notes
1	Identify log type	Registration (.txt.gz/.debug.gz) or auth (.log/.log.gz)
2	Create directory structure	config/, reports/, script file
3	Build LogParser	Correct timestamp format: , for registration ms, . for auth ms
4	Implement SessionAggregator	pandas groupby for key dimensions (token, IP, device, email)
5	Create JSON config	Default thresholds (see Configuration Pattern below)
6	Implement velocity rules	R1-R12 or A1-A13 -- highest signal-to-noise ratio
7	Add bot detection rules	R13-R17 or A14-A19
8	Add behavioral analysis rules	R18-R22 or A20-A25
9	Enable PIIScanner	GDPR compliance pass on all log lines
10	Test in discover mode	--mode discover against example data
11	Tune thresholds	Reduce false positives, verify known fraud patterns surface
12	Cross-node correlation	Merge by token/session_id before aggregation

---

Quick Reference

Architecture (4 Layers)

Every fraud detection script follows this pattern:

Log Files (.gz, .log)
    |
    v
[1] LogFileReader       -- Walk dirs, handle .gz decompression, iterate lines
    |
    v
[2] LogParser           -- Regex extraction -> dataclass (RegistrationEvent / AuthEvent)
    |
    v
[3] SessionAggregator   -- Group by token/IP/device/email, compute features
    |
    v
[4] RuleEngine + Report -- Evaluate rules, produce Markdown + CSV alerts

Detection Rule Categories

Category	Registration (R)	Authentication (A)	Reference
Fraud velocity	R1-R12	A1-A13	references/registration-fraud-rules.md, references/auth-fraud-rules.md
Bot vs human	R13-R17	A14-A19	references/bot-detection-patterns.md
Behavioral analysis	R18-R22	A20-A25	references/behavioral-analysis-rules.md
GDPR PII scanning	Both scripts	Both scripts	references/gdpr-pii-scanning.md

Rule Severity Quick Map

Severity	Registration Examples	Auth Examples
CRITICAL	JNDI injection (R10), national ID exposure	Personal data API response leaks
HIGH	Email/device velocity (R1-R2), IP hopping (R6)	Brute force (A1), credential stuffing (A2), session hijack (A4)
MEDIUM	Partial phone masking, confirmation brute force (R8)	Captcha trigger rate (A8), off-hours surge (A10)
LOW	Sequential email patterns (R17)	Auth method escalation (A21)

---

Decision Tree

New fraud detection task:
    |
    +-- Registration logs?
    |   +-- .txt.gz / .debug.gz format?
    |   |   -> Use RegistrationEvent parser (references/log-parser-architecture.md)
    |   +-- What signals available?
    |       +-- Token, IP, DeviceSerial, Email, Phone -> R1-R12 velocity rules
    |       +-- Timing data -> R13-R15 bot detection
    |       +-- Platform field -> R12, R16 device fingerprinting
    |
    +-- Authentication logs?
    |   +-- .log / .log.gz format?
    |   |   -> Use AuthEvent parser (references/log-parser-architecture.md)
    |   +-- What signals available?
    |       +-- user_id, IP, device_id -> A1-A6 velocity rules
    |       +-- Fraud check weights -> A5, A11 risk scoring
    |       +-- Country field -> A4, A20 impossible travel
    |       +-- Auth type field -> A12, A21 method switching
    |
    +-- GDPR compliance audit?
        -> Run PIIScanner pass on both log types
        -> See references/gdpr-pii-scanning.md

---

CLI Interface Pattern

# Discover mode: analyze example logs, output pattern statistics
python registration_fraud.py examples/epa-registration/ --mode discover --output reports/

# Detect mode: apply rules to new logs, generate alerts
python registration_fraud.py /path/to/new-day-logs/ \
  --config config/registration_rules.json --output reports/

# Auth fraud (same pattern)
python auth_fraud.py examples/epa-identity-auth-publicapi/ --mode discover --output reports/

Output Format

Output	Filename	Contents
Markdown report	report_YYYYMMDD_HHMMSS.md	Summary table, severity breakdown, detailed alerts with log line evidence
CSV export	alerts_YYYYMMDD_HHMMSS.csv	One row per alert, importable into SIEM/ticketing

---

Tech Stack

Component	Tool	Notes
Runtime	Python 3.10+	Standard library: re, gzip, json, csv, argparse, dataclasses, collections, datetime, pathlib, statistics
Data analysis	pandas	Time-window grouping and aggregation (only pip dependency)
Report formatting	tabulate (optional)	Pretty markdown tables

---

Configuration Pattern

Rules use external JSON configs for tunable thresholds (no code changes needed):

{
  "gdpr_pii_scanner": {
    "enabled": true,
    "check_emails": true,
    "check_phones": true,
    "check_names": true,
    "check_national_ids": true
  },
  "bot_detection": {
    "timing_variance_threshold_ms": 50,
    "min_human_step_interval_seconds": 2,
    "known_emulator_serials": ["000000000000000", "emulator-5554"],
    "scripting_user_agents": ["python-requests", "curl", "Go-http-client"]
  },
  "behavioral": {
    "impossible_travel_speed_kmh": 900,
    "burst_silence_ratio_threshold": 5.0,
    "session_abandonment_rate_threshold": 0.8
  }
}

---

Common Anti-Patterns

Anti-Pattern	Why It Fails	Instead
Hardcoded thresholds in code	Cannot tune without redeployment	External JSON config per rule
Single-dimension rules only	Easy to evade by changing one variable	Cross-correlate IP + device + email + timing
No deduplication	Duplicate log lines inflate counts	Deduplicate by (timestamp, request_id, message hash)
Ignoring multi-line entries	Auth logs have stack traces across lines	Parser must detect continuation lines
Treating all timestamps alike	Registration uses , for ms; auth uses .	Normalize timestamp parsing per log type
Cross-node blind spots	Same session spans K8s nodes	Merge by token/session_id before aggregation
PII in fraud reports	GDPR violation in the detection output itself	Mask PII in report output, reference by hash/ID

---

Known Challenges

Challenge	Impact	Mitigation
Multi-line log entries	Auth logs have stack traces across lines	Detect continuation lines (leading whitespace, at , Caused by:)
Duplicate log lines	Registration logs inflate counts	Deduplicate by (timestamp, request_id, message hash)
Masked data (***MASKED***)	Auth logs limit email/phone correlation	IP/device/user_id analysis still works
Different timestamp formats	Registration , for ms; auth . for ms	Normalize parsing per log type
Cross-node correlation	Same session spans K8s nodes	Merge by token/session_id before aggregation
Internal scanner noise	Qualys scanner IP 10.7.2.171 triggers R10	Flag but annotate as likely internal scan

---

Trend Awareness Protocol

When users ask about current fraud detection approaches, search before answering:

#	Search Query	Domain
1	"fintech fraud detection patterns 2026"	Fraud patterns
2	"application log fraud analysis tools 2026"	Tooling
3	"GDPR log compliance requirements 2026"	Compliance
4	"bot detection registration abuse 2026"	Bot detection

---

Navigation

Reference Guides

File	Coverage
references/registration-fraud-rules.md	Registration fraud rules R1-R12: thresholds, signals, detection logic
references/auth-fraud-rules.md	Auth fraud rules A1-A13: thresholds, signals, detection logic
references/bot-detection-patterns.md	Bot vs human: timing analysis, UA fingerprinting, speed checks, emulators
references/behavioral-analysis-rules.md	Behavioral: impossible travel, session abandonment, burst-then-silence
references/gdpr-pii-scanning.md	GDPR PII scanner: regex patterns, severity levels, config, report format
references/log-parser-architecture.md	4-layer architecture: LogFileReader, LogParser, SessionAggregator, RuleEngine
data/sources.json	18 curated antifraud, OWASP, GDPR, and log analysis resources

Related Skills

Skill	Use For
software-security-appsec	Application security patterns, OWASP Top 10
ai-ml-data-science	ML-based fraud classification (when rule-based is insufficient)
data-analytics-engineering	Data pipeline patterns for log aggregation
qa-observability	Observability, structured logging, SIEM integration