Scientific Critical Thinking

Systematic evaluation of research rigor through methodology assessment, bias detection, and evidence quality frameworks.

Triggers

• User asks to evaluate a study's quality
• User needs to assess evidence strength
• User is reviewing trial methodology
• User wants to identify limitations or biases
• User is critiquing research for an editorial

Core Capabilities

1. Methodology Critique

Validity Assessment:

Type	Question	Red Flags
Internal	Did the study measure what it intended?	Confounders, selection bias
External	Can results generalize?	Narrow population, artificial setting
Construct	Do measures capture the concept?	Surrogate endpoints, proxy measures
Statistical	Are conclusions supported by data?	Underpowered, multiple testing

Study Design Hierarchy:

1. Systematic reviews/meta-analyses of RCTs
2. Individual RCTs
3. Cohort studies
4. Case-control studies
5. Cross-sectional studies
6. Case series/reports
7. Expert opinion

2. Bias Detection

Cognitive Biases in Research:

• Confirmation bias: Interpreting data to support hypothesis
• HARKing: Hypothesizing after results known
• Publication bias: Positive results published more
• Spin: Overstating or misrepresenting findings

Selection Biases:

• Sampling bias (non-representative)
• Volunteer bias (healthier participants)
• Attrition bias (differential dropout)
• Survivorship bias (only studying survivors)

Measurement Biases:

• Observer/detection bias
• Recall bias
• Social desirability bias
• Hawthorne effect

Analysis Biases:

• P-hacking (multiple testing)
• Outcome switching
• Selective reporting
• Data dredging

3. Statistical Evaluation Checklist

• Sample size adequate? (power analysis done?)
• Statistical test appropriate for data type?
• Multiple comparison correction applied?
• Effect sizes reported (not just p-values)?
• Confidence intervals provided?
• Missing data handled appropriately?
• Assumptions of tests verified?

4. Evidence Quality Assessment (GRADE)

Quality Levels:

Level	Meaning	Implications
High	Very confident in estimate	Strong recommendation
Moderate	Moderately confident	Conditional recommendation
Low	Limited confidence	Further research likely
Very Low	Little confidence	Estimate highly uncertain

Downgrade Factors:

• Risk of bias
• Inconsistency across studies
• Indirectness (surrogate outcomes)
• Imprecision (wide CIs)
• Publication bias

Upgrade Factors:

• Large effect size
• Dose-response relationship
• Residual confounding would reduce effect

5. Logical Fallacy Detection

Causation Fallacies:

• Post hoc ergo propter hoc (after = because of)
• Correlation ≠ causation
• Reverse causation
• Confounding as causation

Generalization Errors:

• Hasty generalization (small sample)
• Ecological fallacy (group to individual)
• Exception fallacy (individual to group)

Statistical Fallacies:

• Texas sharpshooter (finding patterns in noise)
• Base rate neglect
• Regression to mean confusion
• Multiple endpoints fishing

6. Research Design Questions

When evaluating a study, ask:

1. Question: Is the research question clear and answerable?
2. Design: Is the study design appropriate for the question?
3. Population: Is the sample representative of target population?
4. Intervention: Was the intervention clearly defined and consistent?
5. Comparison: Was the control group appropriate?
6. Outcome: Were outcomes clinically meaningful and measured reliably?
7. Follow-up: Was follow-up long enough and complete enough?
8. Analysis: Was the analysis appropriate and pre-specified?

7. Claim Evaluation Framework

For any scientific claim:

1. Identify the assertion - What exactly is being claimed?
2. Evaluate supporting evidence - What studies support it?
3. Check logical connection - Does evidence actually support claim?
4. Assess proportionality - Is strength of claim proportional to evidence?
5. Detect overgeneralization - Are limits of findings respected?
6. Flag red flags - Conflicts of interest, spin, p-hacking?

Application to Cardiology Content

Evaluating Trial Results

1. Check randomization and blinding adequacy
2. Assess primary endpoint clinical relevance
3. Evaluate intention-to-treat vs per-protocol
4. Look for protocol changes mid-trial
5. Examine subgroup analyses critically
6. Consider funding source influence

For Editorials/Newsletters

• Acknowledge study limitations explicitly
• Don't overstate findings
• Note where evidence is weak
• Distinguish association from causation
• Highlight what questions remain

Critique Output Format

When critiquing research:

1. Summary: Brief overview of what study did
2. Strengths: What was done well
3. Critical concerns: Major methodological issues
4. Important limitations: Secondary concerns
5. Minor issues: Small points for completeness
6. Overall assessment: Balanced conclusion on reliability