19 Critically Evaluating AI Research
Chapter 13a taught you how to read AI papers. This chapter teaches you how to evaluate them. These are different skills. Reading is about extracting information; evaluation is about deciding whether to believe it. In a field where thousands of papers appear every month and hype cycles distort reality, the ability to critically assess AI claims is among the most valuable skills you can develop.
In the age of arXiv preprints and Twitter announcements, there is no gatekeeper ensuring that claims are true. Papers claim “state-of-the-art” results with unfair baselines. Press releases describe incremental improvements as breakthroughs. Even peer-reviewed work sometimes contains errors. Your job as a critical reader is to separate signal from noise.
19.1 The Anatomy of an AI Claim
Every empirical AI paper makes claims of this general form: “Our method X achieves result Y on benchmark Z.” To evaluate this claim, you must interrogate each component:
- The method (X): Is it actually novel, or is it a minor variation of existing work? Does the paper clearly describe what is new versus what is borrowed?
- The result (Y): How was it measured? Is the metric appropriate? Are confidence intervals or error bars provided? Was the result selected from multiple runs?
- The benchmark (Z): Is it representative of real-world performance? Is it saturated (all methods score above 95%)? Could the model have seen the test data during pre-training (data contamination)?
19.2 Red Flags in Experimental Design
19.2.1 Unfair Baselines
This is the most common and most damaging flaw in ML papers. Watch for:
- Baselines from different eras with different compute budgets
- Baselines that use smaller models or less training data
- Baselines without proper hyperparameter tuning while the proposed method is carefully optimized
- Missing strong baselines that the community considers standard
Ask yourself: “If the authors applied the same compute budget and tuning effort to the baselines, would the gap still exist?” Many claimed improvements vanish when baselines are properly tuned. Lipton and Steinhardt's “Troubling Trends in Machine Learning Scholarship” (Lipton and Steinhardt 2019) documented these patterns systematically.
19.2.2 Cherry-Picked Results
Papers may show a “best run” from many attempts. Look for:
- Results without standard deviations across multiple seeds
- A suspiciously narrow set of benchmarks (only the ones where the method excels)
- Missing ablation studies that would reveal which component actually helps
- Qualitative examples hand-selected to look impressive
19.2.3 Evaluation Gaming
Some evaluation issues are subtle:
- Train/test contamination: Especially for LLMs trained on internet-scale data, the test set may have appeared in training data. This inflates benchmark scores without improving real capability.
- Metric hacking: Optimizing for one specific metric (e.g., BLEU score for translation) while ignoring aspects that matter in practice (fluency, adequacy, cultural appropriateness).
- Goodhart's Law: When a measure becomes a target, it ceases to be a good measure. This is pervasive in AI benchmarking.
19.3 Statistical Reasoning in AI Papers
Most ML practitioners are not trained statisticians, and it shows. Common issues:
- Missing significance tests: A 0.3% improvement without error bars is meaningless. It could be random variation.
- Multiple comparisons: Testing ten hypotheses and reporting the one that “worked” inflates false positive rates.
- Small evaluation sets: Evaluation on a few hundred examples produces noisy estimates. A 2% improvement on 200 test items has wide confidence intervals.
- Confounding variables: Improvements may come from more data, more compute, or better hyperparameters rather than the proposed method.
If a paper reports improvements without confidence intervals and does not discuss statistical significance, be skeptical. If the improvement is under 1% absolute on any metric, it is likely within noise unless supported by very large evaluation sets.
19.4 Evaluating Scaling Claims
Scaling results are particularly tricky to evaluate:
- Are the scaling curves extrapolated? Many papers fit a trend line to small-scale experiments and extrapolate to much larger scales. These extrapolations often break down.
- Fixed vs. compute-optimal comparisons: A bigger model is nearly always better if compute is unlimited. The relevant question is whether the method is better at the same compute budget.
- Emergent abilities: Claims about “emergent” capabilities (abilities that appear suddenly at scale) have been challenged by work showing that emergence often depends on the choice of metric rather than being a true phase transition (Schaeffer et al. 2023).
19.5 Evaluating LLM Benchmarks
LLM evaluation is a minefield. Key considerations:
- Open vs. closed benchmarks: Public benchmarks like MMLU are widely used but suffer from contamination. Private benchmarks (available only to evaluators) are more trustworthy but less reproducible.
- Human evaluation: Chatbot Arena (Chiang et al. 2024) uses Elo ratings from blind human comparisons, which may be the most trustworthy LLM evaluation method.
- LLM-as-judge: Using one LLM to evaluate another is convenient but introduces systematic biases (e.g., preferring longer or more verbose responses, preferring responses in a similar style to the judge model).
- Reasoning benchmarks: Tasks like GSM8K (grade-school math), HumanEval (code generation), and ARC (reasoning) test specific capabilities but can be gamed through targeted training.
LMSYS's Chatbot Arena solved many evaluation problems at once: real users, blind comparisons, diverse queries, Elo ratings with confidence intervals, and no test set to contaminate. When you see benchmark claims, check how the model performs on Chatbot Arena for a reality check.
19.6 Reading Between the Lines
Experienced readers develop an intuition for what papers do not say:
- The Related Work tells a story: Authors position their work by choosing which papers to cite and how. Notice whose work is missing.
- Limitations sections are gold: Most papers now include limitations sections. These often contain the most honest assessment of the work. Read them carefully.
- Appendices hide important details: Page limits force authors to move crucial details (hyperparameters, failure cases, additional results) to appendices. Always skim the appendix.
- Code availability: Papers with released code are more credible. If code is “available upon request,” treat claims with more skepticism.
19.7 Building a Systematic Literature Review Practice
Beyond reading individual papers, you need a system for managing the firehose:
- Follow curated sources: Newsletters (The Batch, NLP News), podcasts (Machine Learning Street Talk, Gradient Dissent), and curated Twitter/X lists provide filtered signal.
- Track citation networks: When you find an important paper, trace its citations forward (who cites it?) and backward (what does it cite?). Connected Papers and Semantic Scholar make this easy.
- Maintain a reading log: For each paper, record: one-sentence summary, key claims, strengths, weaknesses, and whether you would build on this work.
- Discuss with others: Paper reading groups force you to articulate your evaluation. Explaining why you trust or distrust a result sharpens your critical thinking.
After reading a paper, write one paragraph explaining why you should or should not believe the main claim. If you cannot articulate the reasons, you have not read critically enough. This practice, done consistently, will make you a much better researcher and practitioner.
19.8 Exercises
- Find a recent paper (2024 or later) that claims state-of-the-art results. Analyze the baselines: Are they fairly compared? Are the strongest existing methods included? Write a one-page critique.
- Take a paper with strong benchmark results and check the Chatbot Arena rankings for the same model (if available). Do the benchmark results align with human preferences?
- Read a paper's limitations section and appendix carefully. List three things from these sections that change your interpretation of the main results.
- Find two papers that make contradictory claims about the same phenomenon (e.g., whether scaling improves reasoning, whether chain-of-thought helps small models). Analyze why they reach different conclusions: different benchmarks? Different model sizes? Different evaluation methodology?
- Start a reading log. For the next two weeks, read two papers per week and write a one-paragraph critical evaluation of each, explicitly stating whether you believe the main claims and why.