Matthew Scott
I write about measuring language models: how to read an eval leaderboard, what quantization costs, and what an agentic coding benchmark can and cannot tell you.
Series: reading eval results
Nine Models, Two Results
A ranking table with nine sorted rows looks like nine findings. This one supports two. How to read an eval leaderboard by its diagnostics instead of its order — and why the diagnostic that looks fatal is fine while the one that looks clean says almost nothing.
The Model Ranked Third Never Performs Third
Two models finished 0.01 apart on the same leaderboard. One scored between 2 and 4 on eight of twelve prompts; the other was above 4 six times and below 2 four times. Why a mean over a prompt set destroys the property that decides whether you can ship.
Your Ten-Dimension Rubric Measures One Thing
A radar chart with ten axes promises ten independent signals about a model. In this study 97.6% of the variance sits on a single factor — and the rubric had explicitly instructed the judge not to let that happen.
Latest writing
One Axis, Three Benchmarks
A chart of Terminal-Bench 4.0 scores by reasoning effort looks like one benchmark run seventeen times. It is three different evaluations sharing a y-axis. What the chart supports, what it does not, and how to tell the difference from the numbers alone.
What Quantization Really Does to LLM Accuracy
FP8, Q4, GGUF — the bit count on the label tells you less than you think. What actually happens to model quality when you compress an LLM, worked through DeepSeek-V4-Flash-0731 and Llama 3.3 70B.
About
I'm Matthew Scott. These are my notes on building software, with the sources and the working left in.