LLM Accuracy vs Reproducibility: Are We Measuring Capability or Sampling Luck?

Why identical prompts can produce different reasoning paths — and why that matters for evaluation LLM Accuracy vs Reproducibility: Are We Measuring Capability or Sampling Luck? When working with LLMs, we often rely on metrics like accuracy, pass rates, or benchmark scores to evaluate performance. But a simple experiment reveals something that’s easy to overlook. The Setup Same prompt Same model snapshot Same temperature Same sampling configuration Run the same input multiple times. The Observation The outputs don’t just vary slightly. They often follow completely different reasoning paths. In some cases, the structure of the response changes significantly — different intermediate steps, different logic, different phrasing. And yet: The final answer may still be the same. Why This Matters Most evaluation frameworks implicitly assume: Same input → consistent reasoning process → comparable outputs But what we actually observe looks more like: Same input → multiple competing generation pat