How many examples do I need in an LLM eval set?

For a ±3 point margin of error at 95% confidence on a score near 70%, about 897 independent items: n = 1.96² × 0.7 × 0.3 / 0.03². Halving the margin roughly quadruples the items.

Why use the Wilson interval instead of p ± 1.96·SE?

The normal (Wald) interval can extend below 0% or above 100% and undercovers when accuracy is near the extremes or n is small. The Wilson score interval keeps better coverage in those cases.

Do these formulas hold if eval items are clustered?

No. If items share a passage or template, they are not independent and the effective sample size is smaller. Miller (2024) recommends clustered standard errors in that case.

Free tool · runs in your browser

Eval Sample Size & Label Count Calculator

Free calculator for LLM evaluation sets: how many labeled items you need for a target margin of error, how many to detect a difference between two models, and the Wilson confidence interval for an observed score.

Confidence level

z = 1.960

1. Items needed to estimate accuracy within ±E

Expected accuracy p%Margin of error E± ptsPool size N (0 = unlimited)

897 labeled eval items

n₀ = z²·p(1−p)/E²; with a finite pool, n = n₀ / (1 + (n₀−1)/N).

2. Items per model to detect a difference between two models

Model A accuracy%Model B accuracy%Power%

1,251 items per model (independent samples, two-sided)

n = [z₁₋α/₂·√(2p̄(1−p̄)) + z₁₋β·√(p₁(1−p₁)+p₂(1−p₂))]² / (p₁−p₂)², p̄ = (p₁+p₂)/2. If both models answer the same items, a paired analysis (McNemar test) usually needs fewer items; this figure is conservative.

3. Confidence interval for an observed score (Wilson)

Items evaluated nObserved accuracy%

65.8% – 73.9%

Wilson score interval: (p̂ + z²/2n ± z·√(p̂(1−p̂)/n + z²/4n²)) / (1 + z²/n).

Frequently asked questions

How many examples do I need in an LLM eval set?: For a ±3 point margin of error at 95% confidence on a score near 70%, about 897 independent items: n = 1.96² × 0.7 × 0.3 / 0.03². Halving the margin roughly quadruples the items.
Why use the Wilson interval instead of p ± 1.96·SE?: The normal (Wald) interval can extend below 0% or above 100% and undercovers when accuracy is near the extremes or n is small. The Wilson score interval keeps better coverage in those cases.
Do these formulas hold if eval items are clustered?: No. If items share a passage or template, they are not independent and the effective sample size is smaller. Miller (2024) recommends clustered standard errors in that case.

Assumptions and references

Items are treated as independent Bernoulli trials (correct/incorrect). Nothing you enter leaves your browser. See also confidence intervals for eval accuracy and evaluation datasets and contamination.