Direct answer: There is no universal sample size for an AI quality-inspection POC. It depends on defect prevalence, target recall/precision, confidence level, and line/shift/changeover variation. A rare defect may need thousands to tens of thousands of parts; a common defect may need a few hundred—but you must use an independent holdout set and per-type metrics.
This report is for quality, production, IT, procurement and AI project leads at manufacturers, and discusses how many samples an AI inspection POC needs, how to calculate confidence intervals, and how acceptance sampling differs from model evaluation. Information is current to 2026-10-29. It is a statistical-method summary, not statistical consulting, a test report or an acceptance conclusion; sample sizes must be confirmed against defect rates, risk, production lines and a statistician's advice.
Overall accuracy is masked by class imbalance; sample size for rare defects is calculated by “number of defects”, not “total parts”; AQL acceptance sampling and model recall/precision are not the same thing.
Why can “99% accuracy” mean zero detection?
When the defect rate is low, a model that predicts every part as good can still show high overall accuracy, while recall is zero; acceptance must use the confusion matrix and per-type metrics, not one aggregate number.
Assume a lot of 10,000 parts with a 1% true defect rate (100 defective, 9,900 good). If the model predicts all 10,000 as good, overall accuracy is 99.0%, but all 100 defects are missed: recall is 0 and escape rate is 100%. That is why “99% accuracy” cannot be an acceptance conclusion, especially for safety parts, customer-complaint parts and batch scrap.
| Metric | Calculation | This example | Note |
|---|---|---|---|
| Overall accuracy | (TP+TN) ÷ total | 9,900 ÷ 10,000 = 99.0% | Inflated by the many good parts |
| Recall/detection rate | TP ÷ (TP+FN) | 0 ÷ 100 = 0% | Defects are not found |
| Escape/miss rate | FN ÷ (TP+FN) | 100 ÷ 100 = 100% | Share flowing to the next process |
| Precision | TP ÷ (TP+FP) | No TP, undefined | Must be read with false positives |
| False-positive rate | FP ÷ (FP+TN) | 0 ÷ 9,900 = 0% | But the cost is missing everything |
TP=true positive, FN=false negative/miss, FP=false positive/false alarm, TN=true negative. The example illustrates definitions, not any model's performance.
Define the metrics first: recall, precision, false-positive and escape rates
Acceptance should specify recall, precision, false-positive/overkill, escape, repeatability and cycle time; writing only “accuracy” hides class imbalance and false-positive cost, so it cannot support a go-live decision.
“Good” AI inspection has at least two sides: whether defects are found (recall) and whether good parts are wrongly rejected (precision/false positives). Recall alone can stop the line with false alarms; precision alone can miss defects. Quality, production and the supplier should write the definition, sample source, threshold and owner for each metric before the POC.
| Metric | Definition | Acceptance note |
|---|---|---|
| Recall/detection | TP ÷ (TP+FN), by defect type/severity | Safety/complaint defects may need a higher bar; do not use one aggregate |
| Precision | TP ÷ (TP+FP) | Read with false positives/overkill to avoid excessive recheck |
| False-positive/overkill | FP ÷ (FP+TN) | Calculate on good parts; convert to recheck hours/line-stop cost |
| Escape/miss rate | FN ÷ (TP+FN), i.e. 1−recall | Use an independent holdout, not training/demo samples |
| Repeatability | Share of repeated inspections of the same part with the same result | Retest after angle/shift/restart changes |
| Cycle time/P95 | Per-part inspection time, P95 latency, hourly capacity | Accurate but too slow still cannot go live |
| Human takeover | Tasks requiring human review/intervention ÷ total tasks | Too high turns “automatic full inspection” into “human plus system” |
Definitions can reference ISO 3534-1 statistical terms and the NIST/SEMATECH handbook; thresholds depend on defect severity, customer requirements and false-positive cost.
How should sample size be estimated?
First estimate total parts to see at least one defect, then estimate defect samples for recall precision; rare defects need far more parts, so do not set a number by production volume alone.
Sample size has at least two questions: first, can you see enough defects to estimate recall; second, how precise is the recall/false-positive estimate (how wide is the confidence interval). For low-prevalence defects, the total number of parts can be very large; for common defects, a few hundred may be enough, but you still need line/shift/changeover/lighting coverage. The tables below are planning assumptions, not statistical guarantees; a formal plan should be adjusted by a statistician using binomial/Wilson/Clopper-Pearson methods and stratified sampling.
| True defect rate | Planning total to see at least one defect (95% confidence, approximate) | Meaning |
|---|---|---|
| 10% | About 29 parts | Common defect; still cover type/shift |
| 1% | About 299 parts | Moderate; recall precision still needs more defect samples |
| 0.1% | About 2,995 parts | Rare; “total parts” quickly becomes full inspection |
| 0.01% | About 29,956 parts | Very rare; use targeted sampling/historical defect library/synthetic data and validate separately |
The “see at least one defect” approximation comes from the 95% upper bound for zero defects (about 3/n); it is not recall precision and does not guarantee coverage of every defect type.
| Target (assume recall around 90%) | Defect samples needed (approximate) | Total parts if defect rate is 1% |
|---|---|---|
| Recall ±10% (95% confidence) | About 35 defects | About 3,500 parts |
| Recall ±5% (95% confidence) | About 138 defects | About 13,800 parts |
| Recall ±3% (95% confidence) | About 384 defects | About 38,400 parts |
Approximation n ≈ z²p(1−p)/E² (p≈0.9, z≈1.96); defect samples must come from an independent holdout, not training data. False-positive estimation also needs enough good samples.
How does acceptance sampling differ from AI model evaluation?
AQL acceptance sampling decides whether a lot is accepted; AI model evaluation decides whether the system can reliably detect defects. The former is lot/attribute-based, the latter uses an independent holdout and per-type metrics; they cannot replace each other.
ISO 2859-1 acceptance sampling (AQL) is an attribute, lot-by-lot tool for incoming/outgoing/process sampling; it does not directly tell you model recall, false-positive rate or changeover stability. An AI inspection POC needs model metrics on an independent holdout plus shadow running and production sampling; the two can be combined, but an AQL table cannot prove a model is “99% accurate”.
| Dimension | AQL acceptance sampling | AI model evaluation |
|---|---|---|
| Object | Lot/incoming/outgoing | Model/line/defect type |
| Unit | Part/sample, attribute decision | Part/image/task, prediction vs ground truth |
| Risk | Producer/consumer risk | False positive/miss, changeover/drift, human takeover |
| Data | Random/stratified sample | Independent holdout, time/line/shift split |
| Conclusion | Accept/reject lot | Continue/narrow/pause/go live |
AQL sampling: ISO 2859-1; model confidence intervals: NIST/SEMATECH handbook. They are not the same acceptance system.
How should the POC sample plan be structured?
Stratify by defect type/severity, line/shift, lighting/angle, changeover, operator and rare defects; split training/validation/test by time or line, freeze the test set before acceptance.
More samples is not automatically better; coverage matters, leakage must be prevented, and rare cases must be supplemented. Build the defect dictionary and stratification plan before the POC; run shadow mode on the real line during the POC; after the POC, report metrics only from the frozen independent holdout. The table below is an operational stratification plan; quantities come from the previous section.
| Stratum | What to cover | Evidence |
|---|---|---|
| Defect type/severity | Scratch, dirt, deformation, missing part, dimension/colour; safety/complaint types separately | Defect dictionary, samples per type |
| Line/shift | Day/night, different machines, different operators | Stratified sampling sheet, shift record |
| Lighting/angle | Lights on/off, reflection, occlusion, part pose | Image samples, imaging parameters |
| Changeover | Different model/colour/material/lot | Changeover record, samples per model |
| Rare defects | Historical complaints, scrap, rework, trial-production defects | Historical library, targeted sampling, synthetic-data tag |
| Independent test set | Split by time/line/shift; training never touches test | Split script, frozen version |
Stratification is an editorial suggestion; sample size, sampling method and statistical power depend on defect rate, risk and a statistician's advice.
What if there are not enough samples?
With insufficient samples, the POC conclusion should be “inconclusive”, not “pass”; you can add rare defects, narrow scope or extend shadow running, but cannot force a go-live with one overall accuracy number.
Many POCs fail not because the model is bad, but because the sample is too small to support a conclusion. If only a few rare defects appear, the recall confidence interval will be wide; the correct action is to add samples, narrow to one defect/one line, extend shadow running, or keep a manual/rule fallback. The matrix below is for the POC review, not a statistical ruling.
| Decision | When it applies | Next step |
|---|---|---|
| Continue | Enough defect samples, per-type metrics pass, shadow run stable, false-positive cost acceptable | Accept per contract; move to single-line rollout |
| Narrow | One defect type/line works, others have insufficient samples or high false positives | Start with one defect/line; keep the rest manual |
| Add samples | Direction is workable but confidence intervals are wide/rare defects too few | Targeted collection, historical library, longer shadow run |
| Pause | Data leakage, unstable imaging, changeover not reproducible, human takeover too high | Freeze launch; fix data and fixtures first |
| Exit | Samples remain insufficient, false-positive cost exceeds benefit, supplier cannot provide independent validation | Stop the project; keep manual/rule-based inspection |
The decision matrix is an editorial suggestion; “enough samples” should be judged by defect type/severity and confidence intervals, not total part count.
Pre-launch acceptance checklist
Before launch, check the defect dictionary, sample stratification, train/test isolation, per-type metrics, confidence intervals, shadow run, false-positive cost, changeover and exit conditions; do not sign if any item is missing.
The checklist can be shared by quality, production, IT and the supplier. The point is reproducible evidence, not a good demo: where the samples came from, whether the test set was frozen, how metrics were calculated, how wide the confidence interval is, whether changeover/shift were covered, and who handles false positives.
| Check | Pass standard | Evidence |
|---|---|---|
| Defect dictionary | Type/severity/acceptance boundary frozen | Dictionary version, sign-off |
| Sample stratification | Covers defect, line, shift, lighting, changeover | Stratification sheet, sample list |
| Train/test isolation | Split by time/line/shift; test set frozen | Split script, hash/version |
| Per-type metrics | Recall, precision, false positive, escape, repeatability, cycle time | Metric table, calculation script |
| Confidence intervals | Key metrics have intervals/sample-size notes | Statistical report |
| Shadow run | 2–4 weeks, covering changeover/abnormal shifts | Run record, discrepancy closure |
| False-positive cost | Recheck hours/line stop/scrap acceptable | Cost table, production sign-off |
| Exit conditions | Drift/changeover/supplier exit has a plan | Contract, operations/retraining plan |
The checklist is an editorial suggestion, not a substitute for statistical/quality/contract review; sample size and acceptance thresholds are project-specific.
Who should read this, who should not, and what decision does it support?
It fits manufacturers running an AI inspection POC, acceptance or line expansion; it does not fit teams without a defect dictionary, independent test set, or those wanting to accept with one accuracy screenshot.
Who should read: quality/production/IT/procurement and AI inspection project leads at manufacturers; teams evaluating vision inspection, defect recognition, AOI/recheck or incoming/outgoing sampling; project leads who must explain “is the sample enough, are the metrics credible” to the owner/customer. Who should not: teams without a clear defect definition, ground truth/review process, or those buying a device and expecting it to go live; teams expecting a fixed sample size or a guaranteed acceptance; projects needing formal statistical consulting/test reports. Decisions supported: how many samples a POC needs, how to calculate recall/false positives, how wide the confidence interval is, how AQL and model metrics differ, whether to continue/narrow/pause/exit when samples are insufficient, and what evidence to collect before launch.
FAQ
Is there a universal sample size?
No. It depends on defect rate, target recall/precision, confidence level, line/shift/changeover variation and false-positive cost. A common defect may need a few hundred parts; a rare defect may need thousands to tens of thousands. Estimate “see at least one defect” and “recall precision” separately.
Why is 99% accuracy not enough?
Because overall accuracy is inflated by the many good parts. With a 1% defect rate, predicting everything as good still gives 99% accuracy, but recall is 0. Acceptance should use the confusion matrix, per-type recall, precision, false-positive/escape rates and confidence intervals.
How should training and test sets be split?
Split by time, line or shift; the test set must be independent and frozen, not mixed with training. Do not let multiple images of the same part/lot cross train and test, which leaks information. Cover changeover, lighting and shifts.
Can AQL sampling replace model acceptance?
No. AQL sampling decides lot accept/reject; model evaluation judges recall/false positives/stability on an independent sample. They can be combined, but an AQL table cannot prove model accuracy, and model metrics cannot replace outgoing sampling.
What if there are too few rare defects?
You can collect historical defects, extend shadow running, narrow to one defect/line, or use synthetic data with separate validation. If still insufficient, the POC conclusion should be “inconclusive”; do not force a launch with overall accuracy. Be especially careful with safety/complaint defects.
What false-positive rate is acceptable?
There is no universal value. Convert false positives into recheck hours, line stops, scrap and delivery impact, and compare with miss risk; thresholds vary by defect severity, line cycle time and customer requirements.
Bottom line
For an AI inspection POC, calculate sample size from defect rate, recall/precision targets, confidence intervals and stratification; start with the confusion matrix and per-type metrics, then accept with an independent holdout and shadow run. If samples are insufficient, write “inconclusive”—do not replace detection rate with 99% overall accuracy.
Method and limits
This article is a public statistical-method, standard-vocabulary and editorial summary, not statistical consulting, a test report or a quality conclusion for any company, line, model, sample or acceptance project. The sample-size formulas and tables are planning approximations and do not account for all finite-population, stratification/clustering, repeated-measurement and multiple-comparison effects; a formal plan should be confirmed by statistical/quality professionals against defect rate, risk and sampling design. Standards/handbooks change; the latest official text governs.
Sources
Information is current to 2026-10-29; each official/primary page governs its own formulas, assumptions and version.
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.2 Sample sizes required: itl.nist.gov. Sample-size formulas and assumptions for proportion tests/confidence intervals.
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.1 Confidence intervals: itl.nist.gov. Wilson/exact confidence intervals for proportions, useful for understanding recall/false-positive uncertainty.
- ISO 2859-1:2026, Sampling procedures for inspection by attributes — Part 1: Sampling schemes indexed by acceptance quality limit (AQL) for lot-by-lot inspection: iso.org. Attribute/lot acceptance sampling; not the same as AI model evaluation.
- ISO 3534-1:2006, Statistics — Vocabulary and symbols — Part 1: General statistical terms and terms used in probability: iso.org. Statistical terms/symbols reference.