How many samples does an auditor actually test?
- No standard sets a SOC 2 sample size. The numbers firms use are convention, not rule.
- The number comes from how often the control runs. Annual is one item. Daily is around twenty five.
- A Type I tests design at a point in time, so it usually looks at one instance of each control rather than a sample.
- Which items get picked matters more than how many. When your population is small, testing all of it beats sampling any of it.
No standard sets a SOC 2 sample size
If you go looking for the required sample size in the SOC 2 rulebook, you will not find one. The attestation standards tell the auditor to obtain sufficient appropriate evidence and leave the sizing to professional judgment.
So where do the numbers come from. Convention, mostly inherited from financial statement audit practice, where sampling has a long history and a well worn set of tables. Every firm you talk to will be working from something close to the same ladder, and none of them are quoting a rule.
That is worth knowing before somebody tells you a number is mandatory.
Frequency decides the number
The logic is straightforward. The more often a control runs, the more instances there are, and the more you need to look at before you can say anything about the whole.
| How often the control runs | Items typically tested |
|---|---|
| Annually | 1 |
| Quarterly | 2 |
| Monthly | 2 to 5 |
| Weekly | 5 to 15 |
| Daily | 20 to 40 |
| Many times a day | 25 to 60 |
Read that against your own controls and the total stops being mysterious. Your annual risk assessment is one item. Your quarterly access review is two. Your code deployments, if you ship daily, are somewhere in the twenties.
Notice what this means for a small company. A lot of your controls run annually or quarterly, so a lot of your testing is one or two items.
A Type I usually tests one of each
A Type I reports on whether controls are suitably designed and implemented at a point in time. There is no period to sample across, so testing is generally one instance of each control: does this exist, is it configured the way the description says.
Sampling proper belongs to a Type II, which reports on whether controls operated over a period. That is where the table above starts to matter.
The picking matters more than the counting
Here is the part that took me years of fieldwork to fully appreciate.
Twenty five is twenty five whether the auditor picks the twenty five that look tidy or twenty five chosen at random. Sampling only means something if the selection is not steered, and the person doing the selecting is the same person whose evening gets longer when an item fails.
I wrote separately about what that incentive does to an audit. The short version is that the honest fix is to take the selection away from the person with the incentive. Random selection with a recorded seed does it. So does testing everything.
When the population is small, test all of it
This is the bit that changes for a company of five people.
If you deployed forty times last year, there is no reason to sample forty. Test forty. If you onboarded six employees, test six. Sampling exists because looking at everything used to be prohibitively expensive, and for a small population it is not expensive any more.
That is the part AI changed, and what it changed is the economics rather than the standard. Reading forty deployment records was hours of somebody's afternoon, so firms looked at a handful and reasoned about the rest. A model reads all forty in the time it used to take to decide which five to open. What you get out of that is a stronger conclusion. A sample supports an inference about the population. A census supports a statement about it. Judgment on each item that comes back flagged is still a human's job, and that is where an auditor's hours should go.
A report that says every item in the population was tested is a stronger statement than one that says twenty five items were selected, and for most small companies it is also the easier one to produce.
Worth asking your auditor directly: for each of my controls, are you sampling or testing the full population, and how are the samples selected. The answer is a good read on how the engagement will actually be run.
Frequently asked questions
What is the sample size for a SOC 2 audit?
Does SOC 2 require a minimum sample size?
How many samples are tested for a quarterly control?
Does a SOC 2 Type I use sampling?
Can an auditor test the whole population instead of sampling?
Keep reading
Do you need an evidence collection tool?
Nothing in the standards requires one. Evidence is records your systems already produce, and the auditor should be the one collecting them.
How many controls does SOC 2 require?
None, as a number. The rulebook holds 61 criteria and no control list at all. What actually decides how many you end up writing.
How much evidence does a SOC 2 audit need?
About twenty sources, and one export often answers several criteria at once. What auditors ask for, and what does not count.
How long does a SOC 2 audit take?
Three clocks run and only one is your auditor's. Where the time actually goes, and the part nothing can speed up.
Sources
- AT-C section 205 sets the performance and reporting requirements for examination engagements such as SOC 2; it does not prescribe a required number of controls, a required volume of evidence, or a required sample size.
- Audit sampling is formally defined as the application of an audit procedure to less than 100 percent of the items within a population.
- AICPA SOC for Service Organizations: a Type II report covers a period of time, while a Type I describes controls at a point in time.
- Trust Services Criteria are evaluated at the criterion level; points of focus are illustrative, not requirements (2017 TSC with revised points of focus, 2022).