Sampling and Uncertainty

Why do analysts use samples?

Collecting data from every member of a population can be:

• Too expensive (surveying every customer)
• Too slow (manual review of every transaction)
• Impossible (future customers)

A sample gives information about the larger population, but the result depends on:

• How the sample was selected
• How large it is
• How much random variation it contains

Population and sample

Example: A survey of app users who voluntarily answer a pop-up may not represent all users. People who respond can differ from people who ignore it.

Sampling bias

Sampling methods:

A sample can be large and still biased. Size reduces random error, not bias.

Uncertainty

• Sample estimates vary from sample to sample. This is sampling variability.
• Standard error (SE) measures how much an estimate is expected to vary. For a mean: SE = SD ÷ √n.
• Larger samples → smaller SE → more precise estimates. Quadrupling the sample size roughly halves the SE.
• Confidence intervals communicate a range of values compatible with the data under a statistical procedure and its assumptions. A common approximation for a 95% interval for a mean, with a reasonably large sample, is: estimate ± 1.96 × SE.
• A 95% confidence interval does not mean "there is a 95% chance the true value is in this specific interval". It means the procedure captures the true value in about 95% of repeated samples.

Key takeaways

• Ask: How was the data collected? Who was excluded? Does the sample represent the group you want to describe?
• These questions often matter more than the choice of formula.
• Report uncertainty alongside estimates.