Confidence Intervals and Applying Statistics to Data Science

Statistical conclusions are more useful when they include information about uncertainty. A confidence interval provides a range of plausible values for a population parameter under the assumptions of the method.

Calculating a Confidence Interval for a Mean

For a sample with an approximately normal sampling distribution, a t-based confidence interval can be calculated using SciPy.

import numpy as np

from scipy.stats import t

data = np.array([52, 55, 57, 60, 62, 58, 64, 59])

n = len(data)

mean = np.mean(data)

standard_error = np.std(data, ddof=1) / np.sqrt(n)

confidence_level = 0.95

degrees_of_freedom = n - 1

critical_value = t.ppf(

(1 + confidence_level) / 2,

degrees_of_freedom

)

lower = mean - critical_value * standard_error

upper = mean + critical_value * standard_error

print("Sample mean:", mean)

print("95% confidence interval:", (lower, upper))

This code calculates a 95% t-confidence interval for the population mean. Its reliability depends on the sampling design and assumptions, including whether the data is sufficiently well-behaved for the method.

A 95% confidence level refers to the long-run coverage of the procedure over repeated samples; it does not mean that 95% of individual observations fall inside the interval.

Applying Statistics to a Dataset

Consider a dataset containing customer purchase amounts.

import pandas as pd

from scipy import stats

df = pd.DataFrame({

"Customer": ["A", "B", "C", "D", "E", "F", "G", "H"],

"Purchase_Amount": [450, 520, 480, 600, 550, 700, 490, 850]

})

amounts = df["Purchase_Amount"]

print("Mean purchase:", amounts.mean())

print("Median purchase:", amounts.median())

print("Standard deviation:", amounts.std())

print("First quartile:", amounts.quantile(0.25))

print("Third quartile:", amounts.quantile(0.75))

These calculations summarize typical spending and its variability. If the mean is noticeably higher than the median, investigate whether a small number of large purchases are influencing the average.

Connecting Statistics to Machine Learning

Statistics supports several stages of a data science workflow:

• Descriptive statistics help identify distributions and unusual values.
• Probability helps model uncertainty and outcomes.
• Sampling concepts help assess whether data represents the target population.
• Correlation analysis helps explore relationships between variables.
• Hypothesis tests help evaluate specific statistical claims.
• Confidence intervals communicate uncertainty around estimates.

These methods complement machine learning rather than replace it. Before applying a statistical test or model, verify the assumptions, understand how the data was collected, and select an approach that matches the question being investigated.