Descriptive Statistics for Data Science
A dataset containing thousands of numerical observations can be difficult to interpret without statistical summaries. Individual records provide specific measurements, but they do not clearly describe the overall distribution, central values, or variability of the dataset.
Descriptive statistics converts raw numerical information into summary measurements that help explain the dataset's characteristics.
Descriptive statistics is the branch of statistics concerned with organising, summarising, and describing observed data using numerical measures and distributional summaries.
In data science, descriptive statistics supports exploratory analysis, feature engineering, anomaly detection, and model evaluation.
What Are the Main Types of Descriptive Statistics?
Descriptive statistics can be divided into several categories.
| Category | Purpose | Common Measures |
|---|---|---|
| Central Tendency | Describes typical values | Mean, median, mode |
| Dispersion | Measures variability | Range, variance, standard deviation |
| Position | Identifies relative location | Quartiles, percentiles |
| Distribution Shape | Describes asymmetry and tails | Skewness, kurtosis |
| Association | Describes relationships | Covariance, correlation |
Each category reveals a different characteristic of the dataset.
For example, average monthly revenue describes central tendency, while standard deviation indicates how much revenue varies across months.
What Are Measures of Central Tendency?
Measures of central tendency identify representative values within a distribution.
The three commonly used measures are mean, median, and mode.
1. Arithmetic Mean
The mean is calculated by dividing the sum of observations by the total number of observations.
\[ \bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i \]
Here:
- \(x_i\) represents an individual observation.
- \(n\) represents the number of observations.
- \(\bar{x}\) represents the sample mean.
Consider the following daily sales values:
₹1,000, ₹1,500, ₹2,000, ₹2,500, and ₹3,000.
\[ \bar{x}=\frac{1000+1500+2000+2500+3000}{5} \]
\[ \bar{x}=2000 \]
The average daily sales value is ₹2,000.
Calculating Mean Using Python
import numpy as npsales = np.array([1000, 1500, 2000, 2500, 3000])print(np.mean(sales))Output:
2000.0The mean is useful when observations are reasonably representative of the distribution.
However, it is sensitive to extreme values.
2. Median
The median is the middle observation after sorting numerical values.
For an odd number of observations, it is the middle value.
For an even number, it is the average of the two middle values.
Consider:
\[ [10,15,20,25,100] \]
The median is:
\[ Median=20 \]
The mean is:
\[ Mean=34 \]
The unusually large value of 100 increases the mean but does not change the median substantially.
Calculating Median in Python
values = np.array([10, 15, 20, 25, 100])print(np.median(values))Output:
20.0Median is commonly used for skewed distributions, including income and transaction values.
3. Mode
The mode is the most frequently occurring value.
Consider:
\[ [2,3,3,4,5,5,5,6] \]
The mode is 5 because it appears three times.
Python implementation:
import pandas as pdvalues = pd.Series([2, 3, 3, 4, 5, 5, 5, 6])print(values.mode())Output:
0 5
dtype: int64Mode is useful for categorical variables, such as the most frequently purchased product category.
A dataset may contain multiple modes when several values share the highest frequency.
Mean vs Median vs Mode
| Measure | How It Is Calculated | Main Advantage | Limitation |
|---|---|---|---|
| Mean | Sum divided by count | Uses all numerical observations | Sensitive to extremes |
| Median | Middle ordered value | Resistant to extreme values | Does not directly describe magnitude of all observations |
| Mode | Most frequent value | Supports categorical analysis | May not be unique |
The appropriate measure depends on the distribution and analytical question.
What Are Measures of Dispersion?
Measures of dispersion describe how widely observations vary.
Two datasets may have the same mean but different degrees of variability.
For example:
Dataset A:
\[ [18,19,20,21,22] \]
Dataset B:
\[ [5,10,20,30,35] \]
Both datasets have a mean of 20.
However, Dataset B contains much greater variation.
Dispersion measures quantify this difference.
What Is Range?
Range measures the difference between the largest and smallest observations.
\[ Range=Maximum-Minimum \]
For Dataset A:
\[ Range=22-18=4 \]
For Dataset B:
\[ Range=35-5=30 \]
Dataset B has a much larger range.
However, the range is highly sensitive to extreme observations.
What Is Variance?
Variance measures the average squared deviation from the mean.
For a population:
\[ \sigma^2=\frac{\sum_{i=1}^{N}(x_i-\mu)^2}{N} \]
For a sample:
\[ s^2=\frac{\sum_{i=1}^{n}(x_i-\bar{x})^2}{n-1} \]
Here:
- \(\sigma^2\) represents population variance.
- \(s^2\) represents sample variance.
- \(\mu\) represents the population mean.
- \(\bar{x}\) represents the sample mean.
The sample formula uses \(n-1\) as the denominator for the conventional unbiased estimate of population variance under suitable sampling assumptions.
Calculating Variance Using Python
import numpy as npdata = np.array([10, 20, 30, 40, 50])population_variance = np.var(data)sample_variance = np.var(data, ddof=1)print(population_variance)print(sample_variance)Output:
200.0
250.0The ddof=1 parameter changes the denominator from \(n\) to \(n-1\).
Variance is expressed in squared units, which can make direct interpretation difficult.
What Is Standard Deviation?
Standard deviation is the square root of variance.
For a population:
\[ \sigma=\sqrt{\frac{\sum_{i=1}^{N}(x_i-\mu)^2}{N}} \]
Standard deviation expresses variability in the same units as the original variable.
For example, if daily revenue has a mean of ₹10,000 and a standard deviation of ₹2,000, it indicates the scale of variation around the average.
However, standard deviation alone does not specify the shape of the distribution.
Calculating Standard Deviation
data = np.array([10, 20, 30, 40, 50])print(np.std(data))Output:
14.142135623730951NumPy uses the population-style denominator by default.
A sample standard deviation can be calculated using ddof=1.
What Are Quartiles and Percentiles?
Quartiles divide ordered numerical observations into four parts using three cut points.
- First Quartile (Q1): 25th percentile
- Second Quartile (Q2): 50th percentile or median
- Third Quartile (Q3): 75th percentile
Percentiles describe the position of values within an ordered distribution.
For example, a 90th-percentile transaction amount represents a threshold at or below which approximately 90% of observations fall, subject to the percentile calculation method.
Calculating Percentiles Using NumPy
data = np.array([ 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000])print(np.percentile(data, 25))print(np.percentile(data, 50))print(np.percentile(data, 75))Output:
325.0
550.0
775.0These results use NumPy's default percentile calculation method.
Different statistical software or interpolation methods may produce slightly different estimates for small datasets.
What Is the Interquartile Range?
The Interquartile Range measures the spread between the first and third quartiles.
\[ IQR=Q_3-Q_1 \]
For the previous example:
\[ IQR=775-325=450 \]
The middle 50% of observations span 450 units under this quartile definition.
IQR is less sensitive to extreme observations than the overall range.
It is commonly used in box plots and preliminary outlier detection.
What Is Skewness?
Skewness describes the asymmetry of a distribution.
A positively skewed distribution has a longer or heavier right tail.
A negatively skewed distribution has a longer or heavier left tail.
A symmetric distribution has approximately balanced tails.
| Distribution | Typical Characteristic | Example |
|---|---|---|
| Positive Skew | Long right tail | Customer spending |
| Negative Skew | Long left tail | Scores concentrated near a maximum |
| Approximately Symmetric | Similar behaviour on both sides | Some measurement distributions |
Calculating Skewness
import pandas as pdvalues = pd.Series([100, 150, 200, 250, 5000])print(values.skew())The resulting statistic describes sample skewness using Pandas' estimator.
A positive value indicates right-skewed behaviour under this measure.
However, skewness near zero does not prove that the distribution is normal.
What Is Kurtosis?
Kurtosis is a statistical measure associated with the tail behaviour of a distribution.
Distributions with heavier tails may produce more extreme observations than a normal distribution.
Pandas calculates excess kurtosis, for which a normal distribution has a theoretical value of zero.
print(values.kurt())Kurtosis can help investigate extreme-value behaviour, but estimates may be unstable for very small datasets.
It should be interpreted alongside visualisations and other distributional measures.
How Do You Calculate Descriptive Statistics Using Pandas?
Pandas provides describe() for generating several descriptive measures together.
import pandas as pddf = pd.DataFrame({ "sales": [1000, 1500, 2000, 2500, 3000], "orders": [2, 3, 4, 5, 6]})print(df.describe())The output includes count, mean, sample standard deviation, minimum, quartiles, and maximum for numerical columns.
Additional statistics can be calculated separately.
print(df["sales"].median())print(df["sales"].var())print(df["sales"].skew())These methods provide a more detailed statistical description of the dataset.
How Are Descriptive Statistics Used in Data Science?
Customer Behaviour Analysis
Retailers calculate average order value, median spending, and spending variability to understand customer purchasing patterns.
Fraud Detection
Financial institutions examine unusual transaction amounts and deviations from historical activity.
Model Preprocessing
Means, medians, and standard deviations support imputation and feature scaling.
Operational Monitoring
Businesses use statistical summaries to monitor delivery duration, production output, and service performance.
Machine Learning Evaluation
Descriptive statistics helps compare model residuals, prediction distributions, and performance across groups.
Common Mistakes in Descriptive Statistics
- Using the mean without examining extreme observations.
- Confusing population variance with sample variance.
- Assuming standard deviation establishes normality.
- Interpreting percentiles without checking the calculation method.
- Drawing conclusions from small or unrepresentative samples.
Data Science Tutorial for Beginners
Vaishali Ardhana








