Menu

Descriptive Statistics for Data Science

A dataset containing thousands of numerical observations can be difficult to interpret without statistical summaries. Individual records provide specific measurements, but they do not clearly describe the overall distribution, central values, or variability of the dataset.

Descriptive statistics converts raw numerical information into summary measurements that help explain the dataset's characteristics.

Descriptive statistics is the branch of statistics concerned with organising, summarising, and describing observed data using numerical measures and distributional summaries.

In data science, descriptive statistics supports exploratory analysis, feature engineering, anomaly detection, and model evaluation.

What Are the Main Types of Descriptive Statistics?

Descriptive statistics can be divided into several categories.

Category Purpose Common Measures
Central Tendency Describes typical values Mean, median, mode
Dispersion Measures variability Range, variance, standard deviation
Position Identifies relative location Quartiles, percentiles
Distribution Shape Describes asymmetry and tails Skewness, kurtosis
Association Describes relationships Covariance, correlation

Each category reveals a different characteristic of the dataset.

For example, average monthly revenue describes central tendency, while standard deviation indicates how much revenue varies across months.

What Are Measures of Central Tendency?

Measures of central tendency identify representative values within a distribution.

The three commonly used measures are mean, median, and mode.

1. Arithmetic Mean

The mean is calculated by dividing the sum of observations by the total number of observations.

\[ \bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i \]

Here:

  • \(x_i\) represents an individual observation.
  • \(n\) represents the number of observations.
  • \(\bar{x}\) represents the sample mean.

Consider the following daily sales values:

₹1,000, ₹1,500, ₹2,000, ₹2,500, and ₹3,000.

\[ \bar{x}=\frac{1000+1500+2000+2500+3000}{5} \]

\[ \bar{x}=2000 \]

The average daily sales value is ₹2,000.

Calculating Mean Using Python

import numpy as npsales = np.array([1000, 1500, 2000, 2500, 3000])print(np.mean(sales))

Output:

2000.0

The mean is useful when observations are reasonably representative of the distribution.

However, it is sensitive to extreme values.

2. Median

The median is the middle observation after sorting numerical values.

For an odd number of observations, it is the middle value.

For an even number, it is the average of the two middle values.

Consider:

\[ [10,15,20,25,100] \]

The median is:

\[ Median=20 \]

The mean is:

\[ Mean=34 \]

The unusually large value of 100 increases the mean but does not change the median substantially.

Calculating Median in Python

values = np.array([10, 15, 20, 25, 100])print(np.median(values))

Output:

20.0

Median is commonly used for skewed distributions, including income and transaction values.

3. Mode

The mode is the most frequently occurring value.

Consider:

\[ [2,3,3,4,5,5,5,6] \]

The mode is 5 because it appears three times.

Python implementation:

import pandas as pdvalues = pd.Series([2, 3, 3, 4, 5, 5, 5, 6])print(values.mode())

Output:

0    5
dtype: int64

Mode is useful for categorical variables, such as the most frequently purchased product category.

A dataset may contain multiple modes when several values share the highest frequency.

Mean vs Median vs Mode

Measure How It Is Calculated Main Advantage Limitation
Mean Sum divided by count Uses all numerical observations Sensitive to extremes
Median Middle ordered value Resistant to extreme values Does not directly describe magnitude of all observations
Mode Most frequent value Supports categorical analysis May not be unique

The appropriate measure depends on the distribution and analytical question.

What Are Measures of Dispersion?

Measures of dispersion describe how widely observations vary.

Two datasets may have the same mean but different degrees of variability.

For example:

Dataset A:

\[ [18,19,20,21,22] \]

Dataset B:

\[ [5,10,20,30,35] \]

Both datasets have a mean of 20.

However, Dataset B contains much greater variation.

Dispersion measures quantify this difference.

What Is Range?

Range measures the difference between the largest and smallest observations.

\[ Range=Maximum-Minimum \]

For Dataset A:

\[ Range=22-18=4 \]

For Dataset B:

\[ Range=35-5=30 \]

Dataset B has a much larger range.

However, the range is highly sensitive to extreme observations.

What Is Variance?

Variance measures the average squared deviation from the mean.

For a population:

\[ \sigma^2=\frac{\sum_{i=1}^{N}(x_i-\mu)^2}{N} \]

For a sample:

\[ s^2=\frac{\sum_{i=1}^{n}(x_i-\bar{x})^2}{n-1} \]

Here:

  • \(\sigma^2\) represents population variance.
  • \(s^2\) represents sample variance.
  • \(\mu\) represents the population mean.
  • \(\bar{x}\) represents the sample mean.

The sample formula uses \(n-1\) as the denominator for the conventional unbiased estimate of population variance under suitable sampling assumptions.

Calculating Variance Using Python

import numpy as npdata = np.array([10, 20, 30, 40, 50])population_variance = np.var(data)sample_variance = np.var(data, ddof=1)print(population_variance)print(sample_variance)

Output:

200.0
250.0

The ddof=1 parameter changes the denominator from \(n\) to \(n-1\).

Variance is expressed in squared units, which can make direct interpretation difficult.

What Is Standard Deviation?

Standard deviation is the square root of variance.

For a population:

\[ \sigma=\sqrt{\frac{\sum_{i=1}^{N}(x_i-\mu)^2}{N}} \]

Standard deviation expresses variability in the same units as the original variable.

For example, if daily revenue has a mean of ₹10,000 and a standard deviation of ₹2,000, it indicates the scale of variation around the average.

However, standard deviation alone does not specify the shape of the distribution.

Calculating Standard Deviation

data = np.array([10, 20, 30, 40, 50])print(np.std(data))

Output:

14.142135623730951

NumPy uses the population-style denominator by default.

A sample standard deviation can be calculated using ddof=1.

What Are Quartiles and Percentiles?

Quartiles divide ordered numerical observations into four parts using three cut points.

  • First Quartile (Q1): 25th percentile
  • Second Quartile (Q2): 50th percentile or median
  • Third Quartile (Q3): 75th percentile

Percentiles describe the position of values within an ordered distribution.

For example, a 90th-percentile transaction amount represents a threshold at or below which approximately 90% of observations fall, subject to the percentile calculation method.

Calculating Percentiles Using NumPy

data = np.array([    100, 200, 300, 400, 500,    600, 700, 800, 900, 1000])print(np.percentile(data, 25))print(np.percentile(data, 50))print(np.percentile(data, 75))

Output:

325.0
550.0
775.0

These results use NumPy's default percentile calculation method.

Different statistical software or interpolation methods may produce slightly different estimates for small datasets.

What Is the Interquartile Range?

The Interquartile Range measures the spread between the first and third quartiles.

\[ IQR=Q_3-Q_1 \]

For the previous example:

\[ IQR=775-325=450 \]

The middle 50% of observations span 450 units under this quartile definition.

IQR is less sensitive to extreme observations than the overall range.

It is commonly used in box plots and preliminary outlier detection.

What Is Skewness?

Skewness describes the asymmetry of a distribution.

A positively skewed distribution has a longer or heavier right tail.

A negatively skewed distribution has a longer or heavier left tail.

A symmetric distribution has approximately balanced tails.

Distribution Typical Characteristic Example
Positive Skew Long right tail Customer spending
Negative Skew Long left tail Scores concentrated near a maximum
Approximately Symmetric Similar behaviour on both sides Some measurement distributions

Calculating Skewness

import pandas as pdvalues = pd.Series([100, 150, 200, 250, 5000])print(values.skew())

The resulting statistic describes sample skewness using Pandas' estimator.

A positive value indicates right-skewed behaviour under this measure.

However, skewness near zero does not prove that the distribution is normal.

What Is Kurtosis?

Kurtosis is a statistical measure associated with the tail behaviour of a distribution.

Distributions with heavier tails may produce more extreme observations than a normal distribution.

Pandas calculates excess kurtosis, for which a normal distribution has a theoretical value of zero.

print(values.kurt())

Kurtosis can help investigate extreme-value behaviour, but estimates may be unstable for very small datasets.

It should be interpreted alongside visualisations and other distributional measures.

How Do You Calculate Descriptive Statistics Using Pandas?

Pandas provides describe() for generating several descriptive measures together.

import pandas as pddf = pd.DataFrame({    "sales": [1000, 1500, 2000, 2500, 3000],    "orders": [2, 3, 4, 5, 6]})print(df.describe())

The output includes count, mean, sample standard deviation, minimum, quartiles, and maximum for numerical columns.

Additional statistics can be calculated separately.

print(df["sales"].median())print(df["sales"].var())print(df["sales"].skew())

These methods provide a more detailed statistical description of the dataset.

How Are Descriptive Statistics Used in Data Science?

Customer Behaviour Analysis

Retailers calculate average order value, median spending, and spending variability to understand customer purchasing patterns.

Fraud Detection

Financial institutions examine unusual transaction amounts and deviations from historical activity.

Model Preprocessing

Means, medians, and standard deviations support imputation and feature scaling.

Operational Monitoring

Businesses use statistical summaries to monitor delivery duration, production output, and service performance.

Machine Learning Evaluation

Descriptive statistics helps compare model residuals, prediction distributions, and performance across groups.

Common Mistakes in Descriptive Statistics

  1. Using the mean without examining extreme observations.
  2. Confusing population variance with sample variance.
  3. Assuming standard deviation establishes normality.
  4. Interpreting percentiles without checking the calculation method.
  5. Drawing conclusions from small or unrepresentative samples.

Data Science Tutorial for Beginners

VA

Vaishali Ardhana



Learn via Video Course

undefined Logo

0 hrs



Get in Touch For Details