Correlation, Covariance, and Hypothesis Testing

Statistical methods help data scientists examine relationships and evaluate whether observed differences provide evidence against a proposed assumption.

Understanding Correlation

Correlation measures the strength and direction of a linear relationship between two numerical variables.

import pandas as pd

df = pd.DataFrame({

"Study_Hours": [1, 2, 3, 4, 5, 6],

"Exam_Score": [45, 50, 60, 65, 78, 88],

"Sleep_Hours": [8, 7, 8, 6, 7, 6]

})

print(df.corr(numeric_only=True))

Pearson correlation ranges from -1 to +1. A positive value indicates that higher values of one variable tend to accompany higher values of another, while a negative value indicates an opposite linear pattern.

Correlation does not establish causation. Confounding variables, outliers, and nonlinear relationships can affect interpretation.

Understanding Covariance

Covariance indicates whether two variables tend to move in the same direction or in opposite directions.

print(df["Study_Hours"].cov(df["Exam_Score"]))

Positive covariance indicates that the variables tend to vary in the same direction, while negative covariance indicates that they tend to vary in opposite directions. Unlike correlation, covariance depends on the scales of the variables, making direct comparisons across different datasets more difficult.

Introduction to Hypothesis Testing

Hypothesis testing uses sample data to evaluate a claim about a population.

• Null hypothesis ((H_0)): The default claim being tested, such as no difference between two population means.
• Alternative hypothesis ((H_1)): The competing claim, such as a difference between the means.
• Significance level ((\alpha)): A threshold selected before testing, often 0.05.
• P-value: The probability, assuming the null hypothesis and the test's assumptions are true, of obtaining a result at least as extreme as the observed result.

Consider comparing the scores of two independent groups.

from scipy.stats import ttest_ind

group_a = [70, 75, 80, 72, 78, 74]

group_b = [82, 85, 79, 88, 84, 81]

statistic, p_value = ttest_ind(

group_a,

group_b,

equal_var=False

)

print("Test statistic:", statistic)

print("P-value:", p_value)

The code uses Welch's independent-samples t-test, which does not assume equal population variances.

If the p-value is below the chosen significance level, the result provides evidence against the null hypothesis. Otherwise, there is insufficient evidence to reject it. This does not prove that the null hypothesis is true.

Statistical significance also does not automatically imply practical importance. The size of the observed difference, uncertainty, sample quality, and study design should all be considered.