Exploratory Data Analysis in R: Distributions, Statistics and Outliers

Understanding Data Distributions

A distribution shows how the values in a column are spread out. A histogram shows it best.

Look at three things:

• Centre: Where values tend to fall.
• Spread: How widely values vary.
• Shape: How the values are distributed.

Skewness describes the shape. A positive value means a longer tail on the high side.

The maths marks here have a small positive skew of 0.33, which means most students score in the middle, with a few high scores. Look at the histogram before you trust any average, since an average alone can hide the shape.

round(mean(((dmath-mean(dmath)) / sd(d$math))^3), 2)   # 0.33

Descriptive Statistics

Descriptive statistics summarise data in a few numbers.

• Mean: The average.
• Median: The middle value.
• Standard deviation: Shows how far values usually are from the mean.
• Quartiles: Split the data into four equal parts.

The summary() function gives most of these at once, so use it first on every numeric column.

Compare the mean and the median: if they differ a lot, the data is probably skewed.

Identifying Patterns and Outliers

An outlier is a value far from the others. A common rule marks a value as an outlier if it is more than 1.5 times the interquartile range beyond the quartiles.

Outliers are not always mistakes. Check each one before you decide to keep or remove it. An outlier may also be a real and important case, such as a top student, so look before you act.

Here, the rule finds two unusual maths marks.