Hypothesis Testing

Establishing and evaluating research hypotheses

Beatrice Taylor - beatrice.taylor@ucl.ac.uk

8th October 2025

Last week

Overview of lecture 2

Continued concepts from exploratory data analysis, in particular probability distributions.

• Representative data
• Normal distribution
• Binomial distribution
• Poisson distribution
• Exponentials
• Logarithms

This week

Motivation

How do we understand how likely events are to occur?

Question

What is the probability of someone at UCL being over 190cm?

Answer

Try to understand the distribution of heights.

Last week we asked this question and we thought about the distribution of the data. How to describe the distribution of the data mathematically.

This week we look at how to properly ask the research question, and how to answer it.

The scientific method

This lecture thinking about posing and answering research questions.

Last week we looked at exploratory data analysis - now the formal process of coming up with and evaluating questions - very much part of the scientific method.

Research science is about coming up with hypotheses and evaluating them.

It’s iterative - as we find out new information we update our method and processes accordingly. (Gelman et al. 2020. Bayesian Workflow)

Learning Objectives

By the end of this lecture you should be able to:

1. Establish a hypothesis for a given research project.
2. Define the Type I and Type II errors.
3. Evaluate a hypothesis using appropriate statistical tests.

We see stories in the news, how do we know what to believe and what not to believe? This is where having an understanding of statistical tests can help us - we can evaluate claims to try and figure out which are/ aren’t statistically significant.

Statisitical tests are frequently used in science to figure out if our findings are important or not - essentially a way to formally define a threshold of what is an interesting result vs, what isn’t an interesting result.

A clearly defined, and reporducible way to evaluate findings. ::: –>

How do you come up with a hypothesis?

Research question vs. hypothesis

Research question

A research question focuses on a specific problem.

Hypothesis

A formal statement that you will seek to prove or disprove.

The hypothesis is a statement you can explicitly test. It can either be true or false - and hence using evidence you can come to a judgement about it’s truth value.

Flip a coin

• You have a coin.
• You think it’s a fair coin.
• You toss it 10 times.
• It comes up heads 7 times.

What do you think is the hypothesis here?

WHich of these is something you can test?

The hypothesis

• You have a coin.
• You think it’s a fair coin.
• You toss it 10 times.
• It comes up heads 7 times.

The hypothesis is that it is a fair coin.

What research question can you ask?

• You have a coin.
• You think it’s a fair coin.
• You toss it 10 times.
• It comes up heads 7 times.

Is it a fair coin?

What’s the probability that it’s fair?

If the coin is fair, how likely would it be to see 7 heads out of 10 flips?

What research question should you ask?

• You have a coin.
• You think it’s a fair coin.
• You toss it 10 times.
• It comes up heads 7 times.

Correct formulation:

If the coin is fair, how likely would it be to see 7 heads out of 10 flips or an even more extreme result?

Interested in the probability of seeing the event or something even more extreme.

Establishing and evaluating a Hypothesis

In five simple steps.

Step 1

Define the null and alternative hypothesis

\(H_0\) - the null hypothesis

• this is the “status quo”
• it is assumed to be true

\(H_1\) - the alternative hypothesis

• your hypothesis
• it requires some evidence (i.e. data) to verify
• it directly contradicts the null hypothesis

Be careful in how the two are defined. H_1 needs to contradict H_0.

For example if my H_0 was everyone likes chocolate. H_1 couldn’t be some people prefer biscuits - since in this situation both H_0 and H_1 can be true at the same time. H_1 would have to be that not everyone likes chocolate.

Step 2

Set your significance level \(\alpha\)

The significance level is the threshold below which you reject the null hypothesis.

• Decide what “too unlikely” means
• Common choice is 5% significance
  • \(\alpha = 0.05\)
  • This means that if we see evidence that would have less than a 5% chance of occurring under the null hypothesis, then we reject the null hypothesis.

WARNING

Decide what “too unlikely” means before you do the test

• otherwise considered ‘HARKing’
  • Hypothesising
  • After
  • the
  • Results
  • are
  • Known

It’s important to decide on your significance level prior, as otherwise you might find an event which is significant at the 10% threshold and not at 5% - which could lead to bad scientific practise.

There’s a balance - often don’t know what will be interesting - so you do some exploratory data analysis. But you shouldn’t be fishing around for stuff - needs to be grounded in theory and literature.

Step 3

Identify the evidence

• This could mean collecting the data
• Or identifying a suitable exisiting dataset

In quant methods this normally means finding an open public dataset to use.

The evidence needs to be suitable to answer the question - think about biased/representtaive dtaa from last week. Soemthign they will be assessed on in the assessment.

Step 4

Calculate the p-value

The p-value is the probability of seeing the evidence, or something even more extreme, if the null hypothesis is true.

• Calculated according to the appropriate statistical test
• The choice of test is determined by the research question and the data

We’ll come back to different types of statistical test later in the lecture

Step 5

Compare p-value with significance level

• p-value \(> \alpha\)
  • Evidence not that unlikely.
  • Not enough evidence to reject \(H_0\).
• p-value \(\leq \alpha\)
  • Evidence very unlikely.
  • Reject \(H_0\) and accept \(H_1\).

The steps

In order to evaluate our hypothesis we just have to do the five steps:

1. Define the null and alternative hypothesis
2. Set you significance level
3. Identify the evidence
4. Calculate the p-value
5. Compare p-value with hypothesis level

Types of error

… where things can go wrong.

Type I error

The true null hypothesis is incorrectly rejected.

The null hypothesis is true, but you get a false positive leading to you rejecting the null hypothesis.

This is also called a false positive.

Example: In court a defendant is found guilty despite being innocent.

Type II error

The false null hypothesis is incorrectly accepted.

The null hypothesis is false, but you get a false negative result, leading you to accepting the null hypothesis.

This is also called a false negative.

Example: In court a defendant is found innocent despite being guilty.

Example: Mammograms

NHS offers breast cancer screening for all people with breasts between the ages of 50 and 70.

The idea is to screen all those in the population who are at high risk of breast cancer - in the hope they pick up results better.

Whilst the tests are good they’re not 100% accurate.

Example: Screening outcomes

\(H_0:\) The individual doesn’t have breast cancer.

\(H_1:\) The individual does have breast cancer.

Example: Evaluating the evidence

From NHS digital.

False positive

• In 2020-2021, \(4.0%\) of those screened had an abnormal results and were referred for assessment.
• Of these, \(77.1%\) were found to not have breast cancer at follow up assessment.
• Leading to a false positive rate of \(3.1%\).

False negative

• Those who had a negative screening but did in fact have breast cancer.
• Harder to calculate the false negative rate, as they might be diagnosed with breast cancer at any later point in time.
• Studies suggest the false negative rate could be as high as \(20%\).

False positive is bad as you might pursue a healthcare treatment which is unecessary.

False negative - in healthcare context this is the worst outcome - as goes undiagnosed and hence untreated.

Matrix of errors

Possible outcomes form a matrix. Have two good outcomes, and two bad outcomes which we want to look out for (these are the Type I and Type II errors).

This is something to bear in mind when we look at evaluating our hypotheses. How good is our data, so how reliable is our outcome?

Often this language is used when evaluating the outcomes of an ML classification model. Where we report the quality of a model in terms of its rate of false positives and false negatives.

A good hypothesis or a bad hypothesis?

Understanding the literature and the context

The hypothesis should not come out of thin air.

Should consider:

• What do you know about the context?
• What research have other people done?

Asking ethical hypothesis questions

It’s important to not make unethical assumptions in choosing the hypothesis.

Example

Police profiling - assumes a correlation between ethnicity and crime

• Use contextual knowledge
• Is this causation?
• Or correlation linked to other factors?

Correlation vs. Causation

Correlation: Two variables are statistically related, as one changes so does the other.

Causation: One variable influences the other variable to occur.

Causation implies correlation. BUT correlation does not imply causation!!!

Lots of things can be correlated BUT it doesn’t mean one event caused another.

Aliens and librarians

Image credit: [Spurious Correlations](https://www.tylervigen.com/spurious/correlation/19598_google-searches-for-report-ufo-sighting_correlates-with_the-number-of-librarians-in-hawaii)

It’s a dumb example but it’s quite easy to find nonsense correlations between random variables. Can generate your own at the website spurious correlations.

Correlation IS NOT causation

You might not know whether events are correlated, or causing each other.

BUT

The point of the hypothesis test is to test your idea – but you should use your contextual understanding to come up with plausible (and ethical) initial questions.

Will return to looking at the correlation between variables in week 5.

The point of the scientific method

It’s a process

• question
• test
• evaluate
• REPEAT!

Example: students height

Research question
Are male and female students similar heights?

Research hypothesis
Male and female students are different heights on average.

Example - step 1

Define the null and alternative hypothesis

\(H_0\): The mean height of male and female students is the same.

\(H_1\): The mean height of male and female students is different.

Example - step 2

Set your significance level

\(\alpha = 0.05\)

Example - step 3

Identify the evidence

I’ve collected data from 198 students, as follows:

Group	Sample Size	Mean (cm)	std (cm)
Female students	95	170	5
Male students	103	180	6

Example - step 4

Calculate the p-value

Aha!

We need to know what statistical test to use!

Statistical tests

Parametric vs. Non-parametric tests

Parametric tests

• Evaluate hypothesis for specific parameters
• Assumptions about the distribution
  • assumed normal distribution
  • independent and unbiased samples
  • equal/comparable variances
• Continuous data

Non-parametric tests

• Evaluate hypothesis for entire population distribution
• Typcially less assumptions on the distribution
• Continuous or discrete data

Deciding on a test

NOTE: want a table here that guides you on how to choose an appropriate statistical test

Parametric tests

Student’s T-test

Student’s T-test is used to compare the mean of a dataset.

• parametric statistical test
• assumes the data is normally distributed

This is William Sealy Gosset - he was **not** a student.
Image credit: https://en.wikipedia.org/wiki/William_Sealy_Gosset#/media/File:William_Sealy_Gosset.jpg

William Gosset was working for Guinness brewing company when he came up with the t-test - but his company wanted his to publish under a pseudonym - hence ‘student’. He was comparing the chemical properties of different samples of barley.

Student’s T-test: one sample

Tests whether the population mean is equal to a specific value or not

The test statistic is calculated as:

\[\begin{align} t = \frac{\bar{x} - \mu_{0}}{s / \sqrt{n}} \end{align}\]

where

• \(\bar{x}\) is the sample mean
• \(\mu_{0}\) is the hypothesised population mean
• \(s\) is the sample standard deviation
• \(n\) is the sample size

Student’s T-test: two sample

Tests if the population means for two different groups are equal or not.

The test statistic is:

\[\begin{align} t = \frac{\bar{x}_1 - \bar{x}_2}{s_p \sqrt{\frac{1}{n_1} + \frac{1}{n_2}}} \end{align}\]

• \(\bar{x}_1, \bar{x}_2\) are the sample means of groups 1 and 2
• \(n_1, n_2\) are the sample sizes of groups 1 and 2
• \(s_p\) is the pooled standard deviation

\[\begin{align} s_p = \sqrt{\frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}} \end{align}\]

with \(s_1, s_2\) the sample standard deviations.

Student’s T-test: paired

Tests if the difference between paired measurements for a population is zero or not - normally used with longitudinal data.

The test statistic is:

\[\begin{align} t = \frac{\bar{d}}{s_d / \sqrt{n}} \end{align}\]

where

• \(\bar{d}\) is the mean of the paired differences
• \(s_d\) is the standard deviation of the paired differences
• \(n\) is the number of pairs

For example used when I have data from different years, and want to figure out if there’s a difference across the years.

You need to be able to pair the data - i.e. the same things being observed at time point 1 and time point 2.

How many tails?

Tests can be one-tailed or two-tailed - which you want is determined when you define the hypothesis.

One tailed: if you only care is the mean is significant in one direction Two tailed: if you care about the mean being different regardless of direction

Non-parametric tests

Kolmogorov-Smirnov

NOTE: want a diagram showing this.

• Compares two probability distributions
• Can be used to test whether an observed sample came from a given distribution
• Or to test whether two samples both came from the same distribution

Named after two Russian soviet mathematicians working in the mid 20th century

K-S test: one sample test

The Kolmogorov–Smirnov test statistic is:

\[\begin{align} D_n = \sup_x \, | F_n(x) - F(x) | \end{align}\]

where

• \(F_n(x)\) is the empirical distribution function (EDF) of the sample
• \(F(x)\) is the cumulative distribution function (CDF) of the reference distribution

Note

‘\(sup\)’ is the suprenum - think of it as the smallest upper bound.

K-S: empirical distribution function

The empirical distribution function (EDF) is:

\[\begin{align} F_{n}(x) = \frac{1}{n} \sum_{i=1}^{n} 1_{(-\infty ,x]}(X_{i}) \end{align}\]

where

• \(n\) is the number of observations
• \(X_i\) are the ordered sample values
• \(1_{(-\infty ,x]}(X_{i})\) is an indicator function (1 if \(X_i \leq x\), else 0)

K-S test: two sample test

For the two-sample test:

\[\begin{align} D_{n,m} = \sup_x \, | F_n(x) - G_m(x) | \end{align}\]

where

• \(F_n(x)\) and \(G_m(x)\) are the EDFs of the two samples.

K-S test: decision rule

The hypotheses would be:

\(H_0\): the distributions are the same \(H_1\): the distributions differ

Larger values of the test statistic \(D\) is stronger evidence against \(H_0\).

Kernel density estimate (KDE)

• Used to generate a smooth PDF for a random variable dataset.
• Useful for understanding the underlying distribution of a sample .
• Think of it as getting a smooth function to describe a histogram of data.
• There are no assumptions about the prior distribution.

KDE of simulated heights

It’s easy to fit a KDE to data in Python:

import numpy as np
import pandas as pd 
from scipy.stats import gaussian_kde

# Supposing you have some data 
data = pd.read_csv('/path_to_data')

# Kernel Density Estimation
kde = gaussian_kde(data)
x_vals = np.linspace(100, 200, 100)
y_vals = kde(x_vals)

KDE of simulated heights

KDE use case

NOTE: want a diagram to show this.

• fit the KDE to two samples
• compare visually
• carry out non-paremetric test - such as Kolmogorov-Smirnov

Example: students height

Example - step 1, 2, 3

Define the null and alternative hypothesis

\(H_0\): The mean height of male and female students is the same.

\(H_1\): The mean height of male and female students is different.

Set your significance level

\(\alpha = 0.05\)

Identify the evidence

Group 1 – female students
\(\bar{x}_1 = 170\), \(s_1 = 5\), \(n_1\) = 95

Group 2 – male students
\(\bar{x}_2 = 180\), \(s_2 = 6\), \(n_2\) = 103

Example - step 4

Calculate the p-value

• Use Student’s T-test: two sample
• don’t care if students are taller or shorter - so use two-tailed test

Example - step 4 - calculation

NOTE: want to add how to calculate this in Python

Substituting values:

\[\begin{align} s_p &= \sqrt{\frac{(95-1)\cdot 5^2 + (103-1)\cdot 6^2}{95+103-2}} \approx 5.55 \end{align}\]

Now compute \(t\):

\[\begin{align} t &= \frac{170 - 180}{5.55 \cdot \sqrt{\tfrac{1}{95} + \tfrac{1}{103}}} \approx -12.7 \end{align}\]

Example - step 5

Compare p-value with hypothesis level

Now we need to compare the test statistic to the critical t-value.

Example - step 5 - calculation

NOTE: need to add a description of degrees of freedom

For Student’s T-Test we need degrees of freedom:

\[\begin{align} df = n_1 + n_2 - 2 = 95 + 103 - 2 = 196 \end{align}\]

Then use Python to calculate the two-tailed critical t-value at \(\alpha = 0.05\).

from scipy.stats import t

alpha = 0.05
df = 196

# two-tailed: split alpha
t_crit = t.ppf(1 - alpha/2, df)
print("Critical t-value:", t_crit.round(2))

Critical t-value: 1.97

Example - conclusion

Test statistic: \(t \approx -12.7\).

Critical t-value at \(\alpha=0.05\): \(t_{crit} = 1.97\)

Since

\[\begin{align} t= |-12.7| \gg 1.97= t_crit \end{align}\]

we reject \(H_0\). Male and female students have significantly different heights.

Overview

We’ve covered:

• What makes a good hypothesis
• How to formally state a hypothesis
• Types of statistical tests

Practical

The practical will focus on establishing and evaluating a research hypothesis.

Have questions prepared!