31 Likelihood Ratio Tests

31.1 General Set-up

Most hypothesis testing procedures can be formulated so that a test statistic, \(S(\boldsymbol{x})\) is applied to the data \(\boldsymbol{x} = (x_1, x_2, \ldots, x_n)^T\) so that:

1. \(S(\boldsymbol{x}) \geq 0\)
2. The larger \(S(\boldsymbol{x})\) is, the more significant the test is (i.e., the more evidence against the null in favor of the alternative)
3. The p-value is \(p(\boldsymbol{x}) = \Pr(S(\boldsymbol{X^*}) \geq S(\boldsymbol{x}))\) where \(S(\boldsymbol{X^*})\) is distributed according to the null distribution

31.2 Significance Regions

A level \(\alpha\) test is a signficance rule (i.e., a rule for calling a test statistically significant) that results in a false positive rate (i.e., Type I error rate) of \(\alpha\). Under our set-up, significance regions take the form:

\[\Gamma_\alpha = \left\{\boldsymbol{x}: S(\boldsymbol{x}) \geq c_{1-\alpha} \right\},\]

where \(c_{1-\alpha}\) is the \((1-\alpha)\) percentile of \(S(\boldsymbol{X}^*)\) so that \(\Pr(S(\boldsymbol{X}^*) \geq c_{1-\alpha}) = \alpha\). We restrict \(0 < \alpha < 1\).

Note that if \(\alpha' \leq \alpha\) then \(\Gamma_{\alpha'} \subseteq \Gamma_\alpha\).

31.3 P-values

A p-value can be defined in terms of significance regions:

\[p(\boldsymbol{x}) = \min \left\{\alpha: \boldsymbol{x} \in \Gamma_\alpha \right\}\]

31.4 Example: Wald Test

Consider the hypothesis test, \(H_0: \theta=\theta_0\) vs \(H_1: \theta \not= \theta_0\). Let \(\hat{\theta}_n(\boldsymbol{x})\) be the MLE of \(\theta\). We have

\[S(\boldsymbol{x}) = \frac{\left| \hat{\theta}_n(\boldsymbol{x}) - \theta_0 \right|}{\hat{{\operatorname{se}}}\left(\hat{\theta}_n(\boldsymbol{x})\right)},\]

\[\Gamma_\alpha = \left\{\boldsymbol{x}: S(\boldsymbol{x}) \geq |z_{\alpha/2}| \right\},\]

where \(z_{\alpha/2}\) is the \(\alpha/2\) percentile of the Normal\((0,1)\) distribution.

31.5 Neyman-Pearson Lemma

Suppose we are testing \(H_0: \theta = \theta_0\) vs \(H_1: \theta = \theta_1\) where in practice \(\theta_0\) and \(\theta_1\) are known, fixed quantities. The most powerful test has statistic and significance regions:

\[ S(\boldsymbol{x}) = \frac{f(\boldsymbol{x}; \theta_1)}{f(\boldsymbol{x}; \theta_0)} = \frac{L(\theta_1; \boldsymbol{x})}{L(\theta_0; \boldsymbol{x})} \]

\[\Gamma_\alpha = \left\{\boldsymbol{x}: S(\boldsymbol{x}) \geq c_{1-\alpha} \right\},\]

where \(c_{1-\alpha}\) is the \((1-\alpha)\) percentile of \(S(\boldsymbol{X}^*)\) so that \(\operatorname{Pr}_{\theta_0}(S(\boldsymbol{X}^*) \geq c_{1-\alpha}) = \alpha\).

31.6 Simple vs. Composite Hypotheses

A simple hypothesis is defined in terms of a single value, e.g.,

• \(H_0: \mu=0\)
• \(H_0: p = p_0\) where \(p_0\) is a placehold for a known, fixed number in practice
• \(H_1: \lambda=5\)

A composite hypothesis is defined by multiple values, e.g.,

• \(H_0: \mu \leq 0\) vs \(H_1: \mu > 0\)
• \(H_0: p_1 = p_2\), where \(p_1\) and \(p_2\) are two unknown parameters corresponding to two populations
• \(H_1: \mu \not= 0\)

31.7 General Hypothesis Tests

Let \(X_1, X_2, \ldots, X_n \stackrel{{\rm iid}}{\sim} F_\theta\) where \(\theta \in \Theta\). Let \(\Theta_0, \Theta_1 \subseteq \Theta\) so that \(\Theta_0 \cap \Theta_1 = \varnothing\) and \(\Theta_0 \cup \Theta_1 = \Theta\). The hypothesis test is:

\(H_0: \theta \in \Theta_0\) vs \(H_1: \theta \in \Theta_1\)

If \(\Theta_0\) or \(\Theta_1\) contain more than one value then the corresponding hypothesis is composite.

31.8 Composite \(H_0\)

The significance regions indexed by their level \(\alpha\) are determined so that:

\[\Gamma_\alpha = \left\{\boldsymbol{x}: S(\boldsymbol{x}) \geq c_{1-\alpha} \right\},\]

where \(c_{1-\alpha}\) is such that \[\max_{\theta \in \Theta_0} \Pr(S(\boldsymbol{X^*}) \geq c_{1-\alpha}) = \alpha.\]

In this case,

\[\begin{align*} p(\boldsymbol{x}) & = \min \left\{\alpha: \boldsymbol{x} \in \Gamma_\alpha \right\} \\ & = \max_{\theta \in \Theta_0} \operatorname{Pr}_\theta (S(\boldsymbol{X}^*) \geq S(\boldsymbol{x})) \end{align*}\]

31.9 Generalized LRT

Let \(X_1, X_2, \ldots, X_n \stackrel{{\rm iid}}{\sim} F_\theta\) where \(\theta \in \Theta\) and we are testing \(H_0: \theta \in \Theta_0\) vs \(H_1: \theta \in \Theta_1\).

The generalized LRT utilizes test statistic and significance regions:

\[\lambda(\boldsymbol{x}) = \frac{\max_{\theta \in \Theta} L(\theta; \boldsymbol{x})}{\max_{\theta \in \Theta_0} L(\theta; \boldsymbol{x})} = \frac{L\left(\hat{\theta}; \boldsymbol{x}\right)}{L\left(\hat{\theta}_0; \boldsymbol{x}\right)} \]

\[\Gamma_\alpha = \left\{\boldsymbol{x}: \lambda(\boldsymbol{x}) \geq c_{1-\alpha} \right\}\]

31.10 Null Distribution of Gen. LRT

The null distribution of \(\lambda(\boldsymbol{x})\) under “certain regularity assumptions” can be shown to be such that, as \(n \rightarrow \infty\),

\[ 2 \log \lambda(\boldsymbol{x}) \stackrel{D}{\longrightarrow} \chi^2_v \]

where \(v = \operatorname{dim}(\Theta) - \operatorname{dim}(\Theta_0)\).

The significance regions can be more easily written as \(\Gamma_\alpha = \left\{\boldsymbol{x}: 2 \log \lambda(\boldsymbol{x}) \geq c_{1-\alpha} \right\}\) where \(c_{1-\alpha}\) is the \(1-\alpha\) percentile of the \(\chi^2_v\) distribution.

31.11 Example: Poisson

Let \(X_1, X_2, \ldots, X_n \stackrel{{\rm iid}}{\sim} \mbox{Poisson}(\theta)\) where \(\theta > 0\) and we are testing \(H_0: \theta = \theta_0\) vs \(H_1: \theta \not= \theta_0\). The unconstrained MLE is \(\hat{\theta} = \overline{x}\). The generalized LRT statistic

\[ 2 \log \lambda(\boldsymbol{x}) = 2 \log \frac{e^{-n \hat{\theta}} \hat{\theta}^{\sum x_i} }{e^{-n \theta_0} \theta_0^{\sum x_i} } = 2 n \left[ (\theta_0 - \hat{\theta}) - \hat{\theta} \log(\theta_0 / \hat{\theta}) \right] \]

which has an asymptotic \(\chi^2_1\) null distribution.

31.12 Example: Normal

Let \(X_1, X_2, \ldots, X_n \stackrel{{\rm iid}}{\sim} \mbox{Normal}(\mu, \sigma^2)\) and we are testing \(H_0: \mu = \mu_0\) vs \(H_1: \mu \not= \mu_0\). The generalized LRT can be applied for multidimensional parameter spaces \(\boldsymbol{\Theta}\) as well. The statistic, which has asymptotic null distribution \(\chi^2_1\), is

\[ 2 \log \lambda(\boldsymbol{x}) = 2 \log \left( \frac{\hat{\sigma}^2_0}{\hat{\sigma}^2} \right)^{n/2} \]

where

\[ \hat{\sigma}^2_0 = \frac{\sum_{i=1}^n (x_i - \mu_0)^2}{n}, \quad \hat{\sigma}^2 = \frac{\sum_{i=1}^n (x_i - \overline{x})^2}{n}. \]