6 classical hypothesis testing theory

Classical Hypothesis

Testing Theory

Review

•

5 steps of classical hypothesis testing

(Ch. 3)

1. Declare null hypothesis H0 and alternate

hypothesis H1

2. Fix a threshold α for Type I error (1% or 5%)

• Type I error (α): reject H0 when it is true

• _{Type II error (β): accept H0 when it is false}

Review

4. Determine what observed values of the test statistic should lead to rejection of H0

• _{Significance point K (determined by α)}

5. Test to see if observed data is more extreme than significance point K

• If it is, reject H0

Overview of Ch. 9

– _{Simple Fixed-Sample-Size Tests}

– _{Composite Fixed-Sample-Size Tests} – _{The -2 log λ Approximation}

– _{The Analysis of Variance (ANOVA)}

– _{Multivariate Methods}

– _{ANOVA: the Repeated Measures Case}

– _{Bootstrap Methods: the Two-sample t}

The Issue

• _{In the simplest case, everything is} specified

– Probability distribution of H0 and H1 • _{Including all parameters}

– _{α (and K)}

– _{But: β is left unspecified}

Most Powerful Procedure

•

Neyman-Pearson Lemma

– _{States that the likelihood-ratio (LR) test is the}

most powerful test for a given α

– _{The LR is defined as:}

– _where

• f0, f1 are completely specified density functions for

H0,H1

• X1, X2, … Xn are iid random variables

Neyman-Pearson Lemma

– _{H0 is rejected when LR ≥ K}

– _{With a constant K chosen such that:}

P(LR ≥ K when H0 is true) = α

– _{Let’s look at an example using the}

Neyman-Pearson Lemma!

Example

• _{Basketball players seem to}

be taller than average

– Use this observation to

formulate our hypothesis H1:

• “Tallness is a factor in the recruitment of KU basketball players”

– The null hypothesis, H0, could

be:

• “No, the players on KU’s team are a just average height compared to the population in the U.S.”

Example

•

Setup:

– _{Average height of males in the US: 5’9 ½“} – _{Average height of KU players in 2008:}

6’04 ½”

• _{Assumption: both populations are}

normal-distributed centered on their respective

averages (μ0 = 69.5 in, μ1 = 76.5 in) and σ = 2

• _{Sample size: 3}

Example

•

The two populations:

height (inches)

p

f_{0 f}

Example

– _{Our test statistic is the Likelihood Ratio, LR}

– _{Now we need to determine a significance}

point K at which we can reject H , given α =

) ( ) ( ) ( ) ( ) ( ) ( ) ( 3 0 2 0 1 0 3 1 2 1 1 1 x f x f x f x f x f x f x         2 2 2 2 2 2 2 2 2 2 2 2 8 ) 5 . 69 ( 8 ) 5 . 69 ( 8 ) 5 . 69 ( 8 ) 5 . 76 ( 8 ) 5 . 76 ( 8 ) 5 . 76 ( 2 3 2 2 2 1 2 3 2 2 2 1              x x x x x x e e e e e e       3 1 2 2 _{( 76.5)}

Example

– _{So we just need to solve for K’ and calculate K:}

• _{How to solve this? Well, we only need one set of}

values to calculate K, so let’s pick two and solve for

the third:

• We get one result: K3’=71.0803



  



'

1 2' 3'

05 . 0 ) ( ) ( )

( _{1 0 2 0 3 1 2 3}

0

K K K

Example

– _{Then we can just plug it in to Λ and}

calculate K:

      3 1 2 ' 2

' _{69.5) ( 76.5)}

( 8 1

i i i

K K

e K

_{(68 69.5)}2 _{(68 76.5)}2 _{(71 69.5)}2 _{(71 76.5)}2 _{(71.0803 69.5)}2 _{(71.0803 76.5)}2

Example

– _{With the significance point K = 1.663*10}-7 we

can now test our hypothesis based on observations:

• _{E.g.: Sasha = 83 in, Darrell = 81 in, Sherron = 71 in}

• _1.446*1012 > 1.663*10-7

• _{Therefore, our hypothesis that tallness is a factor in}

the recruitment of KU basketball players is true.

Neyman-Pearson Proof

•

Let A

define region in the joint range

of

X

1

,

X

2

, …

X

n

such that LR ≥

K

.

A

is

the

critical region

.

– _{If A is the only critical region of size α}

we are done

– _{Let’s assume another critical region of}











Proof

– _{H0 is rejected if the observed vector (x1,}

x2, …, xn) is in A or in B.

– _{Let A and B overlap in region C}

– _Power of the test: rejecting H0 when H1 is

true

• _{The Power of this test using A is:}

n n

A L(H1)







f1(u1) f1(u2) f1(u )du1du2du





Proof

– _{Define: Δ = ∫AL(H1) - ∫BL(H1)}

• _{The power of the test using A minus using B}

• _{Where A\C is the set of points in A but not in}

C

• _{And B\C contains points in B but not in C}

 

 





  f₁(u₁) f₁(u_n)du₁du_n  f₁(u₁) f₁(u_n)du₁du_n

A B

 

 



  f₁(u₁) f₁(u_n)du₁du_n  f₁(u₁) f₁(u_n)du₁du_n

C

Proof

– _{So, in A\C we have:}

– _{While in B\C we have:}

) ( ) ( ) ( )

( _{1 1 0 1 0}

1 u f un Kf u f un

f   

) ( ) ( ) ( )

( _{1 1 0 1 0}

1 u f un Kf u f un

f   

Proof

– _Thus

– _{Which implies that the power of the test}

 

 





  Kf₀(u₁) f₀(u_n)du₁du_n  Kf₀(u₁) f₀(u_n)du₁du_n

 

 



  Kf₀(u₁) f₀(u_n)du₁du_n  Kf₀(u₁) f₀(u_n)du₁du_n

C

A\ B\C

A B



 K

K  

0

Not Identically Distributed

•

In most cases, random variables are not

identically distributed, at least not in

H

1

– _{This affects the likelihood function, L}

– For example, H1 in the two-sample t-test is:

– Where μ1 and μ2 are different





       n i x m i

x _{i i}

e e L 1 2 ) ( 1 2 ) ( 2 2 2 2 2 2 1 1 2 1 2

1 _  _ 

  

Composite

– _{Further, the hypotheses being tested do}

not specify all parameters

– _{They are composite}

– _{This chapter only outlines aspects of}

Parameter Spaces

– _{The set of values the parameters of interest}

can take

– _{Null hypothesis: parameters in some region ω} – _{Alternate hypothesis: parameters in Ω}

– _{ω is usually a subspace of Ω} • _{Nested hypothesis case}

– Null hypothesis nested within alternate hypothesis – This book focuses on this case

• _{“if the alternate hypothesis can explain the data}

λ Ratio

•

Optimality theory for composite tests

suggests this as desirable test statistic:

• Lmax(ω): maximum likelihood when parameters

are confined to the region ω

• Lmax(Ω): maximum likelihood when parameters

are confined to the region Ω, defined by H1

• H0 is rejected when λ is sufficiently small (→

Type I error)

)

(

)

(

max max





L

L



Example: t-tests

•

The next slides calculate the

λ

-ratio

for the two sample

t

-test (with the

likelihood)

– _{t-tests later generalize to ANOVA and T}2

tests





       n i x m i

x _{i i}

e e L 1 2 ) ( 1 2 ) ( 2 2 2 2 2 2 1 1 2 1 2

1 _  _ 

  

Equal Variance Two-Sided

t-test

• _Setup

– Random variables X11,…,X1m in group 1 are

Normally and Independently Distributed (μ1,σ2)

– Random variables X21,…,X2n in group 2 are

NID (μ2,σ2)

– X1i and X2j are independent for all i and j

– Null hypothesis H0: μ1= μ2 (= μ, unspecified)

Equal Variance Two-Sided

t-test

•

Setup (continued)

– _σ2 is unknown and unspecified in H₀ and

H1

• _{Is assumed to be the same in both}

distributions

– _{Region ω is:}

– _Region {_{Ω is:} , ,0 }

2 2

1         

  

   

} 0

,

{ ₁  ₂  2  

   

Equal Variance Two-Sided

t-test

•

Derivation

– _H0: writing μ for the mean, when μ1= μ2,

the maximum over likelihood ω is at

– _{And the (common) variance σ}2 is n m X X X X X X

X m n

        

 11 12 1 21 22 2

ˆ    n m X X X

X _in _i

m i i    









1

Equal Variance Two-Sided

t-test

– _{Inserting both into the likelihood}

function, L

2 2

0 max

2

) ˆ 2 (

1 )

(

n m

e

L m n

 





Equal Variance Two-Sided

t-test

– _{Do the same thing for region Ω}

– _{Which produces this likelihood Function,}

L

m

X X

X

X 11 12 1m

1 1 ˆ      n X X X

X 21 22 2n

2 2 ˆ      n m X X X

X _in _i

m i i    









1

2 2 2 1 2 1 1 2 1 ) ( ) ( ˆ  2 2 1 max 2 ) ˆ 2 ( 1 ) ( n m e

L m n

Equal Variance Two-Sided

t-test

– _{The test statistic λ is then}

Equal Variance Two-Sided

t-test

– _{We can then use the algebraic identity}

– _{To show that}

– _{Where t is (from Ch. 3)}

2 2 1 2 1 2 2 2 1 1 1 2 1 2 2 1

1 ) ( ) ( ) ( ) ( )

( X X

n m mn X X X X X X X X n i i m i i n i i m i i                   2 2 1 2 1 n m n m t              n m S mn X X T  

Equal Variance Two-Sided

t-test

– _{t is the observed value of T} – _{S is defined in Ch. 3 as}

2

) (

) (

1

2 2 2

1

2 1 1

2

 

 













n m

X X

X X

S

n

i

i m

i

i

We can plot λ as a

Equal Variance Two-Sided

t-test

– _{So, by the monotonicity argument, we can}

use t2 or |t| instead of λ as test statistic

– _{Small values of λ correspond to large}

values of |t|

– Sufficiently large |t| lead to rejection of H0

– The H₀ distribution of t is known

• _{t-distribution with m+n-2 degrees of freedom}

– _{Significance points are widely available}

• Once α has been chosen, values of |t|

Equal Variance Two-Sided

t-test

.s

oc

r.u

cl

a.

ed

u/

A

pp

le

ts

.d

ir/

T-ta

bl

e.

ht

m

Equal Variance One-Sided

t-test

•

Similar to Two-Sided t-test case

– Different region Ω for H1:

• Means μ1 and μ2 are not simply different, but

one is larger than the other μ1 ≥ μ2

• _{If then maximum likelihood}

estimates are the same as for the two-sided case

} 0

,

{ ₁  ₂  2   

   

2

1 x

Equal Variance One-Sided

t-test

• _{If then the unconstrained maximum}

of the likelihood is outside of ω

• _{The unique maximum is at , implying}

that the maximum in ω occurs at a boundary point in Ω

• At this point estimates of μ1 and μ2 are equal

• At this point the likelihood ratio is 1 and H0 is

not rejected

• Result: H is rejected in favor of H (μ ≥ μ )

2

1 x

x 

) , (x1 x2

Example - Revised

•

This scenario fits with our original

example:

– _H1 is that the average height of KU

basketball players is bigger than for the general population

– _{One-sided test}

– _{We could assume that we don’t know the}

averages for H0 and H1

– _{We actually don’t know σ (I just guessed 2 in}

Example - Revised

• _{Updated example:}

– Observation in group 1 (KU): X1 = {83, 81, 71}

– Observation in group 2: X2 = {65, 72, 70}

– Pick significance point for t from a table: tα =

2.132

• _{t-distribution, m+n-2 = 4 degrees of freedom, α =}

0.05

– Calculate t with our observations

185 . 2 7673 .

12 9 . 27 6

2122 .

5

9 ) 69 3

. 78 (

 

 

Comments

• _{Problems that might arise in other cases}

– _{The λ-ratio might not reduce to a function of a}

well-known test statistic, such as t

– There might not be a unique H0 distribution of λ

– _{Fortunately, the t statistic is a pivotal quantity}

• Independent of the parameters not prescribed by H0

– e.g. μ, σ

– _{For many testing procedures this property does}

Unequal Variance Two-Sided

t-test

• _{Identical to Equal Variance Two-Sided t}

-test

– _{Except: variances in group 1 and group 2 are}

no longer assumed to be identical

• _{Group 1: NID(μ1, σ12)}

• _{Group 2: NID(μ2, σ22)}

• _{With σ12 and σ22 unknown and not assumed}

identical

• Region ω = {μ1 = μ2, 0 < σ12, σ22 < +∞}

Unequal Variance Two-Sided

t-test

– _{The likelihood function of (X11, X12, …,}

X1m, X21, X22, …, X2n) then becomes

– Under H0 (μ1 = μ2 = μ), this becomes:





      n i x m i

x _{i i}

e e 1 2 ) ( 2 1 2 ) ( 1 2 2 2 2 21 2 1 2 1 1 2 1 2 1        





      n i x m i

x _{i i}

Unequal Variance Two-Sided

t-test

– _{Maximum likelihood estimates ,}

and satisfy the simultaneous equations: 0 ˆ ) ˆ ( ˆ ) ˆ ( 2 2 2 2 1 1           _i i x x m x _i    2 1 2 1 ) ˆ ( ˆ   n x _i    2 2 2 2 ) ˆ ( ˆ  

Unequal Variance Two-Sided

t-test

–  cubic equation in

– _{Neither the λ ratio, nor any monotonic}

function has a known probability distribution when H0 is true!

– _{This does not lead to any useful testing}

statistic

• _{The t-statistic may be used as reasonably close} • However H0 distribution is still unknown, as it

depends on the unknown ratio σ12/σ22

• _{In practice, a heuristic is often used (see Ch. 3.5)}

The -2 log λ Approximation

•

Used when the

λ

-ratio procedure does

not lead to a test statistic whose

H

0

distribution is known

– _{Example: Unequal Variance Two-Sided t}

-test

•

Various approximations can be used

The -2 log λ Approximation

•

Best known approximation:

– If H0 is true, -2 log λ has an asymptotic

chi-square distribution,

• _{with degrees of freedom equal to the}

difference in parameters unspecified by H0

and H1, respectively.

• _{λ is the likelihood ratio}

• _{“asymptotic” = “as the sample size → ∞”}

The -2 log λ Approximation

– _{Restrictions:}

• Parameters must be real numbers that can take on values in some interval

• The maximum likelihood estimator is found at a turning point of the function

– _{i.e. a “real” maximum, not at a boundary point}

• H0 is nested in H1 (as in all previous slides)

– _{These restrictions are important in the}

proof

The -2 log λ Approximation

•

Instead:

– _{Our original basketball example, revised}

again:

• _{Let’s drop our last assumption, that the variance}

in the population at large is the same as in the group of KU basketball players.

• _{All we have left now are our observations and}

the hypothesis that μ1 > μ2

– Where μ1 is the average height of Basketball players

Example – Revised Again

– _{Using the Unequal Variance One-Sided t}

-Test

The Analysis of Variance

(ANOVA)

•

Probably the most frequently used

hypothesis testing procedure in

statistics

•

This section

– _{Derives of the Sum of Squares}

– _{Gives an outline of the ANOVA procedure} – _{Introduces one-way ANOVA as a}

generalization of the two-sample t-test

Sum of Squares

•

New variables (from Ch. 3)

– _{The two-sample t-test tests for equality} of the means of two groups.

– _{We could express the observations as:}

– Where the Eij are assumed to be

NID(0,σ2)

ij i

ij

E

Sum of Squares

– _{This can also be written as:}

• _{μ could be seen as overall mean} • αj as deviation from μ in group j

– _{This model is overparameterized}

• _{Uses more parameters than necessary} • _{Necessitates the requirement}

• _{(always assumed imposed)}

ij i

ij E

X 







 i _1,2

0

2

1   

 n

Sum of Squares

– _{We are deriving a test procedure similar}

to the two-sample two-sided t-test

– _{Using |t| as test statistic}

• _{Absolute value of the T statistic}

– _{This is equivalent to using t}2

• _{Because it’s a monotonic function of |t|}

– _{The square of the t statistic (from Ch. 3)}

mn X

X  )

Sum of Squares

– _{…can, after algebraic manipulations, be}

written as F

– _where

) 2 (  

 m n

W B F    m j j m X X 1 1 1    n j j n X X 1 2 2 n m X n X m X    1 2

2 2 2 1 2 2

1 ) ( ) ( )

(X X m X X n X X n

m mn

B     

 





      n j j m j

j X X X

X W 1 2 2 2 1 2 1

1 ) ( )

Sum of Squares

– _{B: between (among) group sum of squares} – _{W: within group sum of squares}

– _{B + W: total sum of squares} • _{Can be shown to be:}

– _{Total number of degrees of freedom: m + n}

– 1

• _{Between groups: 1}





 

 



n

i

i m

i

i X X X

X

1

2 2

1

2

1 ) ( )

Sum of Squares

– _{This gives us the F statistic}

– _{Our goal is to test the significance of the}

difference between the means of two groups

• _{B measures the difference}

– _{The difference must be measured relative to}

the variance within the groups

• _{W measures that}

– _{The larger F is, the more significant the}

difference

) 2 (  

 m n

The ANOVA Procedure

•

Subdivide observed total sum of

squares into several components

– _{In our case, B and W}

•

Pick appropriate significance point for

a chosen Type I error

α

from an

F

table

•

Compare the observed components to

F-Statistic

•

Significance points depend on

degrees of freedom in

B

and

W

Comments

•

The two-group case readily generalizes

to any number of groups.

•

ANOVAs can be classified in various

ways, e.g.

– _{fixed effects models} – _{mixed effects models} – _{random effects model}

Comments

•

Terminology

– _{Although ANOVA contains the word}

‘variance’

– _{What we actually test for is a equality in}

means between the groups

• The different mean assumptions affect the variance, though

•

ANOVAs are special cases of regression

One-Way ANOVA

• _{One-Way fixed-effect ANOVA} • _{Setup and derivation}

– _{Like two-sample t-test for g number of groups} – Observations (ni observations, i=1,2,…,g)

– _{Using overparameterized model for X}

– _E assumed NID(0,σ2), Σn α = 0, α fixed in

in i

i X X

X ₁, ₂,,

ij i

ij E

One-Way ANOVA

– Null Hypothesis H0 is: α1 = α2 = … = αg =

0

– _{Total sum of squares is}

– _{This is subdivided into B and W}

– _with



   g i n j ij i X X 1 1 2 ) (



   g i i

i X X

n B 1 2 ) (



    g i n j i ij i X X W 1 1 2 ) (    i n j i ij i n X X

1   

One-Way ANOVA

– _{Total degrees of freedom: N – 1}

• Subdivided into dfB = g – 1 and dfW = N - g

– _{This gives us our test statistic F}

– _{We can now look in the F-table for these}

degrees of freedom to pick significance points for B and W

– _{And calculate B and W from the observed data}

1 *

  

g g N W

Example

•

Revisiting the Basketball example

– _{Looking at it as a One-Way ANOVA}

analysis

• _{Observation in group 1 (KU):} X1 = {83, 81,

71}

• Observation in group 2: X2 = {65, 72, 70}

– _{Total Sum of Squares:}

– _{B (between groups sum of squares)}

3336 . 239 ) 70 66 . 73 ( ) 72 66 . 73 ( ) 65 66 . 73 ( ) 71 66 . 73 ( ) 81 66 . 73 ( ) 83 66 . 73

( _ 2_{ } 2_{ } 2_{ } 2_{ } 2_{ } 2 _

57 . 130 ) 33 . 76 69 ( 3 ) 33 . 76 33 . 78 ( 3 )

( 2 2

1

2 _{    }

   g i i i X X

Example

– _{W (within groups sum of squares)}

– _{Degrees of freedom}

• _{Total: N-1 = 5}

• dfB = g – 1 = 2 - 1 = 1

• dfW = N – g = 6 – 2 = 4

Example

– _{Table lookup for df 1 and 4 and α = 0.05:} – _{Critical value: F = 7.71}

– _{Calculate F from our data:}

– _{So… 4.806 < 7.71}

– With ANOVA we actually accept H0!

• _{Seems to be the large variance in group 1}

806 . 4 1

2 2 6 * 667 . 108

57 . 130 1

* 

  

  

g g N W

Same Example – with Excel

Excel

Two-Way ANOVA

•

Two-Way Fixed Effects ANOVA

•

Overview only (in the scope of this book)

•

More complicated setup; example:

– Expression levels of one gene in lung cancer patients

– _a different risk classes

• _{E.g.: ultrahigh, very high, intermediate, low}

Two-Way ANOVA

– Expression levels (our observations): Xijk

• _i is the risk class (i = 1, 2, …, a) • _j indicates the age group

• _k corresponds to the individual in each group (k = 1, …, n)

– Each group is a possible risk/age combination

• The number of individuals in each group is the same, n

• This is a “balanced” design

Two-Way ANOVA

– _{The Xijk can be arranged in a table:}

1 2 3 4

1 n n n n

2 n n n n

3 n n n n

4 n n n n

5 n n n n

i j

Risk category

A

ge

g

ro

Two-Way ANOVA

– The model adopted for each Xijk is

• Where Eijk are NID(μ, α2)

• The mean of Xijk is μ + αi + βi + δij

• αi is a fixed parameter, additive for risk class i

• βi is a fixed parameter, additive for age group i

• δij is a fixed risk/age interaction parameter

– _{Should be added is a possible group/group interaction} exists

ijk ij

j i

ijk E

X      

a

Two-Way ANOVA

– _{These constraints are imposed} • Σiαi = Σiβi = 0

• Σiδij = 0 for all j

• Σjδij = 0 for all i

– _{The total sum of squares is then subdivided}

into four groups:

Two-Way ANOVA

– _{Associated with each sum of squares}

• _{Corresponding degrees of freedom}

• _{Hence also a corresponding mean square}

– Sum of squares divided by degrees of freedom

– _{The mean squares are then compared using}

F ratios to test for significance of various effects

• _{First – test for a significant risk/age interaction} • _{F-ratio used is ratio of interaction mean square}

Two-Way ANOVA

• _{If such an interaction is used, it may not be}

reasonable to test for significant risk or age differences

• _{Example, μ in two risk classes, two age}

groups:

– No evidence of interaction

1 2

1 4 12 2 7 15

1 2

1 4 15 Risk

A

ge

A

Multi-Way ANOVA

•

One-way and two-way fixed effects

ANOVAs can be extended to

multi-way ANOVAs

•

Gets complicated

•

Example: three-way ANOVA model:

ijkm ijk

jk ik

ij k

j i

ijkm E

Further generalizations of

ANOVA

•

The 2

m

factorial design

– _{A particular form of the one-way ANOVA}

• _{Interactions between main effects}

– _{m “factors” taken at two “levels”}

• _{E.g. (1) Gender, (2) Tissue (lung, kidney), and}

(3) status (affected, not affected)

– ₂m possible combinations of levels/groups

Further generalizations of

ANOVA

– _{Example, m = 3, denoted by A, B, C}

• _{8 groups, {abc, ab, ac, bc, a, b, c, 1}}

• Write totals of n observations Tabc, Tab, …, T1 • _{The total between sum of squares can be}

subdivided into seven individual sums of squares

– _{Three main effects (A, B, C)}

– _{Three pair wise interactions (AB, AC, BC)} – _{One triple-wise interaction (ABC)}

– _{Example: Sum of squares for A, and for BC,}

respectively _n T T T T T T T

T_{abc ab ac a bc b c}

8 ) ( 2 1        T T T T T T T

T_{abc ab ac a bc b c} )

Further generalizations of

ANOVA

– If m ≥ 5 the number of groups becomes large

– Then the total number of observations, n2m is

large

– _{It is possible to reduce the number of}

observations by a process …

• _Confounding

– Interaction ABC probably very small and not interesting

Further generalizations of

ANOVA

•

Fractional Replication

– _{Related to confounding}

– _{Sometimes two groups cannot be}

distinguished from each other, then they are aliases

• E.g. A and BC

– _{This reduces the need to experiments and}

data

– _{Ch. 13 talks more about this in the context}

Random/Mixed Effect

Models

• _{So far: fixed effect models}

– E.g. Risk class, age group fixed in previous example

• Multiple experiments would use same categories • But: what if we took experimental data on several

random days?

• The days in itself have no meaning, but a

“between days” sum of squares must be extracted

– What if the days turn out to be important?

Random/Mixed Effect

Models

•

Mixed Effect Models

– _{If some categories are fixed and some}

are random

– _{Symbols used:}

• _{Greek letters for fixed effects}

• _{Uppercase Roman letters for random effects} • _{Example: two-way mixed effect model with}

– _{Risk class a and days d and n values collected}

each day, the appropriate model is written:

ikl il

l i

ikl D G E

Random/Mixed Effect

Models

• _{Random effect model have no fixed}

categories

• _{The details on the ANOVA analysis depend}

on which effects are random and which are fixed

• _{In a microarray context (more in Ch. 13)}

Multivariate Methods

ANOVA: the Repeated

Measures Case

Bootstrap Methods: the

Two-sample t-test

Sequential Analysis

•

Sequential Probability Ratio

– _{Sample size not known in advance} – _{Depends on outcomes of successive}

observations

– _{Some of this theory is in BLAST}

• _{Basic Local Alignment Search Tool}

– _{The book focuses on discreet random}

Sequential Analysis

– _Consider:

• _{Random variable Y with distribution P(y;ξ)} • _{Tests usually relate to the value of}

parameter ξ

• H0: ξ is ξ0 • H1: ξ is ξ1

• _{We can choose a value for the Type I error α} • _{And a value for the Type II error β}

Sequential Analysis

– _A and B are chosen to correspond to an α

and β

– Sampling continues until the ratio is less

than A (accept H0) or greater than B (reject

H0)

– Because these are discreet variables,

boundary overshoot usually occurs

• _{We don’t expect to exactly get values α and β}

– _{Desired values for α and β approximately}

achieved by using

 

 

1 A

 



1

Sequential Analysis

– _{It is also convenient to take logarithms,}

which gives us:

– _Using

– _{We can write}

Sequential Analysis

• _{Example: sequence matching}

– H0: p0 = 0.25 (probability of a match is 0.25)

– H1: p1 = 0.35 (probability of a match is 0.35)

– Type I error α and Type II error β chosen 0.01

– Yi: 1 if there is a match at position i,

otherwise 0

– Sampling continues while

– with



  i i Y

S ( ) log99 99

1

log _1,0

Sequential Analysis

– _{S can be seen as the support offered by}

Yi for H1

– _{The inequality can be re-written as}

– _{This is actually a random walk with step}

sizes 0.7016 for a match and -0.2984 for a mismatch



 

 

i i

Y 0.2984) 9.581 (

Sequential Analysis

•

Power Function for a Sequential Test

– _{Suppose the true value of the parameter}

of interest is ξ

– _{We wish to know the probability that H1}

is accepted, given ξ

– _{This probability is the power Ρ(ξ) of the}

Sequential Analysis

– Where θ* is the unique non-zero solution

to θ in

– _R is the range of values of Y

– Equivalently, θ* is the unique non-zero

solution to θ in



        R

y P y

y P y P 1 ) ; ( ) ; ( ) ; ( 0 1    



  R y y S e y

Sequential Analysis

– _{This is very similar to Ch. 7 – Random}

Walks

– _{The parameter θ}* is the same as in Ch. 7

– _{And it will be the same in Ch 10 – BLAST}

Sequential Analysis

•

Mean Sample Size

– _{The (random) number of observations}

until one or the other hypothesis is accepted

– _{Find approximation by ignoring}

boundary overshoot

– _{Essentially identical method used to find}

Sequential Analysis

– _{Two expressions are calculated for}

ΣiS1,0(Yi)

• _{One involves the mean sample size}

• _{By equating both expressions, solve for}

mean sample size _

                 



 _  

 ( )log 1

Sequential Analysis

– _{So, the mean sample size is:}

– _{Both numerator and denominator}

depend on Ρ(ξ), and so also on θ*

– _{A generalization applies if Q(y) of Y has}

different distribution than H0 and H1 –

relevant to BLAST



      R

y P y

y P

y

P ₍( _;; )₎

Sequential Analysis

•

Example

– _{Same sequence matching example as}

before

• H0: p0 = 0.25 (probability of a match is 0.25)

• H1: p1 = 0.35 (probability of a match is 0.35)

• _{Type I error α and Type II error β chosen 0.01} – _{Mean sample size equation is:}

– Mean sample size is when H0 is true: 194

– Mean sample size is when H is true: 182

15 13 7

5 _{(1 )log}

log

595 . 4 ) ( 190 . 9

p p

p

 

Sequential Analysis

• _{Boundary Overshoot}

– So far we assumed no boundary overshoot

– In practice, there will almost always be, though

• Exact Type I and Type II errors different from α and β

– _{Random walk theory can be used to assess how}

significant the effects of boundary overshoot are

– It can be shown that the sum of Type I and Type II errors is always less than α + β (also individually)