Copyright is owned by the Author of the thesis. Permission is given for
a copy to be downloaded by an individual for the purpose of research and
private study only. The thesis may not be reproduced elsewhere without
the permission of the Author.
GEN ERAL ALGORITHMS BA SED ON LEAST SQUAR ES C ALCULATIONS FOR MAXIMUM L IKELI HOOD ESTIMATION
IN MULTI PARAMETER MODELS
A thes is pres ented i n partial fulf i lment of the requ irements f or the degree of Ph. D . in St at i s ti c s at Massey Univer s i ty .
by
W. Dou glas St i r l i ng 1 98 5
ABSTR AC T
Th i s thes i s develops algori thms for max imum l i k eli hood est imat ion that can be i mpl emented us i ng a sequence of we ighted least squares compu tat i on s and examines the ir proper t i es .
St andar d l east squares algori thms are f irst descr i bed and the ir execut i on t imes , stor age requir ement s and ac curac ies ar e compared. The G i vens QR algor i thm uses l ess s torage than o ther algor i thms of compar abl e accuracy and i n good impl ementations is v ir tually as f ast as them if there are several explanatory var i abl es . A vers ion that can be used f or constr ained least squar es is desc r ibed ; it is used for l east squares calcul ati ons i n the remai nder of the thes i s .
In many max imum l i k el i hood probl ems , the l i k el i hood can be wr it ten as a sum of f unct i o ns called l og-l ikel ihood components and these of ten depend on the unknown par ameters only through one or two quant i t ies cal l e d systemat i c parts . For these mode l s , a class of algor i thms call ed NRL algor ithms appro aches the max imum l i k el i hood est imate with a s equenc e of least squares calculations . For many common models , the Newton-Raphson algor ithm and Fi sher 's scor i ng techn ique ar e par t i c ular NRL algor i thms . Implementation of NRL algor i thms i s descr ibed i n det a i l an d the r e l at i ve mer its o f the var ious NRL algor ithms ar e d i scuss e d . If the NR algor i thm i s i n the clas s , i t converges bes t near the max imum l i k el i hood est i mate , but other NRL algor ithms may perform better i n the f ir s t few i terat ions . Sev eral exampl es are analy sed to i l lu st r at e the var ious poss ible methods .
When the max imum l ikelihood est imates of some parame ters c an be wr i tten as expl i c i t functions of the rest , the c onver genc e of the NR and NRL algori thms can often be i mpr oved by adjust i ng these paramet ers between iter at i ons . The relat i onsh ip of th is techn i que to elim i nat ion of thes e parame ters from the l ikel ihood is investigat e d . In several typ es of model , including non l inear l e ast squares , adj ustment can be
performed wi thout s l ow i ng the NRL i terat i o ns . A rel ate d more general method i s also descr i b ed for improv ing NRL i terat ions wh en some parameter s are l i near and some are non l i near in the systematic par t s .
Another general algor ithm call e d the EM algorithm i s descr i b e d . I t c a n be app l i e d t o several types of model for wh i ch the NR L algor i thm cannot be use d . I n some mo dels , i t can also be implemented using a s equenc e of l east s quares calculat i ons , but f or appl i cations wher e b oth
EM and NRL algor it hms can be use d , the l at t er u sually converge f aster . Final ly , i n two appendices , Fortran subr out ines that can be used to implement the algor ithms in the the s i s ar e descr ibed and l i s ted .
'
ACKNOWLEDGEMENTS
I would like to thank Dr R J Brook for supervising this thesis.
I am also grateful to the Department of Mathematics and
Statistics for allowing me time to complete this thesis and to Massey University for making available computer facilities for typing it.
I finally wish to acknowledge the support my wife Sue has given me and to thank her and my children David and George for
tolerating the evening and weekends I have spent on the thesis in the last few y ears.
TABLE OF CONT ENTS
Chapter 1 . I ntr oduct ion
1 . 1 Purpose and Scope of the Thes i s 1 . 2 Structur e o f the Thesis
Chap ter 2 . Least Squar es Al gor i thms
3
2 . 1 The Normal Li near Model 5
2 . 2 Algori thms Based on the Normal Equat ions 7
2 . 3 QR Al g or ithms 9
2 .� A N umer i cal Compar i son of the Accuracy of the Al gori thms 1 7
2 . 5 L i n ear Cons traints 25
2 . 6 D ependenc i es Between Explanatory Var i ables 27 2 . 7 A Set o f For tran Subroutines f or Le ast Squares 29
Chapter 3 . Mode l s wi th a Si ngle Systemati c Part
3 . 1 General iz ed L inear Models 3 1
3 . 2 Some Models Outs i de the Cl ass o f Gener al i z ed Li near Models 35 3 . 3 General Optimizat ion Algori thms for Maximum Likel ihood ��
3 .� Impl ementation of Ummodif i ed NRL Al g orithms 52 3 . 5 Asymptot i c Performance of NRL Algori thms 60 3 . 6 T.�e FS algor ithm and i ts Relat i onsh ip to NRL 7�
3 . 1 Implement ation of I terat i vely Rewe i ghted Least Squares
3 . 8 C o nstra ints
3 . 9 Vari a nc es and Tests
Al gor ithms i n the I nit i al Iter at i ons 77 8�
86
Chapter 4 . Models wi th Two or More Systemati c P arts
14 . 1 F i t t i ng Models wi th Iterat i vely Rewe ighted Least Squ ar es 14 . 2 Var i ances and Tests
14 . 3 Examples Normal Models wi th Var i anc e a Funct ion of Me an 14 . 4 Examples Normal Models with Var i ance a Funct ion of
Expl anatory Vari abl es 14 . 5 Exampl es Negative Binomial Models
14 . 6 Examples Robust Es timat i on in Li near Models
Chapter 5 . Elimination and Adj ustment of Parameters .
5 . 1 General Des cr ipt ion of Eliminat ion and Adj ustment
5 . 2 El imination and Adj ustment Appl i ed to the NR , NRL a nd FS
90 99 1 02
1 1 3 1 22 1 28
1 38
Algor i thms 1 4 0 5 . 3 Nonl inear Least Squares
5 . 4 A Nonli near Least Squares Exampl e
5 . 5 Other Appl i cat ions of Elim inat ion and Adj ustment 5 . 6 Systemat i c Parts w ith Li near and Nonlinear Parameters 5 . 7 Conclud i ng Remarks about Eliminat ion and Adj ustment
Chapter 6 . The EM Al gor i thm
6 . 1 Gener al Descr iption of the EM Al g or ithm 6 . 2 Examples Miss i ng D at a
6 . 3 Examples Mixtures a n d Totals 6 . 4 Exampl es Var i ance Components
6 . 5 Exampl es Hyperpar ameter Estimat i on
6 . 6 Examples Robust Es timat ion in L i near Models 6 . 7 Conclud i ng Remar ks About the EM Al gorithm
Chapter 7 . Conclusion
1 4 5 1 50 1 5 4 1 59 1 62
1 6 4 1 70 1 7 4 1 76 1 7 9 1 8 1 1 87
1 88
Append ices
A B
Fortran Subroutine s for Model Fit t i ng Fortran Subrout i n es for Model Fitting
R e ferences
Parameters Code
1 9 2 2 02
21 5