[ Team LiB ]
• Table of Cont ent s
Un ix ™ Syst e m s Pr ogr a m m in g: Com m u n ica t ion , Con cu r r e n cy, a n d Th r e a ds
By Kay A. Robbins, St even Robbins
Publisher: Prent ice Hall PTR
Pub Dat e: June 17, 2003
I SBN: 0- 13- 042411- 0
Pages: 912
This com plet ely updat ed classic ( originally t it led Pract ical UNI X Program m ing) dem onst rat es how t o design com plex soft w are t o get t he m ost from t he UNI X operat ing syst em . UNI X
Syst em s Program m ing provides a clear and easy- t o- underst and int roduct ion t ot he essent ials of UNI X program m ing. St art ing w it h short code snippet st hat illust rat e how t o use syst em calls, Robbins and Robbins m ovequickly t o hands- on proj ect s t hat help readers expand t heir skill levels.
This pract ical guide t horoughly explores com m unicat ion, concurrency,and m ult it hreading. Known for it s com prehensive and lucid explanat ions of com plicat ed t opics such as signals and concurrency, t he book feat ures pract ical exam ples, exercises, reusable code, and sim plified libraries for use in net w ork com m unicat ion applicat ions.
A self- cont ained reference t hat relies on t he lat est UNI X st andards,UNI X Syst em s Program m ing provides t horough coverage of files, signals,sem aphores, POSI X t hreads, and client - server com m unicat ion. Thisedit ion feat ures all- new chapt ers on t he Web, UDP, and server
perform ance. The sam ple m at erial has been t est ed ext ensively in t heclassroom .
[ Team LiB ]
• Table of Cont ent s
Un ix ™ Syst e m s Pr ogr a m m in g: Com m u n ica t ion , Con cu r r e n cy, a n d Th r e a ds
By Kay A. Robbins, St even Robbins
Publisher: Prent ice Hall PTR
Pub Dat e: June 17, 2003
I SBN: 0- 13- 042411- 0
Pages: 912
Copyright
About t he Web Sit e Preface
Acknowledgm ent s Part I : Fundam ent als
Chapt er 1. Technology's I m pact on Program s Sect ion 1.1. Term inology of Change Sect ion 1.2. Tim e and Speed
Sect ion 1.3. Mult iprogram m ing and Tim e Sharing Sect ion 1.4. Concurrency at t he Applicat ions Level Sect ion 1.5. Securit y and Fault Tolerance
Sect ion 1.6. Buffer Overflow s for Breaking and Ent ering Sect ion 1.7. UNI X St andards
Sect ion 1.8. Addit ional Reading
Chapt er 2. Program s, Processes and Threads Sect ion 2.1. How a Program Becom es a Process Sect ion 2.2. Threads and Thread of Execut ion Sect ion 2.3. Layout of a Program I m age Sect ion 2.4. Library Funct ion Calls
Appendix B. Rest art Library Appendix C. UI CI I m plem ent at ion
Sect ion C.1. Connect ion- Orient ed UI CI TCP I m plem ent at ion Sect ion C.2. Nam e Resolut ion I m plem ent at ions
Sect ion C.3. Connect ionless UI CI UDP I m plem ent at ion Appendix D. Logging Funct ions
Sect ion D.1. Local At om ic Logging Sect ion D.2. Rem ot e Logging Appendix E. POSI X Ext ensions Bibliography
[ Team LiB ]
Copyright
Robbins, St even,
1947-UNI X syst em s program m ing: com m unicat ion, concurrence, and t hreads / St even Robbins, Kay Robbins
p. cm .
Previously published under t he t it le: Pract ical UNI X Program m ing / Kay Robbins. Upper Saddle River, NJ: Prent ice Hall, c1996.
I SBN 0- 13- 0424110
1. UNI X ( Com put er file) 2. Operat ing syst em s ( Com put ers) I . Robiins, Kay A. I I . Robbins, Kay A. Pract ical UNI X program m ing. I I I . Tit le
Product ion Supervisor: Wil Mara
Acquisit ions Edit or: Greg Doench
Cover Design: Nina Scuderi and Talar Booruj y
Cover Design Direct or: Jerry Vot t a
Edit orial Assist ant : Brandt Kenna
Market ing Manager: Dan DePasquale
Manufact uring Manager: Alexis Heydt - Long
© 2003 Pearson Educat ion, I nc.
Publishing as Prent ice Hall Professional Technical Reference
Upper Saddle River, New Jersey 07458
Prent ice Hall books are w idely used by corporat ions and governm ent agencies for t raining, m arket ing, and resale.
Pr e n t ice H a ll PTR offe r s e x ce lle n t discou n t s on t h is book w h e n or de r e d in qu a n t it y for bu lk pu r ch a se s or spe cia l sa le s. For m or e in for m a t ion , ple a se con t a ct : U.S. Cor por a t e a n d Gove r n m e n t Sa le s, 1 - 8 0 0 - 3 8 2 - 3 4 1 9 , cor psa le s@pe a r son t e ch gr ou p.com. For sa le s ou t side t h e U.S., ple a se con t a ct : I n t e r n a t ion a l Sa le s, 1 - 3 1 7 - 5 8 1 - 3 7 9 3 ,
Com pany and product nam es m ent ioned herein are t he t radem arks or regist ered t radem arks of t heir respect ive owners.
Allright s reserved. No part of t his book m ay be reproduced, in any form or by any m eans, w it hout perm ission in w rit ing from t he publisher.
Print ed in t he Unit ed St at es of Am erica
First Print ing
Pearson Educat ion LTD.
Pearson Educat ion Aust ralia PTY, Lim it ed
Pearson Educat ion Singapore, Pt e. Lt d.
Pearson Educat ion Nort h Asia Lt d.
Pearson Educat ion Canada, Lt d
Pearson Educat ión de Mexico, S.A. de C.V.
Pearson Educat ion —Japan
Pearson Educat ion Malaysia, Pt e. Lt d.
Dedication
To Nicole and Thom as
[ Team LiB ]
About the Web Site
The ht t p: / / usp.cs.ut sa.edu/ usp w eb sit e offers addit ional resources for t he book, including all of
t he program s in dow nloadable form . These program s are freely available wit h no resrict ions ot her t han acknow ledgem ent of t heir source. The sit e also has links t o sim ulat ors, t est ing t ools, course m at erial prepared by t he aut hors, and an errat a list .
[ Team LiB ]
Preface
UNI X Syst em s Program m ing: Com m unicat ion, Concurrency and Threads is t he second edit ion of
Pract ical UNI X Program m ing: A Guide t o Com m unicat ion, Concurrency and Mult it hreading, w hich was published by Prent ice Hall in 1995. We changed t he t it le t o bet t er convey what t he book is about . Several t hings have changed, besides t he t it le, since t he last edit ion.
The I nt ernet has becom e a dom inant aspect of com put ing and of societ y. Our privat e
inform at ion is online; our soft w are is under const ant at t ack. Never has it been so im port ant t o w rit e correct code. I n t he new edit ion of t he book, we t ried t o produce code t hat correct ly handles errors and special sit uat ions. We realized t hat saying handle all errors but giving code exam ples wit h t he error handling om it t ed was not effect ive. Unfort unat ely, error handling m akes code m ore com plex. We have w orked hard t o m ake t he code clear.
Anot her im port ant developm ent since t he last edit ion is t he adopt ion of a Single UNI X Specificat ion, which w e refer t o as POSI X in t he book. We no longer have t o decide which
vendor's version of a library funct ion t o use—t here is an official version. We have done our best t o com ply w it h t he st andard.
The exercises and proj ect s m ake t his book unique. I n fact , t he book began as a proj ect w orkbook developed as part of a Nat ional Science Foundat ion Grant . I t becam e clear t o us, aft er prelim inary developm ent , t hat t he m at erial needed t o do t he proj ect s was scat t ered in m any places—oft en found in reference books t hat provide m any det ails but lit t le concept ual overview. The book has since evolved int o a self- cont ained reference t hat relies on t he lat est UNI X st andards.
The book is organized int o four part s, each of w hich cont ains t opic chapt ers and proj ect chapt ers. A t opic chapt er covers t he specified m at erial in a work- along fashion. The t opic chapt ers have m any exam ples and short exercises of t he form " t ry t his" or " what happens if." The t opic chapt ers close w it h one or m ore exercise sect ions. The book provides program m ing exercises for m any fundam ent al concept s in process m anagem ent , concurrency and
com m unicat ion. These program m ing exercises sat isfy t he sam e need as do laborat ory
experim ent s in a t radit ional science course. You m ust use t he concept s in pract ice t o have real underst anding. Exercises are specified for st ep- by- st ep developm ent , and m any can be
im plem ent ed in under 100 lines of code.
I Fu n da m e n t a ls
UNI X I / O 4
Files and Direct ories 5
UNI X Special Files 6
The Token Ring 7
I I Asyn ch r on ou s Eve n t s
Signals 8
Tim es and Tim ers 9
Virt ual Tim ers 10
Cracking Shells 11
I I I Con cu r r e n cy
POSI X Threads 12
Thread Synchronizat ion 13
Sem aphores 14
POSI X I PC 15
Producer Consum er 16
Virt ual Machine 17
I V Com m u n ica t ion Connect ion- Orient ed Com m un. 18
WWW Redirect ion 19
Connect ionless Com m un. 20
I nt ernet Radio 21
Server Perform ance 22
Proj ect chapt ers int egrat e m at erial from several t opic chapt ers by developing a m ore ext ensive applicat ion. The proj ect s w ork on t w o levels. I n addit ion t o illust rat ing t he program m ing ideas, t he proj ect s lead t o underst anding of an advanced t opic relat ed t o t he applicat ion. These proj ect s are designed in st ages, and m ost full im plem ent at ions are a few hundred lines long. Since you don't have t o w rit e a large am ount of code, you can concent rat e on underst anding concept s rat her t han debugging. To sim plify t he program m ing, we m ake libraries available for net w ork com m unicat ion and logging of out put . For a professional program m er, t he exercises at t he end of t he t opic chapt ers provide a m inim al hands- on int roduct ion t o t he m at erial.
Typically, an inst ruct or using t his book in a course w ould select several exercises plus one of t he m aj or proj ect s for im plem ent at ion during a sem est er course. Each proj ect has a num ber of variat ions, so t he proj ect s can be used in m ult iple sem est ers.
Part I . The except ion is t he discussion at t he end of lat er chapt ers about int eract ions ( e.g., how t hreads int eract w it h signals) .
We have assum ed t hat you are a good C program m er t hough not necessarily a UNI X C
program m er. You should be fam iliar wit h C program m ing and basic dat a st ruct ures. Appendix A
covers t he bare essent ials of program developm ent if you are new t o UNI X.
This book includes synopsis boxes for t he st andard funct ions. The relevant st andards t hat specify t he funct ion appear in t he lower- right corner of t he synopsis box.
A book like t his is never done, but w e had t o st op som ewhere. We welcom e your com m ent s and suggest ions. You can send em ail t o us at aut hors@usp.cs.ut sa.edu. We have done our best t o produce an error- free book. How ever, should you be t he first t o report an error, we will grat efully acknow ledge you on t he book w eb sit e. I nform at ion on t he book is available on t he WWW sit e ht t p: / / usp.cs.ut sa.edu/ usp. All of t he code included in t he book can be downloaded from t he WWW sit e.
[ Team LiB ]
Acknowledgments
We are very grat eful t o Mike Speciner and Bob Lynch for reading t he ent ire m anuscript and m aking m any useful suggest ions. We are especially grat eful t o Mary Lou Nohr for her careful and int elligent copy- edit ing. We w ould also like t o express our appreciat ion t o Neal Wagner and Radia Perlm an for t heir encouragem ent and suggest ions.
We have t aught undergraduat e and graduat e operat ing syst em s courses from 1988 t o dat e ( 2003) , and m uch of t he m at erial in t he book has been developed as part of t eaching t hese courses. The st udent s in t hese courses have suffered t hrough draft s in various st ages of developm ent and have field- t est ed em erging proj ect s. Their program bugs, com m ent s, com plaint s, and suggest ions m ade t he book a lot bet t er and gave us insight int o how t hese t opics int errelat e. Som e of t he st udent s w ho found errors in an early draft include Joseph Bell, Carlos Cadenas, I gor Grinshpan, Jason Jendrusch and Jam es Manion. We would like t o
acknow ledge t he Nat ional Science Foundat ion for providing support t hrough t he NSFI LI grant USE- 0950497 t o build a laborat ory so t hat w e had t he opport unit y t o develop t he original
curriculum upon which t his book is based. NSF ( DUE- 975093, DUE- 9752165 and DUE- 0088769) also support ed developm ent of t ools for explorat ion and analysis of OS concept s.
We w ould like t o t hank Greg Doench, our edit or at Prent ice Hall, for guiding us t hrough t he process and William Mara our product ion edit or, for bringing t he book t o publicat ion. We
t ypeset t he book using LATEX2 , and we would like t o express our appreciat ion t o it s producers for m aking t his soft w are freely available.
Special t hanks go t o our fam ilies for t heir unfailing love and support and especially t o our children, Nicole and Thom as, w ho have dealt wit h t his arduous proj ect w it h ent husiasm and underst anding.
[ Team LiB ]
Part I: Fundamentals
Chapt er 1. Technology's I m pact on Program s
Chapt er 2. Program s, Processes and Threads
Chapt er 3. Processes in UNI X
Chapt er 4. UNI X I / O
Chapt er 5. Files and Direct ories
Chapt er 6. UNI X Special Files
Chapt er 7. Proj ect : The Token Ring
[ Team LiB ]
Chapter 1. Technology's Impact on Programs
This chapt er int roduces t he ideas of com m unicat ion, concurrency and asynchronous operat ion at t he operat ing syst em level and at t he applicat ion level. Handling such program const ruct s incorrect ly can lead t o failures w it h no apparent cause, even for input t hat previously seem ed t o w ork perfect ly. Besides t heir added com plexit y, m any of t oday's applicat ions run for weeks or m ont hs, so t hey m ust properly release resources t o avoid wast e ( so- called leaks of
resources) . Applicat ions m ust also cope wit h out rageously m alicious user input , and t hey m ust recover from errors and cont inue running. The Port able Operat ing Syst em I nt erface ( POSI X) st andard is an im port ant st ep t ow ard producing reliable applicat ions. Program m ers who writ e for POSI X- com pliant syst em s no longer need t o cont end w it h sm all but crit ical variat ions in t he behavior of library funct ions across plat form s. Most popular UNI X versions ( including Linux and Mac OS X) are rapidly m oving t o support t he base POSI X st andard and various levels of it s ext ensions.
Objectives
● Learn how an operat ing syst em m anages resources
● Experim ent w it h buffer overflows
● Explore concurrency and asynchronous behavior
● Use basic operat ing syst em s t erm inology
● Underst and t he serious im plicat ions of incorrect code
[ Team LiB ]
1.1 Terminology of Change
Com put er pow er has increased exponent ially for nearly fift y years [73] in m any areas including processor, m em ory and m ass- st orage capacit y, circuit densit y, hardware reliabilit y and I / O bandw idt h. The grow t h has cont inued in t he past decade, along wit h sophist icat ed inst ruct ion pipelines on single CPUs, placem ent of m ult iple CPUs on t he deskt op and an explosion in net w ork connect ivit y.
The dram at ic increases in com m unicat ion and com put ing pow er have t riggered fundam ent al changes in com m ercial soft w are.
● Large dat abase and ot her business applicat ions, w hich form erly execut ed on a
m ainfram e connect ed t o t erm inals, are now dist ribut ed over sm aller, less expensive m achines.
● Term inals have given w ay t o deskt op workst at ions wit h graphical user int erfaces and
m ult im edia capabilit ies.
● At t he ot her end of t he spect rum , st andalone personal com put er applicat ions have
evolved t o use net w ork com m unicat ion. For exam ple, a spreadsheet applicat ion is no longer an isolat ed program support ing a single user because an updat e of t he
spreadsheet m ay cause an aut om at ic updat e of ot her linked applicat ions. These could graph t he dat a or perform sales proj ect ions.
● Applicat ions such as cooperat ive edit ing, conferencing and com m on whit eboards
facilit at e group w ork and int eract ions.
● Com put ing applicat ions are evolving t hrough sophist icat ed dat a sharing, realt im e
int eract ion, int elligent graphical user int erfaces and com plex dat a st ream s t hat include audio and video as w ell as t ext .
These developm ent s in t echnology rely on com m unicat ion, concurrency and asynchronous operat ion w it hin soft w are applicat ions.
Asynchronous operat ion occurs because m any com put er syst em event s happen at
unpredict able t im es and in an unpredict able order. For exam ple, a program m er cannot predict t he exact t im e at which a print er at t ached t o a syst em needs dat a or ot her at t ent ion. Sim ilarly, a program cannot ant icipat e t he exact t im e t hat t he user presses a key for input or int errupt s t he program . As a result , a program m ust w ork correct ly for all possible t im ings in order t o be correct . Unfort unat ely, t im ing errors are oft en hard t o repeat and m ay only occur once every m illion execut ions of a program .
Concurrency is t he sharing of resources in t he sam e t im e fram e. When t wo program s execut e on t he sam e syst em so t hat t heir execut ion is int erleaved in t im e, t hey share processor resources. Program s can also share dat a, code and devices. The concurrent ent it ies can be t hreads of execut ion w it hin a single program or ot her abst ract obj ect s. Concurrency can occur in a syst em wit h a single CPU, m ult iple CPUs sharing t he sam e m em ory, or independent syst em s running over a net w ork. A m aj or j ob of a m odern operat ing syst em is t o m anage t he concurrent operat ions of a com put er syst em and it s running applicat ions. However, concurrency cont rol has also becom e an int egral part of applicat ions. Concurrent and asynchronous
operat ions share t he sam e problem s—t hey cause bugs t hat are oft en hard t o reproduce and creat e unexpect ed side effect s.
Wide Web and t he dom inance of net w ork applicat ions, m any program s m ust deal w it h I / O over t he net w ork as w ell as from local devices such as disks. Net work com m unicat ion int roduces a m yriad of new problem s result ing from unpredict able t im ings and t he possibilit y of undet ect ed rem ot e failures.
The rem ainder of t his chapt er describes sim plified exam ples of asynchronous operat ion, concurrency and com m unicat ion. The buffer overflow problem illust rat es how careless program m ing and lack of error checking can cause serious problem s and securit y breaches. This chapt er also provides a brief overview of how operat ing syst em s w ork and sum m arizes t he operat ing syst em st andards t hat are used in t he book.
[ Team LiB ]
1.2 Time and Speed
Operat ing syst em s m anage syst em resources: processors, m em ory and I / O devices including keyboards, m onit ors, print ers, m ouse devices, disks, t apes, CD- ROMs and net w ork int erfaces. The convolut ed way operat ing syst em s appear t o w ork derives from t he charact erist ics of peripheral devices, part icularly t heir speed relat ive t o t he CPU or processor. Table 1.1 list s t ypical processor, m em ory and peripheral t im es in nanoseconds. The t hird colum n shows t hese speeds slowed down by a fact or of 2 billion t o give t he t im e scaled in hum an t erm s. The scaled t im e of one operat ion per second is roughly t he rat e of t he old m echanical calculat ors from fift y years ago.
Ta ble 1 .1 . Typica l t im e s for com pon e n t s of a com pu t e r syst e m . On e
n a n ose con d ( n s) is 1 0
– 9se con ds, on e m icr ose con d (
µ
s) is 1 0
– 6se con ds, a n d on e m illise con d ( m s) is 1 0
– 3se con ds.
it e m t im e
sca le d t im e in h u m a n t e r m s ( 2 billion t im e s slow e r )
processor cycle 0.5 ns ( 2 GHz) 1 second
cache access 1 ns ( 1 GHz) 2 seconds
m em ory access 15 ns 30 seconds
cont ext swit ch 5,000 ns ( 5 µs) 167 m inut es
disk access 7,000,000 ns ( 7 m s) 162 days
quant um 100,000,000 ns ( 100 m s) 6.3 years
Disk drives have im proved, but t heir rot at ing m echanical nat ure lim it s t heir perform ance. Disk access t im es have not decreased exponent ially. The disparit y bet ween processor and disk access t im es cont inues t o grow ; as of 2003 t he rat io is roughly 1 t o 14,000,000 for a 2- GHz processor. The cit ed speeds are a m oving t arget , but t he t rend is t hat processor speeds are increasing exponent ially, causing an increasing perform ance gap bet ween processors and peripherals.
The cont ext - sw it ch t im e is t he t im e it t akes t o sw it ch from execut ing one process t o anot her. The quant um is roughly t he am ount of CPU t im e allocat ed t o a process before it has t o let anot her process run. I n a sense, a user at a keyboard is a peripheral device. A fast t ypist can t ype a keyst roke every 100 m illiseconds. This t im e is t he sam e order of m agnit ude as t he process scheduling quant um , and it is no coincidence t hat t hese num bers are com parable for int eract ive t im esharing syst em s.
A m odem is a device t hat perm it s a com put er t o com m unicat e wit h anot her com put er over a phone line. A t ypical m odem is rat ed at 57,600 bps, w here bps m eans " bit s per second."
Assum ing it t akes 8 bit s t o t ransm it a byt e, est im at e t he t im e needed for a 57,600 bps m odem t o fill a com put er screen w it h 25 lines of 80 charact ers. Now consider a graphics display t hat consist s of an array of 1024 by 768 pixels. Each pixel has a color value t hat can be one of 256 possible colors. Assum e such a pixel value can be t ransm it t ed by m odem in 8 bit s. What
com pression rat io is necessary for a 768- kbps DSL line t o fill a screen wit h graphics as fast as a 57,600- bps m odem can fill a screen w it h t ext ?
An sw e r :
Table 1.2 com pares t he t im es. The t ext display has 80 x 25 = 2000 charact ers so 16,000 bit s
m ust be t ransm it t ed. The graphics display has 1024 x 768 = 786,432 pixels so 6,291,456 bit s m ust be t ransm it t ed. The est im at es do not account for com pression or for com m unicat ion prot ocol overhead. A com pression rat io of about 29 is necessary!
Ta ble 1 .2 . Com pa r ison of t im e e st im a t e s for fillin g a scr e e n .
m ode m t ype bit s pe r se con d
t im e n e e de d t o displa y
t e x t gr a ph ics
1979 t elephone m odem 300 1 m inut e 6 hours
1983 t elephone m odem 2,400 6 seconds 45 m inut es
current t elephone m odem 57,600 0.28 seconds 109 seconds
current DSL m odem 768,000 0.02 seconds 8 seconds
[ Team LiB ]
1.3 Multiprogramming and Time Sharing
Observe from Table 1.1 t hat processes perform ing disk I / O do not use t he CPU very efficient ly: 0.5 nanoseconds versus 7 m illiseconds, or in hum an t erm s, 1 second versus 162 days. Because of t he t im e disparit y, m ost m odern operat ing syst em s do m ult iprogram m ing. Mult iprogram m ing
m eans t hat m ore t han one process can be ready t o execut e. The operat ing syst em chooses one of t hese ready processes for execut ion. When t hat process needs t o wait for a resource ( say, a keyst roke or a disk access) , t he operat ing syst em saves all t he inform at ion needed t o resum e
Most devices are handled by t he operat ing syst em rat her t han by applicat ions. When an applicat ion execut es a disk read, t he call issues a request for t he operat ing syst em t o act ually perform t he operat ion. The operat ing syst em now has cont rol. I t can issue com m ands t o t he disk cont roller t o begin ret rieving t he disk blocks request ed by t he applicat ion. How ever, since t he disk ret rieval does not com plet e for a long t im e ( 162 days in relat ive t im e) , t he operat ing syst em put s t he applicat ion's process on a queue of processes t hat are wait ing for I / O t o com plet e and st art s anot her process t hat is ready t o run. Event ually, t he disk cont roller
int errupt s t he CPU inst ruct ion cycle w hen t he result s are available. At t hat t im e, t he operat ing syst em regains cont rol and can choose whet her t o cont inue wit h t he current ly running process or t o allow t he original process t o run.
UNI X does t im esharing as w ell as m ult iprogram m ing. Tim esharing creat es t he illusion t hat several processes execut e sim ult aneously, even t hough t here m ay be only one physical CPU. On a single processor syst em , only one inst ruct ion from one process can be execut ing at any part icular t im e. Since t he hum an t im e scale is billions of t im es slower t han t hat of m odern com put ers, t he operat ing syst em can rapidly swit ch bet ween processes t o give t he appearance of several processes execut ing at t he sam e t im e.
Consider t he follow ing analogy. Suppose a grocery st ore has several checkout count ers ( t he processes) but only one checker ( t he CPU) . The checker checks one it em from a cust om er ( t he inst ruct ion) and t hen does t he next it em for t hat sam e cust om er. Checking cont inues unt il a price check ( a resource request ) is needed. I nst ead of wait ing for t he price check and doing not hing, t he checker m oves t o anot her checkout count er and checks it em s from anot her
cust om er. The checker ( CPU) is alw ays busy as long as t here are cust om ers ( processes) ready t o check out . This is m ult iprogram m ing. The checker is efficient , but cust om ers probably would not w ant t o shop at such a st ore because of t he long wait w hen som eone has a large order wit h no price checks ( a CPU- bound process) .
for a m axim um of 10 seconds ( t he quant um ) . I f t he t im er expires, t he checker m oves t o anot her cust om er even if no price check is needed. This is t im esharing. I f t he checker is sufficient ly fast , t he sit uat ion is alm ost equivalent t o having one slow er checker at each checkout st and. Consider m aking a video of such a checkout st and and playing it back at 100 t im es it s norm al speed. I t w ould look as if t he checker w ere handling several cust om ers sim ult aneously.
Ex e r cise 1 .3
Suppose t hat t he checker can check one it em per second ( a one- second processor cycle t im e in
Table 1.1) . According t o t his t able, w hat would be t he m axim um t im e t he checker would spend
w it h one cust om er before m oving t o a w ait ing cust om er?
An sw e r :
The t im e is t he quant um t hat is scaled in t he t able t o 6.3 years. A program m ay execut e billions of inst ruct ions in a quant um —a bit m ore t han t he num ber of grocery it em s purchased by t he average cust om er.
I f t he t im e t o m ove from one cust om er t o anot her ( t he cont ext - swit ch t im e) is sm all com pared w it h t he t im e bet ween sw it ches ( t he CPU burst t im e) , t he checker handles cust om ers
efficient ly. Tim esharing w ast es processing cycles by swit ching bet ween cust om ers, but it has t he advant age of not w ast ing t he checker resources during a price check. Furt herm ore, cust om ers w it h sm all orders are not held in abeyance for long periods while wait ing for cust om ers w it h large orders.
The analogy would be m ore realist ic if inst ead of several checkout count ers, t here were only one, wit h t he cust om ers crow ded around t he checker. To swit ch from cust om er A t o cust om er B, t he checker saves t he cont ent s of t he regist er t ape ( t he cont ext ) and rest ores it t o what it w as w hen it last processed cust om er B. The cont ext - swit ch t im e can be reduced if t he cash regist er has several t apes and can hold t he cont ent s of several cust om ers' orders
sim ult aneously. I n fact , som e com put er syst em s have special hardware t o hold m any cont ext s at t he sam e t im e.
Mult iprocessor syst em s have several processors accessing a shared m em ory. I n t he checkout analogy for a m ult iprocessor syst em , each cust om er has an individual regist er t ape and m ult iple checkers rove t he checkout st ands working on t he orders for unserved cust om ers. Many grocery st ores have packers w ho do t his.
[ Team LiB ]
1.4 Concurrency at the Applications Level
Concurrency occurs at t he hardw are level because m ult iple devices operat e at t he sam e t im e. Processors have int ernal parallelism and work on several inst ruct ions sim ult aneously, syst em s have m ult iple processors, and syst em s int eract t hrough net work com m unicat ion. Concurrency is visible at t he applicat ions level in signal handling, in t he overlap of I / O and processing, in com m unicat ion, and in t he sharing of resources bet w een processes or am ong t hreads in t he sam e process. This sect ion provides an overview of concurrency and asynchronous operat ion.
1.4.1 Interrupts
The execut ion of a single inst ruct ion in a program at t he convent ional m achine level is t he result of t he processor inst ruct ion cycle. During norm al execut ion of it s inst ruct ion cycle, a processor ret rieves an address from t he program count er and execut es t he inst ruct ion at t hat address. ( Modern processors have int ernal parallelism such as pipelines t o reduce execut ion t im e, but t his discussion does not consider t hat com plicat ion.) Concurrency arises at t he
convent ional m achine level because a peripheral device can generat e an elect rical signal, called an int errupt, t o set a hardw are flag w it hin t he processor. The det ect ion of an int errupt is part of t he inst ruct ion cycle it self. On each inst ruct ion cycle, t he processor checks hardw are flags t o see if any peripheral devices need at t ent ion. I f t he processor det ect s t hat an int errupt has occurred, it saves t he current value of t he program count er and loads a new value t hat is t he address of a special funct ion called an int errupt service rout ine or int errupt handler. Aft er finishing t he int errupt service rout ine, t he processor m ust be able t o resum e execut ion of t he previous inst ruct ion w here it left off.
An event is asynchronous t o an ent it y if t he t im e at w hich it occurs is not det erm ined by t hat ent it y. The int errupt s generat ed by ext ernal hardware devices are generally asynchronous t o program s execut ing on t he syst em . The int errupt s do not alw ays occur at t he sam e point in a program 's execut ion, but a program should give a correct result regardless of w here it is int errupt ed. I n cont rast , an error event such as division by zero is synchronous in t he sense t hat it always occurs during t he execut ion of a part icular inst ruct ion if t he sam e dat a is present ed t o t he inst ruct ion.
Alt hough t he int errupt service rout ine m ay be part of t he program t hat is int errupt ed, t he processing of an int errupt service rout ine is a dist inct ent it y wit h respect t o concurrency. Operat ing- syst em rout ines called device drivers usually handle t he int errupt s generat ed by peripheral devices. These drivers t hen not ify t he relevant processes, t hrough a soft ware m echanism such as a signal, t hat an event has occurred.
Operat ing syst em s also use int errupt s t o im plem ent t im esharing. Most m achines have a device called a t im er t hat can generat e an int errupt aft er a specified int erval of t im e. To execut e a user program , t he operat ing syst em st art s t he t im er before set t ing t he program count er. When t he t im er expires, it generat es an int errupt t hat causes t he CPU t o execut e t he t im er int errupt service rout ine. The int errupt service rout ine w rit es t he address of t he operat ing syst em code int o t he program count er, and t he operat ing syst em is back in cont rol. When a process loses t he CPU in t he m anner j ust described, it s quant um is said t o have expired. The operat ing syst em put s t he process in a queue of processes t hat are ready t o run. The process wait s t here for anot her t urn t o execut e.
A signal is a soft w are not ificat ion of an event . Oft en, a signal is a response of t he operat ing syst em t o an int errupt ( a hardw are event ) . For exam ple, a keyst roke such as Ct rl- C generat es an int errupt for t he device driver handling t he keyboard. The driver recognizes t he charact er as t he int errupt charact er and not ifies t he processes t hat are associat ed wit h t his t erm inal by sending a signal. The operat ing syst em m ay also send a signal t o a process t o not ify it of a com plet ed I / O operat ion or an error.
A signal is generat ed w hen t he event t hat causes t he signal occurs. Signals can be generat ed eit her synchronously or asynchronously. A signal is generat ed synchronously if it is generat ed by t he process or t hread t hat receives it . The execut ion of an illegal inst ruct ion or a divide- by-zero m ay generat e a synchronous signal. A Ct rl- C on t he keyboard generat es an asynchronous signal. Signals (Chapt er 8) can be used for t im ers (Chapt er 10) , t erm inat ing program s (Sect ion 8.2) , j ob cont rol (Sect ion 11.7) or asynchronous I / O (Sect ion 8.8) .
A process cat ches a signal w hen it execut es a handler for t he signal. A program t hat cat ches a signal has at least t w o concurrent part s, t he m ain program and t he signal handler. Pot ent ial concurrency rest rict s w hat can be done inside a signal handler (Sect ion 8.6) . I f t he signal handler m odifies ext ernal variables t hat t he program can m odify elsewhere, t hen proper execut ion m ay require t hat t hose variables be prot ect ed.
1.4.3 Input and output
A challenge for operat ing syst em s is t o coordinat e resources t hat have great ly differing
charact erist ic access t im es. The processor can perform m illions of operat ions on behalf of ot her processes w hile a program w ait s for a disk access t o com plet e. Alt ernat ively, t he process can avoid blocking by using asynchronous I / O or dedicat ed t hreads inst ead of ordinary blocking I / O. The t radeoff is bet w een t he addit ional perform ance and t he ext ra program m ing overhead in using t hese m echanism s.
A sim ilar problem occurs w hen an applicat ion m onit ors t wo or m ore input channels such as input from different sources on a net work. I f st andard blocking I / O is used, an applicat ion t hat is blocked wait ing for input from one source is not able t o respond if input from anot her source becom es available.
1.4.4 Processes, threads and the sharing of resources
A t radit ional m et hod for achieving concurrent execut ion in UNI X is for t he user t o creat e
m ult iple processes by calling t he fork funct ion. The processes usually need t o coordinat e t heir operat ion in som e way. I n t he sim plest inst ance t hey m ay only need t o coordinat e t heir
execut es, t he CPU uses t he program count er t o det erm ine which inst ruct ion t o execut e next . The result ing st ream of inst ruct ions is called t he program 's t hread of execut ion. I t is t he flow of cont rol for t he process. I f t w o dist inct t hreads of execut ion share a resource wit hin a t im e fram e, care m ust be t aken t hat t hese t hreads do not int erfere w it h each ot her. Mult iprocessor syst em s expand t he opport unit y for concurrency and sharing am ong applicat ions and wit hin applicat ions. When a m ult it hreaded applicat ion has m ore t han one t hread of execut ion concurrent ly act ive on a m ult iprocessor syst em , m ult iple inst ruct ions from t he sam e process m ay be execut ed at t he sam e t im e.
Unt il recent ly t here has not been a st andard for using t hreads, and each vendor's t hread package behaved different ly. A t hread st andard has now been incorporat ed int o t he POSI X st andard. Chapt ers 12 and 13 discuss t his new st andard. t he video cont roller and disk cont roller t ook over som e of t he processing relat ed t o t hese peripherals, relieving t he CPU of t his burden. I n m odern m achines, t hese cont rollers and ot her I / O cont rollers have t heir ow n special purpose CPUs.
What if aft er all t his auxiliary processing has been offloaded, t he CPU is st ill t he bot t leneck? There are t w o approaches t o im proving t he perform ance. Adm iral Grace Murray Hopper, a pioneer in com put er soft w are, oft en com pared com put ing t o t he w ay fields were plowed in t he pioneer days: " I f one ox could not do t he j ob, t hey did not t ry t o grow a bigger ox, but used t w o oxen." I t w as usually cheaper t o add anot her processor or t wo t han t o increase t he speed of a single processor. Som e problem s do not lend t hem selves t o j ust increasing t he num ber of processors indefinit ely. Seym our Cray, a pioneer in com put er hardware, is report ed t o have said, " I f you w ere plow ing a field, w hich w ould you rat her use? Tw o st rong oxen or 1024 chickens?"
The opt im al t radeoff bet w een m ore CPUs and bet t er CPUs depends on several fact ors, including t he t ype of problem t o be solved and t he cost of each solut ion. Machines wit h m ult iple CPUs have already m igrat ed t o t he deskt op and are likely t o becom e m ore com m on as prices drop. Concurrency issues at t he applicat ion level are slight ly different when t here are m ult iple processors, but t he m et hods discussed in t his book are equally applicable in a m ult iprocessor environm ent .
1.4.6 The network as the computer
The obj ect - based m odel is anot her m odel for dist ribut ed com put at ion. Each resource in t he syst em is view ed as an obj ect w it h a m essage- handling int erface, allow ing all resources t o be accessed in a uniform w ay. The obj ect - based m odel allows for cont rolled increm ent al
developm ent and code reuse. Obj ect fram eworks define int eract ions bet w een code m odules, and t he obj ect m odel nat urally expresses not ions of prot ect ion. Many of t he experim ent al dist ribut ed operat ing syst em s such as Argus [74] , Am oeba [124] , Mach [1] , Arj una [106] , Clouds [29] and Em erald [11] are obj ect based. Obj ect - based m odels require obj ect m anagers t o t rack t he locat ion of t he obj ect s in t he syst em .
An alt ernat ive t o a t ruly dist ribut ed operat ing syst em is t o provide applicat ion layers t hat run on t op of com m on operat ing syst em s t o exploit parallelism on t he net work. The Parallel Virt ual Machine ( PVM) and it s successor, Message Passing I nt erface ( MPI ) , are soft ware libraries [10,
43] t hat allow a collect ion of het erogeneous w orkst at ions t o funct ion as a parallel com put er for solving large com put at ional problem s. PVM m anages and m onit ors t asks t hat are dist ribut ed on w orkst at ions across t he net w ork. Chapt er 17 develops a dispat cher for a sim plified version of PVM. CORBA ( Com m on Obj ect Request Broker Archit ect ure) is anot her t ype of soft w are layer t hat provides an obj ect - orient ed int erface t o a set of generic services in a het erogeneous dist ribut ed environm ent [104] .
[ Team LiB ]
1.5 Security and Fault Tolerance
The 1950s and early 1960s brought bat ch processing, and t he m id- t o- lat e 1960s saw deploym ent of operat ing syst em s t hat support ed m ult iprogram m ing. Tim e- sharing and real-t im e program m ing gained popularireal-t y in real-t he 1970s. During real-t he 1980s, parallel processing m oved from t he supercom put er arena t o t he deskt op. The 1990s was t he decade of t he net work—wit h t he w idespread use of dist ribut ed processing, em ail and t he World Wide Web. The 2000s
appears t o be t he decade of securit y and fault - t olerance. The rapid com put erizat ion and t he dist ribut ion of crit ical infrast ruct ure ( banking, t ransport at ion, com m unicat ion, m edicine and governm ent ) over net w orks has exposed enorm ous vulnerabilit ies. We have com e t o rely on program s t hat w ere not adequat ely designed or t est ed for a concurrent environm ent , writ t en by program m ers w ho m ay not have underst ood t he im plicat ions of incorrect ly working program s. The liabilit y disclaim ers dist ribut ed w it h m ost soft w are at t em pt s t o absolve t he m anufact urers of responsibilit y for dam age—soft w are is dist ribut ed as is.
But , lives now depend on soft w are, and each of us has a responsibilit y t o becom e at t uned t o t he im plicat ions of bad soft w are. Wit h current t echnology, it is alm ost im possible t o writ e com plet ely error- free code, but w e believe t hat program m er aw areness can great ly reduce t he scope of t he problem . Unfort unat ely, m ost people learn t o program for an environm ent in which program s are present ed w it h correct or alm ost correct input . Their ideal users behave
graciously, and program s are allow ed t o exit when t hey encount er an error.
Real- world program s, especially syst em s program s, are oft en long- running and are expect ed t o cont inue running aft er an error ( no blue- screen of deat h or reboot allowed) . Long- running program s m ust release resources, such as m em ory, when t hese resources are no longer
needed. Oft en, program m ers release resources such as buffers in t he obvious places but forget t o release t hem if an error occurs.
Most UNI X library funct ions indicat e an error by a ret urn value. However, C m akes no requirem ent t hat ret urn values be checked. I f a program doesn't check a ret urn value,
execut ion can cont inue w ell beyond t he point at which a crit ical error occurs. The consequence of t he funct ion error m ay not be apparent unt il m uch lat er in t he execut ion. C also allows program s t o writ e out of t he bounds of variables. For exam ple, t he C runt im e syst em does not com plain if you m odify a nonexist ent array elem ent —it writ es values int o t hat m em ory ( which probably corresponds t o som e ot her variable) . Your program m ay not det ect t he problem at t he t im e it happened, but t he overw rit t en variable m ay present a problem lat er. Because
overw rit t en variables are so difficult t o det ect and so dangerous, newer program m ing languages, such as Java, have runt im e checks on array bounds.
Even soft w are t hat has been in dist ribut ion for years and has received heavy scrut iny is riddled w it h bugs. For exam ple, an int erest ing st udy by Chou et al. [23] used a m odified com piler t o look for 12 t ypes of bugs in Linux and OpenBSD source code. They exam ined 21 snapshot s of Linux spanning seven years and one snapshot of OpenBSD. They found 1025 bugs in t he code by using aut om at ic scanning t echniques. One of t he m ost com m on bugs was t he failure t o check for a NULL ret urn on funct ions t hat ret urn point ers. I f t he code lat er uses t he ret urned point er, a core dum p occurs.
Anot her problem is t he exponent ial growt h in t he num ber of t ruly m alicious users w ho launch concert ed at t acks on servers and user com put ers. The next sect ion describes one com m on t ype of at t ack, t he buffer overflow .
[ Team LiB ]
1.6 Buffer Overflows for Breaking and Entering
This sect ion present s a sim plified explanat ion of buffer overflows and how t hey m ight be used t o at t ack a com put er syst em . A buffer overflow occurs w hen a program copies dat a int o a variable for which it has not allocat ed enough space.
Exam ple 1.4 shows a code segm ent t hat m ay have a buffer overflow. A user t ypes a nam e in
response t o t he prom pt . The program st ores t he input in a char array called buf. I f t he user ent ers m ore t han 79 byt es, t he result ing st ring and st ring t erm inat or do not fit in t he allocat ed variable.
Ex a m ple 1 .4
The following code segm ent has t he possibilit y of a buffer overflow .
char buf[80];
printf("Enter your first name:"); scanf("%s", buf);
Your first t hought in fixing t his pot ent ial overflow m ight be t o m ake buf bigger, say, 1000 byt es. What user's first nam e could be t hat long? Even if a user decides t o t ype in a very long st ring of charact ers, 1000 byt es should be large enough t o handle all but t he m ost persist ent user. However, regardless of t he ult im at e size t hat you choose, t he code segm ent is st ill
suscept ible t o a buffer overflow . The user sim ply needs t o redirect st andard input t o com e from an arbit rarily large file.
Exam ple 1.5 shows a sim ple w ay t o fix t his problem . The form at specificat ion lim it s t he input
st ring t o one less t han t he size of t he variable, allowing room for t he st ring t erm inat or. The program reads at m ost 79 charact ers int o buf but st ops when it encount ers a w hit e space charact er. I f t he user ent ers m ore t han 79 charact ers, t he program reads t he addit ional charact ers in subsequent input st at em ent s.
Ex a m ple 1 .5
The following code segm ent does not have a buffer overflow .
char buf[80];
printf("Enter your first name:"); scanf("%79s", buf);
1.6.1 Consequences of buffer overflows
variables t hat are aut om at ic. While t he det ails differ from m achine t o m achine, program s generally allocat e aut om at ic variables on t he program st ack.
I n a t ypical syst em , t he st ack grow s from high m em ory t o low m em ory. When a funct ion is called, t he lower part of t he st ack cont ains t he passed param et ers and t he ret urn address. Higher up on t he st ack ( low er m em ory addresses) are t he local aut om at ic variables. The st ack m ay st ore ot her values and have gaps t hat are not used by t he program at all. One im port ant fact is t hat t he ret urn address for each funct ion call is usually st ored in m em ory aft er ( wit h larger address t han) t he aut om at ic variables.
When a program w rit es beyond t he lim it s of a variable on t he st ack, a buffer overflow occurs. The ext ra byt es m ay w rit e over unused space, ot her variables, t he ret urn address or ot her m em ory not legally accessible t o your program . The consequences can range from none, t o a program crash and a core dum p, t o unpredict able behavior.
Program 1.1 shows a funct ion t hat can have a buffer overflow. The checkpass funct ion checks
w het her t he ent ered st ring m at ches "mypass" and ret urns 1 if t hey m at ch, and 0 ot herwise.
Pr ogr a m 1 .1
checkpass.cA funct ion t hat checks a passw ord. This funct ion is suscept ible t o buffer overflow.
#include <stdio.h> #include <string.h>
int checkpass(void){ int x;
char a[9]; x = 0;
fprintf(stderr,"a at %p and\nx at %p\n", (void *)a, (void *)&x); printf("Enter a short word: ");
scanf("%s", a);
if (strcmp(a, "mypass") == 0) x = 1;
return x; }
Figure 1.1 show s a possible organizat ion of t he st ack for a call t o checkpass. The diagram
assum es t hat int egers and point ers are 4 byt es. Not e t hat t he com piler allocat es 12 byt es for array a, even t hough t he program specifies only 9 byt es, so t hat t he syst em can m aint ain a st ack point er t hat is aligned on a w ord boundary.
Figu r e 1 .1 . Possible st a ck la you t for t h e
checkpassfu n ct ion of
Pr ogr a m
I f t he charact er array a is st ored on t he st ack in lower m em ory t han t he int eger x, a buffer overflow of a m ay change t he value of x. I f t he user ent ers a word t hat is slight ly longer t han t he array a, t he overflow changes t he value of x, but t here is no ot her effect . Exact ly how long t he ent ered st ring needs t o be t o cause a problem depends on t he syst em . Wit h t he m em ory organizat ion of Figure 1.1, if t he user ent ers 12 charact ers, t he st ring t erm inat or overwrit es one byt e of x w it hout changing it s value. I f t he user ent ers m ore t han 12 charact ers, som e of t hem overwrit e x, changing it s value. I f t he user ent ers 13 charact ers, x changes t o a nonzero value and t he funct ion ret urns 1, no m at t er what charact ers are ent ered.
I f t he user ent ers a long passw ord, t he ret urn address is overwrit t en, and m ost likely t he funct ion w ill t ry t o ret urn t o a locat ion out side t he address space of t he program , generat ing a segm ent at ion fault and core dum p. Buffer overflows t hat cause an applicat ion program t o exit w it h a segm ent at ion fault can be annoying and can cause t he program t o lose unsaved dat a. The sam e t ype of overflow in an operat ing syst em funct ion can cause t he operat ing syst em t o crash.
Buffer overflows in dynam ically allocat ed buffers or buffers w it h st at ic st orage can also behave unpredict ably. One of our st udent s w rot e a program t hat appeared t o show an error in t he C library. He t raced a segm ent at ion fault t o a call t o malloc and w as able t o show t hat t he
program was working unt il t he call t o malloc. The program had a segm ent at ion fault before t he call t o malloc ret urned. He event ually t raced t he problem t o a t ype of buffer overflow in which t he byt e before a buffer dynam ically allocat ed by a previous malloc call w as overwrit t en. ( This can easily happen if a buffer is being filled from t he back and a count is off by one.) Overwrit ing cont rol inform at ion st ored in t he heap caused t he next call t o malloc t o crash t he program .
Securit y problem s relat ed t o buffer overflow s have been known for over a decade. They first acquired nat ional at t ent ion w hen on Novem ber 2, 1988, Robert Morris released a worm on t he I nt ernet . A worm is a self- replicat ing, self- propagat ing program . This program forced m any syst em adm inist rat ors t o disconnect t heir sit es from t he I nt ernet so t hat t hey would not be cont inually reinfect ed. I t t ook several days for t he I nt ernet t o ret urn t o norm al. One of t he m et hods used by t he Morris w orm w as t o exploit a buffer overflow in t he finger daem on. This daem on ran on m ost UNI X m achines t o allow t he display of inform at ion about users.
I n response t o t his worm , CERT, t he Com put er Em ergency Response Team , w as creat ed [24] . The CERT Coordinat ion Cent er is a federally funded cent er of I nt ernet securit y expert ise t hat regularly publishes com put er securit y alert s.
Program s t hat are suscept ible t o buffer overflow are st ill being writ t en, in spit e of past experiences. The first six CERT advisories in 2002 describe buffer overflow flaws in various com put er syst em s, including Com m on Deskt op Environm ent for t he Sun Solaris operat ing environm ent ( a w indow ing syst em ) , I CQ from AOL ( an inst ant m essaging program used by priorit ies are t o produce code quickly, sloppy code w ill be produced. The effect s of poor coding are exacerbat ed by com pilers and runt im e syst em s t hat don't enforce range checking.
There are m any w ays in w hich buffer overflow s have been used t o com prom ise a syst em . Here Suppose t he funct ion allocat es a buffer of size 100 for t he password. This m ight seem
reasonable, since passw ords in UNI X are at m ost 8 byt es long. I f t he program does not check
som et hing like t he follow ing code.
runs w it h t his user I D, and t he ordinary user now has root privileges. The vulnerabilit y depends on get t ing t he byt es in t he input file exact ly right . Finding t he address of t he execvl is not as difficult as it m ight first appear, because m ost processor inst ruct ion set s support a relat ive addressing m ode.
[ Team LiB ]
1.7 UNIX Standards
Not t oo long ago, t w o dist inct and som ewhat incom pat ible " flavors" of UNI X, Syst em V from AT&T and BSD from Berkeley coexist ed. Because no official st andard exist ed, t here were m aj or and m inor differences bet w een t he versions from different vendors, even w it hin t he sam e flavor. Consequent ly, program s w rit t en for one t ype of UNI X would not run correct ly or som et im es would not even com pile under a UNI X from anot her vendor.
The I EEE ( I nst it ut e of Elect ronic and Elect rical Engineers) decided t o develop a st andard for t he UNI X libraries in an init iat ive called POSI X. POSI X st ands for Port able Operat ing Syst em
I nt erface and is pronounced pahz- icks, as st at ed explicit ly by t he st andard. I EEE's first at t em pt , called POSI X.1, was published in 1988. When t his st andard w as adopt ed, t here was no known hist orical im plem ent at ion of UNI X t hat w ould not have t o change t o m eet t he st andard. The original st andard covered only a sm all subset of UNI X. I n 1994, t he X/ Open Foundat ion published a m ore com prehensive st andard called Spec 1170, based on Syst em V.
Unfort unat ely, inconsist encies bet w een Spec 1170 and POSI X m ade it difficult for vendors and applicat ion developers t o adhere t o bot h st andards.
I n 1998, aft er anot her version of t he X/ Open st andard, m any addit ions t o t he POSI X st andard, specificat ion is referred t o as t he Single UNI X Specificat ion, Version 3, or I EEE St d. 1003.1-2001, POSI X. I n t his book w e refer t o t his st andard m erely as POSI X.
Each of t he st andards organizat ions publishes copies of t he st andard. Print and elect ronic versions of t he st andard are available from I EEE and I SO/ I EC. The Open Group publishes t he st andard on CD- ROM. I t is also freely available on t heir w eb sit e [89] . The copy of t he st andard
AI O asynchronous input and out put yes yes no
FSC file synchronizat ion yes yes yes
RTS realt im e signals ext ension yes yes no
SEM sem aphores yes unnam ed only nam ed only
THR t hreads yes alm ost yes
TMR t im ers yes yes no
TPS t hread execut ion scheduling yes yes yes
TSA t hread st ack address at t ribut e no no no
TSF t hread- safe funct ions yes strtok_r only yes
XSI XSI ext ension yes yes t im ers, getsid, ftok,
no I PC
_POSIX_VERSION 199506 199506 198808
A POSI X- com pliant im plem ent at ion m ust support t he POSI X base st andard. Many of t he int erest ing aspect s of POSI X are not part of t he base st andard but rat her are defined as
ext ensions t o t he base st andard. Table E.1 of Appendix E gives a com plet e list of t he ext ensions in t he 2001 version of POSI X. Appendix E applies only t o im plem ent at ions t hat claim
com pliance w it h t he 2001 version base st andard. These im plem ent at ions set t he sym bol _POSIX_VERSION defined in unistd.h t o 200112L. As of t he w rit ing of t his book, none of t he syst em s w e t est ed used t his value. Syst em s t hat support t he previous version of POSI X have a value of 199506L. Differences bet w een t he 1995 and 2001 st andards for feat ures support ed by bot h are m inor.
The new POSI X st andard also incorporat es t he I SO/ I EC I nt ernat ional St andard 9899, also referred t o as I SO C. I n t he past , m inor differences bet ween t he POSI X and I SO C st andards have caused confusion. Oft en, t hese differences were unint ent ional, but differences in published st andards required developers t o choose bet w een t hem . The current POSI X st andard m akes it clear t hat any differences bet w een t he published POSI X st andard and t he I SO C st andard are unint ent ional. I f any discrepancies occur, t he I SO C st andard t akes precedence.
[ Team LiB ]
1.8 Additional Reading
Most general operat ing syst em s books present an overview and hist ory of operat ing syst em s. Recom m ended int roduct ions include Chapt er 1 of Modern Operat ing Syst em s by Tanenbaum [122] or Chapt ers 1 t o 3 of Operat ing Syst em s Concept s by Silberschat z et al. [107] . Chapt ers 1 and 2 of Dist ribut ed Syst em s: Concept s and Design by Coulouris et al. discuss design issues for dist ribut ed syst em s [26] . Dist ribut ed Operat ing Syst em s by Tanenbaum [121] also has a good overview of dist ribut ed syst em s issues, but it provides fewer det ails about specific
dist ribut ed syst em s t han does [26] . See also Dist ribut ed Syst em s: Principles and Paradigm s by Van St een and Tanenbaum [127] .
Advanced Program m ing in t he UNI X Environm ent by St evens [112] is a key t echnical reference on t he UNI X int erface t o use in conj unct ion w it h t his book. Serious syst em s program m ers should acquire t he POSI X St d. 1003.1 from t he I EEE [50] or t he Open Group w eb sit e [89] . The st andard is surprisingly readable and t horough. The rat ionale sect ions included wit h each
funct ion provide a great deal of insight int o t he considerat ions t hat went int o t he st andard. The final arbit er of C quest ions is t he I SO C st andard [56] .
The CERT w eb sit e [24] is a good source for current inform at ion on recent ly discovered bugs, ongoing at t acks and vulnerabilit ies. The book Know Your Enem y: Revealing t he Securit y Tools, Tact ics, and Mot ives of t he Blackhat Com m unit y edit ed by m em bers of t he Honeynet Proj ect [48] is an int erest ing glim pse int o t he realm of t he m alicious.
[ Team LiB ]
Chapter 2. Programs, Processes and Threads
One popular definit ion of a process is an inst ance of a program whose execut ion has st art ed but has not yet t erm inat ed. This chapt er discusses t he differences bet ween program s and
processes and t he ways in w hich t he form er are t ransform ed int o t he lat t er. The chapt er addresses issues of program layout , com m and- line argum ent s, program environm ent and exit handlers.
Objectives
● Learn about program s, processes and t hreads
● Experim ent w it h m em ory allocat ion and m anipulat ion
● Explore im plicat ions of st at ic obj ect s
● Use environm ent variables for cont ext
● Underst and program st ruct ure and layout
[ Team LiB ]
2.1 How a Program Becomes a Process
A program is a prepared sequence of inst ruct ions t o accom plish a defined t ask. To writ e a C source program , a program m er creat es disk files cont aining C st at em ent s t hat are organized int o funct ions. An individual C source file m ay also cont ain variable and funct ion declarat ions, t ype and m acro definit ions ( e.g., typedef) and preprocessor com m ands ( e.g., #ifdef,
#include, #define) . The source program cont ains exact ly one main funct ion.
Tradit ionally, C source filenam es have a .c ext ension, and header filenam es have a .h
ext ension. Header files usually only cont ain m acro and t ype definit ions, defined const ant s and funct ion declarat ions. Use t he #include preprocessor com m and t o insert t he cont ent s of a header file int o t he source.
The C com piler t ranslat es each source file int o an obj ect file. The com piler t hen links t he individual obj ect files w it h t he necessary libraries t o produce an execut able m odule. When a program is run or execut ed, t he operat ing syst em copies t he execut able m odule int o a program im age in m ain m em ory.
A process is an inst ance of a program t hat is execut ing. Each inst ance has it s own address space and execut ion st at e. When does a program becom e a process? The operat ing syst em reads t he program int o m em ory. The allocat ion of m em ory for t he program im age is not
enough t o m ake t he program a process. The process m ust have an I D ( t he process I D) so t hat t he operat ing syst em can dist inguish am ong individual processes. The process st at e indicat es t he execut ion st at us of an individual process. The operat ing syst em keeps t rack of t he process I Ds and corresponding process st at es and uses t he inform at ion t o allocat e and m anage
resources for t he syst em . The operat ing syst em also m anages t he m em ory occupied by t he processes and t he m em ory available for allocat ion.
When t he operat ing syst em has added t he appropriat e inform at ion in t he kernel dat a st ruct ures and has allocat ed t he necessary resources t o run t he program code, t he program has becom e a process. A process has an address space ( m em ory it can access) and at least one flow of processes m ay reside in m em ory and execut e concurrent ly, alm ost independent ly of each ot her. For processes t o com m unicat e or cooperat e, t hey m ust explicit ly int eract t hrough operat ing syst em const ruct s such as t he filesyst em (Sect ion 5.1) , pipes (Sect ion 6.1) , shared m em ory (Sect ion 15.3) or a net w ork (Chapt ers 18-22) .
[ Team LiB ]
2.2 Threads and Thread of Execution
When a program execut es, t he value of t he process program count er det erm ines which process inst ruct ion is execut ed next . The result ing st ream of inst ruct ions, called a t hread of execut ion, can be represent ed by t he sequence of inst ruct ion addresses assigned t o t he program count er during t he execut ion of t he program 's code.
Ex a m ple 2 .1
Process 1 execut es st at em ent s 245, 246 and 247 in a loop. I t s t hread of execut ion can be represent ed as 2451, 2461, 2471, 2451, 2461, 2471, 2451, 2461, 2471 . . . , where t he subscript s ident ify t he t hread of execut ion as belonging t o process 1.
The sequence of inst ruct ions in a t hread of execut ion appears t o t he process as an
unint errupt ed st ream of addresses. From t he point of view of t he processor, how ever, t he t hreads of execut ion from different processes are int erm ixed. The point at which execut ion sw it ches from one process t o anot her is called a cont ext sw it ch. process. Mult iple t hreads avoid cont ext swit ches and allow sharing of code and dat a. The approach m ay im prove program perform ance on m achines wit h m ult iple processors. Program s w it h nat ural parallelism in t he form of independent t asks operat ing on shared dat a can t ake advant age of added execut ion pow er on t hese m ult iple- processor m achines. Operat ing syst em s have significant nat ural parallelism and perform bet t er by having m ult iple, sim ult aneous
t hreads of execut ion. Vendors advert ise sym m et ric m ult iprocessing support in which t he
operat ing syst em and applicat ions have m ult iple undist inguished t hreads of execut ion t hat t ake advant age of parallel hardw are.
A t hread is an abst ract dat a t ype t hat represent s a t hread of execut ion w it hin a process. A t hread has it s ow n execut ion st ack, program count er value, regist er set and st at e. By declaring m any t hreads w it hin t he confines of a single process, a program m er can writ e program s t hat achieve parallelism w it h low overhead. While t hese t hreads provide low - overhead parallelism , t hey m ay require addit ional synchronizat ion because t hey reside in t he sam e process address space and t herefore share process resources. Som e people call processes heavyw eight because of t he w ork needed t o st art t hem . I n cont rast , t hreads are som et im es called light w eight
[ Team LiB ]
2.3 Layout of a Program Image
Aft er loading, t he program execut able appears t o occupy a cont iguous block of m em ory called a
program im age. Figure 2.1 show s a sam ple layout of a program im age in it s logical address space [112] . The program im age has several dist inct sect ions. The program t ext or code is show n in low - order m em ory. The init ialized and uninit ialized st at ic variables have t heir own sect ions in t he im age. Ot her sect ions include t he heap, st ack and environm ent .
Figu r e 2 .1 . Sa m ple la you t for a pr ogr a m im a ge in m a in m e m or y.
An act ivat ion record is a block of m em ory allocat ed on t he t op of t he process st ack t o hold t he execut ion cont ext of a funct ion during a call. Each funct ion call creat es a new act ivat ion record on t he st ack. The act ivat ion record is rem oved from t he st ack when t he funct ion ret urns, providing t he last - called- first - ret urned order for nest ed funct ion calls.
The act ivat ion record cont ains t he ret urn address, t he param et ers ( whose values are copied from t he corresponding argum ent s) , st at us inform at ion and a copy of som e of t he CPU regist er values at t he t im e of t he call. The process rest ores t he regist er values on ret urn from t he call represent ed by t he record. The act ivat ion record also cont ains aut om at ic variables t hat are allocat ed w it hin t he funct ion w hile it is execut ing. The part icular form at for an act ivat ion record depends on t he hardw are and on t he program m ing language.
and argv and for allocat ions by malloc. The malloc fam ily of funct ions allocat es st orage from a free m em ory pool called t he heap. St orage allocat ed on t he heap persist s unt il it is freed or unt il t he program exit s. I f a funct ion calls malloc, t he st orage rem ains allocat ed aft er t he
funct ion ret urns. The program cannot access t he st orage aft er t he ret urn unless it has a point er t o t he st orage t hat is accessible aft er t he funct ion ret urns.
St at ic variables t hat are not explicit ly init ialized in t heir declarat ions are init ialized t o 0 at run t im e. Not ice t hat t he init ialized st at ic variables and t he uninit ialized st at ic variables occupy different sect ions in t he program im age. Typically, t he init ialized st at ic variables are part of t he execut able m odule on disk, but t he uninit ialized st at ic variables are not . Of course, t he
aut om at ic variables are not part of t he execut able m odule because t hey are only allocat ed w hen t heir defining block is called. The init ial values of aut om at ic variables are undet erm ined unless t he program explicit ly init ializes t hem .
Ex e r cise 2 .3
The execut able m odule for Version 1 should be about 200,000 byt es larger t han t hat of Version 2 because t he myarray of Version 1 is init ialized st at ic dat a and is t herefore part of t he
execut able m odule. The myarray of Version 2 is not allocat ed unt il t he program is loaded in m em ory, and t he array elem ent s are init ialized t o 0 at t hat t im e.
st at ic variable m ay behave in unexpect ed w ays. For t hese reasons, avoid using st at ic variables except under cont rolled circum st ances. Sect ion 2.9 present s an exam ple of when t o use
variables w it h st at ic st orage class.
Alt hough t he program im age appears t o occupy a cont iguous block of m em ory, in pract ice, t he operat ing syst em m aps t he program im age int o noncont iguous blocks of physical m em ory. A com m on m apping divides t he program im age int o equal- sized pieces, called pages. The
operat ing syst em loads t he individual pages int o m em ory and looks up t he locat ion of t he page in a t able when t he processor references m em ory on t hat page. This m apping allows a large logical address space for t he st ack and heap wit hout act ually using physical m em ory unless it is needed. The operat ing syst em hides t he exist ence of such an underlying m apping, so t he
program m er can view t he program im age as logically cont iguous even when som e of t he pages do not act ually reside in m em ory.
