Li n u x Gu i d e

La s t rev i s e d :28/03/2007

By Ke v i n Dr y s d a l e

©2007 iom a r t

Pa g e 1

Table of Contents

1 Int r o d u c t i o n

2 Co n n e c t i n g to a Li n u x se r v e r

2. 1 Th i n g s to kno w be f o r e you con n e c t

2.2 If yo u ar e us i n g W i n d o w s

2.3 If yo u ar e us i n g Lin u x , Ma c OS X, or an y ot h e r UN I X

3 Ba s i c co n c e p t s an d co m m a n d s

3. 1 Li s t i n g the co n t e n t s of a di r e c t o r y

3.2 Fi l e ow n e r s h i p s an d pe r m i s s i o n s

3.3 A fu r t h e r wo r d ab o u t se c u r i t y

3.4 sud o

3.5 Th e con t e n t s of the ro o t di r e c t o r y

3.6 Ch a n g i n g an d dis p l a y i n g the cu r r e n t di r e c t o r y

3.7 Co p y i n g a file fro m on e pl a c e to an o t h e r

3.8 Mo v i n g a file fro m on e pla c e to an o t h e r

3.9 Re m o v i n g a file

3. 10 Cr e a t i n g a ne w di r e c t o r y

3. 1 1 Re m o v i n g a di r e c t o r y

3. 1 2 Vi e w i n g the con t e n t s of a fil e

3. 1 3 En d- of-se c t i o n sum m a r y

4 Fu r t h e r co n c e p t s an d co m m a n d s

4. 1 W o r k i n g wi t h mu l t i p l e fil e s an d di r e c t o r i e s sim u l t a n e o u s l y

4.2 A ca u t i o n a r y wo r d

4.3 Re m o v i n g di r e c t o r i e s wi t h o u t ma n u a l l y em p t y i n g the m firs t

4.4 An o t h e r ca u t i o n a r y wo r d

4.5 Se a r c h i n g the con t e n t s of fil e s

Pa g e 2

4.6 So r t i n g the con t e n t s of fil e s

4.7 Fi n d i n g a file

4.8 Pi p e s

4.9 Inp u t an d ou t p u t re-di r e c t i o n

4. 10 Ch a n g i n g you r pa s s w o r d

4. 1 1 Ch a n g i n g the ow n e r an d gr o u p of a fil e or di r e c t o r y

4. 1 2 Ch a n g i n g the pe r m i s s i o n s on a file or di r e c t o r y

4. 1 3 Sy m b o l i c lin k s

4. 1 4 En d- of-se c t i o n sum m a r y

5 Ad v a n c e d co m m a n d s

5. 1 sort rev i s i t e d5.2 find rev i s i t e d5.3 grep rev i s i t e d5.4 Th e awk com m a n d

5.5 Th e sed com m a n d

5.6 So m e ot h e r us e f u l tex t pr o c e s s i n g com m a n d s

5.7 Th e man com m a n d , an d co m m a n d- lin e he l p

5.8 En d- of-se c t i o n sum m a r y

6 Ad v a n c e d co n c e p t s

6. 1 Pr o c e s s e s

6.2 Pr o c e s s ma n a g e m e n t

6.3 Re s o u r c e mo n i t o r i n g

6.4 Co m m o n se r v e r so f t w a r e

6.4. 1 W e b se r v e r - Ap a c h e

6.4.2 FT P se r v e r - vs f t p d

6.4.3 Da t a b a s e se r v e r – My S Q L

6.4.4 Em a i l se r v e r - se n d m a i l

7 Co n c l u s i o n

Pa g e 3

1. Introduction

Th e pu r p o s e of thi s gu i d e to pr o v i d e the rea d e r wi t h an int r o d u c t i o n to us i n g Lin u x , an d to ba s i c sy s t e m s ad m i n i s t r a t i o n . Yo u do n’ t ne e d an y pr i o r kn o w l e d g e of ei t h e r Lin u x or an y ot h e r UN I X- ba s e d op e r a t i n g sy s t e m to be n e f i t – we wi l l be sta r t i n g wi t h the ve r y ba s i c s , an d wo r k i n g on fro m the r e to mo r e ad v a n c e d co n c e p t s .

Th i s gu i d e is co n s t a n t l y un d e r de v e l o p m e n t , an d so an y an d al l fee d b a c k is ve r y we l c o m e . If the r e ar e an y sec t i o n s tha t in yo u r op i n i o n ar e un c l e a r in an y wa y , or ind e e d wh i c h ar e inc o r r e c t , pl e a s e let me kn o w . Sim i l a r l y if the r e ar e an y top i c s yo u fee l tha t sh o u l d be cov e r e d tha t pr e s e n t l y ar e no t , dr o p me a line wi t h de t a i l s of wh a t yo u thi n k sh o u l d be inc l u d e d an d I wi l l be ha p p y to wr i t e so m e t h i n g up.

Pa g e 4

2. Connecting to a Linux server

2.1 Things to know before you connect

Th e ve r y fir s t st e p in us i n g an y Lin u x sy s t e m is to es t a b l i s h a co n n e c t i o n to it. Al l m o d e r n UN I X- ba s e d op e r a t i n g sy s t e m s us e SS H - the S ec u r e Sh el l – as the so f t w a r e tha t pr o v i d e s thi s co n n e c t i o n . SS H en c r y p t s al l tra f f i c tha t pa s s e s fro m yo u r ow n co m p u t e r to the se r v e r , so it is in the o r y ex t r e m e l y di f f i c u l t if no t imp o s s i b l e for an y o n e to int e r c e p t an y t h i n g yo u typ e as it tra v e l s fro m yo u r ow n PC to the ho s t Li n u x sy s t e m .

In or d e r to co n n e c t , yo u wi l l ne e d a username an d a password . Th e us e r n a m e is un i q u e to yo u, an d ide n t i f i e s yo u to the sy s t e m . Al l fil e s tha t yo u ow n or cr e a t e on the se r v e r wi l l be as s o c i a t e d wi t h yo u r us e r n a m e alo n e . Th e pa s s w o r d is as s o c i a t e d wi t h yo u r us e r n a m e , an d ac t s as a sec u r e me a n s of en s u r i n g tha t yo u ar e the on l y pe r s o n wh o ca n log in wi t h yo u r us e r n a m e .

By wa y of an ex a m p l e , thi n k of us i n g a ba n k ca r d to wi t h d r a w mo n e y fro m a ca s h ma c h i n e . Th e ca r d is yo u r us e r n a m e – the pu b l i c l y-vi e w a b l e res o u r c e tha t ide n t i f i e s yo u to the AT M . Th e pa s s w o r d is yo u r PI N nu m b e r – the ad d i t i o n a l com p o n e n t tha t on l y yo u kn o w , so tha t ev e n if yo u r ca r d fal l s in to the wr o n g ha n d s yo u r mo n e y is no t ac c e s s i b l e by an y thi r d pa r t y .

A s yo u mi g h t ex p e c t fro m the ab o v e , you should NEVER give your password to anyone ! If so m e o n e els e lea r n s yo u r pa s s w o r d , the y ma y be ab l e to log in to the Li n u x sy s t e m , an d the log s on the sy s t e m wo u l d ind i c a t e tha t an y t h i n g the y di d wa s do n e by yo u. Ke e p i n g yo u r us e r n a m e co n f i d e n t i a l is no t so imp o r t a n t .

Yo u r us e r n a m e an d pa s s w o r d wi l l be su p p l i e d to yo u by an o t h e r me m b e r of the sy s t e m s ad m i n i s t r a t i o n tea m . Yo u ca n ch a n g e yo u r pa s s w o r d to an y t h i n g yo u lik e – mo r e ab o u t thi s lat e r , ho w e v e r . Yo u r us e r n a m e wi l l alw a y s st a y the sa m e thr o u g h o u t the co u r s e of yo u r tim e wi t h the co m p a n y .

Ho w ex a c t l y yo u wi l l es t a b l i s h an SS H co n n e c t i o n to the se r v e r de p e n d s on wh e t h e r or no t yo u r ow n PC is alr e a d y run n i n g Li n u x , or if yo u ar e us i n g Wi n d o w s . Ins t r u c t i o n s for bo t h wi l l be pr o v i d e d he r e .

Pa g e 5

2.2 If you are using Windows

1. Ch e c k to se e if yo u r PC alr e a d y ha s a pr o g r a m ca l l e d PuTTY in s t a l l e d . Pu T T Y is an SS H cl i e n t for W i n d o w s , an d is av a i l a b l e for do w n l o a d at no co s t . If yo u r PC al r e a d y ha s Pu T T Y ins t a l l e d , the n yo u do no t ne e d to do w n l o a d it. Ho w e v e r , if yo u do no t se e m to ha v e it al r e a d y , yo u ca n dow n l o a d it he r e :

http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

2. On c e Pu T T Y ha s be e n do w n l o a d e d or loc a t e d on yo u r PC, do u b l e- cli c kon the Pu T T Y ico n. A wi n d o w lik e the fol l o w i n g wi l l the n ap p e a r :

3. In “Host Name (or IP address) ”, typ e in the ho s t n a m e or IP ad d r e s s of the rem o t e Lin u x sy s t e m yo u wa n t to co n n e c t to.

Pa g e 6

4. Ma k e su r e tha t the “SSH ” rad i o bu t t o n is se l e c t e d , ins t e a d of

ei t h e r “Raw ” or “Telnet ”. Te l n e t is an old e r , no w- ob s o l e t e me t h o d of co n n e c t i n g to UN I X- st y l e sy s t e m s , an d is ins e c u r e . Ra w co n n e c t i o n s sim p l y ma k e a dir e c t co n n e c t i o n be t w e e n yo u r com p u t e r an d the rem o t e sy s t e m , wi t h o u t pe r f o r m i n g an y tel n e t or SS H ne g o t i a t i o n be f o r e h a n d . Yo u wi l l al m o s t ne v e r ne e d to us e ei t h e r of the s e op t i o n s .

5. On c e yo u ar e su r e tha t the ho s t n a m e or IP ad d r e s s is the on e yo u

wa n t , an d tha t SS H is yo u r co n n e c t i o n me t h o d , cli c k the “Open ” bu t t o n . A wi n d o w sim i l a r to the fol l o w i n g wi l l the n ap p e a r :

6. At the “login as: “ pr o m p t , typ e yo u r us e r n a m e , an d pr e s s ret u r n . No t e tha t , lik e mo s t thi n g s in the Li n u x wo r l d , yo u r us e r n a m e is ca s e- se n s i t i v e . Th i s me a n s tha t if the r e ar e an y ca p i t a l let t e r s in yo u r us e r n a m e , yo u mu s t typ e the s e in up p e r- ca s e , si n c e Lin u x di s t i n g u i s h e s be t w e e n the low e r- an d up p e r- ca s e va r i a n t s of al l let t e r s .

7. Yo u wi l l ne x t be pr o m p t e d for yo u r pa s s w o r d . Ty p e thi s in al s o , an d pr e s s ret u r n . Ag a i n , lik e yo u r us e r n a m e , thi s is ca s e se n s i t i v e .

Pa g e 7

8. If yo u r ho s t n a m e , us e r n a m e , an d pa s s w o r d we r e al l co r r e c t , yo u wi l l no w ha v e a Pu T T Y wi n d o w tha t loo k s lik e thi s:

9. Th a t ’ s it ! Yo u ar e no w co n n e c t e d via SS H to the rem o t e Li n u x sy s t e m .

Pa g e 8

2.3 If you are using Linux, Mac OS X, or any other UNIX

If yo u ar e us i n g Li n u x on yo u r PC the pr o c e s s is far sim p l e r , si n c e yo u al r e a d y ha v e al l the so f t w a r e yo u ne e d for an SS H co n n e c t i o n ins t a l l e d on yo u r ma c h i n e .

1. B r i n g up yo u r te r m i n a l pr o g r a m of ch o i c e – e.g. xt e r m , ko n s o l e , gt e r m , et c.

2. At the sh e l l pr o m p t , typ e:

ssh <username>@<hostname or IP>

an d pr e s s ret u r n . Fo r ex a m p l e , if yo u r us e r n a m e wa s ‘drysdalk’ an d yo u wa n t e d to co n n e c t to ‘stimpy.iomart.com’, yo u wo u l d typ e:

ssh drysdalk@stimpy.iomart.com

It’s imp o r t a n t to no t e tha t , lik e mo s t thi n g s in Li n u x , yo u r us e r n a m e is ca s e- se n s i t i v e . In ot h e r wo r d s , if it co n t a i n s an y ca p i t a l let t e r s , yo u mu s t typ e the s e in up p e r- ca s e , as Lin u x di s t i n g u i s h e s be t w e e n the up p e r- an d low e r- ca s e va r i a n t s of al l let t e r s .

3. Af t e r yo u pr e s s ret u r n , yo u wi l l re c e i v e a pa s s w o r d pr o m p t su c h as:

drysdalk@stimpy.iomart.com's password:

Ty p e yo u r pa s s w o r d (note tha t , lik e yo u r us e r n a m e , it is ca s e-se n s i t i v e), an d pr e s s ret u r n .

4. Th a t ’ s it ! Yo u ar e no w log g e d in, an d yo u r ter m i n a l wi l l loo k som e t h i n g lik e thi s:

[root@zurg root]# ssh drysdalk@stimpy.iomart.comdrysdalk@stimpy.iomart.com's password:Last login: Thu Feb 15 20:11:10 2007 from 10.7.155.170[drysdalk@stimpy drysdalk]$

Pa g e 9

3. Basic concepts and commandsLi k e al l mo d e r n op e r a t i n g sy s t e m s , Lin u x ke e p s its fil e s ins i d e

di r e c t o r i e s . Th e ve r y top- lev e l di r e c t o r y is ca l l e d the root directory , an d al l ot h e r dir e c t o r i e s br a n c h of f fro m it. Th e r e ar e al s o va r i o u s se c u r i t y me c h a n i s m s for res t r i c t i n g ac c e s s to sp e c i f i c us e r s for ce r t a i n fil e s an d dir e c t o r i e s , if re q u i r e d .

3.1 Listing the contents of a directory

Yo u ca n vie w the co n t e n t s of an y dir e c t o r y on the sy s t e m wi t h the ls com m a n d . At its sim p l e s t , yo u ca n jus t typ e ls on its ow n to se e the co n t e n t s of the dir e c t o r y yo u ar e cu r r e n t l y in:

[drysdalk@stimpy drysdalk]$ lsCopy of Redbus Rack Usage.xls IPInformation.xlsCorporate Customers with Support.xls Redbus Rack Usage.xlsDC1 Assett List.xls Scolocate Rack Usage.xlsDC1 Rack Usage.xls slash/[drysdalk@stimpy drysdalk]$

He r e we ca n se e se v e n Ex c e l sp r e a d s h e e t s , an d so m e t h i n g el s e – an en t r y ca l l e d slash, wi t h (co-inc i d e n t a l l y) a / at the en d. Th e for w a r d-sla s h at the en d of thi s ind i c a t e s tha t it is an o t h e r di r e c t o r y , ins i d e ou r cu r r e n t di r e c t o r y .

S o, as yo u mi g h t ex p e c t , we ca n se e the co n t e n t s of slash as fol l o w s :

[drysdalk@stimpy drysdalk]$ ls slashimport.csv Report.doc script.sh*[drysdalk@stimpy drysdalk]$

He r e we ca n se e thr e e ot h e r fil e s ins i d e the slash di r e c t o r y . On e of the m , script.sh, ha s an as t e r i s k at the en d . Th i s ind i c a t e s tha t it is ex e c u t a b l e – tha t is, it is a pr o g r a m fil e wh i c h ca n be run, rat h e r tha n sim p l y be i n g a da t a fil e.

No w , wh a t if we wa n t mo r e inf o r m a t i o n on the s e fil e s – for ex a m p l e , wh e n we r e the y las t ch a n g e d ? To wh o m do the y be l o n g ? Wh a t so r t s of thi n g s ca n we do wi t h the m ? Th e ls com m a n d ca n tel l us thi s, too, if

we su p p l y it wi t h the ap p r o p r i a t e flags . Fl a g s ar e ex t r a op t i o n s tha t we pa s s to the pr o g r a m s we us e, to tel l the m to fun c t i o n in a di f f e r e n t wa y fro m the de f a u l t .

Pa g e 10

Wh a t we ne e d he r e is to tel l ls to gi v e us a lon g e r , mo r e de t a i l e d

di r e c t o r y lis t i n g . Th i s is do n e wi t h the –l fla g:

[drysdalk@stimpy drysdalk]$ ls -ltotal 924-rwxr-xr-- 1 drysdalk drysdalk 67584 Mar 27 2006 deprov.sh*-rwxr-xr-- 1 drysdalk drysdalk 27136 Mar 27 2006 easyshare.sh*-rwxr-xr-- 1 drysdalk drysdalk 17408 Jan 14 20:10 dns.sh*drwxrwxr-x 2 drysdalk drysdalk 4096 Feb 15 21:12 slash/[drysdalk@stimpy drysdalk]$ ls -l slashtotal 0-rw------- 1 drysdalk sys 12662 Feb 15 21:12 import.csv-rw------- 1 drysdalk sys 24910 Feb 15 21:11 Report.doc-rwx------ 1 drysdalk sys 82 Feb 15 21:12 script.sh*[drysdalk@stimpy drysdalk]$

No w , he r e we ha v e a lot mo r e inf o r m a t i o n ab o u t the s e fil e s . Le t’ s loo k at wh a t ea c h of the s e ex t r a bi t s of da t a me a n s . We wi l l st a r t at the far-rig h t , an d wo r k ou r wa y al o n g to the lef t.

Th e fir s t thi n g yo u wi l l no t i c e is tha t ea c h fil e no w ha s its ow n de d i c a t e d line in the lis t i n g . Th e far- rig h t co l u m n sh o w s the na m e of ea c h file, jus t as it ap p e a r e d be f o r e .

Th e ne x t pie c e of da t a is the da t e an d/or tim e tha t the fil e wa s las t mo d i f i e d . Fi l e s mo d i f i e d be f o r e the st a r t of the cu r r e n t ye a r do no t sh o w the tim e , an d on l y sh o w the da t e . If a fil e ha s be e n ch a n g e d in the cu r r e n t ye a r , yo u wi l l al s o se e the tim e sta m p in ad d i t i o n to the da t e .

Th e ne x t fie l d co n t a i n s a nu m b e r – thi s is the si z e of the fil e, in by t e s . On e by t e is eq u a l to on e sin g l e ch a r a c t e r (so to st o r e the wo r d ‘byt e’, yo u wo u l d ne e d 4 by t e s).

3.2 File ownerships and permissions

So far, so go o d . It ge t s a lit t l e mo r e com p l i c a t e d now . Th e ne x t tw o

fie l d s tel l us wh i c h user and group ow n the fil e (the us e r is the lef t-ha n d fie l d , the gr o u p is the rig h t- ha n d). Fo r se c u r i t y pu r p o s e s , ev e r y fil e on a Lin u x sy s t e m is ow n e d by on e sin g l e us e r on the sy s t e m , an d is al s o as s o c i a t e d wi t h a gr o u p . Al l us e r s on the sy s t e m ar e the m s e l v e s me m b e r s of at lea s t on e gr o u p . Ac c e s s to fil e s ca n be res t r i c t e d to on l y the ow n e r , or to the ow n e r an d me m b e r s of the gr o u p to wh i c h the fil e al s o be l o n g s .Th e ne x t fie l d is a sin g l e- dig i t nu m b e r for al l the fil e s we’ v e loo k e d at. Fo r no w we’ l l ign o r e thi s – we’ l l di s c u s s it ag a i n mu c h lat e r , bu t in alm o s t

Pa g e 1 1

al l ca s e s yo u’ l l ne v e r ne e d to loo k at thi s fie l d or ha v e an y da y- to-da y us e for it.

Th e fin a l fie l d is al s o pa r t of ho w Lin u x de a l s wi t h fil e se c u r i t y . It sh o w s

the permissions on the fil e or di r e c t o r y in qu e s t i o n . In ad d i t i o n to res t r i c t i n g ac c e s s to fil e s ba s e d on the us e r or gr o u p tha t ow n s the m , it is po s s i b l e to sp e c i f y pr e c i s e l y wh a t the ow n e r us e r an d gr o u p ca n do wi t h the fil e, as we l l as wh a t al l ot h e r us e r s on the sy s t e m wh o ar e no t a me m b e r of the gr o u p ca n do.

Th i s fie l d is ar g u a b l y the mo s t co m p l e x in the lis t i n g , so we wi l l loo k at it in mo r e de t a i l .

H e r e ar e the pe r m i s s i o n s for ea c h of the fil e s in the cu r r e n t di r e c t o r y :

-rw-r-xr--

To ma k e un d e r s t a n d i n g thi s ea s i e r , we wi l l br e a k it in to blo c k s as fol l o w s :

1 2 3- rwx r-x r--

Ig n o r i n g the fir s t of the s e fie l d s (the sin g l e- ch a r a c t e r on e), the s e br e a k do w n as fol l o w s :

G r o u p 1 – Th e pe r m i s s i o n s for the owner of the fil e

G r o u p 2 – Th e pe r m i s s i o n s for the group the fil e is as s o c i a t e d wi t h

G r o u p 3 – Th e pe r m i s s i o n s for everyone else on the sy s t e m

Th e ac t u a l ch a r a c t e r s the m s e l v e s alw a y s ap p e a r in the fol l o w i n g or d e r , an d alw a y s ha v e the fol l o w i n g me a n i n g s :

r Re a d ac c e s s . If thi s is se t , the file ca n be op e n e d for rea d- on l y ac c e s s .

w Wr i t e ac c e s s . If thi s is se t , the file ca n be wr i t t e n to.

x Ex e c u t e ac c e s s . If thi s is se t, an d if the fil e is a va l i d bi n a r y ex e c u t a b l e or scr i p t , it ca n be run as a pr o g r a m .

– A da s h ind i c a t e s tha t the ap p r o p r i a t e pe r m i s s i o n is no t se t – in ot h e r wo r d s , ac c e s s of the sp e c i f i c typ e the ch a r a c t e r re p r e s e n t s is de n i e d for the

Pa g e 12

fil e.

Pa g e 13

So, Gr o u p 1:

rwx

te l l s us tha t for the ow n e r of the fil e rea d pe r m i s s i o n is se t, wr i t e pe r m i s s i o n is se t, an d ex e c u t e pe r m i s s i o n is se t. In ot h e r wo r d s , the fil e’s ow n e r ca n do an y t h i n g the y lik e to it – the y ca n op e n it to se e wh a t is in it, the y ca n ch a n g e its co n t e n t s , an d (so lon g as it is a va l i d , wo r k i n g ex e c u t a b l e fil e) the y ca n run it as a pr o g r a m .

Gr o u p 2 sh o w s us the pe r m i s s i o n s tha t ot h e r me m b e r s of the fil e’s gr o u p ha v e :

r-x

Re a d an d ex e c u t e ac c e s s – so me m b e r s of the gr o u p ca n se e wh a t is in the fil e, an d if it is a va l i d pr o g r a m ca n al s o run it. Th e y ca n n o t ma k e an y ch a n g e s to it, ho w e v e r , si n c e the y do no t ha v e wr i t e ac c e s s .

G r o u p 3, wh i c h de t e r m i n e s the pe r m i s s i o n s for ev e r y o n e el s e , rea d s :

r--

So ev e r y o n e el s e on the sy s t e m ca n rea d the fil e, bu t ca n n o t ex e c u t e it, ev e n if it is a va l i d pr o g r a m . No r ca n the y ma k e an y ch a n g e s to it.

Th e up s h o t of al l th i s, the n , is tha t for al l the fil e s in the cu r r e n t di r e c t o r y (whic h yo u wi l l se e ha v e the sa m e pe r m i s s i o n s), the ow n e r ha s rea d , wr i t e an d ex e c u t e ac c e s s to the m , me m b e r s of the fil e’s gr o u p ca n rea d an d ex e c u t e the m , an d ev e r y o n e el s e ha s rea d- on l y ac c e s s .

No w , thi n g s ar e di f f e r e n t wh e n we loo k at the pe r m i s s i o n s on the co n t e n t s of the slash/ dir e c t o r y . To rec a p the re l e v a n t po r t i o n :

[drysdalk@stimpy drysdalk]$ ls -l slashtotal 0-rw------- 1 drysdalk sys 12662 Feb 15 21:12 import.csv-rw------- 1 drysdalk sys 24910 Feb 15 21:11 Report.doc-rwx------ 1 drysdalk sys 82 Feb 15 21:12 script.sh*[drysdalk@stimpy drysdalk]$

Pa g e 14

Ag a i n ign o r i n g the ve r y fir s t ch a r a c t e r , for import.csv we ha v e :

- rw- --- --—

So, for the fil e’s ow n e r , we ha v e rea d an d wr i t e pe r m i s s i o n ; an d for bo t h the me m b e r s of the fil e’s gr o u p an d for ev e r y o n e el s e , we ha v e no rea d , wr i t e or ex e c u t e ac c e s s .

To sum m a r i s e the pe r m i s s i o n s on import.csv, the fil e’s ow n e r , drysdalk, ha s rea d an d wr i t e ac c e s s . No- one el s e on the sy s t e m ca n do an y t h i n g at al l wi t h it, ev e n if the y ar e a me m b e r of the sys gr o u p .

3.3 A further word about security

Li n u x , lik e al l UN I X- st y l e op e r a t i n g sy s t e m s , ma k e s a cle a r di s t i n c t i o n be t w e e n the lev e l of tru s t gi v e n to reg u l a r us e r s of a sy s t e m , an d the ac c o u n t us e d for sy s t e m s ad m i n i s t r a t i o n . Th i s sp e c i a l ac c o u n t is ca l l e d

the superuser account , an d ha s the us e r n a m e of root . Th e roo t ac c o u n t by p a s s e s al l the se c u r i t y ch e c k s tha t the op e r a t i n g sy s t e m no r m a l l y ca r r i e s ou t to pr e v e n t us e r s fro m mo d i f y i n g pa r t s of the sy s t e m tha t the y ar e no t al l o w e d to.

As yo u mi g h t ima g i n e , the r e ar e ce r t a i n sy s t e m s ad m i n i s t r a t i o n tas k s tha t on l y roo t ca n ca r r y ou t . Ho w e v e r , it is jus t as tru e tha t it is en t i r e l y po s s i b l e to ac c i d e n t a l l y do a gr e a t de a l of da m a g e to a Li n u x sy s t e m if yo u ar e no t ve r y ca r e f u l wh e n log g e d in as roo t . As roo t , yo u ca n lit e r a l l y do an y t h i n g – the r e is no fil e, no dir e c t o r y , no com p o n e n t of the sy s t e m tha t yo u ca n n o t ad d to, ch a n g e , or ev e n rem o v e en t i r e l y .

Th e les s o n to tak e fro m thi s is:

Do not log on as root unless you really, REALLY have to !

Un t i l yo u ar e ve r y com f o r t a b l e wi t h the da y- to-da y us a g e of a Lin u x sy s t e m and ca r r y i n g ou t ba s i c sy s t e m s ad m i n i s t r a t i o n tas k s , it is be s t to av o i d the roo t ac c o u n t al t o g e t h e r .

Ho w e v e r , the r e is a wa y to br i d g e the ga p be t w e e n reg u l a r us e r ac c o u n t s an d the su p e r u s e r ac c o u n t . Th e r e is a Lin u x pr o g r a m tha t al l o w s yo u to log on wi t h yo u r ow n us e r ac c o u n t , an d on l y us e the el e v a t e d pr i v i l e g e s of the su p e r u s e r ac c o u n t wh e n yo u wa n t to.

Pa g e 15

3.4 sudo

Th e pr o g r a m tha t ma k e s thi s po s s i b l e is ca l l e d sudo . It al l o w s yo u to ca r r y ou t ce r t a i n sp e c i f i c ta s k s as the su p e r u s e r . Th e ide a is tha t yo u sh o u l d sp e n d alm o s t al l yo u r tim e log g e d in wi t h the rig h t s of a reg u l a r us e r , an d on l y ma k e us e of the pr i v i l e g e s of the roo t ac c o u n t wh e n the r e is a pa r t i c u l a r com m a n d yo u ne e d to us e tha t req u i r e s the m . Th i s gr e a t l y red u c e s the ch a n c e s of yo u ac c i d e n t a l l y do i n g so m e t h i n g tha t co u l d da m a g e the sy s t e m , or ha v e co n s e q u e n c e s tha t yo u di d no t int e n d .

Fo r ex a m p l e : a wh i l e ag o , an o t h e r me m b e r of the sy s t e m s ad m i n i s t r a t i o n tea m co p i e d a fil e in to yo u r ho m e di r e c t o r y on stimpy. Yo u’ r e do n e wi t h it no w , an d no lon g e r ne e d it, so yo u de c i d e to rem o v e it. Th e com m a n d for de l e t i n g fil e s is rm. Ho w e v e r , the fil e is ow n e d by root, so wh e n yo u try to de l e t e it, the fol l o w i n g ha p p e n s :

[drysdalk@stimpy files]$ ls -ltotal 4-rw-r--r-- 1 root root 170 Feb 15 20:50 file.csv[drysdalk@stimpy files]$ rm file.csvrm: remove write-protected regular file `file.csv'? yrm: cannot remove `file.csv': Permission denied[drysdalk@stimpy files]$

As yo u ca n se e fro m the ou t p u t of ls, the fil e is ow n e d by root, as pr e v i o u s l y me n t i o n e d , an d ev e r y o n e el s e ca n on l y rea d fro m it, an d no t wr i t e . Wh e n yo u ar e log g e d on as a reg u l a r us e r , an d yo u try to rem o v e a fil e tha t is no t ow n e d by yo u an d wh i c h yo u do no t ha v e wr i t e pe r m i s s i o n to, yo u ca n n o t – thi s is the wh o l e po i n t of fil e pe r m i s s i o n s .

S o, in or d e r to rem o v e thi s fil e, we wi l l ne e d to ma k e us e of the sp e c i a l pr i v i l e g e s as s o c i a t e d wi t h the root ac c o u n t by us i n g sudo to run rm.

[drysdalk@stimpy files]$ sudo rm file.csv[drysdalk@stimpy files]$ ls -ltotal 0[drysdalk@stimpy files]$

As yo u ca n se e, the fil e is now go n e . Wh e n rm wa s run via sudo, it wa s run as root, an d so wa s ab l e to rem o v e the fil e tha t yo u di d no t ha v e ac c e s s to yo u r s e l f .

Pa g e 16

3.5 The contents of the root directory

As yo u mi g h t rec a l l , al l di r e c t o r i e s on a Li n u x sy s t e m ar e loc a t e d un d e r n e a t h the roo t dir e c t o r y . Th i s is rep r e s e n t e d by a sin g l e / ch a r a c t e r . So if we wa n t to se e the co n t e n t s of the roo t di r e c t o r y , we ca n typ e :

[drysdalk@stimpy drysdalk]$ ls /bin/ dev/ home/ lib/ misc/ opt/ root/ tmp/ var/boot/ etc/ initrd/ lost+found/ mnt/ proc/ sbin/ usr/[drysdalk@stimpy drysdalk]$

He r e we se e ma n y ot h e r di r e c t o r i e s loc a t e d wi t h i n the roo t di r e c t o r y , wh i c h is the ve r y hig h e s t- lev e l di r e c t o r y on the sy s t e m . Th e ma j o r i t y of the s e dir e c t o r i e s ar e com m o n to al l Li n u x an d UN I X- st y l e sy s t e m s , an d ha v e the i r ow n pa r t i c u l a r us e. He r e ar e the int e n d e d pu r p o s e s of the mo s t imp o r t a n t of the m :

( Oh, an d do n’ t be co n f u s e d by the di r e c t o r y ca l l e d root/ un d e r n e a t h the / di r e c t o r y its e l f – thi s is sim p l y a su b- di r e c t o r y tha t ha p p e n s to ha v e the na m e root/, an d is no t the roo t dir e c t o r y , wh i c h is alw a y s rep r e s e n t e d so l e l y by / )

bin/Us e d for st o r i n g the pr o g r a m s yo u wi l l com m o n l y wa n t to us e – thi s is wh e r e thi n g s lik e ls and mv liv e.boot/Al l the cr i t i c a l fil e s tha t a Lin u x se r v e r ne e d s to st a r t up ar e loc a t e d in he r e .

dev/In s i d e he r e ar e fil e s tha t al l o w the sy s t e m to ac c e s s the ha r d w a r e of the ph y s i c a l se r v e r – the ha r d dr i v e s , US B de v i c e s , sy s t e m me m o r y , an d so on.

etc/Th i s is wh e r e al l the ke y co n f i g u r a t i o n fil e s for the sy s t e m ar e st o r e d .

home/Ea c h us e r on a Lin u x sy s t e m ha s wh a t is ca l l e d the i r ow n home directory . Th i s is a dir e c t o r y tha t is as s o c i a t e d wi t h the m sp e c i f i c a l l y , an d is the di r e c t o r y tha t yo u sta r t of f in wh e n yo u fir s t log in. /home is the de f a u l t loc a t i o n un d e r n e a t h wh i c h the s e ho m e dir e c t o r i e s ar e loc a t e d .

Pa g e 17

lib/Th i s is wh e r e sy s t e m lib r a r y fil e s ar e ke p t . Th e s e ar e bi n a r y , no n-hu m a n- rea d a b l e fil e s tha t st o r e inf o r m a t i o n tha t the res t of the pr o g r a m s on the sy s t e m ca n ma k e us e of. Th e y co n t a i n inf o r m a t i o n ab o u t ho w to ca r r y ou t co m m o n tas k s , so tha t wh e n so f t w a r e de v e l o p e r s ar e wr i t i n g pr o g r a m s the y do no t ha v e to wr i t e rou t i n e s to tel l the i r pr o g r a m s ho w to, for ex a m p l e , wr i t e fil e s to di s k or to pr i n t to the sc r e e n . Ra t h e r , the y ca n ca l l fun c t i o n s for the s e pu r p o s e s co n t a i n e d wi t h i n the sy s t e m libr a r i e s . Th i s ke e p s the siz e s of the i r ow n pr o g r a m s do w n , an d ma k e s up d a t i n g the s e fun c t i o n s ac r o s s al l pr o g r a m s ea s i e r .

proc/In f o r m a t i o n ab o u t the run n i n g sy s t e m ca n be fou n d in he r e – tho u g h no t al w a y s in a hu m a n- rea d a b l e fas h i o n . Sy s t e m ut i l i t y pr o g r a m s us e the inf o r m a t i o n in he r e to fin d ou t ho w mu c h me m o r y is av a i l a b l e , wh a t ha r d w a r e is in us e by the sy s t e m , how the di s c dr i v e s in the sy s t e m ar e be i n g ac c e s s e d , an d so on.

sbin/Th i s , lik e /bin, is an o t h e r dir e c t o r y wi t h i n wh i c h co m m a n d- lin e pr o g r a m s ar e loc a t e d . Tr a d i t i o n a l l y , ho w e v e r , thi s wo u l d be us e d for tho s e pr o g r a m s tha t wo u l d be run by root on l y , or sp e c i f i c a l l y for sy s t e m s ad m i n i s t r a t i o n pu r p o s e s . Th i s line ha s be e n blu r r e d som e w h a t in rec e n t ye a r s an d the r e is now qu i t e a bi t of cr o s s o v e r be t w e e n /bin and /sbin, bu t tha t’ s the ide a be h i n d it.

tmp/Te m p o r a r y fil e s ar e ke p t in he r e . Th e pe r m i s s i o n s on /tmp ar e su c h tha t an y o n e ca n cr e a t e an y t h i n g the y lik e in it, bu t the y ca n on l y rem o v e item s tha t the y cr e a t e d . It is me a n t to be a dir e c t o r y tha t pr o g r a m s ca n sa f e l y wr i t e to wi t h o u t pe r m i s s i o n s ge t t i n g in the wa y , if the y ne e d to cr e a t e fil e s in the ba c k g r o u n d in the co u r s e of do i n g wh a t e v e r the sy s t e m ’ s us e r s req u i r e of the m .

usr/Us e r- ins t a l l e d pr o g r a m s tha t ar e no t a co r e pa r t of the sy s t e m ar e ins t a l l e d he r e . Us u a l l y the bin a r y co m p o n e n t s of so f t w a r e su c h as We b se r v e r s , ma i l se r v e r s , or an y ot h e r thi r d- pa r t y pr o g r a m tha t is no t dir e c t l y pa r t of the op e r a t i n g sy s t e m its e l f wi l l be ins t a l l e d he r e .

Pa g e 18

var//var wa s or i g i n a l l y int e n d e d to st o r e tho s e file s wh i c h wo u l d be va r i a b l e – tha t is, wh o s e co n t e n t s or ex i s t e n c e wo u l d ch a n g e du r i n g the run n i n g of the sy s t e m . Mo s t of t e n it is us e d for st o r i n g inc o m i n g e-ma i l for ma i l se r v e r s , We b si t e co n t e n t for We b se r v e r s , pr i n t job s for de s k t o p sy s t e m s , an d sp e c i a l tem p o r a r y fil e s tha t ar e us e d to de t e r m i n e wh a t pr o g r a m s ar e run n i n g at an y gi v e n tim e on the sy s t e m .

So, tha t co v e r s wh a t ea c h of the s e dir e c t o r i e s be n e a t h / ar e for. No w if we wa n t to loo k in on e of the m , we ca n ag a i n us e ls:

[drysdalk@stimpy drysdalk]$ ls /homeadamh/ aroberts/ colinn/ derek/ iainr/ patc/ richard/agoodall/ ben/ colinw/ drysdalk/ ian/ paulj/ tomd/anne/ callum/ davidh/ graemep/ markd/ rcassidy/[drysdalk@stimpy drysdalk]$

He r e we se e the ho m e di r e c t o r i e s for ea c h of the loc a l us e r s se t up on the sy s t e m .

3.6 Changing and displaying the current directory

We ha v e fai r l y we l l co v e r e d ho w to se e wh a t is in an y dir e c t o r y by thi s po i n t , bu t wh a t if we wa n t to ac t u a l l y mo v e fro m on e dir e c t o r y to an o t h e r ? Th i s is ac h i e v e d wi t h the cd com m a n d . Si m i l a r l y , wh a t if we wa n t to kn o w wh a t dir e c t o r y we ar e in ? Th i s is do n e wi t h the pwd com m a n d .

In the fol l o w i n g ex a m p l e , we ch a n g e to the roo t dir e c t o r y , an d tak e a di r e c t o r y lis t i n g . We the n lis t the co n t e n t s of /tmp, ch a n g e in to a di r e c t o r y we fin d wi t h i n it, an d pr i n t ou r cu r r e n t loc a t i o n .

[drysdalk@stimpy drysdalk]$ cd /[drysdalk@stimpy /]$ lsbin/ dev/ home/ lib/ misc/ opt/ root/ tmp/ var/boot/ etc/ initrd/ lost+found/ mnt/ proc/ sbin/ usr/[drysdalk@stimpy /]$ ls tmpfoo/ lost+found/[drysdalk@stimpy /]$ cd /tmp/foo[drysdalk@stimpy foo]$ pwd/tmp/foo[drysdalk@stimpy foo]$

Pa g e 19

Wh y di d we ha v e to sp e c i f y tha t we wa n t e d to ex p l i c i t l y ch a n g e to /tmp/foo wh e n we we r e alr e a d y in the roo t di r e c t o r y - co u l d n ’ t we ha v e om i t t e d the ini t i a l /, si n c e we we r e al r e a d y in the roo t di r e c t o r y ? Co u l d we ha v e ch a n g e d in to tmp, an d the n in to foo ? Th e an s w e r in ea c h cas e is ye s:

[drysdalk@stimpy drysdalk]$ cd /[drysdalk@stimpy /]$ ls tmpfoo/ lost+found/[drysdalk@stimpy /]$ cd tmp/foo[drysdalk@stimpy foo]$ pwd/tmp/foo[drysdalk@stimpy foo]$

[drysdalk@stimpy drysdalk]$ cd /[drysdalk@stimpy /]$ ls tmpfoo/ lost+found/[drysdalk@stimpy /]$ cd tmp[drysdalk@stimpy tmp]$ cd foo[drysdalk@stimpy foo]$ pwd/tmp/foo[drysdalk@stimpy foo]$

Yo u ca n mo v e ba c k to yo u r ho m e di r e c t o r y at an y tim e by typ i n g cd on its ow n :

[drysdalk@stimpy drysdalk]$ pwd/home/drysdalk[drysdalk@stimpy drysdalk]$ cd /tmp[drysdalk@stimpy tmp]$ pwd/tmp[drysdalk@stimpy tmp]$ cd[drysdalk@stimpy drysdalk]$ pwd/home/drysdalk[drysdalk@stimpy drysdalk]$

Yo u ca n al s o ret u r n ho m e by typ i n g cd ~ (cd fol l o w e d by a til d e sy m b o l).

Al s o , at al l tim e s , the dir e c t o r y tha t is on e ‘up’ in the tre e fro m yo u r pr e s e n t loc a t i o n is rep r e s e n t e d by tw o do t s . So if yo u wa n t e d to mo v e up to the roo t di r e c t o r y ins t e a d of ret u r n i n g ho m e , yo u co u l d typ e :

[drysdalk@stimpy drysdalk]$ pwd/home/drysdalk[drysdalk@stimpy drysdalk]$ cd /tmp[drysdalk@stimpy tmp]$ pwd/tmp[drysdalk@stimpy tmp]$ cd ..[drysdalk@stimpy /]$ pwd/

Pa g e 20

[drysdalk@stimpy /]$

So now we kn o w how to mo v e fro m on e di r e c t o r y to an o t h e r , to lis t the co n t e n t s of di r e c t o r i e s , an d to ch e c k wh e r e we ar e at an y gi v e n tim e .

3.7 Copying a file from one place to another

To co p y a fil e to a ne w loc a t i o n , we us e the cp co m m a n d . In the fol l o w i n g ex a m p l e , we co p y the file tes t . t x t in to the fil e/ di r e c t o r y , wh e r e bo t h ar e loc a t e d in ou r ho m e di r e c t o r y :

[drysdalk@stimpy drysdalk]$ lsfiles/ file.txt[drysdalk@stimpy drysdalk]$ cp file.txt files[drysdalk@stimpy drysdalk]$ lsfiles/ file.txt[drysdalk@stimpy drysdalk]$ ls filesfile.txt[drysdalk@stimpy drysdalk]$

3.8 Moving a file from one place to another

To mo v e a file fro m on e dir e c t o r y to an o t h e r , we ca n us e the mv com m a n d . In the fol l o w i n g ex a m p l e , we mo v e the file test.txt in to the files/ di r e c t o r y , wh e r e bo t h ar e loc a t e d in ou r ho m e dir e c t o r y :

[drysdalk@stimpy drysdalk]$ lsfiles/ file.txt[drysdalk@stimpy drysdalk]$ mv file.txt files[drysdalk@stimpy drysdalk]$ lsfiles/[drysdalk@stimpy drysdalk]$ ls filesfile.txt[drysdalk@stimpy drysdalk]$

Th i s is al s o ho w we re-na m e fil e s an d di r e c t o r i e s in Li n u x (if yo u thi n k ab o u t it, re-na m i n g a fil e is jus t ‘mo v i n g ’ it fro m its ol d na m e to its new na m e):

[drysdalk@stimpy drysdalk]$ lsfile.txt[drysdalk@stimpy drysdalk]$ mv file.txt newname.txt[drysdalk@stimpy drysdalk]$ lsnewname.txt[drysdalk@stimpy drysdalk]$

3.9 Removing a file

Pa g e 21

An d as pr e v i o u s l y se e n , we ca n rem o v e fil e s wi t h the rm co m m a n d :

[drysdalk@stimpy drysdalk]$ lsfiles/ file.txt[drysdalk@stimpy drysdalk]$ rm file.txt[drysdalk@stimpy drysdalk]$ lsfiles/[drysdalk@stimpy drysdalk]$

3.10 Creating a new directory

We can cr e a t e new di r e c t o r i e s wi t h the mkdir com m a n d :

[drysdalk@stimpy test]$ lsfiles/[drysdalk@stimpy test]$ mkdir more_files[drysdalk@stimpy test]$ lsfiles/ more_files/[drysdalk@stimpy test]$

3.11 Removing a directory

An d we ca n rem o v e al r e a d y- em p t y dir e c t o r i e s wi t h rmdir:

[drysdalk@stimpy drysdalk]$ lsfiles/ more_files/[drysdalk@stimpy drysdalk]$ rmdir more_files[drysdalk@stimpy drysdalk]$ lsfiles/[drysdalk@stimpy drysdalk]$No w wh a t if a di r e c t o r y is no t alr e a d y em p t y ? In tha t ca s e , wh e n we try to rem o v e it we rec e i v e the fol l o w i n g er r o r :

[drysdalk@stimpy drysdalk]$ ls filesimport.csv[drysdalk@stimpy drysdalk]$ rmdir filesrmdir: `foo': Directory not empty[drysdalk@stimpy drysdalk]$

So we ne e d to ma n u a l l y rem o v e the co n t e n t s of files/ be f o r e we ca n rem o v e the di r e c t o r y its e l f :

[drysdalk@stimpy drysdalk]$ lsfiles/ more_files/[drysdalk@stimpy drysdalk]$ ls filesimport.csv[drysdalk@stimpy drysdalk]$ cd files[drysdalk@stimpy files]$ rm import.csv[drysdalk@stimpy files]$ cd ..

Pa g e 22

[drysdalk@stimpy drysdalk]$ rmdir files[drysdalk@stimpy drysdalk]$ lsmore_files/[drysdalk@stimpy drysdalk]$

3.12 Viewing the contents of a file

Yo u ca n vie w the co n t e n t s of an y fil e on the sy s t e m tha t yo u ha v e rea d ac c e s s to wi t h the cat co m m a n d :

[drysdalk@ioprov2 drysdalk]$ lsfile.txt[drysdalk@ioprov2 drysdalk]$ cat file.txtThis is the text inside file.txt[drysdalk@ioprov2 drysdalk]$

Pa g e 23

3.13 End-of-section summary

So, tha t’ s the ve r y ba s i c s . To rec a p , in thi s se c t i o n we ha v e lea r n e d ho w to:

- Li s t the co n t e n t s of a dir e c t o r y- Se e an d un d e r s t a n d file ow n e r s h i p s an d pe r m i s s i o n s

- Not to use the root account unless you really, REALLY have to !

- Us e sudo to sa f e l y run com m a n d s as root ins t e a d- Se e the co n t e n t s of the roo t dir e c t o r y , an d un d e r s t a n d the i r

pu r p o s e- Ch a n g e fro m one di r e c t o r y to an o t h e r- Co p y a fil e fro m on e pl a c e to an o t h e r- Mo v e a fil e or dir e c t o r y to a ne w loc a t i o n- Re- na m e a fil e or di r e c t o r y- Re m o v e a fil e- Cr e a t e a new di r e c t o r y- Re m o v e an al r e a d y- em p t y di r e c t o r y- Vi e w the co n t e n t s of a fil e

Pa g e 24

4. Further concepts and commandsIn thi s se c t i o n , we wi l l ex p a n d on wh a t we cov e r e d in Se c t i o n 3. Yo u wi l l lea r n how to wo r k wi t h mu l t i p l e fil e s an d di r e c t o r i e s at on c e , ho w to se t the pe r m i s s i o n s , ow n e r an d gr o u p on fil e s an d di r e c t o r i e s , an d mo r e be s i d e s .

4.1 Working with multiple files and directories simultaneously

At tim e s , it is co n v e n i e n t to be ab l e to ref e r to a wh o l e gr o u p of fil e s in on e go, pa r t i c u l a r l y if the r e is on e sp e c i f i c thi n g tha t yo u ne e d to do to ev e r y file in a dir e c t o r y , or to ev e r y file wi t h ce r t a i n ch a r a c t e r s in its na m e .

Li n u x pr o v i d e s a wa y for yo u to do thi s vi a the us e of wildcards . A wi l d c a r d is a sp e c i a l sym b o l us e d at the co m m a n d line to rep r e s e n t on e or mo r e ch a r a c t e r s , wi t h o u t ha v i n g to sp e c i f y in ad v a n c e ex a c t l y wh a t tho s e ch a r a c t e r s sh o u l d be.

Fo r ex a m p l e – in yo u r ho m e dir e c t o r y , yo u ha v e the fol l o w i n g fil e s , an d yo u ne e d to rem o v e the m al l:

[drysdalk@ioprov2 drysdalk]$ lsfile file10 file12 file14 file2 file4 file6 file8file1 file11 file13 file15 file3 file5 file7 file9[drysdalk@ioprov2 drysdalk]$

No w , yo u co u l d rem o v e ea c h of the s e fil e s in tur n wi t h the rm com m a n d , bu t thi s wo u l d be a ve r y lon g an d clu n k y com m a n d to typ e. Ins t e a d , we ca n us e a wi l d c a r d to rem o v e al l the s e fil e s at on c e .

In Lin u x , the wi l d c a r d yo u wi l l us e the mo s t of t e n is * (an as t e r i s k). Th e as t e r i s k in al l UN I X- st y l e op e r a t i n g sy s t e m s ma t c h e s ze r o or mo r e ch a r a c t e r s .

S o, to rem o v e al l the ch a r a c t e r s in ou r ho m e di r e c t o r y , we co u l d typ e:

[drysdalk@ioprov2 drysdalk]$ lsfile file10 file12 file14 file2 file4 file6 file8file1 file11 file13 file15 file3 file5 file7 file9

Pa g e 25

[drysdalk@ioprov2 drysdalk]$ rm *[drysdalk@ioprov2 drysdalk]$ ls[drysdalk@ioprov2 drysdalk]$

An d as yo u ca n se e, the y ar e now al l go n e .

Pa g e 26

No w , ima g i n e a mo r e co m p l i c a t e d sc e n a r i o . We ha v e the fol l o w i n g file s in ou r ho m e di r e c t o r y :

[drysdalk@ioprov2 drysdalk]$ lsfile1.txt file2.txt file3.txt file4.txt file5.txt script1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$

We are do n e wi t h ea c h of the tex t fil e s , bu t we wa n t to ke e p the sh e l l sc r i p t s script1.sh and script2.sh.

It is po s s i b l e to sp e c i f y a se t of ch a r a c t e r s ei t h e r be f o r e or af t e r a wi l d c a r d to qu a l i f y it fur t h e r . Fo r ex a m p l e , in thi s ca s e , we co u l d sp e c i f y file* to rem o v e al l fil e s tha t be g a n wi t h the wo r d ‘file’, an d the n ha d ze r o or mo r e fur t h e r ch a r a c t e r s in the i r na m e fro m tha t po i n t on w a r d .

Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ lsfile1.txt file2.txt file3.txt file4.txt file5.txt script1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$ rm file*[drysdalk@ioprov2 drysdalk]$ lsscript1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$

An d no w the tex t fil e s ar e go n e , bu t the sh e l l sc r i p t s sa f e l y rem a i n .

G o i n g ba c k to ou r fir s t ex a m p l e , let’ s sa y we had on l y wa n t e d to rem o v e al l the fil e s from file10 up w a r d s . We co u l d ha v e do n e so in the fol l o w i n g wa y :

[drysdalk@ioprov2 drysdalk]$ lsfile file11 file13 file15 file3 file5 file7 file9file10 file12 file14 file2 file4 file6 file8[drysdalk@ioprov2 drysdalk]$ rm file1*[drysdalk@ioprov2 drysdalk]$ lsfile file2 file3 file4 file5 file6 file7 file8 file9

So we ar e lef t wi t h al l fil e s ex c e p t tho s e wh o s e na m e s be g i n wi t h file1.

Pa g e 27

Wi l d c a r d s ca n al s o co m e at the st a r t of a fil e n a m e , al l o w i n g yo u to sp e c i f y tha t yo u wa n t to wo r k wi t h al l fil e s wh o s e na m e ha s a ce r t a i n en d i n g , reg a r d l e s s of the i r be g i n n i n g . So ag a i n in ou r ex a m p l e of wa n t i n g to ke e p the sh e l l sc r i p t s bu t di s c a r d ev e r y t h i n g els e , we co u l d ha v e typ e d :

[drysdalk@ioprov2 drysdalk]$ lsfile1.txt file2.txt file3.txt file4.txt file5.txt script1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$ rm *txt[drysdalk@ioprov2 drysdalk]$ lsscript1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$

An d the res u l t is the sam e .

In fac t , wi l d c a r d s ca n go an y w h e r e in a fil e n a m e at al l:

[drysdalk@ioprov2 drysdalk]$ lsfile1.txt file2.txt file3.txt file4.txt file5.txt script1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$ rm fi*txt[drysdalk@ioprov2 drysdalk]$ lsscript1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$

On c e ag a i n , the en d res u l t is ide n t i c a l .

W i l d c a r d s ca n wo r k wi t h an y com m a n d at al l, too. Le t’ s sa y we wa n t e d a lon g dir e c t o r y lis t i n g , bu t on l y for the sh e l l sc r i p t s an d no t the tex t fil e s :

[drysdalk@ioprov2 drysdalk]$ lsfile1.txt file3.txt file5.txt script2.sh*file2.txt file4.txt script1.sh* script3.sh[drysdalk@ioprov2 drysdalk]$ ls -l script*-rwxr-xr-x 1 drysdalk drysdalk 54 Feb 16 14:45 script1.sh*-rwxr-xr-x 1 drysdalk drysdalk 24 Feb 16 14:45 script2.sh*-rw-rw-r-- 1 drysdalk drysdalk 189 Feb 16 14:45 script3.sh[drysdalk@ioprov2 drysdalk]$

We on l y sa w a lon g dir e c t o r y lis t i n g for the sh e l l sc r i p t s an d no t the file s , be c a u s e the us e of the qu a l i f i e d wi l d c a r d on l y ma t c h e d the m , an d no t the tex t fil e s .

Th e r e is on e ot h e r wi l d c a r d tha t yo u mi g h t som e t i m e s ha v e to us e, tho u g h no t qu i t e as of t e n as the as t e r i s k . Th a t ’ s ? (a qu e s t i o n ma r k). A qu e s t i o n ma r k ma t c h e s on e single ch a r a c t e r , un l i k e the as t e r i s k , wh i c h ma t c h e s fro m ze r o to an y nu m b e r of ch a r a c t e r s .

Pa g e 28

Le t’ s loo k at ou r ex a m p l e of fif t e e n fil e s in a dir e c t o r y on c e ag a i n . Wh a t if we wa n t e d to on l y rem o v e the fil e s tha t ha d a sin g l e- di g i t nu m b e r in the i r na m e , an d lea v e the do u b l e- dig i t s ? We co u l d n ’ t do rm file*, be c a u s e tha t wo u l d rem o v e al l of the m . So ins t e a d , we ca n do thi s:

[drysdalk@ioprov2 drysdalk]$ lsfile1 file11 file13 file15 file3 file5 file7 file9file10 file12 file14 file2 file4 file6 file8[drysdalk@ioprov2 drysdalk]$ rm file?[drysdalk@ioprov2 drysdalk]$ lsfile10 file11 file12 file13 file14 file15[drysdalk@ioprov2 drysdalk]$

Th i s gi v e s the de s i r e d res u l t , an d lea v e s us wi t h on l y wh a t we wa n t e d .

No w wh a t if we ha d wa n t e d to do the op p o s i t e – to on l y rem o v e the do u b l e- di g i t fil e s ? Yo u r fir s t tho u g h t mi g h t be to do rm file1*, bu t thi s wo u l d al s o rem o v e file1 its e l f , si n c e an as t e r i s k ma t c h e s ze r o or mo r e ch a r a c t e r s . So ins t e a d :

[drysdalk@ioprov2 drysdalk]$ lsfile1 file11 file13 file15 file3 file5 file7 file9file10 file12 file14 file2 file4 file6 file8[drysdalk@ioprov2 drysdalk]$ rm file??[drysdalk@ioprov2 drysdalk]$ lsfile1 file2 file3 file4 file5 file6 file7 file8 file9[drysdalk@ioprov2 drysdalk]$

So as yo u ca n se e, we ca n com b i n e ? wi l d c a r d s to ma t c h a se q u e n c e of si n g l e- ch a r a c t e r s in a row.

Fi n a l l y , we ca n co m b i n e bo t h typ e s of wi l d c a r d s in on e fai r l y com p l e x com m a n d . Ima g i n e yo u ha v e the fol l o w i n g fil e s in yo u r ho m e di r e c t o r y :

[drysdalk@ioprov2 drysdalk]$ lsfile-aa-keep file-a-keep file-bb-keep file-b-keepfile-aa-wipe file-a-wipe file-bb-wipe file-b-wipe[drysdalk@ioprov2 drysdalk]$

If we wa n t e d to er a s e al l fil e s tha t ha d pr e c i s e l y tw o let t e r s in the mi d d l e of the file n a m e , an d wh o s e na m e en d e d wi t h wi p e , we co u l d do thi s:

[drysdalk@ioprov2 drysdalk]$ lsfile-aa-keep file-a-keep file-bb-keep file-b-keepfile-aa-wipe file-a-wipe file-bb-wipe file-b-wipe[drysdalk@ioprov2 drysdalk]$ rm file-??-w*[drysdalk@ioprov2 drysdalk]$ ls

Pa g e 29

file-aa-keep file-a-keep file-a-wipe file-bb-keep file-b-keep file-b-wipe[drysdalk@ioprov2 drysdalk]$

As yo u ca n se e, thi s ha s lef t us wi t h al l do u b l e- let t e r fil e s wh o s e na m e s en d e d wi t h ‘kee p ’, an d ha s ha d no ef f e c t on sin g l e- let t e r fil e s at al l.

4.2 A cautionary word

As yo u ha v e se e n , wi l d c a r d s ca n be ve r y us e f u l . Ho w e v e r , as yo u ma y

ha v e co m e to rea l i s e , the y ca n al s o be extremely dangerous ! If yo u ar e no t mo s t ca r e f u l , it is very ea s y to rem o v e or ot h e r w i s e af f e c t a fil e or di r e c t o r y tha t yo u di d no t int e n d to. Th e nu m b e r- on e ru l e wh e n wo r k i n g wi t h wi l d c a r d s is:

Stop, check, THINK, and check again !

Sp e c i f i c a l l y , yo u mi g h t wa n t to as k yo u r s e l f the fol l o w i n g be f o r e run n i n g an y co m m a n d inv o l v i n g wi l d c a r d s :

• Wo u l d the wi l d c a r d ac t u a l l y wo r k , an d ca t c h al l the fil e s or di r e c t o r i e s I wa n t ?• Am I su r e tha t no ot h e r fil e s or di r e c t o r i e s wi l l be af f e c t e d by wh a t I’m ab o u t to

do ?• Am I typ i n g the rig h t co m m a n d – e.g. no t mi s t a k e n l y typ i n g rm ins t e a d of mv ?• Ha v e I actually typ e d wh a t I think I’ve typ e d ?

On tha t las t top i c , he r e’ s a cla s s i c m i s t a k e :

[drysdalk@ioprov2 drysdalk]$ lsfile1.txt file2.txt file3.txt file4.txt file5.txt script1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$ rm file *rm: cannot lstat `file': No such file or directory[drysdalk@ioprov2 drysdalk]$ ls[drysdalk@ioprov2 drysdalk]$

Wh a t we n t wr o n g ? Wh a t ha p p e n e d wa s tha t the r e wa s a sp a c e be t w e e n the file and the * in the ar g u m e n t s pa s s e d to the rm com m a n d . rm, lik e mo s t Lin u x com m a n d s , is ca p a b l e of tak i n g mu l t i p l e ar g u m e n t s on the com m a n d lin e at on c e . So, rm fir s t loo k e d for a file sim p l y ca l l e d file, an d tri e d to rem o v e it – it di d n’ t fin d on e , be c a u s e no suc h fil e ex i s t s . Th a t ’ s wh y we go t the er r o r :

Pa g e 30

rm: cannot lstat `file': No such file or directory

So, it mo v e d on to the ne x t ar g u m e n t – ou r wi l d c a r d . Th i s , un d e r s t a n d a b l y , ma t c h e d al l fil e s in the di r e c t o r y , an d so al l fil e s we r e irr e v e r s i b l y de l e t e d . Th i s is a big pr o b l e m , sin c e ins t e a d of rem o v i n g file*, wh i c h wa s ou r int e n t i o n , we’ v e wi p e d the en t i r e co n t e n t s of the di r e c t o r y .

Ex t r a sp a c e s be t w e e n ch a r a c t e r s an d wi l d c a r d s ca n sp e l l di s a s t e r . Al w a y s , always dou b l e- ch e c k yo u r sy n t a x wh e n wo r k i n g wi t h wi l d c a r d s ! Yo u mu s t be certain tha t yo u ha v e no t ma d e an y typ i n g er r o r s , or sim p l y do n e the wr o n g thi n g , be f o r e pr e s s i n g ret u r n . Be c a u s e on c e tho s e fil e s ar e go n e , the y ar e no t co m i n g ba c k . Th e r e ’ s no Re c y c l e Bi n or Tr a s h in Li n u x – wh e n thi n g s ar e de l e t e d , the y va n i s h ins t a n t l y an d pe r m a n e n t l y .

To tha t en d, he r e ar e two go o d tec h n i q u e s for av o i d i n g rm wi l d c a r d di s a s t e r s :

1. Be f o r e run n i n g yo u r rm co m m a n d , run an ls wi t h the ex a c t sam e fla g s an d wi l d c a r d . Lo o k at the fil e lis t tha t is ret u r n e d – is it exactly wh a t yo u ex p e c t e d ? Ar e the r e an y fil e s mi s s i n g ? Ar e the r e an y extra fil e s or di r e c t o r i e s ? If so, fix yo u r wi l d c a r d , run an o t h e r ls, an d ch e c k ag a i n . Wh e n yo u ar e su r e it is sa f e , the n yo u ca n pr o c e e d wi t h yo u r rm com m a n d .

2. Th e rm com m a n d its e l f ha s a sa f e t y fea t u r e , the –i fla g. Th i s ma k e s the sy s t e m pr o m p t yo u to co n f i r m tha t yo u rea l l y do wa n t to rem o v e the fil e s yo u ar e try i n g to de l e t e . Th i s is a go o d thi n g wh e n us i n g wi l d c a r d s , as it wi l l ex p a n d yo u r wi l d c a r d ou t to ma t c h the lis t of file s tha t wo u l d be rem o v e d , an d yo u wi l l be as k e d ab o u t ea c h of the m in tur n . Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ lsfile1.txt file2.txt file3.txt file4.txt file5.txt script1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$ rm -i file*rm: remove regular file `file1.txt'? yrm: remove regular file `file2.txt'? yrm: remove regular file `file3.txt'? yrm: remove regular file `file4.txt'? yrm: remove regular file `file5.txt'? y[drysdalk@ioprov2 drysdalk]$ lsscript1.sh* script2.sh*[drysdalk@ioprov2 drysdalk]$

Pa g e 31

4.3 Removing directories without emptying them manually first

In the pr e v i o u s se c t i o n , yo u we r e sh o w n ho w to rem o v e a dir e c t o r y . Ho w e v e r , it wa s po i n t e d ou t tha t yo u ha v e to rem o v e its co n t e n t s ma n u a l l y fir s t . Th i s ca n be a pa i n – pa r t i c u l a r l y if a di r e c t o r y co n t a i n s ma n y su b- dir e c t o r i e s , wh i c h co n t a i n mo r e fil e s the m s e l v e s . Yo u wo u l d the n ha v e to em p t y ou t the s e su b- dir e c t o r i e s be f o r e rem o v i n g the m , so yo u co u l d fin a l l y rem o v e the pa r e n t dir e c t o r y yo u wa n t e d to de l e t e in the fir s t pl a c e .

As yo u mi g h t ex p e c t , the r e is a wa y to rem o v e an en t i r e dir e c t o r y an d al l its co n t e n t s in on e go. Bu t , lik e wi l d c a r d s , thi s is as po t e n t i a l l y

da n g e r o u s as it is us e f u l . Th i s is do n e by us i n g the –r fla g of rm. Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ lsfiles/[drysdalk@ioprov2 drysdalk]$ rmdir filesrmdir: `files': Directory not empty[drysdalk@ioprov2 drysdalk]$ ls filesfile1 file2 file3[drysdalk@ioprov2 drysdalk]$ rm -r files[drysdalk@ioprov2 drysdalk]$ ls[drysdalk@ioprov2 drysdalk]$

In thi s ex a m p l e , the di r e c t o r y files/ co n t a i n e d fol d e r s , an d so ou r rmdir com m a n d did no t al l o w us to rem o v e it. Ho w e v e r , rm –r wa s suc c e s s f u l l y us e d to rem o v e it an d al l its co n t e n t s in on e sin g l e com m a n d .

Yo u ca n al s o us e rm –r wi t h wi l d c a r d s :

[drysdalk@ioprov2 drysdalk]$ lsfiles1/ files2/ files3/[drysdalk@ioprov2 drysdalk]$ rm -r *[drysdalk@ioprov2 drysdalk]$ ls[drysdalk@ioprov2 drysdalk]$

An d al l thr e e of the s e di r e c t o r i e s ar e sim u l t a n e o u s l y rem o v e d .

4.4 Another cautionary word

As if wi l d c a r d s an d rm –r we r e no t po t e n t i a l l y da n g e r o u s en o u g h on the i r ow n , the com b i n a t i o n of the two ca n be de v a s t a t i n g . Th e y ca n

Pa g e 32

sa v e yo u a lot of tim e an d ha s s l e wh e n us e d co r r e c t l y an d sa f e l y , bu t wi t h on e mi s t a k e yo u co u l d er a s e a wh o l e gr o u p of fil e s yo u di d n’ t int e n d to, or ev e n de s t r o y the sy s t e m en t i r e l y if yo u pe r f o r m a sudo rm –r wh i l s t in the roo t dir e c t o r y or in som e ot h e r vit a l loc a t i o n .

O n c e ag a i n , the go l d e n ru l e is:

Stop, check, THINK, and check again !

Wh e n yo u ar e jus t st a r t i n g ou t in the wo r l d of Li n u x , yo u ar e st r o n g l y en c o u r a g e d to av o i d the com b i n a t i o n of rm an d wi l d c a r d s al t o g e t h e r , or to as k a mo r e ex p e r i e n c e d me m b e r of the sy s t e m s ad m i n i s t r a t i o n tea m to ch e c k ov e r the com m a n d yo u ar e ab o u t to run to ma k e su r e it is sa f e .

Ho w e v e r , if yo u ab s o l u t e l y mu s t us e rm –r wi t h a wi l d c a r d , the n as be f o r e the ke y qu e s t i o n s to as k yo u r s e l f ar e:

• Wo u l d the wi l d c a r d ac t u a l l y wo r k , an d ca t c h al l the fil e s or di r e c t o r i e s I wa n t ?• Am I su r e tha t no ot h e r fil e s or di r e c t o r i e s wi l l be af f e c t e d by wh a t I’m ab o u t to

do ?

Pa g e 33

An d , mo s t imp o r t a n t l y of al l wh e r e rm –r and wi l d c a r d s ar e co n c e r n e d :

• Ha v e I actually typ e d wh a t I think I’ve typ e d ? • Am I in the di r e c t o r y I thi n k I am in ?

• Am I de f i n i t e l y , to t a l l y , ab s o l u t e l y certain I am not in the ro o t di r e c t o r y , or /usr, or.. ?

On c e yo u ar e sa t i s f i e d tha t the an s w e r to al l of the s e qu e s t i o n s is ‘yes’, the n – an d only the n – sh o u l d yo u pr o c e e d .

4.5 Searching the contents of files

Som e t i m e s , yo u wi l l ha v e a co l l e c t i o n of fil e s tha t co n t a i n a lar g e am o u n t of inf o r m a t i o n , an d yo u wi l l wa n t to fin d on e sp e c i f i c item co n t a i n e d wi t h i n the m . Lin u x pr o v i d e s a com m a n d lin e ut i l i t y for thi s pu r p o s e , ca l l e d grep.

Fo r ex a m p l e , let’s sa y yo u ha v e a fil e ca l l e d domains.txt, wh i c h co n t a i n s a nu m b e r of do m a i n na m e s . Yo u ha v e be e n gi v e n the tas k of fin d i n g al l do m a i n s co n t a i n i n g the ph r a s e ‘fur b y ’. He r e ’ s ho w we co u l d do thi s, us i n g grep:

[drysdalk@ioprov2 drysdalk]$ ls -ltotal 888-rw-r--r-- 1 drysdalk drysdalk 903719 Feb 16 16:56 domains.txt[drysdalk@ioprov2 drysdalk]$ grep furby domains.txtfurby.comadoptafurby.comfurbyforum.co.uk[drysdalk@ioprov2 drysdalk]$

As yo u ca n se e, thr e e res u l t s we r e ret u r n e d , in the or d e r tha t the y we r e fou n d in the fil e. So grep ca n al l o w yo u to qu i c k l y se a r c h ve r y lar g e file s for sp e c i f i c pi e c e s of inf o r m a t i o n , an d is qu i t e of t e n an es s e n t i a l too l .

Th e grep ut i l i t y ca n al s o pe r f o r m the op p o s i t e fun c t i o n – it ca n sh o w yo u al l line s in a file tha t do not co n t a i n the ph r a s e yo u su p p l y . Th i s is do n e by us i n g the –v ar g u m e n t , wh i c h inv e r t s the de f a u l t be h a v i o u r :

Pa g e 34

[drysdalk@ioprov2 drysdalk]$ lsdomains.txt file.txt[drysdalk@ioprov2 drysdalk]$ cat file.txtiomart.comeasyspace.comfurby.comnicnames.co.ukadoptafurby.comfurbyforum.co.ukinternetters.co.uk[drysdalk@ioprov2 drysdalk]$ grep -v furby file.txtiomart.comeasyspace.comnicnames.co.ukinternetters.co.uk[drysdalk@ioprov2 drysdalk]$

As yo u ca n se e, grep –v di d ind e e d ret u r n al l line s in the file tha t did no t co n t a i n the st r i n g ‘fur b y ’.

4.6 Sorting the contents of files

Th e co n t e n t s of fil e s ca n be so r t e d wi t h the sort com m a n d . Yo u ca n ei t h e r sim p l y ge t the so r t e d co n t e n t s of a fil e to be di s p l a y e d on the com m a n d lin e:

[drysdalk@ioprov2 drysdalk]$ sort file.txtadoptafurby.comeasyspace.comfurby.comfurbyforum.co.ukinternetters.co.ukiomart.comnicnames.co.uk[drysdalk@ioprov2 drysdalk]$

Yo u ca n sp e c i f y a ne w fil e n a m e tha t the so r t e d ve r s i o n sh o u l d be sto r e d in, wi t h the –o ar g u m e n t . Th i s cr e a t e s a ne w co p y of the fil e wi t h al l the sam e inf o r m a t i o n as the or i g i n a l , bu t in al p h a b e t i c a l l y- so r t e d or d e r . Fo r ex a m p l e :

Pa g e 35

[drysdalk@ioprov2 drysdalk]$ sort -o file-sorted.txt file.txt[drysdalk@ioprov2 drysdalk]$ cat file-sorted.txtadoptafurby.comeasyspace.comfurby.comfurbyforum.co.ukinternetters.co.ukiomart.comnicnames.co.uk[drysdalk@ioprov2 drysdalk]$

4.7 Finding a file

Le t’ s sa y yo u wa n t to fin d a sp e c i f i c file tha t yo u kn o w is so m e w h e r e on the sy s t e m , bu t yo u ca n no lon g e r rem e m b e r wh e r e . Th e Lin u x com m a n d fo r se a r c h i n g fo r a fil e is ca l l e d find.

It’s a bi t mo r e inv o l v e d to us e tha n grep or sort, bu t on c e yo u ha v e us e d it a few tim e s yo u wi l l qu i c k l y ge t us e d to it. It’s we l l- wo r t h pe r s e v e r i n g wi t h , si n c e it ca n be an inv a l u a b l e too l .

Th e ba s i c sy n t a x of find br e a k s do w n as fol l o w s :

find <starting location> -name <filename to search for>

Fo r ex a m p l e , let’s sa y yo u ha v e a nu m b e r of di r e c t o r i e s , ea c h of wh i c h co n t a i n s a qu a n t i t y of fil e s . On e of the s e fil e s is a lis t of pa s s w o r d s tha t yo u ne e d , an d yo u rem e m b e r tha t yo u ca l l e d it passwords.txt. Ho w e v e r , yo u ca n’ t rem e m b e r wh e r e yo u pu t it. So, yo u ca n fin d it lik e thi s:

[drysdalk@ioprov2 drysdalk]$ lsfiles1 files2 files3 [drysdalk@ioprov2 drysdalk]$ find ~ -name passwords.txt/home/drysdalk/files1/passwords.txt[drysdalk@ioprov2 drysdalk]$

Re m e m b e r tha t ~ rep r e s e n t s yo u r ho m e di r e c t o r y – so we tol d find to st a r t its se a r c h in tha t loc a t i o n . Yo u’ l l al s o no t i c e tha t find ret u r n e d the com p l e t e pa t h to the fil e we we r e loo k i n g for, st a r t i n g fro m the roo t di r e c t o r y an d wo r k i n g its wa y do w n .

W e ca n al s o se a r c h for fil e s st a r t i n g wi t h the roo t dir e c t o r y its e l f . Le t’ s sa y yo u wa n t to kn o w wh e r e the bi n a r y fil e for the co m m a n d ls is loc a t e d :

Pa g e 36

[drysdalk@ioprov2 drysdalk]$ sudo find / -name ls/bin/ls[drysdalk@ioprov2 drysdalk]$

Th i s ma k e s se n s e , an d is wh e r e we wo u l d ex p e c t to fin d ls and ot h e r com m o n co m m a n d s – yo u wi l l re c a l l fro m the pr e v i o u s se c t i o n tha t /bin is us e d fo r st o r i n g tho s e sy s t e m bin a r i e s tha t ar e int e n d e d for reg u l a r us e r s to run.

Bu t wh y di d we us e sudo to run find ? Re m e m b e r tha t wh e n run n i n g as a reg u l a r us e r , yo u wi l l on l y be ab l e to se a r c h or ot h e r w i s e ac c e s s di r e c t o r i e s wh o s e pe r m i s s i o n s sp e c i f y yo u ha v e ac c e s s to. Mo s t of the imp o r t a n t pa r t s of the sy s t e m ar e ow n e d by root an d ar e loc k e d do w n to tha t us e r on l y , so to be su r e tha t we fin d som e t h i n g wh e n se a r c h i n g the en t i r e sy s t e m we nee d su p e r u s e r pr i v i l e g e s . So sudo had to be us e d he r e , or els e we wo u l d ha v e rec e i v e d se v e r a l er r o r s su c h as thi s:

[drysdalk@ioprov2 drysdalk]$ find / -name lsfind: /lost+found: Permission deniedfind: /boot/lost+found: Permission deniedfind: /home/lost+found: Permission deniedfind: /home/nonis: Permission deniedfind: /home/adamh: Permission deniedfind: /home/agoodall: Permission deniedfind: /home/aroberts: Permission deniedfind: /home/callum: Permission denied...find is as its mo s t us e f u l wh e n it is co m b i n e d wi t h wi l d c a r d s . Go i n g ba c k to ou r fir s t ex a m p l e , wh e r e we ha v e a co l l e c t i o n of di r e c t o r i e s tha t co n t a i n va r i o u s fil e s , let’ s sa y yo u ha d pr e v i o u s l y us e d on e of the m for st o r i n g di g i t a l ph o t o g r a p h s yo u ha d tak e n . Yo u ca n’ t rem e m b e r wh i c h on e, tho u g h . So we ca n se a r c h for al l fil e s wh o s e na m e s en d wi t h .jpg lik e thi s:

[drysdalk@ioprov2 drysdalk]$ find ~ -name *.jpg/mnt/home/drysdalk/drysdalk/files2/cat1.jpg/mnt/home/drysdalk/drysdalk/files2/cat2.jpg/mnt/home/drysdalk/drysdalk/files2/furby.jpg[drysdalk@ioprov2 drysdalk]$

4.8 Pipes

It is po s s i b l e to link two or mo r e co m m a n d s tog e t h e r , so tha t the ou t p u t pr o d u c e d by on e is tak e n as the inp u t us e d by an o t h e r . Th i s is do n e by

us i n g pipes . Yo u ca n thi n k of a pip e as so m e t h i n g thr o u g h wh i c h inf o r m a t i o n flo w s (henc e the na m e). Th i s is on e of the mo s t po w e r f u l

Pa g e 37

an d us e f u l fea t u r e s of Li n u x an d UN I X- st y l e op e r a t i n g sy s t e m s , an d al l o w s us to do som e ve r y co m p l i c a t e d thi n g s in a fai r l y st r a i g h t f o r w a r d fas h i o n .

Pi p e s ar e us e d by typ i n g the co m m a n d s yo u wa n t to lin k tog e t h e r , se p a r a t e d by a | sym b o l (a ve r t i c a l ba r – us u a l l y loc a t e d on the ba c k s l a s h ke y).

Th e ea s i e s t wa y to ge t to gr i p s wi t h pip e s is to se e som e ex a m p l e s . Fo r ins t a n c e , ima g i n e we ha v e a di r e c t o r y wi t h the fol l o w i n g fil e s :

[drysdalk@ioprov2 drysdalk]$ lsfile-20061201 file-20061214 file-20061222 file-20070109 file-20070129 file-20070207 file-20070215 file-20070223file-20061207 file-20061215 file-20061223 file-20070115 file-20070130 file-20070208 file-20070218 file-20070224file-20061209 file-20061216 file-20061229 file-20070116 file-20070201 file-20070211 file-20070219 file-20070225file-20061210 file-20061217 file-20061230 file-20070123 file-20070204 file-20070212 file-20070220 file-20070226file-20061212 file-20061220 file-20070107 file-20070126 file-20070205 file-20070213 file-20070221 file-20070227file-20061213 file-20061221 file-20070127 file-20070206 file-20070214 file-20070222[drysdalk@ioprov2 drysdalk]$

No w , let’ s sa y at a gl a n c e we wa n t e d to kn o w at a gl a n c e wh a t fil e s we ha d co n t a i n i n g inf o r m a t i o n fro m Ja n u a r y 2007. We co u l d do thi s:

Pa g e 38

[drysdalk@ioprov2 drysdalk]$ ls | grep 200701file-20070107file-20070109file-20070115file-20070116file-20070123file-20070126file-20070127file-20070129file-20070130[drysdalk@ioprov2 drysdalk]$

He r e , we too k the ou t p u t fro m ls, wh i c h co n s i s t e d of the di r e c t o r y lis t i n g tha t we saw abo v e pr e v i o u s l y . Ho w e v e r , rat h e r tha n se n d i n g tha t ou t p u t to the sc r e e n as is the de f a u l t , the ou t p u t wa s fed in to a pip e . Th i s pr e v e n t s it fro m be i n g wr i t t e n to the ter m i n a l , an d ins t e a d pa s s e s it on to the pr o g r a m at the ot h e r en d of the pi p e .

So grep rec e i v e d as its inp u t the co m p l e t e dir e c t o r y lis t i n g , an d the n se a r c h e d tha t inp u t for the st r i n g 20070 1 . As is its de f a u l t be h a v i o u r , it the n pr i n t e d ou t to the ter m i n a l al l lin e s it fou n d tha t ma t c h e d the pa t t e r n it ha d be e n gi v e n . So the en d res u l t wa s to gi v e us a di r e c t o r y lis t i n g on l y for tho s e file s wi t h 20070 1 in the i r fil e n a m e .

Al m o s t an y tw o Li n u x co m m a n d s ca n be at t a c h e d by a pip e . So lon g as a com m a n d is ca p a b l e of tak i n g its inp u t fro m the key b o a r d , an d of wr i t i n g its ou t p u t to the ter m i n a l , it ca n fun c t i o n at ei t h e r en d of a pip e .

He r e ’ s an o t h e r ex a m p l e – let’s sa y tha t af t e r ou r ea r l i e r se a r c h thr o u g h domains.txt for do m a i n s co n t a i n i n g the wo r d ‘fur b y ’, we the n de c i d e d tha t we wa n t e d the res u l t s to be alp h a b e t i c a l l y so r t e d , al l in on e go:

[drysdalk@ioprov2 drysdalk]$ ls -ltotal 888-rw-r--r-- 1 drysdalk drysdalk 903719 Feb 16 16:56 domains.txt[drysdalk@ioprov2 drysdalk]$ grep furby domains.txtfurby.comadoptafurby.comfurbyforum.co.uk[drysdalk@ioprov2 drysdalk]$ grep furby domains.txt | sortadoptafurby.comfurby.comfurbyforum.co.ukAn d the res u l t s ar e no w alp h a b e t i c a l l y so r t e d .

Y o u ca n al s o com b i n e as ma n y pi p e s as yo u lik e in on e go:

[drysdalk@ioprov2 drysdalk]$ grep furby domains.txt | grep –v adopt | sortfurby.comfurbyforum.co.uk

Pa g e 39

So as yo u ca n se e, pip e s ar e a ve r y ve r s a t i l e , an d us e f u l , fea t u r e of Li n u x .

4.9 Input and output re-direction

In ad d i t i o n to lin k i n g up com m a n d s us i n g pip e s , it is al s o po s s i b l e for a com m a n d to ge t its inp u t fro m a pr e- ex i s t i n g fil e, or to se n d ou t p u t tha t wo u l d no r m a l l y be pr i n t e d on yo u r ter m i n a l to a pr e- ex i s t i n g or new fil e.

Th i s is ac c o m p l i s h e d by re-directing the input and output of com m a n d s . Inp u t re-dir e c t i o n is ac c o m p l i s h e d by us i n g a < sym b o l (a les s- tha n sig n), an d ou t p u t re-dir e c t i o n is rep r e s e n t e d by a > sym b o l (a gr e a t e r- tha n sig n).

Ag a i n , the ea s i e s t wa y to un d e r s t a n d how the s e wo r k is to jum p st r a i g h t in wi t h so m e ex a m p l e s . Le t’ s sa y tha t du r i n g ou r pr e v i o u s ex a m p l e of se a r c h i n g for do m a i n na m e s , we wa n t to st o r e the res u l t s in a file, rat h e r tha n sim p l y ha v e the m app e a r on ou r ter m i n a l :

[drysdalk@ioprov2 drysdalk]$ lsdomains.txt[drysdalk@ioprov2 drysdalk]$ grep furby domains.txtfurby.comadoptafurby.comfurbyforum.co.uk[drysdalk@ioprov2 drysdalk]$ grep furby domains.txt > results[drysdalk@ioprov2 drysdalk]$ lsdomains.txt results[drysdalk@ioprov2 drysdalk]$ cat resultsfurby.comadoptafurby.comfurbyforum.co.uk[drysdalk@ioprov2 drysdalk]$

As yo u ca n se e, the fil e results wa s cr e a t e d wh e n we re-dir e c t e d the ou t p u t of the grep co m m a n d to it, an d its co n t e n t s ar e ide n t i c a l to the ou t p u t of ou r or i g i n a l grep.

N o w pe r h a p s we we r e or i g i n a l l y loo k i n g for thi s lis t so tha t we co u l d pe r f o r m a pa r t i c u l a r ta s k on the s e sp e c i f i c do m a i n s . Ima g i n e tha t yo u ha v e alr e a d y wr i t t e n a scr i p t to ca r r y ou t thi s tas k , an d tha t it wi l l pr o m p t yo u to typ e ea c h do m a i n na m e to be pr o c e s s e d . So yo u co u l d do som e t h i n g lik e:

[drysdalk@ioprov2 drysdalk]$ ./script.sh

Pa g e 40

Enter domain names, and press return after each one.furby.comDomain processed - furby.comadoptafurby.comDomain processed - adoptafurby.comfurbyforum.co.ukDomain processed - furbyforum.co.uk[drysdalk@ioprov2 drysdalk]$

Bu t we co u l d al s o us e an inp u t re-dir e c t o r to au t o m a t i c a l l y fee d ou r results fil e in to the sc r i p t to sa v e us typ i n g ma n u a l l y , lik e so:

[drysdalk@ioprov2 drysdalk]$ ./script.sh < resultsEnter domain names, and press return after each one.Domain processed - furby.comDomain processed - adoptafurby.comDomain processed - furbyforum.co.uk[drysdalk@ioprov2 drysdalk]$

In cl o s i n g , he r e’ s a ca u t i o n a r y wo r d ab o u t the ou t p u t re-dir e c t o r – if yo u sp e c i f y a fil e tha t alr e a d y ex i s t s , it wi l l be en t i r e l y ov e r w r i t t e n . Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ lsdomains.txt results[drysdalk@ioprov2 drysdalk]$ cat resultsI contain something entirely un-related to this search[drysdalk@ioprov2 drysdalk]$ grep furby domains.txt > results[drysdalk@ioprov2 drysdalk]$ cat resultsfurby.comadoptafurby.comfurbyforum.co.uk[drysdalk@ioprov2 drysdalk]$

If yo u wa n t to pr e s e r v e the co n t e n t s of the fil e yo u ar e re-dir e c t i n g ou t p u t

to, yo u ne e d to us e the append re-di r e c t o r , wh i c h is >> (two gr e a t e r-tha n sig n s). Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ cat resultsI contain something entirely un-related to this search[drysdalk@ioprov2 drysdalk]$ grep furby domains.txt >> results[drysdalk@ioprov2 drysdalk]$ cat resultsI contain something entirely un-related to this searchfurby.comadoptafurby.comfurbyforum.co.uk[drysdalk@ioprov2 drysdalk]$

Ag a i n , as wi t h pip e s , inp u t an d ou t p u t re-di r e c t i o n wi l l wo r k wi t h an y Li n u x co m m a n d tha t ca n ei t h e r rea d its inp u t from the ke y b o a r d or wr i t e its ou t p u t to the ter m i n a l .

4.10 Changing your password

Pa g e 41

Yo u mi g h t wa n t to ch a n g e the pa s s w o r d tha t yo u we r e fir s t iss u e d wi t h – thi s is fin e, an d Lin u x pr o v i d e s yo u wi t h a sim p l e me c h a n i s m for do i n g thi s. Pa s s w o r d s ar e ch a n g e d wi t h the passwd co m m a n d .

To ch a n g e yo u r pa s s w o r d , yo u sim p l y typ e passwd on its ow n , an d yo u wi l l the n be tak e n thr o u g h the pa s s w o r d- ch a n g i n g pr o c e s s . Yo u wi l l fir s t be pr o m p t e d for yo u r ex i s t i n g pa s s w o r d , in or d e r to ma k e su r e tha t the pe r s o n iss u i n g the pa s s w o r d ch a n g e req u e s t is rea l l y yo u, an d no t so m e st r a n g e r wh o is try i n g to loc k yo u ou t be c a u s e the y’ v e com e wa n d e r i n g pa s t yo u r un l o c k e d ter m i n a l . If yo u en t e r it co r r e c t l y , yo u wi l l the n be as k e d fo r yo u r ne w pa s s w o r d tw i c e , to ma k e su r e yo u ar e typ i n g it co r r e c t l y .

[drysdalk@ioprov2 drysdalk]$ passwdChanging NIS account information for drysdalk on nis01.iomart.com.Please enter old password:Changing NIS password for drysdalk on nis01.iomart.com.Please enter new password:Please retype new password:

The NIS password has been changed on nis01.iomart.com.

[drysdalk@ioprov2 drysdalk]$4.11 Changing the owner and group of a file or directory

We’ v e al r e a d y lea r n e d how to se e the ow n e r an d gr o u p as s o c i a t e d wi t h a pa r t i c u l a r file or dir e c t o r y . No w we’ l l se e how to ac t u a l l y ch a n g e the s e .

Im a g i n e yo u ha v e the fol l o w i n g fil e s , wi t h the fol l o w i n g ow n e r s h i p s :

[drysdalk@ioprov2 drysdalk]$ ls -ltotal 12-rwxr-x--- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*-rwxr-x--- 1 drysdalk drysdalk 48 Feb 19 15:19 script2.sh*-rwxr-x--- 1 drysdalk drysdalk 67 Feb 19 15:19 script3.sh*[drysdalk@ioprov2 drysdalk]$

No w , let’ s sa y we wa n t e d to gi v e al l m e m b e r s of the ‘sys’ gr o u p ac c e s s to the s e fil e s . We us e the chgrp co m m a n d to ch a n g e a fil e’s gr o u p , as fol l o w s :

[drysdalk@ioprov2 drysdalk]$ sudo chgrp sys *[drysdalk@ioprov2 drysdalk]$ ls -ltotal 12-rwxr-x--- 1 drysdalk sys 12 Feb 19 15:19 script1.sh*-rwxr-x--- 1 drysdalk sys 48 Feb 19 15:19 script2.sh*

Pa g e 42

-rwxr-x--- 1 drysdalk sys 67 Feb 19 15:19 script3.sh*[drysdalk@ioprov2 drysdalk]$

As yo u ca n se e, the ‘sy s’ gr o u p now is as s o c i a t e d wi t h the s e fil e s , wh e r e a s it wa s the us e r ’ s ow n pr i v a t e gr o u p be f o r e .

No w , wh a t if we wa n t to gi v e aw a y ow n e r s h i p of the s e fil e s al t o g e t h e r , to an en t i r e l y di f f e r e n t pe r s o n ? Th i s is do n e wi t h the chown com m a n d :

[drysdalk@ioprov2 drysdalk]$ sudo chown aroberts *[drysdalk@ioprov2 drysdalk]$ ls -ltotal 12-rwxr-x--- 1 aroberts sys 12 Feb 19 15:19 script1.sh*-rwxr-x--- 1 aroberts sys 48 Feb 19 15:19 script2.sh*-rwxr-x--- 1 aroberts sys 67 Feb 19 15:19 script3.sh*[drysdalk@ioprov2 drysdalk]$

An d no w the file s ar e no lon g e r ow n e d by the or i g i n a l us e r or gr o u p at al l.

No t e tha t un l e s s the or i g i n a l us e r is als o a me m b e r of the ‘sys’ gr o u p , the y wo u l d no lon g e r ha v e an y ac c e s s to the s e file s . If the y ev e r ne e d e d to us e the m aga i n , the y wo u l d ha v e to us e sudo to ei t h e r ac c e s s the m as the su p e r u s e r , or to ch a n g e the ow n e r s h i p s ba c k to a su i t a b l e va l u e . So be ca r e f u l wh e n ch a n g i n g ow n e r s h i p s an d gr o u p s on file s – it is po s s i b l e to loc k reg u l a r us e r s ou t of fil e s or di r e c t o r i e s en t i r e l y .

S p e a k i n g of sudo, yo u’ l l no t i c e we had to us e it in al l the s e ex a m p l e s – thi s is be c a u s e on l y root ca n ch a n g e gr o u p s an d ow n e r s h i p s on fil e s . So thi s is on e tas k yo u’ d us u a l l y al w a y s ha v e to us e sudo for.

4.12 Changing the permissions on a file or directory

In the las t se c t i o n , we saw ho w to vie w an d un d e r s t a n d the pe r m i s s i o n s tha t ap p e a r in a lon g di r e c t o r y lis t i n g . To rec a p us i n g on e of the fil e s fro m ou r pr e v i o u s ex a m p l e :

-rwxr-x--- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*

He r e we ca n se e tha t the pe r m i s s i o n s br e a k do w n as fol l o w s :

Ow n e r Gr o u p Ev e r y o n e els e- rwx r-x ---

Pa g e 43

So the fil e’s ow n e r ‘dr y s d a l k ’ ha s rea d , wr i t e an d ex e c u t e ac c e s s ; ot h e r me m b e r s of the gr o u p ‘dry s d a l k ’ ha v e rea d an d ex e c u t e ac c e s s on l y ; an d ev e r y o n e el s e on the sy s t e m ha s no ac c e s s to script1.sh at al l.

W h a t if we wa n t e d to ch a n g e the s e pe r m i s s i o n s ? Th i s is do n e wi t h the chmod com m a n d . chmod ha s the fol l o w i n g ba s i c sy n t a x :

chmod <permissions to use> <files to have their permissions changed>

Th e pe r m i s s i o n s to us e ha v e the i r ow n un i q u e syn t a x , of the for m :

<ow n e r , gr o u p , al l> <se t or un s e t> <pe r m i s s i o n s >

Pa g e 44

Wh e r e :

u Ow n e r of the fil e

g Gr o u p as s o c i a t e d wi t h the fil e

o Al l ot h e r us e r s on the sy s t e m

An d :

+ Se t the fol l o w i n g pe r m i s s i o n(s)

– Un s e t the fol l o w i n g pe r m i s s i o n(s)

An d fin a l l y :

rwx Po s s i b l e se t of pe r m i s s i o n s – rea d , wr i t e an d ex e c u t e

Th i s ma y se e m qu i t e da u n t i n g at fir s t , bu t wh e n yo u se e a few ex a m p l e s it wi l l be g i n to be c o m e cle a r .

Le t’ s sa y tha t we wa n t e d to rem o v e ou r ow n wr i t e ac c e s s to the fil e, to en s u r e we co u l d no t ac c i d e n t a l l y ov e r w r i t e it. We wo u l d ne e d to typ e:

[drysdalk@ioprov2 drysdalk]$ ls -l script1.sh-rwxr-x--- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*[drysdalk@ioprov2 drysdalk]$ chmod u-w script1.sh[drysdalk@ioprov2 drysdalk]$ ls -l script1.sh-r-xr-x--- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*[drysdalk@ioprov2 drysdalk]$

We us e d ‘u-w ’ as the pe r m i s s i o n s be c a u s e :

u We are ch a n g i n g pe r m i s s i o n s for the ow n e r , rep r e s e n t e d by

u– We are rem o v i n g pe r m i s s i o n s

w Th e pe r m i s s i o n we wi s h to ch a n g e is wr i t e pe r m i s s i o n

So if we wa n t e d to re-ad d wr i t e pe r m i s s i o n , we wo u l d us e u+w :

[drysdalk@ioprov2 drysdalk]$ chmod u+w script1.sh[drysdalk@ioprov2 drysdalk]$ ls -l script1.sh-rwxr-x--- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*[drysdalk@ioprov2 drysdalk]$

Pa g e 45

If we wa n t e d to tak e aw a y al l pe r m i s s i o n s for me m b e r s of the ‘dry s d a l k ’

gr o u p , we wo u l d ne e d to us e the mo d e g-rwx :

[drysdalk@ioprov2 drysdalk]$ chmod g-rwx script1.sh[drysdalk@ioprov2 drysdalk]$ ls -l script1.sh-rwx------ 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*[drysdalk@ioprov2 drysdalk]$

An d as a las t ex a m p l e , if we wa n t e d to ad d rea d pe r m i s s i o n for al l ot h e r us e r s on the sy s t e m no t a me m b e r of the ‘dry s d a l k ’ gr o u p , we wo u l d us e

o+r :

[drysdalk@ioprov2 drysdalk]$ chmod o+r script1.sh[drysdalk@ioprov2 drysdalk]$ ls -l script1.sh-rwx---r-- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*[drysdalk@ioprov2 drysdalk]$

4.13 Symbolic links

A sym b o l i c lin k is a ‘vir t u a l ’ fil e, mo r e or les s , tha t po i n t s to the ‘rea l’ co p y of the file som e w h e r e el s e on the sy s t e m . Th e ide a be h i n d sym b o l i c link s is to be ab l e to ref e r e n c e fil e s fro m mu l t i p l e loc a t i o n s , wi t h o u t ha v i n g to co p y the fil e its e l f an d thu s wa s t e di s c sp a c e . Th e y al s o ha v e the ad d e d ad v a n t a g e tha t an y ch a n g e s ma d e to the rea l fil e its e l f af f e c t ev e r y t h i n g lin k e d to it as we l l . Yo u do n’ t ha v e to ma i n t a i n lot s of di f f e r e n t co p i e s of a fil e – yo u ca n ke e p on e ce n t r a l co p y , an d ha v e al l ot h e r ins t a n c e s of the fil e as sym b o l i c lin k s po i n t i n g to it. Th i s wa y , al l ‘cop i e s ’ ar e al w a y s po i n t i n g to the lat e s t ve r s i o n of the fil e.

S y m b o l i c link s ar e se t up wi t h the ln co m m a n d . Th e sy n t a x req u i r e d to cr e a t e a sym b o l i c link is as fol l o w s :

ln –s <original file> <new link to be created>

Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ ls -ltotal 4-rw-rw-r-- 1 drysdalk drysdalk 27 Feb 19 17:11 file1.txt[drysdalk@ioprov2 drysdalk]$ ln -s file1.txt file2.txt[drysdalk@ioprov2 drysdalk]$ ls -ltotal 4-rw-rw-r-- 1 drysdalk drysdalk 27 Feb 19 17:11 file1.txtlrwxrwxrwx 1 drysdalk drysdalk 9 Feb 19 17:11 file2.txt -> file1.txt[drysdalk@ioprov2 drysdalk]$He r e we ca n se e in a lon g dir e c t o r y lis t i n g tha t file2.txt is (quit e lit e r a l l y) po i n t i n g to file1.txt. Yo u wi l l al s o no t i c e tha t the ve r y fir s t

Pa g e 46

ch a r a c t e r in file2.txt’s pe r m i s s i o n s (whi c h up un t i l no w ha s a sin g l e da s h), is no w a sin g l e low e r- ca s e let t e r l. Th i s tel l s yo u be y o n d a do u b t tha t it is a sym b o l i c lin k.

If file2.txt is rem o v e d , file1.txt wi l l no t be af f e c t e d , si n c e file2.txt wa s no t a rea l fil e – it wa s on l y a po i n t e r to the ge n u i n e ar t i c l e . If file1.txt is rem o v e d , how e v e r , the da t a wi l l be go n e – file2.txt wi l l st i l l ex i s t , bu t it wi l l no w be a us e l e s s , br o k e n lin k, si n c e it po i n t s to a de s t i n a t i o n tha t ha s be e n rem o v e d .

4.14 End-of-section summary

So, we’ v e ex p a n d e d on the ba s i c s of us i n g Lin u x qu i t e a bi t he r e , an d ha v e int r o d u c e d som e new co n c e p t s tha t ho p e f u l l y ha v e gi v e n yo u som e ide a of how ve r s a t i l e an d pow e r f u l Li n u x ca n be at the com m a n d lin e.

To rec a p , in thi s se c t i o n we ha v e lea r n e d how to:

- Wo r k wi t h mo r e tha n on e fil e or dir e c t o r y at on c e by us i n g wi l d c a r d s

- Re m o v e a dir e c t o r y wh e n it is no t al r e a d y em p t y

- Be very, VERY careful when doing either of the above !

- Se a r c h the co n t e n t s of file s wi t h the grep co m m a n d- So r t file s al p h a b e t i c a l l y wi t h the sort co m m a n d- Fi n d fil e s ba s e d on the i r na m e s us i n g the find co m m a n d- Us e the ou t p u t of on e com m a n d as the inp u t of an o t h e r wi t h

pi p e s ( | )- Re- di r e c t the inp u t or ou t p u t of com m a n d s wi t h re-dir e c t o r s ( < >

>> )- Ch a n g e yo u r ow n pa s s w o r d- Ch a n g e the ow n e r or gr o u p as s o c i a t e d wi t h a fil e or dir e c t o r y- Ch a n g e the pe r m i s s i o n s on a fil e or dir e c t o r y- Cr e a t e an d vie w sym b o l i c link s

Pa g e 47

5. Advanced commandsIn thi s se c t i o n , we’ l l ex p a n d on so m e of the com m a n d s dis c u s s e d in the pr e v i o u s se c t i o n , wh i c h ar e ca p a b l e of far mo r e tha n yo u ha v e se e n so far. We’ l l als o int r o d u c e a few new com m a n d s we ha v e n ’ t tal k e d ab o u t be f o r e .

5.1 sort revisited In the las t se c t i o n , we saw tha t sort ca n be us e d to so r t inf o r m a t i o n al p h a b e t i c a l l y . It ca n do mo r e tha n tha t , ho w e v e r – it ca n wo r k wi t h com p l e x se t s of da t a , pe r f o r m a so r t on on e pa r t i c u l a r po r t i o n of its inp u t , an d mo r e .

Fo r ex a m p l e , ima g i n e yo u ha v e the fol l o w i n g lon g di r e c t o r y lis t i n g ;

[drysdalk@ioprov2 drysdalk]$ ls -ltotal 40-rwxr-xr-- 1 drysdalk drysdalk 176 Feb 19 15:20 script10.sh*-rwxr-xr-- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*-rwxr-xr-- 1 drysdalk drysdalk 48 Feb 19 15:19 script2.sh*-rwxr-xr-- 1 drysdalk drysdalk 67 Feb 19 15:19 script3.sh*-rwxr-xr-- 1 drysdalk drysdalk 36 Feb 19 15:19 script4.sh*-rwxr-xr-- 1 drysdalk drysdalk 4 Feb 19 15:19 script5.sh*-rwxr-xr-- 1 drysdalk drysdalk 47 Feb 19 15:20 script6.sh*-rwxr-xr-- 1 drysdalk drysdalk 107 Feb 19 15:20 script7.sh*-rwxr-xr-- 1 drysdalk drysdalk 30 Feb 19 15:20 script8.sh*-rwxr-xr-- 1 drysdalk drysdalk 20 Feb 19 15:20 script9.sh*[drysdalk@ioprov2 drysdalk]$

No w , wh a t if we wa n t to so r t the s e fil e s in or d e r of the i r si z e ? Th e si z e is the fif t h fie l d in the lis t, co u n t i n g al o n g fro m the lef t. Th e sort com m a n d ca n be tol d to so r t ba s e d on a sp e c i f i c fie l d wi t h the –k ar g u m e n t . So to so r t thi s lon g dir e c t o r y lis t i n g ba s e d on the va l u e s in the fif t h fie l d , we wo u l d typ e:

[drysdalk@ioprov2 drysdalk]$ ls -l | sort -bn -k5total 40-rwxr-xr-- 1 drysdalk drysdalk 4 Feb 19 15:19 script5.sh*-rwxr-xr-- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*-rwxr-xr-- 1 drysdalk drysdalk 20 Feb 19 15:20 script9.sh*-rwxr-xr-- 1 drysdalk drysdalk 30 Feb 19 15:20 script8.sh*-rwxr-xr-- 1 drysdalk drysdalk 36 Feb 19 15:19 script4.sh*-rwxr-xr-- 1 drysdalk drysdalk 47 Feb 19 15:20 script6.sh*-rwxr-xr-- 1 drysdalk drysdalk 48 Feb 19 15:19 script2.sh*-rwxr-xr-- 1 drysdalk drysdalk 67 Feb 19 15:19 script3.sh*-rwxr-xr-- 1 drysdalk drysdalk 107 Feb 19 15:20 script7.sh*-rwxr-xr-- 1 drysdalk drysdalk 176 Feb 19 15:20 script10.sh*[drysdalk@ioprov2 drysdalk]$

Pa g e 48

Th e –bn fla g tel l s sort to pe r f o r m a so r t in nu m e r i c a l or d e r , rat h e r tha n al p h a b e t i c a l . An d no w we ca n se e at a gl a n c e wh i c h of the s e sc r i p t s is the sm a l l e s t , an d wh i c h is the lar g e s t .

W e ca n al s o ea s i l y rev e r s e the or d e r of a sort wi t h the –r fla g:

[drysdalk@ioprov2 drysdalk]$ ls -l | sort -bn -k5 -r-rwxr-xr-- 1 drysdalk drysdalk 176 Feb 19 15:20 script10.sh*-rwxr-xr-- 1 drysdalk drysdalk 107 Feb 19 15:20 script7.sh*-rwxr-xr-- 1 drysdalk drysdalk 67 Feb 19 15:19 script3.sh*-rwxr-xr-- 1 drysdalk drysdalk 48 Feb 19 15:19 script2.sh*-rwxr-xr-- 1 drysdalk drysdalk 47 Feb 19 15:20 script6.sh*-rwxr-xr-- 1 drysdalk drysdalk 36 Feb 19 15:19 script4.sh*-rwxr-xr-- 1 drysdalk drysdalk 30 Feb 19 15:20 script8.sh*-rwxr-xr-- 1 drysdalk drysdalk 20 Feb 19 15:20 script9.sh*-rwxr-xr-- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*-rwxr-xr-- 1 drysdalk drysdalk 4 Feb 19 15:19 script5.sh*total 40[drysdalk@ioprov2 drysdalk]$

Th i s is al s o a go o d op p o r t u n i t y to po i n t ou t som e t h i n g el s e ab o u t Lin u x com m a n d fla g s – yo u ca n com b i n e the m in an y or d e r . Yo u ca n ei t h e r pu t ea c h of the m on the i r ow n as we ha v e do n e ab o v e , or yo u ca n rol l the m al l in to on e an d the res u l t is the sa m e :

[drysdalk@ioprov2 drysdalk]$ ls -l | sort -bnrk5-rwxr-xr-- 1 drysdalk drysdalk 176 Feb 19 15:20 script10.sh*-rwxr-xr-- 1 drysdalk drysdalk 107 Feb 19 15:20 script7.sh*-rwxr-xr-- 1 drysdalk drysdalk 67 Feb 19 15:19 script3.sh*-rwxr-xr-- 1 drysdalk drysdalk 48 Feb 19 15:19 script2.sh*-rwxr-xr-- 1 drysdalk drysdalk 47 Feb 19 15:20 script6.sh*-rwxr-xr-- 1 drysdalk drysdalk 36 Feb 19 15:19 script4.sh*-rwxr-xr-- 1 drysdalk drysdalk 30 Feb 19 15:20 script8.sh*-rwxr-xr-- 1 drysdalk drysdalk 20 Feb 19 15:20 script9.sh*-rwxr-xr-- 1 drysdalk drysdalk 12 Feb 19 15:19 script1.sh*-rwxr-xr-- 1 drysdalk drysdalk 4 Feb 19 15:19 script5.sh*total 40[drysdalk@ioprov2 drysdalk]$Ho w e v e r , for the sak e of rea d a b i l i t y , it’s us u a l l y a go o d ide a to se p a r a t e the m so yo u ca n ea s i l y tel l the m ap a r t .

O h, on e las t thi n g – no t e tha t ls ca n ac t u a l l y au t o m a t i c a l l y so r t its ou t p u t

its e l f by siz e , wi t h the –S fla g (note tha t’ s an up p e r- ca s e S). So to ge t a pr e- so r t e d lon g dir e c t o r y lis t i n g , st a r t i n g wi t h the bi g g e s t fil e an d wo r k i n g

do w n to the sm a l l e s t , ls –lS wo u l d wo r k . Th e –r fla g rev e r s e s the or d e r of the so r t , so ls –lrS wo u l d rev e r s e the or d e r of the so r t , fro m sm a l l e s t to big g e s t .

Pa g e 49

5.2 find revisited As yo u mi g h t ha v e su s p e c t e d , find is al s o ca p a b l e of far mo r e tha n jus t se a r c h i n g for fil e s ba s e d on the i r na m e . Th e -name fla g is the on e yo u wi l l un d o u b t e d l y us e the mo s t , bu t the r e ar e a gr e a t ma n y ot h e r s . He r e we’ l l loo k at so m e of the ot h e r mo r e co m m o n (and us e f u l) typ e s of find com m a n d s yo u mi g h t ne e d to run.

Finding files based on their size

Sy n t a x :fin d <st a r t i n g po i n t> -siz e <siz e sp e c i f i c a t i o n>

Wh e r e the si z e sp e c i f i c a t i o n ha s the fol l o w i n g syn t a x :

+ or – Fi n d file s bi g g e r tha n , or les s tha n , the fol l o w i n g siz e

File size A wh o l e nu m b e r , us u a l l y the nu m b e r of by t e s in qu e s t i o n

c or k Th e nu m b e r is in by t e s , or ki l o b y t e s

In ea c h of the fol l o w i n g ex a m p l e s , we as s u m e the fol l o w i n g co n t e n t s in yo u r ho m e di r e c t o r y :

[drysdalk@ioprov2 drysdalk]$ ls -ltotal 8720-rw-rw-r-- 1 drysdalk drysdalk 2097152 Feb 19 16:47 file1.txt-rw-rw-r-- 1 drysdalk drysdalk 1048576 Feb 19 16:48 file2.txt-rw-rw-r-- 1 drysdalk drysdalk 1572864 Feb 19 16:48 file3.txt-rw-rw-r-- 1 drysdalk drysdalk 4194304 Feb 19 16:48 file4.txt-rw-rw-r-- 1 drysdalk drysdalk 524288 Feb 19 16:50 file5.txt[drysdalk@ioprov2 drysdalk]$

Fo r ex a m p l e , to fin d al l fil e s in yo u r ho m e dir e c t o r y big g e r tha n on e me g a b y t e in siz e , yo u co u l d typ e :

[drysdalk@ioprov2 drysdalk]$ find ~ -size +1048576c/mnt/home/drysdalk/drysdalk/file3.txt/mnt/home/drysdalk/drysdalk/file1.txt/mnt/home/drysdalk/drysdalk/file4.txt

Or:

[drysdalk@ioprov2 drysdalk]$ find ~ -size +1024k/mnt/home/drysdalk/drysdalk/file3.txt

Pa g e 50

/mnt/home/drysdalk/drysdalk/file1.txt/mnt/home/drysdalk/drysdalk/file4.txt[drysdalk@ioprov2 drysdalk]$Yo u’ d us u a l l y wa n t to do 1024 k , bu t if yo u ha p p e n to kn o w the ex a c t nu m b e r of by t e s in a me g a b y t e (1,048,57 6, as it ha p p e n s) the ef f e c t is ide n t i c a l .

To fin d fil e s sm a l l e r tha n a me g a b y t e :

[drysdalk@ioprov2 drysdalk]$ find ~ -size -1024k/mnt/home/drysdalk/drysdalk/file5.txt[drysdalk@ioprov2 drysdalk]$

Yo u ca n al s o sp e c i f y the pr e c i s e siz e of fil e yo u wa n t to se a r c h for, by om i t t i n g the – or + sym b o l :

[drysdalk@ioprov2 drysdalk]$ find ~ -size 1024k/mnt/home/drysdalk/drysdalk/file2.txt[drysdalk@ioprov2 drysdalk]$

Finding files based on their type

Th e tit l e of thi s se c t i o n is ac t u a l l y a sli g h t mi s n o m e r , si n c e wh a t we rea l l y me a n he r e by ‘typ e’ is no t the ex t e n s i o n of a fil e n a m e (e.g. .txt, .jpg, .sh, et c), bu t the typ e as far as Lin u x its e l f is co n c e r n e d – e.g. a fil e, a di r e c t o r y , an d so on.

Th i s mi g h t no t se e m too us e f u l to be g i n wi t h , bu t yo u’ l l se e lat e r on tha t be i n g ab l e to fin d al l di r e c t o r i e s wi t h i n a sp e c i f i c loc a t i o n is qu i t e us e f u l , or lik e w i s e to on l y fin d file s an d no t di r e c t o r i e s .In an y ev e n t , the sy n t a x is:

find <starting location> -type <type to search for>

Wh e r e the typ e s yo u wo u l d mo s t co m m o n l y wa n t to se a r c h for ar e:

d Di r e c t o r y

f Fi l e

l Sy m b o l i c link

Fo r ex a m p l e , to fin d al l su b- di r e c t o r i e s of yo u r ow n ho m e di r e c t o r y , yo u co u l d typ e :

Pa g e 51

[drysdalk@ioprov2 drysdalk]$ lsfiles/ notes/[drysdalk@ioprov2 drysdalk]$ find ~ -type d/mnt/home/drysdalk/mnt/home/drysdalk/files/mnt/home/drysdalk/files/images/mnt/home/drysdalk/files/email/mnt/home/drysdalk/files/email/personal/mnt/home/drysdalk/files/email/business/mnt/home/drysdalk/files/scripts/mnt/home/drysdalk/notes

No t i c e tha t the loc a t i o n of the ho m e di r e c t o r y its e l f (/mnt/home/drysdalk) is ret u r n e d as the ve r y fir s t res u l t . Th i s ma k e s se n s e , si n c e if yo u ar e st a r t i n g yo u r se a r c h for di r e c t o r i e s fro m the ho m e di r e c t o r y , tha t wi l l by de f i n i t i o n be the ve r y fir s t di r e c t o r y tha t yo u fin d.

If we se a r c h for fil e s an d no t dir e c t o r i e s , we fin d on l y the co n t e n t s of the di r e c t o r i e s bu t no res u l t s for the ac t u a l di r e c t o r i e s the m s e l v e s , si n c e the y ar e no t fil e s :

[drysdalk@ioprov2 drysdalk]$ find ~ -type f/mnt/home/drysdalk/files/images/cat1.jpg/mnt/home/drysdalk/files/images/cat2.jpg/mnt/home/drysdalk/files/images/furby.jpg/mnt/home/drysdalk/files/email/business/aroberts/mnt/home/drysdalk/files/email/business/paulj/mnt/home/drysdalk/files/email/business/callum/mnt/home/drysdalk/files/email/business/richard/mnt/home/drysdalk/files/email/personal/furby-list/mnt/home/drysdalk/files/scripts/script1.sh/mnt/home/drysdalk/files/scripts/script2.sh/mnt/home/drysdalk/notes/hosting.txt/mnt/home/drysdalk/notes/email.txt[drysdalk@ioprov2 drysdalk]$

Finding files based on their owner or group

Th e las t typ e of fin d tha t yo u ar e lik e l y to ev e r ne e d to do is to loc a t e a fil e or di r e c t o r y ba s e d on the ow n e r or gr o u p as s o c i a t e d wi t h it. Th i s is

do n e wi t h the –user an d –group fla g s of find.

Fo r ex a m p l e , let’s sa y tha t an o t h e r me m b e r of the sy s t e m s ad m i n i s t r a t i o n tea m ha s co p i e d a fil e co n t a i n i n g ins t r u c t i o n s for a tas k yo u ar e to ca r r y ou t int o so m e loc a t i o n un d e r n e a t h yo u r ho m e di r e c t o r y . Th e y ha d to us e sudo to do thi s, an d so the fil e is ow n e d by root. Ho w e v e r , the y did n’ t tel l yo u ex a c t l y wh e r e a b o u t s un d e r n e a t h yo u r

Pa g e 52

ho m e di r e c t o r y the y pu t it. Yo u co u l d tra c k it do w n wi t h a find com m a n d lik e thi s:

[drysdalk@ioprov2 drysdalk]$ find ~ -user root/mnt/home/drysdalk/files/docs/your-instructions.txt[drysdalk@ioprov2 drysdalk]$

Si m i l a r l y , if yo u kn e w tha t the fil e wa s as s o c i a t e d wi t h the ‘sys’ gr o u p , yo u co u l d ha v e tra c k e d it do w n lik e thi s:

[drysdalk@ioprov2 drysdalk]$ find ~ -group sys/mnt/home/drysdalk/files/docs/your-instructions.txt[drysdalk@ioprov2 drysdalk]$

Asking find to give you more information

Th e r e is on e ot h e r fea t u r e of find tha t yo u ca n us e al o n g wi t h an y typ e of se a r c h yo u ar e ca r r y i n g ou t . It is po s s i b l e for find to au t o m a t i c a l l y pe r f o r m an ls –l on wh a t e v e r it fin d s . Th i s ca n be ve r y us e f u l , an d qu i t e a tim e- sav e r , si n c e qu i t e of t e n the ve r y ne x t thi n g yo u’ l l do on c e fin d ha s fou n d wh a t e v e r yo u’ r e loo k i n g for is pe r f o r m an ls –l on it.

Pa g e 53

Yo u ca n ge t fin d to do thi s wi t h the –ls fla g. Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ find ~ -type f -name *.sh -ls57409890 4 -rw-rw-r-- 1 drysdalk drysdalk 90 Feb 19 20:15 /mnt/home/drysdalk/script.sh50430336 4 -rw-rw-r-- 1 drysdalk drysdalk 40 Feb 19 20:15 /mnt/home/drysdalk/scripts/script.sh[drysdalk@ioprov2 drysdalk]$

No w , yo u’ l l no t i c e tha t the inf o r m a t i o n yo u ge t ba c k isn’ t precisely lik e the lon g di r e c t o r y lis t i n g tha t an ls –l wo u l d gi v e yo u. In pa r t i c u l a r , the r e ’ s tha t lar g e nu m b e r at the ve r y be g i n n i n g – yo u ca n ign o r e thi s for no w , tho u g h . Yo u ca n st i l l cl e a r l y se e the pe r m i s s i o n s , ow n e r , gr o u p , siz e , tim e- st a m p an d na m e of the fil e, ho w e v e r .

5.3 grep revisited So far, yo u’ v e se e n how to us e grep to se a r c h for a sp e c i f i c st r i n g in on e or mo r e fil e s in the cu r r e n t di r e c t o r y , or to ret u r n tho s e line s wh i c h do no t co n t a i n the st r i n g . Li k e sort and find, ho w e v e r , it’s ca p a b l e of a lot mo r e tha n suc h sim p l e se a r c h i n g . He r e ’ s a de s c r i p t i o n of a few of the ex t r a thi n g s tha t grep ca n do, an d how yo u mi g h t lik e to us e the m .

Case- insensitive searching

Le t’ s co n s i d e r the fol l o w i n g sce n a r i o . Ima g i n e ou r do m a i n lis t fro m the las t se c t i o n ag a i n , on l y wi t h a tw i s t . So m e do m a i n na m e s in the file ha v e be e n wr i t t e n al l in low e r- ca s e , bu t so m e ha v e be e n wr i t t e n in a mi x t u r e of low e r- an d up p e r- ca s e , or in up p e r- ca s e al o n e . Wh a t ha p p e n s if we try ou r pr e v i o u s se a r c h the n ?

[drysdalk@ioprov2 drysdalk]$ cat domains.txtAdoptAFurby.comFURBY.COMfurbyforum.co.uk[drysdalk@ioprov2 drysdalk]$ grep furby domains.txtfurbyforum.co.uk[drysdalk@ioprov2 drysdalk]$

As yo u ca n se e, we on l y go t on e res u l t ba c k – on l y on e of the do m a i n s co n t a i n e d the exact st r i n g ‘fur b y ’, wh e r e al l the let t e r s ar e in low e r- ca s e .

So, wh a t we ne e d to ca r r y ou t is a ca s e- ins e n s i t i v e se a r c h – tha t is, ha v e grep rep e a t ou r se a r c h , bu t no t ca r e ab o u t wh e t h e r the st r i n g is

Pa g e 54

fo u n d in up p e r- ca s e , low e r- ca s e , or an y co m b i n a t i o n of the two. We do

thi s wi t h the –i fla g:

[drysdalk@ioprov2 drysdalk]$ grep -i furby domains.txtAdoptAFurby.comFURBY.COMfurbyforum.co.uk[drysdalk@ioprov2 drysdalk]$

Summarising our results when dealing with large files

Le t’ s ima g i n e a new sit u a t i o n . We ha v e in ou r cu r r e n t di r e c t o r y on e hu n d r e d lar g e fil e s , domains-1 thr o u g h domains-100, al l co n t a i n i n g a wi d e va r i e t y of do m a i n na m e s . We wa n t to do the sa m e se a r c h as be f o r e , bu t we jus t wa n t to sum m a r i s e ou r res u l t s . We wa n t to kn o w the na m e s of the file s we fin d, rat h e r tha n se e i n g ev e r y si n g l e lin e in ea c h fil e tha t ma t c h e s the res u l t s . Th i s wa y , we ca n se e at a gl a n c e wh i c h fil e s we sh o u l d pa y mo r e at t e n t i o n to, an d wh i c h we ca n di s c a r d .

Th i s is do n e wi t h the –l fla g (a low e r- ca s e let t e r L) of grep. Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ grep -l furby domains*domains-13domains-36domains-42domains-55domains-78domains-93[drysdalk@ioprov2 drysdalk]$

So rat h e r tha n se e i n g ev e r y si n g l e line in the s e fil e s wi t h the st r i n g ‘fur b y ’, we sim p l y sa w the na m e s of the fil e s tha t ma t c h e d ou r se a r c h .

Pa g e 55

Searching the contents of sub-directories

Th e las t fea t u r e of grep tha t yo u ar e lik e l y to fin d of int e r e s t is its ab i l i t y to se a r c h the co n t e n t s of su b- di r e c t o r i e s , as we l l as the fil e s in the

cu r r e n t di r e c t o r y . Th i s is do n e wi t h the –r fla g.

Fo r ex a m p l e , to se a r c h for ‘fur b y ’ no t on l y in the fil e s in the cu r r e n t di r e c t o r y bu t in al l su b- di r e c t o r i e s of it, we co u l d typ e :

[drysdalk@ioprov2 drysdalk]$ grep -r furby *domains.txt:adoptafurby.comdomains.txt:furby.comdomains.txt:furbyforum.co.ukemail/personal/furby-list:Hey ! Check out furby.com ![drysdalk@ioprov2 drysdalk]$

He r e we se e the na m e of ea c h file, pr e c e d e d by its loc a t i o n if it is in a su b- dir e c t o r y , fol l o w e d by the ac t u a l line s tha t we r e fou n d .

An y an d al l of the fla g s of grep ca n be com b i n e d to ac h i e v e d the de s i r e d ef f e c t , so lon g as the y do no t co n t r a d i c t ea c h ot h e r . Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ grep -ril furby *domains.txtemail/personal/furby-list[drysdalk@ioprov2 drysdalk]$

He r e , we se a r c h e d fil e s in the cu r r e n t di r e c t o r y an d al l its su b- dir e c t o r i e s

wi t h the –r fla g; we pe r f o r m e d a ca s e ins e n s i t i v e se a r c h wi t h the –i fla g; an d we sp e c i f i e d we on l y wa n t e d to se e the na m e s of the file s fou n d

rat h e r tha n the ac t u a l line s wi t h the –l fla g.

5.4 The awk command

No w we com e to a co m m a n d tha t , wh e n co m b i n e d wi t h pi p e s , is ca p a b l e of so m e ve r y us e f u l thi n g s . Th e od d l y- na m e d awk com m a n d is on e of the mo s t ve r s a t i l e an d pow e r f u l av a i l a b l e in Lin u x , an d is in fac t a com p l e t e pr o g r a m m i n g lan g u a g e in an d of its e l f . En t i r e bo o k s co u l d be (and ha v e be e n) wr i t t e n on the su b j e c t of thi s on e com m a n d al o n e .

We do n’ t ne e d to wo r r y ab o u t tha t he r e , tho u g h . Th e r e is on e pa r t i c u l a r fea t u r e of awk tha t is us e d mo r e of t e n tha n an y ot h e r , an d tha t is its ab i l i t y to pr i n t ou t an y sp e c i f i c fie l d or gr o u p of fie l d s fro m the inp u t it

Pa g e 56

re c e i v e s , an d al s o op t i o n a l l y to re-ar r a n g e or re- for m a t the m . It’s thi s si n g l e fea t u r e tha t we’ l l ex a m i n e no w .

Le t’ s sa y tha t we wa n t to au t o m a t i c a l l y pr i n t ou t on l y the ow n e r of a fil e fro m a lon g di r e c t o r y lis t i n g . If we loo k at the ou t p u t of an ls –l no w :

[drysdalk@ioprov2 drysdalk]$ ls -ltotal 24-rw-rw-r-- 1 drysdalk drysdalk 13 Feb 19 21:14 cat.jpg-rw-rw-r-- 1 drysdalk sys 36 Feb 19 21:14 file1.txt-rw-rw-r-- 1 drysdalk sys 62 Feb 19 21:14 file2.txt-rw-rw-r-- 1 drysdalk sys 30 Feb 19 21:14 file3.txt-rw-rw-r-- 1 drysdalk drysdalk 91 Feb 19 21:14 furby.jpg-rwx------ 1 aroberts sys 73 Feb 19 21:15 script.sh[drysdalk@ioprov2 drysdalk]$

He r e we ca n se e a co l l e c t i o n of m i x e d fil e s , wi t h di f f e r i n g ow n e r s h i p s an d pe r m i s s i o n s . No w the ow n e r of a fil e is the thi r d fie l d , co u n t i n g al o n g fro m the lef t. awk, as pr e v i o u s l y me n t i o n e d , ha s the ab i l i t y to pr i n t ou t the co n t e n t s of a sp e c i f i c na m e d fie l d . Th e syn t a x for thi s is as fol l o w s :

awk ‘{print $<number of field>}’

So, for ex a m p l e , to pr i n t the thi r d fie l d , we wo u l d ne e d to us e the syn t a x of:

awk ‘{print $3}‘

S o let’ s se e wh a t ha p p e n s if we pi p e the ou t p u t of an ls –l thr o u g h jus t su c h an awk com m a n d :

[drysdalk@ioprov2 drysdalk]$ ls -l | awk '{print $3}'

drysdalkdrysdalkdrysdalkdrysdalkdrysdalkaroberts[drysdalk@ioprov2 drysdalk]$Wh y tha t ini t i a l bl a n k lin e in the ou t p u t of awk ? Be c a u s e the ve r y fir s t line in the ou t p u t of the ls –l is:

total 24

Pa g e 57

Th i s lin e do e s no t ha v e a thi r d fie l d – it on l y ha s two fie l d s in to t a l . So the be s t tha t awk ca n do in thi s ci r c u m s t a n c e is di s p l a y a bl a n k , em p t y line, si n c e the r e is no da t a for it to pr i n t .

Le a v i n g tha t as i d e , ho w e v e r , yo u wi l l se e tha t we ha v e ind e e d suc c e s s f u l l y pr i n t e d ou t the ow n e r , an d on l y the ow n e r , for ea c h of the fil e s , in the or d e r tha t the y ap p e a r e d in the lon g di r e c t o r y lis t i n g .

W h a t if we wa n t e d to pr i n t ou t on l y the gr o u p for ea c h of the fil e s ? Co u n t i n g al o n g fro m the lef t ag a i n , we ca n se e tha t the gr o u p is the fou r t h fie l d . So we wo u l d ne e d to us e the fol l o w i n g co m m a n d :

[drysdalk@ioprov2 drysdalk]$ ls -l | awk '{print $4}'

drysdalksyssyssysdrysdalksys[drysdalk@ioprov2 drysdalk]$

An d on c e ag a i n , we ha v e the inf o r m a t i o n we we r e af t e r , an d no t h i n g el s e .

No w thi s ca n be us e f u l , bu t it is ev e n mo r e us e f u l if we com b i n e fie l d s tog e t h e r . Fo r ex a m p l e , let’ s sa y we wa n t e d to pr i n t the fil e n a m e , fol l o w e d by the ow n e r an d gr o u p . We ca n se p a r a t e fie l d s wi t h a com m a in awk, to ge t it to pr i n t the m ou t se p a r a t e d by a sin g l e sp a c e :

[drysdalk@ioprov2 drysdalk]$ ls -l | awk '{print $9,$3,$4}'

cat.jpg drysdalk drysdalkfile1.txt drysdalk sysfile2.txt drysdalk sysfile3.txt drysdalk sysfurby.jpg drysdalk drysdalkscript.sh aroberts sys[drysdalk@ioprov2 drysdalk]$So he r e we pr i n t e d ou t the ni n t h fie l d (the fil e n a m e), fol l o w e d by the thi r d fie l d (the ow n e r), an d en d i n g wi t h the fou r t h fie l d (the gr o u p ).

La s t l y (and thi s is wh e r e thi n g s ge t qu i t e a bi t mo r e com p l i c a t e d), we ca n al s o pla c e ou r ow n tex t in the ou t p u t of awk, by pl a c i n g it in the pr i n t com m a n d in be t w e e n do u b l e- qu o t e s .

W e’ l l loo k ag a i n at ou r ve r y fir s t ex a m p l e ab o v e , wh e r e we us e d awk to pr i n t ou t the ow n e r of the fil e s . Le t’ s sa y we wa n t e d to pu t a de s c r i p t i o n

Pa g e 58

of the da t a at the st a r t – so m e t h i n g lik e “Ow n e r : “. Th i s ca n be do n e fai r l y ea s i l y , sim p l y by pu t t i n g “Ow n e r : “ in do u b l e- qu o t e s in the awk pr i n t com m a n d , an d the n pu t t i n g a com m a be t w e e n the tex t an d the fie l d we wa n t to pr i n t , jus t as we did ab o v e .

Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ ls -l | awk '{print “Owner:”,$3}'Owner: Owner: drysdalkOwner: drysdalkOwner: drysdalkOwner: drysdalkOwner: drysdalkOwner: aroberts[drysdalk@ioprov2 drysdalk]$

He r e ’ s a mo r e com p l e x ex a m p l e If we wa n t e d to pr o d u c e a ni c e hu m a n-rea d a b l e rep o r t lis t i n g ea c h of the fil e s in the cu r r e n t di r e c t o r y , wh o ow n s the m , an d wh a t gr o u p the y ar e as s o c i a t e d wi t h , we co u l d do thi s:

[drysdalk@ioprov2 drysdalk]$ ls -l | awk '{print "The file",$9,"is owned by user",$3,"and group",$4}'The file is owned by user and groupThe file cat.jpg is owned by user drysdalk and group drysdalkThe file file1.txt is owned by user drysdalk and group sysThe file file2.txt is owned by user drysdalk and group sysThe file file3.txt is owned by user drysdalk and group sysThe file furby.jpg is owned by user drysdalk and group drysdalkThe file script.sh is owned by user aroberts and group sys[drysdalk@ioprov2 drysdalk]$

Ag a i n , we ha v e a bo g u s fir s t lin e in the ou t p u t of awk, be c a u s e the ve r y fir s t lin e of ls –l do e s n ’ t ha v e en o u g h fie l d s to pr o d u c e the inf o r m a t i o n we’ v e as k e d for. If we wa n t e d to, we co u l d st r i p ou t tha t fir s t lin e fro m the ou t p u t of ls –l be f o r e it ge t s to awk in the pip e l i n e , by fir s t pi p i n g the ou t p u t thr o u g h a grep com m a n d lik e thi s:

[drysdalk@ioprov2 drysdalk]$ ls -l | grep -v total | awk '{print "The file "$9" is owned by user "$3" and group "$4}'The file cat.jpg is owned by user drysdalk and group drysdalkThe file file1.txt is owned by user drysdalk and group sysThe file file2.txt is owned by user drysdalk and group sysThe file file3.txt is owned by user drysdalk and group sysThe file furby.jpg is owned by user drysdalk and group drysdalkThe file script.sh is owned by user aroberts and group sys[drysdalk@ioprov2 drysdalk]$

Pa g e 59

He r e , ou r grep –v str i p p e d ou t al l lin e s in the lon g dir e c t o r y lis t i n g co n t a i n i n g the tex t ‘tot a l ’, an d so ou r bo g u s lin e go t dr o p p e d ou t of the pi p e l i n e be f o r e it rea c h e d awk. No t e , ho w e v e r , tha t if an y of ou r fil e s ha d ‘tot a l ’ in an y pa r t of the i r na m e or ow n e r s h i p s , thi s wo u l d ha v e ca u s e d the m to be dr o p p e d al s o , an d awk wo u l d no t ha v e ha d the op p o r t u n i t y to gi v e us a sum m a r y for the m .

So, tha t’ s awk. It ca n se e m ve r y com p l e x ind e e d wh e n yo u fir s t us e it – an d tha t’ s be c a u s e it is. If thi s is the fir s t tim e yo u’ v e en c o u n t e r e d it, yo u mi g h t we l l be loo k i n g at the s e las t two ex a m p l e s an d fin d i n g the m ve r y di f f i c u l t to fol l o w . Do n’ t wo r r y . Lo o k at the fir s t few ex a m p l e s of pr i n t i n g ou t si n g l e fie l d s , an d the n tak e it fro m the r e .

Ev e n be t t e r – try it ou t on yo u r ow n sy s t e m . Yo u ca n’ t ev e r da m a g e or de s t r o y an y t h i n g wh e n us i n g the pr i n t co m m a n d in awk, si n c e al l it do e s wh e n us e d in thi s wa y is mo d i f y the ou t p u t of ot h e r co m m a n d s . Pl a y ab o u t wi t h pr i n t i n g ou t thi s fie l d or tha t fro m a di r e c t o r y lis t i n g , an d wi t h pu t t i n g yo u r ow n tex t in be t w e e n the m . Ta l k to ot h e r pe o p l e ab o u t it. Fe e l fre e to as k ot h e r me m b e r s of the sy s t e m s ad m i n i s t r a t i o n tea m an y qu e s t i o n s yo u mi g h t ha v e , if the r e ’ s an y t h i n g yo u wa n t to cl a r i f y . We’ l l be ha p p y to he l p . Ge t t i n g to gr i p s wi t h awk is we l l wo r t h the ef f o r t .

Pa g e 60

5.5 The sed command

He r e ’ s an o t h e r com m a n d tha t , in ma n y wa y s , is qu i t e sim i l a r to awk. It is al s o ex t r e m e l y pow e r f u l , it is al s o a pr o g r a m m i n g lan g u a g e in its ow n rig h t , an d it ha s al s o ha d wh o l e bo o k s wr i t t e n ab o u t it. Bu t as be f o r e , we’ r e on l y int e r e s t e d he r e in the on e pa r t i c u l a r bi t of its fun c t i o n a l i t y tha t yo u ar e lik e l y to wa n t to us e of t e n .

A g a i n mu c h lik e awk, sed mo d i f i e s the ou t p u t of ot h e r pr o g r a m s , an d di s p l a y s the m in the i r ne w l y- ch a n g e d for m . On e thi n g it ca n do ve r y we l l an d ve r y qu i c k l y is su b s t i t u t e on e pi e c e of tex t for an o t h e r .

Th e sy n t a x for thi s is as fol l o w s :

sed s/<old text>/<new text>/g

Fo r ex a m p l e . Le t’ s sa y we ha v e a fil e ca l l e d gr e e t i n g . t x t , tha t co n t a i n s the tex t ‘He l l o!’. We wa n t to ch a n g e ev e r y oc c u r r e n c e of ‘He l l o’ to ‘Goo d b y e ’ . We ca n do thi s as fol l o w s :

[drysdalk@ioprov2 drysdalk]$ cat greeting.txtHello![drysdalk@ioprov2 drysdalk]$ sed s/Hello/Goodbye/g < greeting.txtGoodbye![drysdalk@ioprov2 drysdalk]$ cat greeting.txtHello![drysdalk@ioprov2 drysdalk]$

He r e , we us e d the inp u t re-dir e c t o r < to fee d the fil e greeting.txt in to sed, an d it pe r f o r m e d the de s i r e d su b s t i t u t i o n . No t e al s o tha t it di d no t ac t u a l l y ch a n g e the fil e greeting.txt – yo u’ l l se e fro m tha t fin a l cat tha t it wa s st i l l un c h a n g e d . sed sim p l y di s p l a y s on the ter m i n a l the ch a n g e d tex t , un l e s s yo u tel l it ot h e r w i s e .

To sa v e the ou t p u t an d ma k e it pe r m a n e n t , we us e an ou t p u t re-di r e c t o r to se n d the mo d i f i e d tex t to a fil e. No w , he r e ’ s a com m o n ea r l y mi s t a k e tha t ma n y pe o p l e ma k e wh i l s t us i n g sed:

Pa g e 61

[drysdalk@ioprov2 drysdalk]$ cat greeting.txtHello![drysdalk@ioprov2 drysdalk]$ sed s/Hello/Goodbye/g < greeting.txt > greeting.txt[drysdalk@ioprov2 drysdalk]$ cat greeting.txt[drysdalk@ioprov2 drysdalk]$

So, wh a t we n t wr o n g he r e ? Yo u pr o b a b l y ex p e c t e d greeting.txt to co n t a i n the tex t ‘Go o d b y e!’. Ins t e a d , as yo u’ l l se e fro m tha t fin a l cat com m a n d , the fil e is no w em p t y – its co n t e n t s we r e de s t r o y e d .

Th e rea s o n for thi s is tha t an y tim e inp u t or ou t p u t re-di r e c t o r s ar e us e d , the s e ar e the fir s t thi n g s tha t Lin u x de a l s wi t h , be f o r e ac t u a l l y ru n n i n g the co m m a n d tha t us e s the m . Th e sy s t e m saw tha t the ou t p u t of the com m a n d it ha d be e n pa s s e d wa s to go to a fil e ca l l e d greeting.txt, wh i c h wa s to be ov e r w r i t t e n (reme m b e r tha t > sim p l y ov e r w r i t e s a fil e, an d do e s no t ap p e n d to it). So be f o r e sed wa s ac t u a l l y ru n, the fil e greeting.txt wa s em p t i e d ou t an d pr e p a r e d for ne w da t a , si n c e the sy s t e m kn e w tha t it wa s to be de s t i n a t i o n for the ou t p u t of sed.

N o w tha t the ou t p u t fil e is rea d y , sed is ac t u a l l y run – an d tak e s as its inp u t the ne w l y- em p t y greeting.txt. So the r e is no t h i n g fo r it to do, an d it qu i t e rig h t l y wr i t e s its ou t p u t – wh i c h is no t h i n g at al l, si n c e tha t is al l tha t wa s in greeting.txt wh e n sed ope n e d it – ba c k to the fil e.

S o if we ha d us e d ap p e n d ou t p u t re-di r e c t i o n , wo u l d the co n t e n t s ha v e be e n pr e s e r v e d ? Le t’ s try it an d se e:

[drysdalk@ioprov2 drysdalk]$ cat greeting.txtHello![drysdalk@ioprov2 drysdalk]$ sed s/Hello/Goodbye/g < greeting.txt >> greeting.txt[drysdalk@ioprov2 drysdalk]$ cat greeting.txtHello!Goodbye![drysdalk@ioprov2 drysdalk]$

An d the y ar e ind e e d , si n c e thi s tim e wh e n Li n u x pr e p a r e d greeting.txt fo r wr i t i n g , it kn e w we we r e us i n g ap p e n d ou t p u t re-dir e c t i o n , an d so did not wi p e the fil e an d st a r t it af r e s h be f o r e pa s s i n g it to sed.

H o w e v e r , thi s st i l l isn’ t qu i t e wh a t we wa n t e d . We do ha v e the new gr e e t i n g , bu t the old on e is st i l l the r e too. So the pr o p e r wa y to rep l a c e ou r or i g i n a l file wi t h the new gr e e t i n g wo u l d be:

[drysdalk@ioprov2 drysdalk]$ cat greeting.txt

Pa g e 62

Hello![drysdalk@ioprov2 drysdalk]$ sed s/Hello/Goodbye/g < greeting.txt > greeting2.txt[drysdalk@ioprov2 drysdalk]$ cat greeting2.txtGoodbye![drysdalk@ioprov2 drysdalk]$ mv greeting2.txt greeting.txt[drysdalk@ioprov2 drysdalk]$ cat greeting.txtGoodbye![drysdalk@ioprov2 drysdalk]$

Wh a t we ha v e to do is wr i t e ou r ou t p u t to a di f f e r e n t fil e, on e tha t do e s no t ye t ex i s t (or wh i c h we don’ t ca r e ab o u t ov e r- wr i t i n g). Th i s wa y , ou r ou t p u t is sa v e d to a di f f e r e n t loc a t i o n fro m the or i g i n a l . We the n mo v e thi s ne w l y- cr e a t e d fil e on top of the or i g i n a l , fin a l l y rep l a c i n g it in a sa f e ma n n e r . We now ha v e su c c e s s f u l l y up d a t e d greeting.txt to rea d ‘Goo d b y e!’ rat h e r tha n ‘He l l o!’

U s i n g se d do e s no t af f e c t an y t h i n g ex c e p t the tex t tha t it ma t c h e s . Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ cat linux-info.txtThe grep command can be used to search files.The cd command can be used to obtain a directory listing.The awk command is scary.[drysdalk@ioprov2 drysdalk]$ cat linux-info.txt | sed s/cd/ls/gThe grep command can be used to search files.The ls command can be used to obtain a directory listing.The awk command is scary.[drysdalk@ioprov2 drysdalk]$

He r e we suc c e s s f u l l y up d a t e d linux-info.txt to co n t a i n the co r r e c t inf o r m a t i o n , an d as yo u ca n se e al l the ot h e r tex t wa s lef t ex a c t l y as it or i g i n a l l y wa s .

Pa g e 63

So, tha t’ s sed. No t qu i t e as sc a r y as awk, bu t yo u ca n run in to so m e na s t y si t u a t i o n s if yo u’ r e no t ca r e f u l ab o u t the de s t i n a t i o n s of yo u r ou t p u t re-di r e c t o r s . Oh, an d it’s ca s e se n s i t i v e , lik e mo s t Lin u x com m a n d s . So ma k e su r e yo u ge t yo u r low e r- an d up p e r- ca s e s co r r e c t in the tex t yo u wa n t to se a r c h for an d rep l a c e .

5.6 Some other useful text processing commands

We’ l l en d thi s se c t i o n on a ligh t e r no t e . We’ l l loo k at som e ot h e r sim p l e com m a n d s tha t ar e us e f u l fo r tex t pr o c e s s i n g pu r p o s e s , an d the kin d of sit u a t i o n s tha t yo u mi g h t wa n t to us e the m in.

The wc command

Th e am u s i n g l y- na m e d wc co m m a n d is us e d to gi v e yo u a w or d c ou n t for a fil e, or gr o u p of fil e s . Its sy n t a x is fai r l y st r a i g h t f o r w a r d , so we’ l l jum p str a i g h t in wi t h an ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ cat example.txtI am the first line.I am the second line.I am the third line.[drysdalk@ioprov2 drysdalk]$ wc example.txt 3 15 64 example.txt[drysdalk@ioprov2 drysdalk]$

Th e thr e e nu m b e r s we’ v e be e n gi v e n for the fil e br e a k dow n as fol l o w s :

1 st num b e r – nu m b e r of line s2 nd num b e r – nu m b e r of wo r d s3 rd num b e r – nu m b e r of ch a r a c t e r s

So m e t i m e s (mos t tim e s , to be ho n e s t), we on l y wa n t to us e wc to ge t the

nu m b e r of line s . Th a t ’ s do n e wi t h the –l fla g:

[drysdalk@ioprov2 drysdalk]$ wc -l example.txt 3 example.txt[drysdalk@ioprov2 drysdalk]$

An d tha t’ s rea l l y al l the r e is to sa y ab o u t wc.

The uniq command

Pa g e 64

Th e pu r p o s e of uniq is to ret u r n on l y tho s e lin e s in a fil e wh i c h ar e

uniq ue. Li k e wc, the syn t a x is ve r y st r a i g h t f o r w a r d , so we’ l l st a r t wi t h an ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ cat example.txtcatcatcatdogdogfurbyfurbyfurbyfurby[drysdalk@ioprov2 drysdalk]$ cat example.txt | uniqcatdogfurby[drysdalk@ioprov2 drysdalk]$

So far, so go o d . Bu t tak e a loo k at thi s ne x t ex a m p l e , an d se e if yo u ca n gu e s s wh y it do e s n ’ t wo r k :

[drysdalk@ioprov2 drysdalk]$ cat example.txtcatfurbycatfurbycatfurbydogfurbydog[drysdalk@ioprov2 drysdalk]$ cat example.txt | uniqcatfurbycatfurbycatfurbydogfurbydog[drysdalk@ioprov2 drysdalk]$

Th e rea s o n tha t uniq co u l d n ’ t pr o v i d e us wi t h the un i q u e line s in the fil e is tha t thi s tim e , the inf o r m a t i o n in the fil e wa s n ’ t al r e a d y al p h a b e t i c a l l y so r t e d so r t e d . Th e uniq com m a n d ca n on l y wo r k on so r t e d da t a , si n c e it el i m i n a t e s du p l i c a t e s ba s e d on the i r di r e c t pr o x i m i t y to on e an o t h e r . Fo r ex a m p l e , if it se e s thr e e line s sa y i n g ‘cat’ in a row it wi l l re d u c e the m to on l y on e. Ho w e v e r , if the r e ar e thr e e ‘ca t’ line s sca t t e r e d thr o u g h o u t the fil e (as in ou r se c o n d ex a m p l e), the y al l ge t al l o w e d thr o u g h , si n c e the y ar e reg a r d e d as se p a r a t e oc c u r r e n c e s of the st r i n g , an d no t rep e a t i n g du p l i c a t e s .

Pa g e 65

So, for a fil e lik e thi s, we ne e d to pip e it thr o u g h sort fir s t , be f o r e pa s s i n g it to uniq:

[drysdalk@ioprov2 drysdalk]$ cat example.txtcatfurbycatfurbycatfurbydogfurbydog[drysdalk@ioprov2 drysdalk]$ cat example.txt | sort | uniqcatdogfurby[drysdalk@ioprov2 drysdalk]$

Th e r e is on e pa r t i c u l a r fla g of uniq tha t yo u’ r e lik e l y to us e mo r e tha n

an y ot h e r , an d tha t’ s the –c fla g. Th i s ma k e s uniq pr i n t the nu m b e r of tim e s tha t ea c h line oc c u r r e d in a fil e, lik e so:

[drysdalk@ioprov2 drysdalk]$ cat example.txt | sort | uniq -c 3 cat 2 dog 4 furby[drysdalk@ioprov2 drysdalk]$

An d if yo u lik e, we ca n of co u r s e so r t thi s ou t p u t nu m e r i c a l l y , to ge t the m in or d e r , fro m lea s t fre q u e n t to mo s t :

[drysdalk@ioprov2 drysdalk]$ cat example.txt | sort | uniq -c | sort -bn 2 dog 3 cat 4 furby[drysdalk@ioprov2 drysdalk]$

The head and tail commands

No w we’ l l loo k at tw o ot h e r com m a n d s tha t ar e ve r y us e f u l , an d ve r y sim p l e . head an d tail, as the i r na m e s va g u e l y imp l y , gi v e yo u the fir s t few line s of a fil e an d the las t few line s of a fil e, res p e c t i v e l y .

Fo r ex a m p l e :

[drysdalk@ioprov2 drysdalk]$ cat example.txtThe first line.

Pa g e 66

The second line.The third line.The fourth line.The fifth line.The sixth line.The seventh line.The eighth line.The ninth line.The tenth line.The eleventh line.The twelfth line.The thirteenth line.The fourteenth line.The fifteenth line.[drysdalk@ioprov2 drysdalk]$

He r e ’ s wh a t we se e wi t h head:

[drysdalk@ioprov2 drysdalk]$ head example.txtThe first line.The second line.The third line.The fourth line.The fifth line.The sixth line.The seventh line.The eighth line.The ninth line.The tenth line.[drysdalk@ioprov2 drysdalk]$

An d he r e ’ s wh a t we se e wi t h tail:

[drysdalk@ioprov2 drysdalk]$ tail example.txtThe sixth line.The seventh line.The eighth line.The ninth line.The tenth line.The eleventh line.The twelfth line.The thirteenth line.The fourteenth line.The fifteenth line.[drysdalk@ioprov2 drysdalk]$

So in ea c h ca s e , we se e ei t h e r the fir s t ten line s or the las t ten lin e s – thi s is the de f a u l t be h a v i o u r for bo t h the s e com m a n d s .

W h a t if we wa n t a di f f e r e n t nu m b e r of lin e s , tho u g h ? Ea s y – we jus t us e the nu m b e r of line s we wa n t as the fla g we pa s s to the com m a n d . Fo r ex a m p l e :

Pa g e 67

[drysdalk@ioprov2 drysdalk]$ head -3 example.txtThe first line.The second line.The third line.[drysdalk@ioprov2 drysdalk]$

[drysdalk@ioprov2 drysdalk]$ tail -3 example.txtThe thirteenth line.The fourteenth line.The fifteenth line.[drysdalk@ioprov2 drysdalk]$

On the i r ow n , the s e com m a n d s ar e n ’ t ter r i b l y us e f u l . Bu t wh e n com b i n e d wi t h ot h e r co m m a n d s in pi p e s , the y ar e. Fo r ins t a n c e , go i n g ba c k to ou r pr e v i o u s uniq ex a m p l e s , wh a t if we on l y wa n t e d to kn o w the ter m tha t oc c u r r e d the mo s t in example.txt ? We l l , we co u l d do thi s:[drysdalk@ioprov2 drysdalk]$ cat example.txt | sort | uniq -c | sort –bg | tail -1 4 furby[drysdalk@ioprov2 drysdalk]$

Th e r e ’ s an o t h e r fea t u r e of tail sp e c i f i c a l l y tha t’ s ex t r e m e l y us e f u l , an d

tha t is the –f fla g. No w , thi s is a bi t ha r d to de m o n s t r a t e in an ex a m p l e on a wr i t t e n pa g e , bu t wh a t it do e s is at t a c h to a fil e, an d ke e p s up d a t i n g yo u r ter m i n a l ev e r y tim e new line s ap p e a r at the en d of tha t fil e. Th i s is ve r y us e f u l wh e n yo u wa n t to ke e p an ey e on a log fil e, or an y ot h e r file tha t is ch a n g i n g in rea l- tim e tha t yo u wa n t to mo n i t o r .

Fo r ex a m p l e , let’s sa y yo u wa n t to ke e p an ey e on the fil e server.log. Th i s fil e is up d a t e d on c e a mi n u t e . Yo u ca n at t a c h a tail to it as fol l o w s :

[drysdalk@ioprov2 drysdalk]$ tail -f server.log20/02/2007 11:14 Nothing happening.

No w , on e mi n u t e lat e r yo u r ter m i n a l mi g h t loo k lik e:

[drysdalk@ioprov2 drysdalk]$ tail -f server.log20/02/2007 11:14 Nothing happening. 20/02/2007 11:15 Nothing happening.

An d so on. As I sa y, a bi t ha r d to ef f e c t i v e l y de m o n s t r a t e on pa p e r , bu t tha t’ s the ge n e r a l ide a be h i n d tail –f.

5.7 The man command, and command- line help

Pa g e 68

La s t l y , we’ l l loo k at a co m m a n d tha t is us e d to gi v e yo u on- line he l p . Li n u x ma i n t a i n s its ow n bu i l t- in ma n u a l , lis t i n g the syn t a x an d us a g e inf o r m a t i o n for alm o s t al l of its com m a n d s . No w yo u mi g h t we l l be wo n d e r i n g wh y thi s wa s n ’ t the ve r y fir s t thi n g tha t we tal k e d ab o u t – the rea s o n is tha t wh i l s t som e ma n u a l pa g e s ca n be ve r y us e f u l , ot h e r s ca n be ve r y co n f u s i n g , es p e c i a l l y wh e n yo u’ r e ne w to Lin u x .

Th i s on- lin e ma n u a l is ac c e s s e d wi t h the man com m a n d . Th e syn t a x is ve r y sim p l e :

man <name of command whose manual page you want to read>

Fo r ex a m p l e , to rea d the ma n u a l pa g e for ls, yo u wo u l d typ e :

[drysdalk@ioprov2 drysdalk]$ man ls

An d the ma n u a l pa g e wo u l d the n ap p e a r . Yo u ca n na v i g a t e thr o u g h ma n u a l pa g e s on e scr e e n at a tim e by pr e s s i n g the sp a c e b a r to mo v e for w a r d , an d the b key to mo v e ba c k w a r d s . Yo u ca n al s o us e the cu r s o r ke y s to mo v e ba c k an d for t h on e line at a tim e .

So m e co m m a n d s ca n al s o giv e yo u inf o r m a t i o n ab o u t the i r us a g e or sy n t a x , by us i n g on e of two sp e c i f i c fla g s . Mo s t Li n u x co m m a n d s wi l l un d e r s t a n d the fla g s –? and ––help (note tha t the r e ar e tw o da s h e s in thi s las t on e).

Fo r ex a m p l e , let’s sa y yo u wa n t e d ful l inf o r m a t i o n on the us a g e an d sy n t a x of wc. Yo u co u l d typ e :

[drysdalk@ioprov2 drysdalk]$ wc --helpUsage: wc [OPTION]... [FILE]...Print byte, word, and newline counts for each FILE, and a total line ifmore than one FILE is specified. With no FILE, or when FILE is -,read standard input. -c, --bytes print the byte counts -m, --chars print the character counts -l, --lines print the newline counts -L, --max-line-length print the length of the longest line -w, --words print the word counts --help display this help and exit --version output version information and exit

Report bugs to <bug-coreutils@gnu.org>.[drysdalk@ioprov2 drysdalk]$

So if ev e r yo u ar e st u c k , thi s is a go o d thi n g to rem e m b e r – tho u g h som e t i m e s un d e r s t a n d i n g wh a t a com m a n d ’ s so- ca l l e d “hel p” is try i n g to tel l yo u ca n be as di f f i c u l t as fig u r i n g it ou t yo u r s e l f .

Pa g e 69

Pa g e 70

5.7 End-of-section summary

We l l , we’ v e st e p p e d thi n g s up qu i t e a bi t in thi s se c t i o n . Do n’ t wo r r y if an y of it – pa r t i c u l a r l y the us a g e of awk an d sed – isn’ t too cle a r to yo u ye t . Th e be s t wa y to ge t to gr i p s wi t h com m a n d s thi s com p l e x is to try us i n g the m yo u r s e l f , on yo u r ow n sy s t e m .

To rec a p , the n , in thi s se c t i o n we ha v e lea r n e d how to:

- Us e sort to pe r f o r m mo r e com p l e x typ e s of so r t i n g- Us e find to loc a t e a wi d e r va r i e t y of di f f e r e n t fil e s an d dir e c t o r i e s- Us e grep to ca r r y ou t mo r e com p l e x an d in-de p t h se a r c h e s- Us e awk to pr i n t ou t sp e c i f i c fie l d s in the ou t p u t of a com m a n d- Us e sed to se a r c h for an d rep l a c e pa r t i c u l a r ph r a s e s wi t h i n a fil e- Us e wc to ob t a i n a co u n t of the lin e s , wo r d s an d ch a r a c t e r s in a

fil e- Us e uniq to ob t a i n on l y tho s e lin e s in a fil e wh i c h ar e un i q u e- Us e head and tail to vie w the be g i n n i n g an d en d of a file,

res p e c t i v e l y- Co n s u l t on- lin e he l p wi t h the man com m a n d , or the -? An d ––help

fla g s

Pa g e 71

6. Advanced conceptsIn thi s se c t i o n we’ l l di s c u s s so m e ad v a n c e d Lin u x sy s t e m co n c e p t s . Th e ide a s an d com m a n d s we’ l l be loo k i n g at he r e ar e ba s i c a l l y wh a t com p r i s e the fun d a m e n t a l sk i l l s of sy s t e m s ad m i n i s t r a t i o n .

6.1 Processes

Ev e r y run n i n g pr o g r a m on a Li n u x sy s t e m is kn o w n as a process . Yo u ca n thi n k of a pr o c e s s as an ac t i v e co p y in the sy s t e m ’ s me m o r y of a bi n a r y pr o g r a m tha t is st o r e d on the sy s t e m ’ s di s c . Fo r ex a m p l e , wh e n yo u run ls, it is loa d e d in to me m o r y , an d a pr o c e s s is cr e a t e d tha t ca r r i e s ou t wh a t e v e r tas k yo u as k e d ls to pe r f o r m .

Yo u ca n vie w the pr o c e s s e s run n i n g on a sy s t e m wi t h the ps co m m a n d . At its sim p l e s t , yo u ca n jus t typ e ps on its ow n to se e so m e ba s i c inf o r m a t i o n ab o u t al l pr o c e s s e s cu r r e n t l y run n i n g on the sy s t e m tha t be l o n g to yo u:

[drysdalk@ioprov2 drysdalk]$ ps PID TTY TIME CMD 1648 pts/7 00:00:00 bash31574 pts/7 00:00:00 ps[drysdalk@ioprov2 drysdalk]$

Le t’ s loo k at ea c h bi t of thi s pr o c e s s lis t i n g in tur n . We ha v e fou r co l u m n s , PI D, TT Y , TI M E an d CM D . Th e y ea c h ha v e the fol l o w i n g me a n i n g s

PIDTh i s sh o w s the P ro c e s s ID of ea c h of the pr o c e s s e s . Th e PI D is a un i q u e ide n t i f i e r as s o c i a t e d wi t h ev e r y pr o c e s s . No two run n i n g pr o c e s s e s ca n ev e r ha v e the sa m e PID , tho u g h ol d PI D nu m b e r s ca n be re-us e d wh e n wh a t e v e r pr o c e s s wa s us i n g the m is fin i s h e d , an d rel e a s e s the m ba c k to the sy s t e m .

TTYTh i s sh o w s the ter m i n a l tha t the us e r wh o lau n c h e d the pr o c e s s (in thi s ca s e , yo u) wa s co n n e c t e d to wh e n it be g a n run n i n g . Th i s on e is a bi t tr i c k i e r to ex p l a i n , bu t ev e r y rem o t e or loc a l us e r wh o is log g e d on to a Li n u x sy s t e m ha s the i r ow n ter m i n a l as s o c i a t e d wi t h the m . A ter m i n a l ca n be tho u g h t of as a vir t u a l ke y b o a r d an d mo n i t o r – it is the vir t u a l

Pa g e 72

de v i c e tha t is us e d to al l o w the rem o t e Li n u x sy s t e m to ge t its inp u t fro m the ke y b o a r d at t a c h e d to yo u r ow n PC, an d to se n d its ou t p u t ba c k to yo u r ow n PC’ s mo n i t o r .E a c h us e r log g e d on to a sy s t e m ha s a un i q u e ter m i n a l as s o c i a t e d wi t h the m . No two us e r s ca n be co n n e c t e d to the sam e ter m i n a l sim u l t a n e o u s l y . Ag a i n , lik e PI D s , ter m i n a l s ca n be re- iss u e d wh e n the pe r s o n wh o is cu r r e n t l y us i n g the m log s of f, thu s fre e i n g the m up.

TIMETh e TIM E co l u m n doe s no t sh o w yo u the tim e of da y tha t a pr o c e s s wa s lau n c h e d at, wh i c h is wh a t yo u mi g h t fir s t ex p e c t . Ra t h e r , it sh o w s the tot a l am o u n t of cum u l a t i v e CP U tim e tha t a pr o c e s s ha s co n s u m e d thr o u g h o u t its lif e t i m e . No w thi s isn’ t as sim p l e as it so u n d s – if yo u st a r t a pr o g r a m at 1 P M and ch e c k ag a i n at 2P M to se e how mu c h CP U tim e tha t it ha s co n s u m e d , yo u ar e lik e l y to fin d the va l u e is on l y a few sec o n d s , or a few mi n u t e s at mo s t .

Th e rea s o n for thi s is tha t a pr o c e s s do e s no t st e a d i l y co n s u m e al l the CP U ’ s tim e wh e n it is run n i n g . If it di d, the r e wo u l d be no op p o r t u n i t y for an y ot h e r pr o g r a m on the sy s t e m to do an y t h i n g , si n c e the CP U wo u l d sp e n d its en t i r e tim e se r v i n g on e sin g l e pr o c e s s . As a res u l t , it wo u l d ne v e r ge t the ch a n c e to gi v e an y run- tim e to an y ot h e r pr o c e s s e s , inc l u d i n g vi t a l op e r a t i n g sy s t e m on e s . In ot h e r wo r d s , the sy s t e m wo u l d be co m p l e t e l y un- us a b l e .

S o, Li n u x on l y giv e s as mu c h CP U tim e to ea c h pr o c e s s as it ab s o l u t e l y req u i r e s in or d e r to do wh a t it wa s cr e a t e d to do. Mo s t pr o g r a m s sp e n d the va s t ma j o r i t y of the i r tim e by far do i n g no t h i n g – qu i t e lit e r a l l y . Th i s tim e sp e n t in be t w e e n us e f u l op e r a t i o n s do e s no t co u n t tow a r d s the tot a l am o u n t of CP U tim e us e d by a pr o c e s s . Th e TIM E va l u e on l y go e s up wh e n the pr o c e s s is bo t h rec e i v i n g at t e n t i o n from the CP U , an d is ac t u a l l y do i n g som e t h i n g us e f u l wi t h it.

CMDTh i s fie l d is fai r l y st r a i g h t f o r w a r d . It sh o w s the na m e of the pr o g r a m tha t wa s run to cr e a t e the pr o c e s s in qu e s t i o n . In ou r ps lis t i n g , we ca n se e two pr o c e s s e s , on e for bash, an d on e for ps its e l f . No w ps is fai r l y se l f-ex p l a n a t o r y – we wo u l d ce r t a i n l y ex p e c t it to ap p e a r , be c a u s e ps had to cr e a t e a pr o c e s s for its e l f in or d e r to run, an d sh o w us the lis t of pr o c e s s e s .

Pa g e 73

Bu t wh a t is bash ? Th i s wi l l alw a y s ap p e a r , reg a r d l e s s of wh a t yo u ar e do i n g or wh a t Lin u x sy s t e m yo u ar e on. Th e rea s o n for thi s is tha t ba s h

is yo u r shell .

Pa g e 74

A sh e l l is the na m e gi v e n to a sp e c i a l typ e of pr o g r a m wh i c h is run wh e n e v e r yo u log in. Yo u r sh e l l is wh a t gi v e s yo u a co m m a n d pr o m p t , an d al l o w s yo u to en t e r co m m a n d s tha t Lin u x ca n the n run. Ba s h is the de f a u l t sh e l l in Li n u x , an d wi l l al w a y s be run n i n g (beca u s e if it wa s n ’ t , yo u wo u l d n ’ t ha v e a com m a n d pr o m p t).

So, tha t’ s the ba s i c s of pr o c e s s vie w i n g . Ho w e v e r , the ab o v e on l y sh o w e d us the pr o c e s s e s tha t we ou r s e l v e s we r e res p o n s i b l e for cr e a t i n g . Wh a t if we wa n t to se e mo r e inf o r m a t i o n ab o u t the s e pr o c e s s e s be y o n d thi s ba s i c de t a i l ?

As yo u mi g h t ex p e c t , ps ha s a gr e a t ma n y fla g s wh i c h ca n be us e d to mo d i f y or ex t e n d its op e r a t i o n in a wi d e va r i e t y of wa y s . We ca n se e

mo r e inf o r m a t i o n ab o u t the run n i n g pr o c e s s e s wi t h the -u fla g, as fol l o w s :

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDdrysdalk 22868 0.0 0.2 4272 1376 pts/3 S 10:32 0:00 -bashdrysdalk 23038 0.0 0.1 2728 680 pts/3 R 10:37 0:00 ps -u

No w he r e we ha v e som e ex t r a co l u m n s , wi t h the fol l o w i n g inf o r m a t i o n :

USERTh i s sh o w s the us e r n a m e of the pe r s o n wh o is res p o n s i b l e for the ini t i a l run n i n g of thi s pr o c e s s . In thi s ca s e , si n c e we ar e on l y se e i n g ou r ow n pr o c e s s e s , the us e r n a m e di s p l a y e d is tha t of drysdalk, the cu r r e n t l y-log g e d- in us e r .

%CPUTh i s ind i c a t e s the cu r r e n t pe r c e n t a g e of the CP U ' s tim e tha t the pr o c e s s e s ar e us i n g . As yo u ca n se e fro m thi s lis t i n g , the us a g e is ne g l i g i b l e , wh i c h is wh a t we wo u l d ex p e c t – ne i t h e r ou r sh e l l no r ou r ps com m a n d sh o u l d be us i n g an y t h i n g mo r e tha n the ba r e mi n i m u m of CP U res o u r c e s , un d e r no r m a l ci r c u m s t a n c e s .

%ME MSi m i l a r to % C P U , thi s sh o w s the pe r c e n t a g e of the sy s t e m ' s me m o r y tha t the s e pr o c e s s e s ar e us i n g . Ag a i n , the nu m b e r s ar e tin y for ou r two pr o c e s e s s he r e , an d thi s sh o u l d alw a y s be the cas e .

Pa g e 75

VSZTh i s sh o w s the am o u n t of vi r t u a l me m o r y tha t the pr o c e s s e s ar e us i n g , in kil o b y t e s . Vi r t u a l me m o r y is the tot a l am o u n t of me m o r y al l o c a t e d by a pr o c e s s , tak i n g in to ac c o u n t bo t h the ph y s i c a l me m o r y us a g e of the pr o c e s s an d the am o u n t of sw a p sp a c e it is us i n g as we l l .

RSSTh i s is the res i d e n t me m o r y us a g e of the pr o c e s s e s . Th i s fig u r e is the am o u n t of ac t u a l ph y s i c a l me m o r y cu r r e n t l y be i n g us e d by the pr o c e s s e s . So an y di f f e r e n c e be t w e e n the vir t u a l si z e an d the res i d e n t siz e is the am o u n t of sw a p sp a c e be i n g us e d by the pr o c e s s .

STATTh i s is the cu r r e n t st a t u s of the pr o c e s s . Th e r e ar e a va r i e t y of va l u e s for thi s, bu t the mo s t com m o n on e s tha t yo u wi l l en c o u n t e r ar e:

R Ru n n i n g – thi s ind i c a t e s tha t thi s pr o c e s s is the on e cu r r e n t l ybe i n g ex p e c t e d by the CP U . No t e tha t a sin g l e CP U or CP U

co r eca n on l y ex e c u t e on e pr o g r a m at a tim e – sy s t e m s cr e a t e theill u s i o n of mo r e tha n on e pr o g r a m run n i n g at on c e by rap i d l ysw i t c h i n g be t w e e n al l the pr o c e s s e s tha t re q u i r e the CP U ' sat t e n t i o n . So yo u wi l l on l y us u a l l y se e as ma n y pr o c e s s e s in

theR st a t e as the r e ar e pr o c e s s o r s or CP U co r e s in the sy s t e m .

S Sl e e p i n g – Sl e e p i n g pr o c e s s e s ar e on e s tha t the CP U is no tcu r r e n t l y gi v i n g its at t e n t i o n to, bu t wh i c h ar e st i l l va l i d

run n i n gpr o g r a m s tha t wi l l be ex e c u t e d wh e n it is the i r tur n.

D Th i s ap p e a r s wh e n a pr o c e s s is in the mi d s t of ex i t i n g an d de l e t i n g its e l f fro m me m o r y . Us u a l l y the s e pr o c e s s e s wi l l

on l ybe se e n ve r y br i e f l y as the y ar e on l y in thi s st a t e for as lon g

asit tak e s the m to ex i t , bu t so m e t i m e s pr o b l e m s ca n oc c u r on asy s t e m tha t st o p pr o c e s s e s fro m ex i t i n g cl e a n l y , an d ca u s e

the mto rem a i n in thi s st a t e ind e f i n i t e l y un t i l the pr o b l e m is

res o l v e d .

Pa g e 76

Z Zo m b i e . A zom b i e is a pr o c e s s wh i c h ha s be e n se n t a ki l l si g n a l

te l l i n g it to qu i t , bu t wh i c h ha s no t ac t u a l l y res p o n d e d an d isst i l l in me m o r y af t e r its su p p o s e d de a t h . He n c e , the na m e .

Th e s edo no t ap p e a r of t e n an d, mu c h lik e the rea l (so to sp e a k)

thi n g ,wh e n yo u se e on e, it' s us u a l l y a sig n of so m e ve r y se r i o u s

tro u b l e .

Pa g e 77

STARTTh i s sh o w s the tim e of da y (or da t e , if the pr o c e s s ha s be e n run n i n g for mo r e tha n 24 ho u r s) tha t the pr o c e s s in qu e s t i o n wa s st a r t e d .

Th e ot h e r co l u m n s ar e ide n t i c a l to tho s e se e n in a sim p l e ps lis t i n g , an d ha v e alr e a d y be e n ex p l a i n e d .

So, now we ha v e se e n how to ge t mo r e inf o r m a t i o n on ou r ow n pr o c e s s e s . Bu t on an y gi v e n sy s t e m the r e ar e al w a y s mo r e pr o c e s s e s run n i n g tha n tho s e tha t we ou r s e l v e s ha v e st a r t e d up sin c e log g i n g in. Be a r in mi n d tha t ev e r y si n g l e pr o g r a m run n i n g on a sy s t e m wi l l re q u i r e at lea s t on e pr o c e s s , an d so ot h e r so f t w a r e on the sy s t e m wi l l ap p e a r in a pr o c e s s lis t i n g too.

W e ca n se e the pr o c e s s e s be l o n g i n g to al l us e r s on the sy s t e m by

com b i n i n g the -u fla g wi t h the -a and -x fla g s , lik e so:

[drysdalk@stimpy drysdalk]$ ps -auxUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDroot 1 0.0 0.0 1524 496 ? S Mar13 0:04 init [3]root 2 0.0 0.0 0 0 ? SW Mar13 0:00 [keventd]root 3 0.0 0.0 0 0 ? SW Mar13 0:00 [kapmd]root 4 0.0 0.0 0 0 ? SWN Mar13 0:00 [ksoftirqd/0]root 7 0.0 0.0 0 0 ? SW Mar13 0:00 [bdflush]root 5 0.0 0.0 0 0 ? SW Mar13 0:04 [kswapd]root 6 0.0 0.0 0 0 ? SW Mar13 0:00 [kscand]root 8 0.0 0.0 0 0 ? SW Mar13 0:00 [kupdated]root 9 0.0 0.0 0 0 ? SW Mar13 0:00 [mdrecoveryd]root 13 0.0 0.0 0 0 ? SW Mar13 0:00 [kjournald]root 402 0.0 0.0 0 0 ? SW Mar13 0:00 [kjournald]root 403 0.0 0.0 0 0 ? SW Mar13 0:00 [kjournald]root 404 0.0 0.0 0 0 ? SW Mar13 0:00 [kjournald]root 405 0.0 0.0 0 0 ? SW Mar13 0:00 [kjournald]root 406 0.0 0.0 0 0 ? SW Mar13 0:01 [kjournald]root 752 0.0 0.1 1592 580 ? S Mar13 0:00 syslogd -m 0root 756 0.0 0.0 1528 456 ? S Mar13 0:00 klogd -xroot 794 0.0 0.0 1580 404 ? S Mar13 0:00 mdadm --monitor --scan -froot 850 0.0 0.0 1520 504 ? S Mar13 0:00 /usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-scripts/apmscriptroot 890 0.0 1.7 12748 9072 ? S Mar13 0:06 /usr/bin/perl -w? /usr/local/bin/blocksshd --startroot 975 0.0 0.2 3652 1516 ? S Mar13 0:00 /usr/sbin/sshdroot 994 0.0 0.1 2132 800 ? S Mar13 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pidntp 1036 0.0 0.5 2584 2576 ? SL Mar13 0:00 ntpd -U ntp -p /var/run/ntpd.pid -groot 1165 0.0 0.4 6096 2492 ? S Mar13 0:00 sendmail: accepting connectionssmmsp 1176 0.0 0.4 5980 2244 ? S Mar13 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueueroot 1212 0.0 0.1 4480 1000 ? S Mar13 0:00 cronddaemon 1233 0.0 0.1 1588 564 ? S Mar13 0:00 /usr/sbin/atdroot 1251 0.0 0.0 1520 424 tty1 S Mar13 0:00 /sbin/mingetty tty1root 1252 0.0 0.0 1516 424 tty2 S Mar13 0:00 /sbin/mingetty tty2root 1253 0.0 0.0 1520 424 tty3 S Mar13 0:00 /sbin/mingetty tty3root 1254 0.0 0.0 1512 424 tty4 S Mar13 0:00 /sbin/mingetty tty4

Pa g e 78

root 1255 0.0 0.0 1520 424 tty5 S Mar13 0:00 /sbin/mingetty tty5root 1259 0.0 0.0 1524 420 tty6 S Mar13 0:00 /sbin/mingetty tty6root 22718 0.0 0.4 6852 2064 ? S 09:14 0:00 sshd: tomd [priv]tomd 22720 0.0 0.4 6996 2284 ? S 09:15 0:00 sshd: tomd@pts/0tomd 22721 0.0 0.2 4256 1364 pts/0 S 09:15 0:00 -bashtomd 22763 0.0 0.3 3520 1632 pts/0 S 09:15 0:00 ssh 10.1.1.66root 22766 0.0 0.4 6852 2060 ? S 10:00 0:00 sshd: tomd [priv]tomd 22773 0.0 0.4 6868 2260 ? S 10:01 0:00 sshd: tomd@pts/1tomd 22774 0.0 0.2 4272 1368 pts/1 S 10:01 0:00 -bashtomd 22816 0.0 0.3 3520 1636 pts/1 S 10:01 0:00 ssh 10.1.1.66root 22819 0.0 0.4 6856 2064 ? S 10:25 0:00 sshd: paulj [priv]paulj 22821 0.0 0.4 7000 2268 ? S 10:25 0:00 sshd: paulj@pts/2paulj 22822 0.0 0.2 4272 1372 pts/2 S 10:25 0:00 -bashroot 22865 0.0 0.4 6856 2232 ? S 10:32 0:00 sshd: drysdalk [priv]drysdalk 22867 0.0 0.4 6868 2424 ? S 10:32 0:00 sshd: drysdalk@pts/3drysdalk 22868 0.0 0.2 4272 1376 pts/3 S 10:32 0:00 -bashdrysdalk 23079 0.0 0.1 2740 716 pts/3 R 10:53 0:00 ps -aux[drysdalk@stimpy drysdalk]$

No w , thi s gi v e s us the sa m e co l u m n s we saw be f o r e wi t h the -u fla g, bu t al s o sh o w s us ev e r y si n g l e pr o c e s s run n i n g on the sy s t e m . We ca n now se e su c h thi n g s as se n d m a i l (a ma i l se r v e r), ssh d (the se r v e r pr o c e s s res p o n s i b l e for al l o w i n g SS H co n n e c t i o n s to a sy s t e m), ot h e r ' s pe o p l e ' s bash sh e l l se s s i o n s an d SS H co n n e c t i o n s , an d ple n t y mo r e be s i d e s .

In c i d e n t a l l y , no t e tha t in thi s fa r lon g e r ps lis t i n g we st i l l on l y se e on e pr o c e s s in the R sta t e , an d tha t wa s ou r ow n ps com m a n d . By de f i n i t i o n , thi s ha d to be the pr o c e s s run n i n g on the sy s t e m at the tim e the ou t p u t of ps wa s pr o d u c e d , an d so it wa s the on l y on e cu r r e n t l y be i n g ac t i v e l y gi v e n at t e n t i o n by the CP U .

Th e r e ar e ple n t y of ot h e r ar g u m e n t s to ps, bu t the s e ar e the mo s t com m o n on e s yo u wi l l ne e d to us e. Ha v e a loo k at the ma n pa g e an d fee l fre e to pla y ar o u n d – yo u ca n ne v e r do an y kin d of da m a g e to a sy s t e m by us i n g ps, si n c e it on l y di s p l a y s inf o r m a t i o n rat h e r tha n ch a n g i n g an y t h i n g .

Pa g e 79

6.2 Process management

No w we ' v e se e n ho w to vie w pr o c e s s e s tha t ar e cu r r e n t l y run n i n g , the ne x t qu e s t i o n se e m s to be ho w to ac t u a l l y ma n a g e the s e pr o c e s s e s an d co n t r o l the m . Mu c h of pr o c e s s ma n a g e m e n t is do n e wi t h the kill com m a n d .

Fo r ex a m p l e – let ' s sa y tha t on yo u r ow n loc a l ma c h i n e , yo u r Fi r e f o x We b br o w s e r ha s st o p p e d res p o n d i n g . Fi r s t , we ne e d to loc a t e its PI D :

[drysdalk@localhost drysdalk]$ ps aux | grep firefoxdrysdalk 27833 3.3 5.5 58568 28544 pts/0 R 11:12 0:09 firefox-bindrysdalk 24530 0.0 0.0 4764 676 pts/0 S 12:15 0:00 grep firefox

No w we kn o w tha t it ha s a PI D of 27833. No w we ca n us e the kill com m a n d to at t e m p t to ma k e it qu i t , as fol l o w s :

[drysdalk@localhost drysdalk]$ kill 27833[drysdalk@localhost drysdalk]$ ps aux | grep firefoxdrysdalk 24533 0.0 0.0 4764 676 pts/0 S 12:15 0:00 grep firefox

Som e t i m e s ho w e v e r pr o c e s s e s wo n ' t ex i t , ei t h e r be c a u s e the y ar e un a b l e or unw i l l i n g res p o n d to a req u e s t to qu i t . In the s e cas e s , we ha v e to us e the -9 fla g to kill to for c e a pr o c e s s to ter m i n a t e :

[drysdalk@localhost drysdalk]$ ps aux | grep firefoxdrysdalk 27833 3.3 5.5 58568 28544 pts/0 R 12:16 0:09 firefox-bindrysdalk 24530 0.0 0.0 4764 676 pts/0 S 12:17 0:00 grep firefox[drysdalk@localhost drysdalk]$ kill 27833[drysdalk@localhost drysdalk]$ ps aux | grep firefoxdrysdalk 27833 3.3 5.5 58568 28544 pts/0 R 12:16 0:09 firefox-bindrysdalk 24533 0.0 0.0 4764 676 pts/0 S 12:17 0:00 grep firefox[drysdalk@localhost drysdalk]$ kill -9 27833[drysdalk@localhost drysdalk]$ ps aux | grep firefoxdrysdalk 24536 0.0 0.0 4764 676 pts/0 S 12:17 0:00 grep firefox

An d as we ca n se e, wh e n the -9 op t i o n wa s us e d , Fi r e f o x di d ind e e d ex i t .

A l s o no t e tha t in the ps us a g e in thi s ex a m p l e we did no t ha v e to pu t a – sym b o l in fro n t of the fla g s . Si n c e ps is on l y ca p a b l e of tak i n g fla g s an d do e s no t ac c e p t an y ar g u m e n t s or wo r k wi t h fil e n a m e s at al l, the r e is no ne e d to pu t a – in fro n t of its fla g s . Ind e e d , yo u ma y fin d on mo r e rec e n t Li n u x di s t r i b u t i o n s tha t yo u rec e i v e a wa r n i n g me s s a g e if yo u try to us e ps thi s wa y .

Pa g e 80

We ha v e alr e a d y se e n in a pr e v i o u s se c t i o n tha t it is po s s i b l e to st o p an y pr o c e s s cu r r e n t l y run n i n g in yo u r ter m i n a l wi t h the Ct r l-C ke y com b i n a t i o n . Th e r e is al s o a wa y to st o p a pr o c e s s , so tha t it is pa u s e d an d ca n be res u m e d at a lat e r st a g e . Th i s is do n e wi t h Ct r l- Z as fol l o w s :

[drysdalk@stimpy drysdalk]$ vi(Note: Ctrl-Z was pressed at this point)[1]+ Stopped vi[drysdalk@stimpy drysdalk]$ ps uUSER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMANDdrysdalk 22868 0.0 0.2 4272 1380 pts/3 S 10:32 0:00 -bashdrysdalk 23103 0.2 0.2 4144 1080 pts/3 T 12:30 0:00 vidrysdalk 23104 0.0 0.1 2740 680 pts/3 R 12:30 0:00 ps u[drysdalk@stimpy drysdalk]$

We can se e tha t the vi pr o c e s s is st i l l run n i n g , bu t wi t h a st a t u s of T. Th i s me a n s tha t it is st o p p e d – it ha s be e n fro z e n do i n g ex a c t l y wh a t it wa s do i n g wh e n we pr e s s e d Ct r l- Z, an d if an d wh e n it is res u m e d it wi l l pi c k up ex a c t l y wh e r e it lef t of f.

W e ca n res u m e it in on e of two wa y s . We ca n typ e bg to re-st a r t thi s pr o c e s s in to the ba c k g r o u n d an d ret u r n us to ou r com m a n d pr o m p t , or we ca n typ e fg to run it as a for e g r o u n d pr o g r a m – tha t is to sa y , on e tha t is dir e c t l y at t a c h e d to ou r te r m i n a l an d rea d y for inp u t on c e ag a i n .

Fo r ex a m p l e :

[drysdalk@stimpy drysdalk]$ vi(Note: Ctrl-Z was pressed at this point)[1]+ Stopped vi[drysdalk@stimpy drysdalk]$ bg[1]+ vi &[drysdalk@stimpy drysdalk]$

An d vi co n t i n u e s run n i n g , bu t we ge t ou r co m m a n d pr o m p t ba c k . Or:

[drysdalk@stimpy drysdalk]$ vi(Note: Ctrl-Z was pressed at this point)[1]+ Stopped vi[drysdalk@stimpy drysdalk]$ fgvi

He r e we do no t ge t ou r com m a n d pr o m p t ba c k un t i l vi is fin i s h e d , an d ex i t s .

Pa g e 81

Th i s is qu i t e ha r d to de m o n s t r a t e me a n i n g f u l l y on pa p e r , bu t try it yo u r s e l f . A goo d ca n d i d a t e for se e i n g ho w thi s wo r k s is the xeyes com m a n d . Tr y run n i n g xeyes an d sto p p i n g it wi t h Ct r l- Z, an d res u m i n g it wi t h bg an d fg – yo u sh o u l d fai r l y qu i c k l y se e pr e c i s e l y wh a t st o p p i n g an d sta r t i n g a pr o g r a m me a n s , an d wh a t the di f f e r e n c e be t w e e n run n i n g in the ba c k g r o u n d an d run n i n g in the for e g r o u n d is.

La s t l y , the r e is on e ot h e r wa y to run a pr o g r a m in the ba c k g r o u n d fro m the ve r y be g i n n i n g , an d thi s is to pu t an am p e r s a n d at the en d of the com m a n d wh e n yo u ar e typ i n g it. Fo r ex a m p l e :

[drysdalk@stimpy drysdalk]$ xeyes &[1] 23118[drysdalk@stimpy drysdalk]$

Th i s st a r t s xeyes in the ba c k g r o u n d imm e d i a t e l y , an d al s o tel l s yo u the PI D tha t wa s al l o c a t e d to it. Yo u r sh e l l the n ret u r n s yo u to yo u r com m a n d pr o m p t so yo u ca n co n t i n u e wo r k i n g as xeyes run s .

6.3 Resource monitoring

Ju s t as imp o r t a n t as ke e p i n g tra c k of the pr o c e s s e s on a sy s t e m is ke e p i n g tra c k of ot h e r res o u r c e s in us e, pr i m a r i l y CP U an d me m o r y ut i l i s a t i o n , as we l l as di s c sp a c e . Fo r t u n a t e l y Li n u x pr o v i d e s a va r i e t y of fai r l y st r a i g h t f o r w a r d wa y s to ch e c k the s e thi n g s .

Disc space

Di s c sp a c e ca n be ea s i l y ch e c k e d wi t h the df co m m a n d , lik e so:

[drysdalk@stimpy drysdalk]$ dfFilesystem 1K-blocks Used Available Use% Mounted on/dev/hda3 5036316 199284 4581200 5% //dev/hda1 101089 38421 57449 41% /boot/dev/hda7 1004024 299480 653540 32% /homenone 255452 0 255452 0% /dev/shm/dev/hda6 1004024 16432 936588 2% /tmp/dev/hda2 6048352 624436 5116676 11% /usr/dev/hda8 24264504 10742920 12289000 47% /var[drysdalk@stimpy drysdalk]$

Pa g e 82

Th i s sh o w s us al l cu r r e n t l y- mo u n t e d fil e s y s t e m s , an d the am o u n t of sp a c e bo t h us e d an d fre e on ea c h of the m . We ca n ge t a mo r e hu m a n-

rea d a b l e for m a t for the s e nu m b e r s wi t h the -h fla g:

[drysdalk@stimpy drysdalk]$ df -hFilesystem Size Used Avail Use% Mounted on/dev/hda3 4.9G 195M 4.4G 5% //dev/hda1 99M 38M 57M 41% /boot/dev/hda7 981M 293M 639M 32% /homenone 250M 0 250M 0% /dev/shm/dev/hda6 981M 17M 915M 2% /tmp/dev/hda2 5.8G 610M 4.9G 11% /usr/dev/hda8 24G 11G 12G 47% /var[drysdalk@stimpy drysdalk]$

La s t l y if we wa n t to kn o w how mu c h di s c sp a c e a pa r t i c u l a r di r e c t o r y or fil e is us i n g , thi s is do n e wi t h the du co m m a n d :

[drysdalk@stimpy drysdalk]$ du /bin/ls68 /bin/ls[drysdalk@stimpy drysdalk]$ du -h /bin/ls68K /bin/ls[drysdalk@stimpy drysdalk]$ du -h /bin4.8M /bin[drysdalk@stimpy drysdalk]$

Memory

We can ch e c k the cu r r e n t me m o r y us a g e on a Li n u x sy s t e m wi t h the free com m a n d :

[drysdalk@stimpy drysdalk]$ free total used free shared buffers cachedMem: 510908 464720 46188 0 35716 399980-/+ buffers/cache: 29024 481884Swap: 1020088 0 1020088[drysdalk@stimpy drysdalk]$

Pa g e 83

Th e am o u n t s he r e ar e al l gi v e n in ki l o b y t e s . Th e ou t p u t of free ca n ap p e a r qu i t e com p l e x at fir s t , bu t the imp o r t a n t pa r t s of it br e a k dow n as fol l o w s :

total Th i s co l u m n lis t s the tot a l am o u n t of me m o r y an d sw a p sp a c e

ins t a l l e d on the sy s t e m , res p e c t i v e l y . So ou r ex a m p l e sy s t e m ha s

51 2 M B of me m o r y , an d ar o u n d 1 G B of sw a p sp a c e co n f i g u r e d .

used Ro w 1: Th e am o u n t of ph y s i c a l me m o r y us e d by pr o g r a m s , an d

for di s c ca c h i n gRo w 2: Th e am o u n t of ph y s i c a l me m o r y us e d by pr o g r a m s

on l yRo w 3: Th e am o u n t of sw a p sp a c e in us e

free Ro w 1: Th e am o u n t of ph y s i c a l me m o r y cu r r e n t l y fre e , reg a r d i n g

the di s c ca c h e as us e d me m o r yRo w 2: Th e am o u n t of ph y s i c a l me m o r y cu r r e n t l y fre e , no t

co u n t i n g the di s c ca c h eRo w 3: Th e am o u n t of sw a p sp a c e cu r r e n t l y fre e

A rea s o n a b l e qu e s t i o n he r e wo u l d be – wh a t is the di s c ca c h e ? Li n u x is ca p a b l e of al l o c a t i n g a po r t i o n of the sy s t e m ' s me m o r y to ho l d i n g a co p y of fre q u e n t l y- ac c e s s e d file s in me m o r y , so as to sp e e d up ac c e s s to the m . Ho w e v e r , the siz e of thi s ca c h e is dy n a m i c a l l y ad j u s t e d by the op e r a t i n g sy s t e m ac c o r d i n g to how mu c h me m o r y pr o g r a m s the m s e l v e s ne e d to run – the s e ar e us u a l l y gi v e n pr i o r i t y . So un d e r no r m a l ci r c u m s t a n c e s it is the fig u r e s in the mi d d l e row tha t ind i c a t e the ab s o l u t e am o u n t of me m o r y tha t is av a i l a b l e to an y pr o g r a m s tha t mi g h t req u i r e it.

N o t e tha t to se e how mu c h me m o r y an y gi v e n pr o g r a m is us i n g , yo u on l y ha v e to ch e c k the VS Z and RS S va l u e s for it in the ou t p u t of ps, as pr e v i o u s l y no t e d .

Processor usage

Pa g e 84

We ' v e alr e a d y se e n tha t we ca n ch e c k the am o u n t of CP U tim e an d the pe r c e n t a g e of the CP U ' s res o u r c e s tha t an ind i v i d u a l pr o g r a m is co n s u m i n g by ch e c k i n g its lis t i n g in ps (the % C P U and TIM E co l u m n s , res p e c t i v e l y). Bu t wh a t if we wa n t an ov e r a l l pi c t u r e – to ge t an ide a of ho w bu s y the sy s t e m ' s CP U gen e r a l l y is at the mo m e n t , or ha s be e n ov e r a l l rec e n t l y ? Th e r e ar e, as yo u wo u l d ex p e c t , a few wa y s to ge t inf o r m a t i o n of thi s so r t .

Pa g e 85

Th e ea s i e s t wa y to ge t an ide a of how bu s y a sy s t e m is at an y gi v e n

mo m e n t is to ch e c k its load average . Wi t h o u t ge t t i n g ov e r l y tec h n i c a l , the loa d av e r a g e of a sy s t e m is a nu m b e r tha t ex p r e s s e s ho w ma n y pr o c e s s e s ar e cu r r e n t l y vy i n g for the at t e n t i o n of the CP U . Th e r e ar e va r i o u s wa y s to ch e c k the cu r r e n t loa d av e r a g e , bu t the ea s i e s t wa y is wi t h the w co m m a n d :

[root@vhost5-1 root]# w 13:18:21 up 149 days, 3:57, 1 user, load average: 0.75, 0.80, 0.80USER TTY FROM LOGIN@ IDLE JCPU PCPU WHATroot pts/0 stimpy.iomart.co 1:18pm 0.00s 0.02s 0.00s w[root@vhost5-1 root]#

As yo u ca n se e, the w co m m a n d do e s so m e t h i n g el s e as we l l – it gi v e s us a sh o r t su m m a r y of ev e r y o n e cu r r e n t l y log g e d in to the sy s t e m , an d wh e r e the y log g e d in fro m . Bu t we ar e int e r e s t e d he r e in tha t fir s t lin e of ou t p u t tha t wa s ret u r n e d :

13:18:21 up 149 days, 3:57, 1 user, load average: 0.75, 0.80, 0.80

Th e va l u e s he r e ha v e the fol l o w i n g me a n i n g s :

13:18:21Th e cu r r e n t loc a l sy s t e m tim e .

up 149 days, 3:57Th e up t i m e of the sy s t e m – the am o u n t of tim e sin c e it wa s las t reb o o t e d . So in ou r ca s e , 149 da y s , 3 ho u r s an d 57 mi n u t e s .

1 userTh e nu m b e r of us e r s cu r r e n t l y log g e d in.

load average: 0.75, 0.80, 0.80He r e we ha v e the ac t u a l loa d av e r a g e fig u r e s the m s e l v e s . Th e s e thr e e nu m b e r s ar e the sy s t e m loa d av e r a g e ov e r the pa s t 1 mi n u t e , 5 mi n u t e s an d 15 mi n u t e s res p e c t i v e l y .

Pa g e 86

So wh a t co n s t i t u t e s a hig h sy s t e m loa d ? St r i c t l y sp e a k i n g , si n c e the loa d av e r a g e ind i c a t e s how ma n y pr o c e s s e s ar e cu r r e n t l y ac t i v e l y req u i r i n g CP U at t e n t i o n , the n if the loa d av e r a g e ex c e e d s the nu m b e r of CP U co r e s on a sy s t e m the n it is too hi g h . Re m e m b e r tha t a CP U ca n on l y do on e thi n g at a tim e – the ill u s i o n of mu l t i- tas k i n g is cr e a t e d by the CP U rap i d l y sw i t c h i n g from one pr o g r a m to an o t h e r .

S o for a sin g l e- pr o c e s s o r si n g l e- co r e sy s t e m , a loa d av e r a g e of gr e a t e r tha n 1.00 ov e r fif t e e n mi n u t e s wo u l d ind i c a t e tha t it wa s too bu s y – in ot h e r wo r d s , the r e wo u l d be too ma n y thi n g s ha p p e n i n g on the sy s t e m for the m all to be res p o n d i n g in a tim e l y fas h i o n . Th e sy s t e m wo u l d pr o b a b l y st i l l be us a b l e , bu t its pe r f o r m a n c e wo u l d be no t i c e a b l y de g r a d e d .

Lo a d av e r a g e sp i k e s ov e r a sh o r t e r pe r i o d , su c h as on l y ov e r the on e- or fiv e- mi n u t e va l u e s , ar e no r m a l l y no t so wo r r y i n g . Su d d e n su r g e s in pr o c e s s o r ac t i v i t y ar e a fai r l y co m m o n oc c u r r e n c e at pe a k tim e s , bu t if a sy s t e m co n s i s t e n t l y ha s a loa d av e r a g e hi g h e r tha n the nu m b e r of CP U s or CP U co r e s it ha s ins t a l l e d , the n it is too bu s y an d wi l l no t be pe r f o r m i n g at its be s t .

Fi n a l l y , ho w do yo u go ab o u t de t e r m i n i n g the ca u s e of a hig h sy s t e m loa d ? We l l , yo u ca n ch e c k the ou t p u t of a ps -aux an d se e wh i c h pr o c e s s e s ar e co n s i s t e n t l y us i n g hi g h pe r c e n t a g e s of the CP U ' s res o u r c e s or wh i c h ha v e tak e n up the hig h e s t va l u e s of its tim e , bu t the r e is a co m m a n d tha t wi l l gi v e yo u a hu m a n- rea d a b l e sn a p s h o t ov e r v i e w of sy s t e m pr o c e s s o r us a g e . If yo u typ e top at the co m m a n d pr o m p t , yo u wi l l the n se e a sc r e e n suc h as thi s:

1:47pm up 72 days, 6:13, 1 user, load average: 0.22, 0.18, 0.1760 processes: 58 sleeping, 2 running, 0 zombie, 0 stoppedCPU states: 0.2% user, 0.0% system, 0.0% nice, 0.5% idleMem: 1030840K av, 1018336K used, 12504K free, 0K shrd, 111628K buffSwap: 2096472K av, 48464K used, 2048008K free 603900K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND19059 apache 17 0 5204 5204 1908 S 37.2 0.5 0:00 store_stats.cgi18236 smmsp 10 0 2400 2276 1460 S 1.9 0.2 0:12 sendmail10755 root 9 0 185M 142M 3616 S 0.9 14.1 2:37 httpd 1 root 8 0 492 464 440 S 0.0 0.0 0:05 init 2 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd 3 root 9 0 0 0 0 SW 0.0 0.0 0:00 kapmd 4 root 19 19 0 0 0 SWN 0.0 0.0 0:03 ksoftirqd_CPU0 5 root 9 0 0 0 0 SW 0.0 0.0 0:18 kswapd 6 root 9 0 0 0 0 SW 0.0 0.0 0:01 kscand/DMA 7 root 9 0 0 0 0 SW 0.0 0.0 548:59 kscand/Normal 8 root 9 0 0 0 0 SW 0.0 0.0 123:19 kscand/HighMem 9 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush 10 root 9 0 0 0 0 SW 0.0 0.0 0:04 kupdated 11 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd 15 root 9 0 0 0 0 SW 0.0 0.0 6:29 kjournald

Pa g e 87

70 root 9 0 0 0 0 SW 0.0 0.0 0:00 khubd 140 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald 471 root 9 0 556 520 476 S 0.0 0.0 2:21 syslogd 475 root 9 0 424 376 372 S 0.0 0.0 0:00 klogd 492 rpc 9 0 564 536 492 S 0.0 0.0 0:01 portmap 511 rpcuser 9 0 688 608 604 S 0.0 0.0 0:00 rpc.statd 568 root 9 0 0 0 0 SW 0.0 0.0 4:24 rpciod 569 root 9 0 0 0 0 SW 0.0 0.0 0:00 lockd 586 root 9 0 744 708 620 S 0.0 0.0 0:05 ypbind 654 root 9 0 528 472 452 S 0.0 0.0 0:00 automount 701 root 9 0 892 768 668 S 0.0 0.0 8:21 sshd 716 root 9 0 812 728 652 S 0.0 0.0 1:20 xinetd 730 ntp 9 0 2068 2068 1868 S 0.0 0.2 0:07 ntpd 753 root 8 0 276 244 204 S 0.0 0.0 0:28 crond 763 root 8 0 452 424 404 S 0.0 0.0 1:17 rhnsd 770 root 9 0 44 4 0 S 0.0 0.0 0:00 mingetty 771 root 9 0 44 4 0 S 0.0 0.0 0:00 mingetty 772 root 9 0 44 4 0 S 0.0 0.0 0:00 mingetty 773 root 9 0 44 4 0 S 0.0 0.0 0:00 mingetty 774 root 9 0 44 4 0 S 0.0 0.0 0:00 mingetty 775 root 9 0 44 4 0 S 0.0 0.0 0:00 mingetty 7854 root 9 0 1428 956 800 S 0.0 0.0 0:05 sendmail 7863 smmsp 9 0 1192 816 724 S 0.0 0.0 0:00 sendmail18993 root 8 0 2488 2344 1680 S 0.0 0.2 0:00 sendmail19013 root 9 0 1524 1524 1148 S 0.0 0.1 0:00 bash19060 root 9 0 1092 1092 872 R 0.0 0.1 0:00 top

So we ca n se e tha t on thi s sy s t e m (one of the sh a r e d We b ho s t i n g se r v e r s) the he a v i e s t us e r of CP U res o u r c e s is store_stats.cgi, a Pe r l sc r i p t ru n n i n g in a cu s t o m e r ' s We b sp a c e . Ne x t hig h e s t is an ins t a n c e of sendmail tha t is pr o b a b l y bu s y se n d i n g ou t an e-ma i l , an d the n we ha v e an httpd pr o c e s s , wh i c h is pa r t of the We b se r v e r so f t w a r e an d is mo s t lik e l y in the pr o c e s s of se r v i n g up a We b pag e .

No t e tha t by de f a u l t top so r t s thi s lis t by pr o c e s s o r us a g e , from hea v i e s t to lig h t e s t us a g e , to ma k e fin d i n g ou t wh a t mi g h t be us i n g mo s t of yo u r sy s t e m ' s CP U tim e ev e n ea s i e r . It al s o di s p l a y s st a t i s t i c s ab o u t me m o r y us a g e as we l l , so it is a ha n d y on e- sto p- sh o p for sy s t e m res o u r c e us a g e inf o r m a t i o n (with the ex c e p t i o n of di s c sp a c e).

Pa g e 88

6.4 Common server software

In thi s se c t i o n , we wi l l loo k at com m o n pi e c e s of se r v e r so f t w a r e , an d ex a m i n e the tas k s yo u wo u l d mo s t typ i c a l l y ha v e to ca r r y ou t wh e n wo r k i n g wi t h the m .

6.4.1 Web server – Apache

Ap a c h e is mo r e or les s the de fac t o st a n d a r d We b se r v e r on mo s t Li n u x sy s t e m s . Th e r e ar e ot h e r s av a i l a b l e – an d we do in fac t ma k e us e of som e of the m ou r s e l v e s in ce r t a i n pla c e s – bu t Ap a c h e is the on e yo u ar e mo s t lik e l y by far to en c o u n t e r tha n an y ot h e r .

Apache at a glance

Th e fol l o w i n g inf o r m a t i o n ap p l i e s to the de f a u l t ou t- of-the- bo x ins t a l l of Ap a c h e on a Re d Ha t st y l e sy s t e m (i.e. Fe d o r a , Ce n t O S , Wh i t e Bo x , Re d Ha t En t e r p r i s e).

• Th e Ap a c h e bi n a r y ex e c u t a b l e is ins t a l l e d as /usr/sb i n/h t t p d• Ap a c h e ' s ma i n co n f i g u r a t i o n fil e is /etc/ht t p d/co n f/h t t p d . c o n f• Th e us e r tha t Ap a c h e is co n f i g u r e d to run as is 'ap a c h e '• Th e gr o u p the Ap a c h e us e r be l o n g s to is 'ap a c h e '• Th e We b tra f f i c log fil e for Ap a c h e is /var/log/ht t p d/a c c e s s _ l o g• Th e Ap a c h e er r o r log is /var/log/ht t p d/e r r o r _ l o g• Th e de f a u l t W e b sit e ' s co n t e n t is loc a t e d be n e a t h /var/ww w/h t m l• In a ps lis t i n g , Ap a c h e no r m a l l y ap p e a r s as /usr/sb i n/h t t p d• Ap a c h e lis t e n s on Po r t 80 on al l co n f i g u r e d IP ad d r e s s e s for HT T P

req u e s t s

No t e tha t an y on e of the s e va l u e s or as s u m p t i o n s ca n be ch a n g e d by ed i t i n g Ap a c h e ' s co n f i g u r a t i o n ap p r o p r i a t e l y , so do no t as s u m e tha t the ab o v e wi l l be tru e on ev e r y si n g l e sy s t e m tha t yo u com e ac r o s s . Ho w e v e r , a cle a n ins t a l l of Ap a c h e wi l l be h a v e ex a c t l y as ou t l i n e d ab o v e , as wi l l on e tha t ha s no t be e n cu s t o m i s e d in an y sig n i f i c a n t wa y .

Pa g e 89

Task 1: Checking to see if Apache is running

To ch e c k to se e if Ap a c h e is run n i n g , the ea s i e s t thi n g to do is to se e if it ap p e a r s in a ps lis t i n g :

[drysdalk@vhost6-1 vhost6-1]$ ps aux | grep httpdroot 19037 0.0 0.2 20340 9108 ? S Jan17 0:48 /usr/sbin/httpdapache 32188 0.0 0.4 28772 18576 ? S 10:10 0:07 /usr/sbin/httpdapache 312 0.0 0.4 29512 19476 ? S 10:32 0:08 /usr/sbin/httpdapache 1546 0.0 0.4 28956 18816 ? S 11:17 0:06 /usr/sbin/httpdapache 1688 0.0 0.4 29480 19328 ? S 11:22 0:07 /usr/sbin/httpdapache 1689 0.0 0.4 27576 17336 ? S 11:22 0:07 /usr/sbin/httpdapache 2108 0.0 0.4 26964 16748 ? S 11:36 0:05 /usr/sbin/httpdapache 2207 0.0 0.4 29448 19100 ? S 11:39 0:05 /usr/sbin/httpdapache 2208 0.0 0.4 29084 18708 ? S 11:39 0:07 /usr/sbin/httpdapache 2210 0.0 0.4 27200 16820 ? S 11:39 0:04 /usr/sbin/httpdapache 2267 0.0 0.4 27052 16940 ? S 11:42 0:07 /usr/sbin/httpdapache 2284 0.0 0.4 28860 18816 ? S 11:42 0:06 /usr/sbin/httpdapache 2385 0.0 0.4 28068 17932 ? S 11:45 0:05 /usr/sbin/httpdapache 2406 0.0 0.4 27344 17352 ? S 11:46 0:07 /usr/sbin/httpdapache 2793 0.0 0.4 28824 18724 ? S 12:02 0:07 /usr/sbin/httpdapache 3588 0.0 0.4 27940 17608 ? S 12:33 0:04 /usr/sbin/httpdapache 3589 0.0 0.4 30016 19872 ? S 12:33 0:04 /usr/sbin/httpdapache 3590 0.0 0.4 27556 17424 ? S 12:33 0:05 /usr/sbin/httpdapache 3592 0.0 0.4 26984 16812 ? S 12:33 0:03 /usr/sbin/httpdapache 3593 0.0 0.4 28676 18436 ? S 12:33 0:05 /usr/sbin/httpdapache 6152 0.0 0.4 27156 16696 ? S 14:11 0:00 /usr/sbin/httpdapache 6635 0.0 0.2 20476 9400 ? S 14:28 0:00 /usr/sbin/httpdapache 6649 0.3 0.3 25776 15048 ? S 14:29 0:00 /usr/sbin/httpddrysdalk 6754 0.0 0.0 3696 660 pts/0 S 14:31 0:00 grep httpd[drysdalk@vhost6-1 vhost6-1]$

As yo u ca n se e, thi s pa r t i c u l a r se r v e r (vhos t 6- 1, on e of the sh a r e d We b ho s t i n g se r v e r s , an d the on e we wi l l us e thr o u g h o u t thi s se c t i o n) is ind e e d run n i n g Ap a c h e . Wh i c h is jus t as we l l .

Pa g e 90

Task 2: Starting Apache if it is not running

If yo u ar e rec e i v i n g ale r t s fro m a We b se r v e r ind i c a t i n g tha t Ap a c h e is no t run n i n g , an d yo u ha v e ve r i f i e d yo u r s e l f us i n g the ab o v e me t h o d tha t thi s is ind e e d the ca s e , yo u ca n st a r t Ap a c h e fro m co l d wi t h the fol l o w i n g com m a n d :

service httpd start

Fo r ex a m p l e :

[root@ren ~]# ps aux | grep httproot 31917 0.0 0.0 4536 568 pts/2 S+ 14:23 0:00 grep http[root@ren ~]# service httpd startStarting httpd: [ OK ][root@ren ~]# ps aux | grep httproot 31929 0.0 0.1 7604 2400 ? Ss 14:23 0:00 /usr/sbin/httpdapache 31931 0.0 0.0 7604 1768 ? S 14:23 0:00 /usr/sbin/httpdapache 31933 0.0 0.0 7604 1768 ? S 14:23 0:00 /usr/sbin/httpdapache 31934 0.0 0.0 7604 1768 ? S 14:23 0:00 /usr/sbin/httpdapache 31935 0.0 0.0 7604 1768 ? S 14:23 0:00 /usr/sbin/httpdapache 31936 0.0 0.0 7604 1768 ? S 14:23 0:00 /usr/sbin/httpdapache 31937 0.0 0.0 7604 1768 ? S 14:23 0:00 /usr/sbin/httpdapache 31938 0.0 0.0 7604 1768 ? S 14:23 0:00 /usr/sbin/httpdapache 31939 0.0 0.0 7604 1768 ? S 14:23 0:00 /usr/sbin/httpdroot 31941 0.0 0.0 3976 568 pts/2 S+ 14:23 0:00 grep http[root@ren ~]#

No t e tha t yo u ha v e to be roo t to st o p an d/or st a r t Ap a c h e , so yo u wo u l d be be s t to ei t h e r do thi s wi t h sudo or to pe r f o r m a sudo su – pr i o r to at t e m p t i n g thi s.

Task 3: Stopping Apache in the proper fashion

Yo u ha v e be e n as k e d to st o p Ap a c h e on on e of the We b se r v e r s . Th i s is do n e , as yo u mi g h t ex p e c t , wi t h :

service httpd stop

Fo r ex a m p l e :

[root@ren ~]# service httpd stopStopping httpd: [ OK ][root@ren ~]# ps aux | grep httproot 32089 0.0 0.0 5216 568 pts/2 S+ 14:25 0:00 grep http[root@ren ~]#Task 4: What to do if there are problems re-starting Apache

Pa g e 91

Som e t i m e s , yo u ' l l log on to a se r v e r an d se e tha t the r e se e m s st i l l to be an Ap a c h e pr o c e s s run n i n g , ev e n tho u g h yo u ar e rec e i v i n g al e r t s ind i c a t i n g tha t the We b se r v e r ha s fai l e d . Th i s ca n ha p p e n if Ap a c h e ha s die d of f for som e rea s o n , bu t on e of its pr o c e s s e s is st i l l al i v e wh e n it sh o u l d no t be. In thi s si t u a t i o n , try i n g to st a r t Ap a c h e ma y no t wo r k , si n c e the ol d pr o c e s s wi l l be blo c k i n g a ne w ins t a n c e of Ap a c h e fro m st a r t i n g .

In ca s e s su c h as thi s, yo u fir s t ha v e to ki l l of f the Ap a c h e pr o c e s s tha t ha s be e n lef t be h i n d . Fo r ex a m p l e :

[root@ren ~]# ps aux | grep httpapache 32109 0.0 0.1 8976 2284 ? S 14:25 0:00 /usr/sbin/httpdroot 32385 0.0 0.0 5720 568 pts/2 S+ 14:31 0:00 grep http[root@ren ~]# service httpd startStarting httpd: [ FAILED ][root@ren ~]# ps aux | grep httpapache 32109 0.0 0.1 8976 2284 ? S 14:25 0:00 /usr/sbin/httpdroot 32397 0.0 0.0 5552 568 pts/2 S+ 14:31 0:00 grep http[root@ren ~]# kill -9 32109[root@ren ~]# ps aux | grep httproot 32399 0.0 0.0 4336 568 pts/2 S+ 14:31 0:00 grep http[root@ren ~]# service httpd startStarting httpd: [ OK ][root@ren ~]# ps aux | grep httproot 32411 0.0 0.1 9248 2404 ? Ss 14:31 0:00 /usr/sbin/httpdapache 32413 0.0 0.0 9248 1772 ? S 14:31 0:00 /usr/sbin/httpdapache 32415 0.0 0.0 9248 1772 ? S 14:31 0:00 /usr/sbin/httpdapache 32416 0.0 0.0 9248 1772 ? S 14:31 0:00 /usr/sbin/httpdapache 32417 0.0 0.0 9248 1772 ? S 14:31 0:00 /usr/sbin/httpdapache 32418 0.0 0.0 9248 1772 ? S 14:31 0:00 /usr/sbin/httpdapache 32419 0.0 0.0 9248 1772 ? S 14:31 0:00 /usr/sbin/httpdapache 32420 0.0 0.0 9248 1772 ? S 14:31 0:00 /usr/sbin/httpdapache 32421 0.0 0.0 9248 1772 ? S 14:31 0:00 /usr/sbin/httpdroot 32423 0.0 0.0 4424 568 pts/2 S+ 14:31 0:00 grep http[root@ren ~]#

Th i s is by far the mo s t com m o n rea s o n wh y Ap a c h e wo n ' t st a r t , ev e n if at fir s t gl a n c e it do e s n ' t ap p e a r to be run n i n g . An o t h e r ca u s e ca n be if Pe r l sc r i p t s or sim i l a r thi n g s ha v e be e n lef t-ov e r from the pr e v i o u s ins t a n c e of Ap a c h e tha t fai l e d . Th e s e too wi l l bl o c k a new ins t a n c e fro m st a r t i n g , so if yo u se e an y Pe r l sc r i p t s ow n e d by the us e r 'ap a c h e ' on a ps lis t i n g , an d Ap a c h e wo n ' t st a r t , the n ki l l i n g the s e of f wi l l pr o b a b l y fix the pr o b l e m .

Pa g e 92

Task 5: Diagnosing problems with CGI scripts

An o t h e r com m o n tas k is ha v i n g to tra c k do w n wh y a pa r t i c u l a r sc r i p t on a We b si t e is giv i n g an 'In t e r n a l Se r v e r Er r o r ' . Th i s is a ca t c h- al l W e b se r v e r ter m for a fai l u r e of the ca l l e d scr i p t to ac t u a l l y ret u r n su c c e s s f u l l y .

If Ap a c h e rec e i v e s an y me a n i n g f u l ou t p u t fro m the sc r i p t ot h e r tha n wh a t it wa s ex p e c t i n g , it wi l l wr i t e tha t ou t p u t to its er r o r log. Th e de f a u l t er r o r log fil e in a Re d Ha t ins t a l l of Ap a c h e is, as no t e d pr e v i o u s l y , /var/log/ht t p d/e r r o r _ l o g .

S o, if we tak e a loo k at the tai l-en d of thi s file on on e of the bu s i e r sh a r e d We b se r v e r s , we se e:

[root@vhost5-1 root]# tail -2 /var/log/httpd/error_log[Wed Mar 21 15:04:05 2007] [error] [client 64.208.172.172] Premature end of script headers: amazon_products_feed.cgi[Wed Mar 21 15:04:06 2007] [error] [client 65.55.209.227] (13)Permission denied: exec of '/vhost/vhost5/d/e/s/designer-info.com/cgi-bin/apf4/amazon_products_feed.cgi' failed

So he r e we ca n se e tha t the sc r i p t am a z o n _ p r o d u c t s _ f e e d . c g i is fai l i n g be c a u s e of a 'Pe r m i s s i o n de n i e d ' er r o r wh e n Ap a c h e tri e s to ex e c u t e it. It's alm o s t ce r t a i n tha t thi s cu s t o m e r sim p l y for g o t to se t ex e c u t e pe r m i s s i o n on the fil e wh e n the y up l o a d e d it to the i r cg i- bin/ di r e c t o r y , an d tha t do i n g so wo u l d fix the pr o b l e m .

Pa g e 93

6.4.2 FTP server – vsftpd

Th e FT P se r v e r tha t is us e d by de f a u l t on mo s t of ou r sh a r e d We b ho s t i n g pla t f o r m is vs f t p d . Th i s is the st a n d a r d Re d Ha t FT P se r v e r the s e da y s , tho u g h yo u ma y fro m tim e- to-tim e en c o u n t e r an o t h e r FT P se r v e r ca l l e d Pr o F T P D , al t h o u g h thi s is rar e r no w tha n it us e d to be.

vsftpd at a glance

As be f o r e , the fol l o w i n g ap p l i e s on l y to the ou t- of-the- bo x de f a u l t ins t a l l a t i o n of vs f t p d on a Re d Ha t- st y l e sy s t e m :

• Th e vs f t p d bin a r y is ins t a l l e d as /usr/sb i n/vs f t p d• Th e co n f i g u r a t i o n fil e for vs f t p d is /etc/vs f t p d/v s f t p d . c o n f• Th e lis t of us e r s wh o ar e no t al l o w e d to log in vi a FT P is

/etc/vs f t p d . f t p u s e r s

Task 1: Starting vsftpd

As yo u mi g h t we l l ex p e c t by now , thi s is sim p l y do n e wi t h

service vsftpd start

Fo r ex a m p l e :

[root@vhost5-1 root]# ps -aux | grep vsftpdroot 16587 0.0 0.0 4768 676 pts/0 S 17:01 0:00 grep vsftpd[root@vhost5-1 root]# service vsftpd startStarting vsftpd: [ OK ][root@vhost5-1 root]# ps -aux | grep vsftpdroot 8124 0.0 0.0 1788 484 ? S 2006 3:01 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.confftp 15236 0.0 0.0 1924 968 ? S 10:22 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.confroot 16592 0.0 0.0 4768 676 pts/0 S 17:01 0:00 grep vsftpd[root@vhost5-1 root]#

Pa g e 94

Task 2: Stopping vsftpd

Th i s is do n e sim p l y wi t h :

service vsftpd stop

Fo r ex a m p l e :

[root@vhost5-1 root]# ps -aux | grep vsftpdroot 8124 0.0 0.0 1788 484 ? S 2006 3:01 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.confftp 15236 0.0 0.0 1924 968 ? S 10:22 0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.confroot 16592 0.0 0.0 4768 676 pts/0 S 17:01 0:00 grep vsftpd[root@vhost5-1 root]# service vsftpd stopStopping vsftpd: [ OK ][root@vhost5-1 root]# ps -aux | grep vsftpdroot 16587 0.0 0.0 4768 676 pts/0 S 17:01 0:00 grep vsftpd[root@vhost5-1 root]#

Task 3: Diagnosing problems with vsftpd

Th e mo s t co m m o n pr o b l e m wi t h vs f t p d oc c u r s wh e n the se r v e r ha s rea c h e d its ma x i m u m num b e r of al l o w e d sim u l t a n e o u s co n n e c t i o n s . Th i s ca n ha p p e n ei t h e r as a res u l t of the se r v e r ge n u i n e l y be i n g bu s y , or mo r e of t e n wh e n on e pa r t i c u l a r us e r de c i d e s to es t a b l i s h do z e n s of co n n e c t i o n s un d e r the i r ow n log i n to sp e e d up the i r FT P tra n s f e r s .

Th e sym p t o m s of thi s wo u l d be the mo n i t o r i n g sy s t e m rep o r t i n g tha t FT P wa s ei t h e r tim i n g ou t or ref u s i n g co n n e c t i o n s on on e of the We b se r v e r s , an d wh e n yo u log in to ch e c k yo u se e a gr e a t ma n y vs f t p d pr o c e s s e s al l ow n e d by the sa m e us e r .

In thi s ca s e , yo u sim p l y st o p vs f t p d , wa i t a few sec o n d s an d ch e c k to se e tha t it ha s st o p p e d su c c e s s f u l l y , an d sta r t it ba c k up ag a i n . Th i s wi l l cl e a r al l cu r r e n t l y- es t a b l i s h e d co n n e c t i o n s an d al l o w ne w us e r s in on c e ag a i n .

If it is a ca s e of ge n u i n e ab u s e rat h e r tha n ac c i d e n t a l foo l i s h n e s s , ho w e v e r , the pr o b l e m wi l l qu i c k l y re-ap p e a r an d req u i r e es c a l a t i o n to ou r ne t w o r k tea m in or d e r to bl o c k the IP ad d r e s s of the of f e n d i n g us e r an d to ar r a n g e to ha v e the m co n t a c t e d .

6.4.3 Database server – MySQL

Pa g e 95

In the Li n u x wo r l d the tw o ma i n da t a b a s e se r v e r s yo u ar e lik e l y to en c o u n t e r ar e My S Q L , an d Po s t g r e S Q L (quit e of t e n jus t ref e r r e d to as Po s t g r e s). On ou r sh a r e d ho s t i n g pla t f o r m we us e My S Q L , an d so thi s is the on e tha t we wi l l fo c u s on he r e in thi s gu i d e .

MySQL at a glance

Th e de f a u l t co n f i g u r a t i o n for My S Q L on a Re d Ha t sy s t e m is as fol l o w s :

• Th e My S Q L bi n a r y is ins t a l l e d as /usr/sb i n/m y s q l d• Da t a b a s e s ar e st o r e d in /var/lib/my s q l , ea c h in its ow n dir e c t o r y . So a

da t a b a s e ca l l e d 'my d b ' wo u l d be sto r e d in the dir e c t o r y :/var/lib/my s q l/m y d b

• Th e co n f i g u r a t i o n fil e for My S Q L is /etc/my. c n f• Th e de f a u l t er r o r log file wi l l ha v e a na m e of the for m :

/var/lib/my s q l/<h o s t n a m e > . e r r So for a da t a b a s e se r v e r my s q l . i o m a r t . c o m , the er r o r log wo u l d be:

/var/lib/my s q l/m y s q l . i o m a r t . c o m . e r r• By de f a u l t the r e is no qu e r y log – on l y er r o r s an d st a r t u p/s h u t d o w n

me s s a g e s ar e log g e d

As wi t h Ap a c h e an y an d al l of the ab o v e ca n be ch a n g e d , al t h o u g h the s e wi l l be the de f a u l t ou t- of- the- bo x se t t i n g s .

W h e n en a b l e d , the qu e r y log for My S Q L rec o r d s ea c h ind i v i d u a l SQ L da t a b a s e qu e r y run, mu c h in the ma n n e r tha t Ap a c h e ' s ac c e s s _ l o g rec o r d s ev e r y si n g l e HT T P req u e s t ma d e . Th e rea s o n tha t it is no t en a b l e d by de a f u l t is tha t su c h a log qu i c k l y be c o m e s ve r y lar g e , an d ca n slo w dow n the My S Q L se r v e r co n s i d e r a b l y .

Y o u ca n en a b l e qu e r y log g i n g , bu t yo u wo u l d no r m a l l y on l y wa n t to do so if yo u ha d to ca r r y ou t de b u g g i n g tha t ab s o l u t e l y req u i r e d yo u ha v i n g qu e r y log inf o r m a t i o n . Yo u ' d al s o us u a l l y wa n t to tur n it of f ag a i n as so o n as yo u no lon g e r ne e d e d it, so as no t to co n s u m e unn e c e s s a r y di s c sp a c e or sl o w the My S Q L se r v e r dow n .

Pa g e 96

Task 1: Starting MySQL

As yo u ' v e pr o b a b l y com e to ex p e c t :

service mysql start

No t e tha t it' s 'service mysql' an d no t 'service mysqld' – the ot h e r se r v i c e com m a n d s we ' v e se e n so far us e the sa m e se r v i c e na m e as the na m e of the bi n a r y its e l f (e.g. ht t p d , vs f t p d), bu t My S Q L doe s no t .

Fo r ex a m p l e :

[root@claudius root]# ps -aux | grep mysqlroot 27767 0.0 0.0 3692 680 pts/0 S 11:49 0:00 grep mysql[root@claudius root]# service mysql startStopping mysql: [ OK ][root@claudius root]# ps -aux | grep mysqlmysql 1671 23.0 7.1 185864 69296 ? S Mar16 3893:43 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/claudius.iomart.com.pid --skip-locking --socket=/var/lib/mysql/mysql.sockroot 1636 0.0 0.0 4248 420 ? S Mar16 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/claudius.iomart.com.pidroot 27768 0.0 0.0 4768 676 pts/0 S 17:01 0:00 grep mysql[root@claudius root]#

Task 2: Stopping MySQL

As for ot h e r s :

service mysql stop

Fo r ex a m p l e :

[root@claudius root]# ps -aux | grep mysqlmysql 1671 23.0 7.1 185864 69296 ? S Mar16 3893:43 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/claudius.iomart.com.pid --skip-locking --socket=/var/lib/mysql/mysql.sockroot 1636 0.0 0.0 4248 420 ? S Mar16 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/claudius.iomart.com.pidroot 27767 0.0 0.0 3692 680 pts/0 S 11:49 0:00 grep mysql[root@claudius root]# service mysql stopStopping mysql: [ OK ][root@claudius root]# ps -aux | grep mysqlroot 27768 0.0 0.0 4768 676 pts/0 S 17:01 0:00 grep mysql[root@claudius root]#

Pa g e 97

Task 3: Checking the error log

If ev e r y My S Q L is ex p e r i e n c i n g pr o b l e m s of som e kin d , be it wi t h run n i n g ou t of me m o r y or di s c sp a c e or if a pa r t i c u l a r da t a b a s e on the sy s t e m ha s be c o m e co r r u p t , it wi l l log thi s inf o r m a t i o n to its er r o r log. To rec a p , the er r o r log by de f a u l t is loc a t e d in a file wi t h a na m e of the for m a t :

/var/lib/mysql/<hostname>.err

He r e , for ex a m p l e , ar e the las t few lin e s fro m a My S Q L se r v e r ca l l e d s 1 d b m y s q l 2 :

[root@s1dbmysql2 root]# cd /var/lib/mysql[root@s1dbmysql2 mysql]# ls -l s1dbmysql2.iomart.com.err-rw-rw---- 1 mysql root 625534 Mar 26 17:19 s1dbmysql2.iomart.com.err[root@s1dbmysql2 mysql]# tail -10 s1dbmysql2.iomart.com.err070326 16:08:05 [ERROR] /usr/sbin/mysqld: Can't open file: 'sms_status.MYI' (errno: 145)070326 16:08:05 [Warning] Checking table: './s1homes/sms_status'070326 17:11:52 [ERROR] Got error 134 when reading table './s1homes/property_images'070326 17:12:14 [ERROR] Got error 134 when reading table './s1homes/property_images'070326 17:12:42 [ERROR] Got error 134 when reading table './s1homes/property_images'070326 17:14:20 [ERROR] /usr/sbin/mysqld: Can't open file: 'messageIDgenerator.MYI' (errno: 145)070326 17:14:20 [Warning] Checking table: './s1homes/messageIDgenerator'070326 17:14:20 [ERROR] /usr/sbin/mysqld: Can't open file: 'sms_messages_sent.MYI' (errno: 145)070326 17:14:20 [Warning] Checking table: './s1homes/sms_messages_sent'070326 17:19:52 [Note] Found 1206874 of 1206876 rows when repairing './s1homes/property_images'[root@s1dbmysql2 mysql]#

Th i s er r o r log sh o w s us a va r i e t y of pr o b l e m s wi t h the tab l e s in the 's 1 h o m e s ' da t a b a s e . Us u a l l y , My S Q L ca n tak e ca r e of the s e iss u e s its e l f the s e da y s , bu t som e t i m e s we nee d to ma n u a l l y run fur t h e r ch e c k s on an y tab l e s tha t ar e ha v i n g tro u b l e .

Pa g e 98

Task 4: Checking tables for errors

If ev e r My S Q L is rep o r t i n g er r o r s wi t h a tab l e in a pa r t i c u l a r da t a b a s e tha t it's no t be e n ab l e to fix its e l f , the r e is a com m a n d- lin e ut i l i t y wi t h wh i c h we ca n ca r r y ou t fur t h e r tab l e ch e c k i n g ou r s e l v e s to ve r i f y tha t the r e is a pr o b l e m .

Th e com m a n d for pe r f o r m i n g tab l e ch e c k s is myisamchk. It ca n pe r f o r m saf e ch e c k s on an y tab l e in an y st a n d a r d My S Q L da t a b a s e wi t h the -c, -e an d -T fla g s .

Fo r ex a m p l e , let ' s sa y we wa n t e d to fur t h e r inv e s t i g a t e thi s pa r t i c u l a r tab l e tha t wa s me n t i o n e d in the My S Q L er r o r log in the pr e v i o u s se c t i o n :

070326 17:11:52 [ERROR] Got error 134 when reading table './s1homes/property_images'

Re m e m b e r tha t al l da t a b a s e s ar e st o r e d in a di r e c t o r y ca l l e d /var/lib/my s q l/<d a t a b a s e na m e>. In thi s ex a m p l e the da t a b a s e is 's 1 h o m e s ' , an d so we nee d to ch e c k the tab l e pr o p e r t y _ i m a g e s in the di r e c t o r y /var/lib/my s q l/s 1 h o m e s .

Fo r ex a m p l e :

[root@s1dbmysql2 root]# cd /var/lib/mysql/s1homes[root@s1dbmysql2 s1homes]# ls -l property_images*-rw-rw---- 1 mysql mysql 8698 May 2 2006 property_images.frm-rw-rw---- 1 mysql mysql 105488948 Mar 28 11:40 property_images.MYD-rw-rw---- 1 mysql mysql 15803392 Mar 28 11:40 property_images.MYI[root@s1dbmysql2 s1homes]# myisamchk -c -e -T property_imagesChecking MyISAM file: property_imagesData records: 1206073 Deleted blocks: 20092myisamchk: warning: 1 client is using or hasn't closed the table properly- check file-size- check record delete-chain- check key delete-chain- check index reference- check data record references index: 1myisamchk: error: Found 1206094 keys of 1206073- check records and index referencesmyisamchk: error: Record at: 102568272 Can't find key for index: 1MyISAM-table 'property_images' is corruptedFix it using switch "-r" or "-o"[root@s1dbmysql2 s1homes]#An d we ca n se e tha t we do ind e e d ha v e an er r o r in thi s tab l e – its st r u c t u r e is co r r u p t , an d it ne e d s to be rep a i r e d . At thi s po i n t yo u wo u l d es c a l a t e thi s ma t t e r to an o t h e r me m b e r of the sy s t e m s ad m i n i s t r a t i o n tea m to ca r r y ou t the rep a i r of the tab l e , or to no t i f y the cus t o m e r wh o s e My S Q L se r v e r it wa s , as ap p r o p r i a t e .

Pa g e 99

6.4.4 E-mail server – sendmail

On mo s t Re d Ha t- de r i v e d Li n u x sy s t e m s , se n d m a i l is the de f a u l t ma i l tra n s f e r ag e n t (or MT A). MT A s ar e pi e c e s of se r v e r so f t w a r e tha t ar e res p o n s i b l e for tra n s f e r r i n g e-ma i l vi a SM T P from one se r v e r to an o t h e r . Ty p i c a l l y an SM T P se r v e r pe r f o r m s on e of the fol l o w i n g ro l e s :

• Re c e i v e s e-ma i l vi a SM T P fro m the ou t s i d e wo r l d , an d sa v e s it in loc a l PO P 3 or IMA P 4 ma i l b o x e s

• Re c e i v e s e-ma i l vi a SM T P fro m the ou t s i d e wo r l d an d pa s s e s it on to an o t h e r loc a l ma i l se r v e r wh i c h ho l d s the ac t u a l PO P 3 or IMA P 4 ma i l b o x e s

• Re c e i v e s e-ma i l vi a SM T P fro m cus t o m e r s se n d i n g ma i l fro m the i r e-ma i l cl i e n t s to be pa s s e d on to ot h e r de s t i n a t i o n s el s e w h e r e

• An y com b i n a t i o n of the ab o v e

So an MT A ca n ei t h e r be rec e i v i n g e-ma i l from the ou t s i d e wo r l d tha t ha s be e n se n t to the cu s t o m e r s it se r v e s , or it ca n be fo r w a r d i n g on e-ma i l se n t by tho s e cus t o m e r s to the ou t s i d e wo r l d .

sendmail at a glance

As be f o r e , the s e ar e the de f a u l t as s u m p t i o n s on a Re d Ha t- de r i v e d sy s t e m .

• Th e se n d m a i l bi n a r y is us u a l l y ref e r r e d to as /usr/sb i n/se n d m a i l• Ma i l b o x e s for loc a l us e r s ar e ke p t in /var/sp o o l/m a i l• Th e co n t e n t s of the cu r r e n t ma i l qu e u e ar e st o r e d in

/var/sp o o l/m q u e u e• Th e de f a u l t log fil e for se n d m a i l is /var/log/ma i l l o g• Mo s t co n f i g u r a t i o n fil e s ar e st o r e d in the di r e c t o r y /etc/ma i l• Th e lis t of IP ad d r e s s e s tha t ar e al l o w e d to se n d ma i l thr o u g h the

se r v e r is ke p t in the fil e /etc/ma i l/ac c e s s• Th e lis t of do m a i n s tha t the se r v e r ca n rec e i v e e-ma i l for is ke p t in

/etc/ma i l/lo c a l- ho s t- na m e s• Th e lis t of co n f i g u r e d e-ma i l ad d r e s s e s is ke p t in

/etc/ma i l/vi r t u s e r t a b l e• Th e ma s t e r co n f i g u r a t i o n fil e is /etc/ma i l/se n d m a i l . c f

Task 1: Starting and stopping sendmail

Pa g e 100

We wo n ' t dw e l l too mu c h on the st a r t i n g an d st o p p i n g at thi s po i n t – yo u ge t the ide a by now (hope f u l l y !).

To sta r t :

service sendmail start

An d to st o p :

service sendmail stop

If yo u ' v e st o p p e d se n d m a i l an d ar e wa n t i n g to re-st a r t it, it's al w a y s a go o d ide a to ch e c k tha t no ol d se n d m a i l pr o c e s s e s ar e st i l l ha n g i n g ar o u n d . Us u a l l y the r e wi l l be – ne a r l y al w a y s , in fac t , be c a u s e e-ma i l tha t is in the mi d s t of be i n g tra n s f e r r e d wh e n yo u as k se n d m a i l to st o p wi l l fin i s h pr o c e s s i n g be f o r e it ac t u a l l y ex i t s . So alw a y s ma k e su r e tha t al l ol d se n d m a i l pr o c e s s e s ar e de a d an d go n e be f o r e st a r t i n g se n d m a i l ba c k up, or el s e yo u ' l l jus t ge t an er r o r .

Task 2: Checking the mail queue

On e of the mo s t com m o n tas k s yo u wa n t to pe r f o r m wi t h se n d m a i l is to ch e c k ho w ma n y e-ma i l s ar e cu r r e n t l y sit t i n g in its ma i l qu e u e . If it's no n e an d it's a se r v e r tha t sh o u l d be bu s y , tha t ' s a pr o b l e m . If it's se v e r a l ten d s of tho u s a n d s , tha t ' s pr o b a b l y a pr o b l e m too, si n c e the se r v e r mi g h t be too ful l to ev e r ge t thr o u g h its qu e u e an d req u i r e at t e n t i o n to fix wh a t e v e r ha s go n e wr o n g .

Th e ma i l qu e u e is ch e c k e d wi t h the 'ma i l q ' com m a n d as fol l o w s :

[root@mta4 root]# mailq /var/spool/mqueue/q1 (31 requests)-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------l2SAtUoH022193 4915 Wed Mar 28 11:55 <> 8BITMIME (Deferred: Connection timed out with mail3.bitsmart.com.) <sidelinefats@bitsmart.com>l2SAnl4S010249 9424 Wed Mar 28 11:49 <hwffppclotz@versanet.de> (Deferred: 451 <bgm2@btconnect.com>... Requested mail action ) <bgm2@btconnect.com>l2SAijV8000555- 1501 Wed Mar 28 11:44 <leota18@speedy.com.ar> (Deferred) <marcus@startrecruitment.com>..l2S2biiq024719 8801 Wed Mar 28 03:37 <istchoices@graphicworksatlanta.com> (Deferred: Connection timed out with aerospaceuk.com.) <catchall@aerospaceuk.com>l2S4Xato027998 16851 Wed Mar 28 05:33 <magnadrylafayette.net@jefferylee.com> (Deferred: 451 <magnadrylafayette.net@jefferylee.com> unable ) <cliveyandrews@tiscali.co.uk> Total requests: 995

Pa g e 10 1

[root@mta4 root]#

In thi s ab b r e v i a t e d lis t (you wo u l d rea l l y ha v e se e n a ful l lis t of al l 995 e-ma i l s), yo u ca n se e the qu e u e ID for ea c h e-ma i l (ever y ma i l cu r r e n t l y in the qu e u e ha s a un i q u e ide n t i f i e r – a bi t lik e PI D s for pr o c e s s e s in te r m s of wh a t it do e s), the siz e of the e-ma i l in by t e s , wh e n it wa s su b m i t t e d to the se r v e r , an d the e-ma i l ad d r e s s e s it is co m i n g fro m and go i n g to.

No r m a l l y yo u wo n ' t ne e d to wo r r y too mu c h ab o u t thi s da t a , ex c e p t to ch e c k tha t the qu e u e is of an ac c e p t a b l e siz e , an d tha t the r e ar e no ma s s i v e e-ma i l s st u c k in the qu e u e tha t mi g h t be ca u s i n g pr o b l e m s . Al s o , if ev e r yo u se e tho u s a n d s of e-ma i l s al l com i n g fro m the sam e ad d r e s s or go i n g to the sa m e rec i p i e n t , tha t ' s pr o b a b l y a sp a m run or som e o n e ' s ge t t i n g ma i l- bo m b e d .

Task 3: Finding out more about a particular message in the queue

Som e t i m e s , yo u wa n t mo r e de t a i l e d inf o r m a t i o n on on e pa r t i c u l a r me s s a g e in the qu e u e . Yo u ca n do thi s by us i n g gr e p to se a r c h for its un i q u e qu e u e ID in /var/log/ma i l l o g , the de f a u l t se n d m a i l log f i l e .

Le t ' s sa y we wa n t e d ful l e r inf o r m a t i o n on tha t las t e-ma i l in the ma i l qu e u e , the on e fro m ma g n a d r y l a f a y e t t e . n e t @ j e f f e r y l e e . c o m . We ca n se e it ha s a un i q u e qu e u e ID of 12 S 4 X a t o 0 2 7 9 9 8 , so we ca n se a r c h for thi s in the se n d m a i l log fil e:

[root@mta4 root]# grep l2S4Xato027998 /var/log/maillogMar 28 05:33:36 mta4 sendmail-rx[27946]: l2S4XZ61027946: from=<magnadrylafayette.net@jefferylee.com>, size=17311, class=0, nrcpts=1, msgid=<000001c770f2$1a5c4680$0100007f@localhost>, proto=SMTP, daemon=MTA-RX, relay=[12.107.194.76]Mar 28 05:33:37 mta4 sendmail[27998]: l2S4Xato027998: from=<magnadrylafayette.net@jefferylee.com>, size=17488, class=0, nrcpts=1, msgid=<000001c770f2$1a5c4680$0100007f@localhost>, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]Mar 28 05:33:49 mta4 sendmail[28002]: l2S4Xato027998: to=<cliveyandrews@tiscali.co.uk>, delay=00:00:13, xdelay=00:00:12, mailer=esmtp, pri=47488, relay=mx5.uk.tiscali.com. [212.74.100.151], dsn=4.0.0, stat=Deferred: 451 <magnadrylafayette.net@jefferylee.com> unable to verify addressMar 28 06:51:53 mta4 sendmail[1918]: l2S4Xato027998: to=<cliveyandrews@tiscali.co.uk>, delay=01:18:17, xdelay=00:00:11, mailer=esmtp, pri=137488, relay=mx1.uk.tiscali.com. [212.74.100.150], dsn=4.0.0, stat=Deferred: 451 <magnadrylafayette.net@jefferylee.com> unable to verify addressMar 28 08:01:39 mta4 sendmail[27734]: l2S4Xato027998: to=<cliveyandrews@tiscali.co.uk>, delay=02:28:03, xdelay=00:00:11, mailer=esmtp, pri=227488, relay=mx3.uk.tiscali.com. [212.74.100.148], dsn=4.0.0, stat=Deferred: 451 <magnadrylafayette.net@jefferylee.com> unable to verify addressMar 28 10:01:18 mta4 sendmail[7887]: l2S4Xato027998: to=<cliveyandrews@tiscali.co.uk>, delay=04:27:42, xdelay=00:00:09, mailer=esmtp, pri=317488, relay=mx5.uk.tiscali.com. [212.74.100.147], dsn=4.0.0, stat=Deferred: 451 <magnadrylafayette.net@jefferylee.com> unable to verify address

Pa g e 102

Mar 28 12:01:36 mta4 sendmail[1192]: l2S4Xato027998: to=<cliveyandrews@tiscali.co.uk>, delay=06:28:00, xdelay=00:00:12, mailer=esmtp, pri=407488, relay=mx5.uk.tiscali.com. [212.74.100.150], dsn=4.0.0, stat=Deferred: 451 <magnadrylafayette.net@jefferylee.com> unable to verify address

No w , the r e ' s a lot of da t a he r e to be su r e , bu t we ca n se e tha t the me s s a g e is be i n g rej e c t e d by its fin a l de s t i n a t i o n (mx5.u k . t i s c a l i . c o m) be c a u s e the ad d r e s s of the se n d e r ca n n o t be ve r i f i e d . In ot h e r wo r d s , it' s alm o s t ce r t a i n l y a sp a m e-ma i l or som e ot h e r sim i l a r jun k .

It's a sa d fac t tha t pr a c t i c a l l y al l e-ma i l tha t pa s s e s thr o u g h mo s t ma i l se r v e r s on the Int e r n e t the s e da y s is ei t h e r sp a w n e d by vir u s- inf e c t e d PC s or sp a m m e r s . So yo u ' r e lik e l y to se e far mo r e e-ma i l s tha t ar e jun k tha n no t in yo u r tra v e l s .

Pa g e 103

7. ConclusionWe l l , we ' v e now com e to the en d of ou r Lin u x Gu i d e . We ' v e go n e in thi s do c u m e n t fro m the ve r y ba s i c s of co n n e c t i n g to a Li n u x sy s t e m an d na v i g a t i n g its fil e s y s t e m , up to mo r e ad v a n c e d sy s t e m s ad m i n i s t r a t i o n co n c e p t s de a l i n g wi t h ind i v i d u a l se r v e r so f t w a r e pr o d u c t s su c h as Ap a c h e an d My S Q L .

It's a lot to tak e in, an d yo u sh o u l d n ' t wo r r y if at thi s po i n t the r e ar e pa r t s tha t yo u ar e st i l l un c l e a r on, or if yo u fee l yo u ne e d to rea d ba c k ov e r pr e v i o u s se c t i o n s of the Gu i d e . By al l m e a n s do so – tha t ' s wh a t it's the r e for.

Si m i l a r l y , if yo u ha v e an y qu e s t i o n s at al l, fee l fre e to as k – ot h e r me m b e r s of the tea m wi l l be mo r e tha n ha p p y to he l p yo u.

Go o d luc k !

Pa g e 104