Introduction to Unix

VIJAY.CHERUKURI
INFOLOB SOLUTIONS INC

Course Outline
Introduction Logging On and Off Unix File System Structure Dealing with Directories Handling Files

Course Outline
Shell Basics Shell Programming Customisation The Internet And more...

Introduction
Unix: a family of operating systems
Initial version (~1969) DEC PDP-7 Rewritten version (~1972) DEC PDP-11 Currently available on machines ranging from notebook PCs to Cray supercomputers

Uses
From embedded control systems (Boeing 777) to weather forecasting

History
The Multics Project
A computer utility Funded by (D)ARPA

Bell Labs Involvement Bell Labs Withdrawal Ritchie & Thompsons Unics

Flavours of Unix
AT&T
Early versions numbered V1, V2 etc. Last being V7

AT&T: System III AT&T: System V UCB: BSD Releases System V release 4

Unix Standards
Theres One for Everybody! System V release 4 (USL)
SVID the System V Interface Definition

IEEE Posix
A series of emerging standards Adopted as a FIPS (Federal Information Processing Standard)

X/Open

Unix Availability
IBM compatibles
Linux FreeBSD386 Solarisx86 SCO Unix

Apple Macintosh
AUX Linux

Unix Availability
Workstations
Sun SPARC
Solaris Linux

Digital Alpha
Digital Unix Linux

Supercomputer
Cray UniCOS

Features of Unix
A multiuser system
Each user has his/her own username or id Many users may be logged on simultaneously Access to files and programs is based on username

A preemptive multitasking system

Each user may run multiple tasks CPU time is shared equitably between tasks

Why Learn/Use Unix?

Portability
Of programs and skills

Familiarity Efficiency Scaleability Applications

Software Availability
You name it weve got it! Programming Languages Text Editors Databases Document Preparation The Net And much, much more

Logging On
Users must identify themselves to the system this is called logging on At the login: prompt enter your id then press Enter (or return) At the password: prompt enter your password note that you cannot see what you are typing Change your password with the passwd (note spelling) command

Logging Off
Disconnecting from the system is called logging off Varies from system to system exit logout ^D

Typing and Correcting Errors

System allows for user-correction before return is pressed Characters used for deleting previous character and whole line are systemdependent If in doubt check with your system administrator Case does matter

Commands and Arguments

Commands are entered in the form:
command [arg1 ...]

Tabs and spaces (aka whitespace) separate command and arguments Quoting arguments
try echo Im a * - a real * and echo Im a * - a real * The * character has a special meaning which is ignored if it is inside quotes

Unix File System

Organised as an inverted tree of directories Begins at the root / Directories contain files and other directories Every user allocated a home directory

A Sample Tree
Details will differ from system to system
/ usr smith .login jones ls test.c bin (root) etc fstab

vi passwd

Paths and Pathnames

A pathname (or path) is a route from the root to the file or directory in question Directory names are separated by /s (DOS users please note) A path beginning with a / is an absolute path A path beginning with anything but a / is a relative path

Working Directories and Pathnames

Relative paths are based upon the current working directory Change the working directory with the cd command Show the working directory with the pwd command List the contents of the working directory with the ls command

Special Names
Two special entries in very directory Created and deleted automatically . the directory itself
cd ./abc is equivalent to cd abc

.. the parent directory

If the working directory is /home/comp/abc/jane then the command cd .. makes the working directory /home/comp/abc

More Special Names

~ my home directory
cd ~/jane

~username usernames home directory

ls ~abc/bin

This tilde expansion is a feature of the shell . and .. a part of the Unix file system Not all shells support tilde expansion

Creating and Removing Directories

A new directory can be created with the mkdir command
Note that this cannot be shortened to md

A directory can be removed using the rmdir command

The directory must be empty (no files or subdirectories) The name cannot be shortened to rd

Manipulating Files
Files are usually created by editors, compilers etc. rm removes files (and directories) cp copies files (and directories) mv moves (renames) files (and directories) ls lists files (not the contents of files) cat displays (con)cat(enates) files

The Shell
The shell is equivalent to COMMAND.COM on DOS On other systems known as the command processor Several alternative shells Most of what is considered the Unix interface is really the shell interface

Shell Flavours
sh the Bourne shell (after its creator): the basic, no-frills shell ksh the Korn Shell: sh plus visual editing csh the C-shell: introduced C-style programming features tcsh: csh plus visual editing bash the Bourne Again SHell: sh plus visual editing

Shell Basics
Shells provide:
Multiple commands on a single line Redirection of input and output Expansion of wildcard filenames Background execution Pipelines Script-programming facility

Wildcard Expressions
We often wish to apply a command to a selection of files or directories The shell wildcard (often known as globbing) feature allows this ls *.c
lists all files whose names end in .c Note that on Unix a . is just another filename character

Wildcard Expressions
When we type in a wildcard expression it is expanded by the shell before the command runs If we type is ls *.c and the working directory contains prog1.c, abc.c and xyz.c then the actual command is
ls abc.c prog1.c xyz.c The wildcard expression has been expanded in place

Wildcard Characters
? matches any character * matches any number of characters [abc] matches any of the characters in the brackets [!abc] matches anything except the characters in the brackets [a-z] matches the range of characters in the brackets

Wildcard Examples
abc?def
matches abcXdef, abc.def, abc3def but not abcdef

abc*
matches abc. abcd, abcdef, abc.another.part

*.*
matches any name with a . Note that this is not the same as DOS

Wildcard Examples
*
matches any name (except a name beginning with a .)

[abc]def
matches adef, bdef and cdef

[a-z]*
matches any name beginning with a lower-case letter

Wildcard Examples
[a-zA-Z]*
any name beginning with an alphabetic character

[!a-zA-Z]*
any name beginning with a non-alphabetic character

[a-z]*[!A-Z]
any name beginning with a lower-case letter and ending with anything except an uppercase letter

Avoiding Wildcard Expansion

Sometimes it is necessary to enter a wildcard character on a command line, but not have it expanded Quote the argument containing the character
ls abc*def
ls the file whose name is abc*def (if the system allows such a name)

ls abc*def
the same

Avoiding Wildcard Expansion

Escape the character
ls abc\*def
again lists the file whose name is abc*def

These can be used in combination

echo Im a * and \* matches * * inside the double quotes is not expanded * after the \ is not expanded only the final asterisk is expanded

Redirection of I/O
Every program run under Unix is provided with three I/O streams stdin, stdout and stderr
stdin stdout Program stderr

Redirection of I/O
Shell allows any or all of these streams to be diverted prog >file1 sends stdout to file1 prog >>file1 appends stdout to file1 prog 2>file2 sends stderr to file2 prog <file3 takes stdin from file3

Redirection of I/O
Shell also allows chaining of commands together in a pipeline ls > temp; sort < temp >sorted; rm temp
Works, but requires temporary intermediate file, which must be manually removed

ls | sort >sorted
No intermediate file, processes run in parallel

Job Control
Normally the shell waits for a command to complete before prompting for another command Can run a job in the background and not wait for it gcc -o bigprog bigprog.c&

Job Control
Some shells (not Bourne) allow the user to interrupt a running program Use ^Z while the program is running Suspends the job Can resume the job several ways
fg %1 bg %1

jobs command lists all jobs

Filters
A filter is a command which reads from stdin performs some processing and write to stdout Unixs pipe feature allows us to create pipelines of filters more sort head tail

Access Control
As Unix is a multiuser system some form of access control is essential Unix divides the user community into three categories:
the owner of a file or directory the owners group everybody else

Each file/directory has nine protection bits grouped into three groups

File Permissions
Unix provides three distinct permissions for files:
read permission to examine the contents of a file write permission to alter the contents of a file execute permission to run a file as a program or shell script

Directory Permissions
The same permissions apply to directories, but with different meanings:
read permission to look at the contents of the directory, i.e. to see file names write permission to create or delete files within the directory execute permission to access the directory. Note that read is needed to do filename expansion

Listing Permissions
the -l (long) argument to the ls command lists permissions
-rw-rw-rw- 1 smith comp Jan 20 1:24 file1 a file with read and write for the owner, group and world drwxr-xr-x 1 jones staff Jun 14 8:29 dir1 a directory with read/write/execute for the owner, read/execute for group and world

Setting Permissions chmod

The chmod (change mode) command sets and unsets file and directory permissions chmod mode filename(s)
where mode can be:
who op permission op2 permission2 ... octalnumber

Permissions the Easy Way

chmod o-w file
Remove write permission for other

chmod g=rx
Set group permission to read/write

chmod u+rwx
Add read/write/execute for user (owner)

chmod a+rx
Add read/execute for all (user, group and other)

Permissions the Quick Way

chmod also allows octal representation chmod 755 directory
Set rwx for owner, rx for group and world

chmod 644 file

Set rw for owner, r for group and world

chmod 711 file

Set rwx for owner, x for group and world

Default Permissions
Every time a directory or file is created it must immediately have some permissions The umask instruction sets this default umask ddd
ddd is a three digit octal number The default permissions are 777 the umask value

The SuperUser
Every Unix system has at least on userid which is special This is referred to as root although the name may be different root has a user number (in the /etc/passwd file) of 0 root may access any files or directories, whatever their protection bits

Trusted Software
Consider the mail system Each user wants to be able to send mail to any other user No user wants any other user to be able to read his/her mail How can we construct a protection mechanism to allow this?

The setuid mechanism

The setuid mechanism allows a program to run as if it were running under the userid of its owner Allows a constrained access to private or privileged data chmod allows its setting:
chmod u+s filename

The setgid mechanism

Similarly a file can have its setgid bit When it is run, it runs with its effective group the same as the owners Also set via chmod:
chmod g+s filename

Both setuid and setgid are potential security holes

Creating Files
Most human-readable files on Unix are created with a text editor Unix has many, many different editors ed a very old line-oriented editor ex an enhanced version of ed vi the most ``popular editor a Visual Interface to ex

Editors
emacs full-screen, fully-extensible editor created in the MIT AI Labs pico the pine editor joe Joes Own Editor jove Jonathans Own Version of Emacs And doubtless many others Choose one you like, but learn vi

The vi Editor
Created by Bill Joy at UCB Visual Interface to the ex line editor Holds file being edited as an internal buffer Screen acts as a window onto the buffer Operates in two basic modes
Command mode Insert mode

Starting vi
vi
Starts vi with an empty buffer

vi file
Starts vi opens file for editing

vi +n file
Starts vi opens file at line n

vi +/pattern file
Starts vi opens file at the first line matching pattern

vi always starts in command mode

vi command mode
In command mode every character typed is a command You can move about in the buffer Characters/lines can be deleted or moved To enter text shift to insert mode

vi insert mode
In insert mode characters typed are entered into the buffer Shift from command to insert move using:
i insert text before cursor a append text after cursor

Shift from insert back to command mode by typing ESCape

vi moving around the file

Only part of the buffer is usually visible onscreen In command mode can navigate through the file Can choose to move:
to a position within the buffer to a position on the screen to a particular text string

vi positioning the cursor

Examples:
H L M h j k l Go to top line on screen Go to last line on screen Go to middle line on screen Move cursor left Move cursor down Move cursor up Move cursor right

Or you could use the arrow keys....

vi more movement
nG Go to line n and centre it in the screen /textCR Find text (forward search) ?textCR Find text (backward search) n Find next occurrence of text (same direction) N Find next occurrence of text (opposite direction) See Quick Reference Sheet for more

vi complex commands
Typing the : character in command mode causes a prompt to appear at the bottom of the screen ex commands can be typed here :q
quit vi will complain if there are unsaved changes

:q! or ZZ
quit and lose any unsaved changes

vi reading and writing files

:r filename
read the named file into the buffer

:w filename
write the buffer to named file

:w
rewrite the buffer to an already named file

Special Files
Unix treats every device as a file Special files can refer to
floppy disk CD-ROM hard disk partitions

Special files all live in the /dev directory

/dev/fd0 first floppy disk /dev/hda1 first partition in first hard disk /dev/modem

More on Filesystems
Unix stores files in filesystems A filesystem lives in a hard disk partition, on a floppy or on a CD-ROM or on a networked computer A filesystem is created using a special command, typically newfs or something similar Before a filesystem can be used it must be mounted

Mounting Filesystems
Only root may mount a filesystem The mount command requires three pieces of information
The special file which refers to the device where the filesystem lives The type of filesystem which will be found there The place to make it appear in the filesystem, the mount point

Mounting Filesystems
Assuming: /dev/fd0 refers to a floppy drive There is a DOS formatted disk in the drive There already exists a directory called /mnt/floppy The command (Linux version)

mount -t

msdos /dev/fd0 /mnt/floppy

Attaches the floppy filesystem A file abc.txt on the floppy is now accessable as /mnt/floppy/abc.txt

Checking on Mounted Filesystems

The mount command with no arguments will displayed the currently mounted filesystems The df command can be used to see how much space is utilised/remains on each mounted filesystem

Unmounting Filesystems
The umount command is required to un-mount a filesystem Note the filesystem must not be busy
No files open No users with any directory in the filesystem as their current working directory

Customising the Unix Environment

Most aspects of the environment can be customised
By the system administrator By the individual user

Examples:
Which shell? Which editor? Which mailer?

Customising Logging In
The login program runs as root After you supply userid and password it:
Switches userid to your userid Runs the shell configured for you in the system password file, which
Sources the system login file Sources your personal login file Displays the prompt

The System Login File

Exact location/name depends on the shell you use Usually named something like /etc/profile Is sourced (i.e. run) for every user when they log in Maintained by the system administrator

The Personal Login File

Name depends on the shell
~/.profile ~/.login ~/.bash_login for the Bourne/Korn shells for the C-shell for the BASH shell

Will be sourced every time you log in Can set up your environment to your own particular requirements

Starting New Shells

Some shells have two flavours of startup file One run when you log in One run every time you start the shell program ~/.cshrc ~/.bashrc

Shell Programming
Each shell defines a simple language While many shell features can be used interactively they find greater utility when used in shell scripts Using shell scripts you can create new commands A shell script is simply a text (ASCII) file containing shell commands

Invoking a Shell Script

. scriptname
Run the script within the current invocation of the shell

shellname arguments scriptname

Run a separate shell to run the script

scriptname
Also runs a separate shell to run the script

Script basics
Comments
A shell script is a program Programs should be commented Anything after a # character on a line is ignored by the shell

Examples:
# This whole line is a comment ls -lsa $1 # List the specified directory

The Script Header

A script will only run reliably if it is run under the correct shell The first line of a script can specify which shell to use No # sh script # but no ! csh script #! remainder of the line is the path of the program used to interpret this script

Script Header Examples

#!/usr/bin/ksh
Invoke the Korn shell on this script

#!/bin/cat
Cat the rest of this script (Note that the program doesnt have to be a shell)

#!/bin/csh -f
Invoke the C-shell with the -f argument on this script

Basic Shell Constructs

cmd & cmd1 ; cmd2 (cmd1; cmd2) cmd1 | cmd2 cmd1 `cmd2` cmd1 && cmd2 cmd1 || cmd2

Positional Parameters
You can pass parameters to scripts which can be referred to by position within the script myscript first second third If myscript contains the line
echo P1 = $1, P2 = $2, P3 = $3

the output from that line is

P1 = first, P2 = second, P3 = third

The shell substitutes the values of the positional parameters before running the command line

Shifting Positional Parameters

We can only refer directly to 9 positional parameter $1 to $9 Korn shell allows ${10} etc We can pull parameters after the 9th into view with the shift command After a shift
$1 -> $0 $2 -> $1 .... ${10} -> $9

Environment Variables
Used for local storage in shell scripts Also used for setting default options for various packages Value established with the set command
set NAME=value NAME=value set NAME set

Environment Variables in Shell Scripts

Changes to the environment by default only affect the current shell Shell scripts are usually run in a subshell To make permanent changes,use the export command in the script TEST=A test value;export TEST This also applies to user profiles and login scripts

Using Environment Variables

Can use environment variable NAME by using the expression $NAME or ${NAME} in command line or script echo $NAME = $NAME Can avoid expansion of $NAME by using single quotes echo $NAME = $NAME Note that $ variable expansion is not suppressed by double quotes echo $NAME = $NAME

Buit-in Shell Variables

A number of variables are supplied by the shell $# $? $* $@ And many others

Command Exit Values

When a command completes it sets an integer called the return or exit status Conventionally: zero means no error non-zero means an error of some kind The shell control flow constructs will all take a zero status to mean true and a non-zero status to mean false Shell scripts may use the exit command to set this status value

Control Flow Constructs

Every programming language needs repetition and selection constructs The shells provide the usual set:
if while for

Together with an extensive set of test functions

Testing Conditions
The if command is built in to the shell(s) Usage: if condition1 then commands1 [elif condition2 then commands2] . [else commands3]

fi

The test Command

If the condition is a regular command then, as noted, a zero status indicates true, non-zero indicates false Shells also provide the test command Examples: test -d name is name a directory? test -f name is name a file? test s1 = s2 does string s1 = s2? test n1 -eq n2 does number n1 = n2? Can use ! to reverse a condition (not)

The while Command

For an indeterminate number of repetitions Usage: while condition do commands done The commands will be executed as long as the condition is true

The for Command

For iterating over a list Usage: for item [in list] do commands done Each pass through the loop assigns the next item in the list to $item If no list specified then $@ is assumed

Interactive Input
The read command allows input to be taken interactively Usage: read var1 [var2...] Reads one line of input Assigns each input word to successive variable Residue goes to last named variable

Files as links
Unix also allows creation of a link to a file A link is essentially a pseudonym Can view as a pointer to a file Two types of link
hard symbolic or soft

Hard links
A hard link is a directory entry which points to the disk space of another file It is like another name for a file There is only a single copy of the file on disk A file may have numerous links A link may only be to a file on the same file system Changing either name has no side effects

Symbolic Links
A symbolic link is a directory entry which contains the ASCII text string of the pathname of another file or directory It is like a pointer to the other file Changing the name of the pointed-at file will break the link

Patterns
Many Unix tools use the concept of a pattern matching Similar to, but not exactly the same as, the wildcard or glob patterns Examples:
editors grep sed awk

Pattern Basics
A pattern matching string is called a regular expression
. (dot, period) match any single character (except newline) * (star, asterisk) match any number (including zero) of precedingcharacter ^ (caret) match start of line $ (dollar) match end of line

More Regex Metacharacters

[ ] match any of enclosed characters [^ ] match anything but the enclosed + match any number (greater than zero) of preceding regex ? match zero or one instances of preceding regex \ turn off special meaning of next character

Example Pattern Matches

Some sample regular expressions and what they match:
abc matches the string abc ^abc abc at the beginning of a line abc$ abc at the end of a line ^abc$ abc as the entire line [Aa]bc abc or Abc a[aeiuo]c a lowercase vowel c a[^aeiou]c a not lowercase vowel c

Useful Commands
Among the many Unix commands a few stand out as being particularly essential These include
grep find sed tar compress

Searching Files with grep

The grep command is actually a family
grep egrep fgrep

All search files for strings which match specified patterns grep ^From: Deryk Mail/*

More on grep
greps behaviour is modifiable via command line options
-c display only count of matching lines -i ignore case when matching -l display only names of files with matching lines -n display line numbers -v display only non-matching lines

grep examples
grep -c t687v1 /etc/passwd
display count of users on this course

grep -l [Uu]nix ~/courses/*

display names of all files in my courses directory which refer to Unix (or unix) note quotes

who | grep t687v1

which members of this course are currently online

The find Command

The find command walks down a subtree of the file system looking for files which match certain creteria, such as:
wildcards file type (e.g. file, directory, special, link) files owner access time modification time

find ~ -name *.c -print

Using find

display names of all files in my hierarchy whose names end in .c note quotes

find /etc -user root -print

display names of all files under /etc owned by root

find /tmp -atime +7 -exec rm {} \;

find all files under /tmp which have not been accessed in more than seven days and delete them

find /tmp -atime +7 -ok rm {} \;

as above, but prompt for each file first

An Editor as filter
The sed (stream editor) acts as a filter Reads a line from stdin or specified file(s) Applies any commands which match Writes line to stdout Command set is drawn from the original ed editor

Using sed
sed [options] command [file(s)] Options:
-n only output lines as specified by printing commands -e cmd specify a command (only needed to specify more than one) -f file file contains editing commands

If no file specified then read from stdin

Commands in sed
Commands have the general form
[address][,address][!]command[arguments]

Addressing:
if no addresses command is applied to all input lines if one address apply to all matching lines if two addresses apply to every line beginning with first and ending with last address! apply to all non-matching lines

Example sed commands

s/abc/def/
substitute def for abc (on every line)

/^$/d
delete every empty line

/^[ ]*$/d
delete every line consisting of nothing but whitespace

/^begin/,/^end/p
print lines from begin to end

Using sed in a pipeline

sed is often most useful in a pipeline grep t687v1 /etc/passwd
displays all usernames beginning t687v1 also includes unassigned ids - which have none in the name field

grep t687v1 /etc/passwd | sed /none/d

removes all lines matching none from the output of grep

Archiving files with tar

Unlike other systems, Unix has one tool for archiving multiple files within one file tar and another for compressing the archive compress A compressed tar archive is a common format for distributing Unix software Look for:
name.tar.Z name.tar.gz compressed tar file gzipped tar file

Using tar
Format:
tar [options] files Functions:
c r t x create new archive append files to archive list contents of archive (or files if specified) extract files from archive

Examples of tar usage

tar cf test.tar ~/test
create new archive file, test.tar, comprising the entire test subtree

tar tf test.tar
list contents of test.tar

tar tf - test.tar | more

list contents to stdout and pipe to more

tar xvf test.tar

extract files (with verify) from archive

Compressing archives
Whereas pkzip compresses and archives, on Unix use a pipeline:
tar cvf - |compress >test.tar.Z tar cvf - |gzip >test.tar.gz

Likewise to access a compressed archive:

uncompress -c test.tar.Z | tar tf gunzip -c test.tar.gz | tar xvf -

Where do I go from here?

Windowing
X-Window system
Designed at MIT (Project Athena) Freely available Inter-platform

Internet
TCP/IP suite largely developed on Unix telnet, ftp, mail

World Wide Web

lynx, mosaic, netscape, hotjava Web servers by the dozen

How do I find out more?

Books
Too many to mention

Journals
Several magazines devoted to Unix

Usenet News
Many newsgroups devoted to Unix in (nearly) all its many flavours

World Wide Web

Many, many sites featuring:
Unix tutorials/references Freely-available software and documentation