Discussion — Red Hat Academy 2.0 https://osu.redhat.com/content/courses/rha230-50-trial1/tag_netserv/tag_l...

Discussion

Linux developers look to the Unix operating system as inspiration for much of Linux's design. As a result, most
Linux network server applications are implemented as daemons. This lesson focuses on the traditional Unix
techniques used to manage network daemons. While these techniques are not often used in day to day
management of Red Hat Enterprise Linux, they provide a useful model which promotes understanding and is
invaluable for troubleshooting.

The lesson begins with a quick review of TCP/IP networking, and then follows along as an administrator installs,
starts, reconfigures, and stops the vsftpd network application.

Course Introduction

Originally, computers were used to crunch numbers. Typical computer users in the 1970's probably wrote their
own programs, submitted their work when there was an opening, and came back for the results. Today,
computers are more often used to produce, edit, organize, and otherwise manipulate words, images, and sounds.
Typical computer users today open a web browser on a home computer, enter a URL or click on a few links, and
have a wealth of information at their fingertips. Time of day is seldom considered; The following story could take
place at 2:00 in the morning just as easily as at 2:00 in the afternoon.

Susan is planning a trip to New York. She sits down at her computer, opens a browser, and searches for hotels in
Manhattan. She looks up a a map of the neighborhood she will be visiting, and prints it out. She searches for
guidebooks about New York restaurants, and purchases one online. Finally, she types up her itinerary, and emails
it to a friend. She then exits the browser, shuts down the computer, and leaves the room.

This course isn't about Susan's computer - or at least not the computer you're thinking of. This course is about the
dozen other computers she used - the ones not directly mentioned in the story. This course is about the
computers that Susan's computer talked to. There are the obvious ones, such as the server which performed her
web searches (a Web server). There are also the more subtle ones - the one that assigned her computer an IP
address as it booted (a DHCP server), the one which translated all of her domain names into IP addresses (a DNS
server), and the one which relayed her email out to its destination (a SMTP server).

These unseen computers probably don't have a mouse, a keyboard, or a monitor - at least not exactly. They
probably live in various closets spread around the world. They definitely don't get shut off when someone leaves
the room. And in the example given above, chances are as good as not that the various web, DHCP, DNS, and
SMTP servers are running Linux. We don't know what type of machine Susan was sitting at, but if she performed
her search on Google.com, or bought her book from Amazon.com, she was using Linux.

This course is about the installation, basic configuration, and management of network applications - the ones
mentioned above, and more. Upon completion, the student should be able to maintain these network services,
whether they exist on a desktop or on an unseen server in a closet.

Review of TCP/IP Networking Basics

The IP protocol

As its name implies, the Internet Protocol (IP) has been adopted as the standard protocol for communication
between machines. Two machines using the IP protocol must each have an identity known as an IP address, which
is usually represented as a series of four integers between 0 and 255, such as "192.168.0.4". This familiar
representation of an IP address is often informally referred to as a dotted quad.

IP addresses can be compared to telephone numbers. Packets of data labeled with a computer's IP address can be
routed to that computer, just as a telephone call can be routed to the right telephone. How this routing occurs is a
wonderful topic, but beyond the scope of this course. For our purposes, just know that the IP protocol, with its

1 of 9 22-Sep-10 7:13 PM
Discussion — Red Hat Academy 2.0 https://osu.redhat.com/content/courses/rha230-50-trial1/tag_netserv/tag_l...

associated IP addresses, is responsible for routing data from one computer to another over the Internet.

Because people can remember names more easily than numbers, IP addresses can be assigned a name, such as
<hostname>www.redhat.com</hostname> or <hostname>www.kernel.org</hostname>. Hostnames can be
converted into an associated IP address using techniques that will be covered later in detail. For now, just assume
that knowing a computer's hostname is essentially the same as knowing its IP address.

The TCP and UDP Protocols

IP addresses and the IP protocol are used to route packets of information from one computer to another, but
that's not the end of the story. Usually, that information needs to be routed to and from particular applications on
the various computers. One of two similar yet distinct protocols are often used, either the TCP or UDP protocol.
Both of these protocols are based on the concepts of ports and sockets.

In order to communicate using the network, applications must ask the Linux kernel to open a socket. The
application can then send information across the network by writing to the socket, and receive information by
reading from the socket. Because a machine may have many different applications carrying on simultaneous
conversations, each socket is assigned a port number, which can range from 1 to 65535 (which mathematicians or
computer geeks might recognize as 2^16, almost).

In the following diagram, applications running on running on two machines,

<hostname>server.isp.net</hostname> with an IP address of 123.45.67.89, and
<hostname>client.example.com</hostname> with an IP address of 192.168.0.1, are having conversations. The
diagram illustrates processes running on each of the machines, and the port numbers which they were assigned.
For example, the httpd process on <hostname>server.isp.net</hostname> has been assigned port number 80,
while the firefox process on <hostname>client.example.com</hostname> has been assigned the port number
44985.

Figure 1. TCP/IP Ports and Sockets

The TCP and UDP protocols are not alternatives to the IP protocol, but are used in addition to the IP protocol.
While the IP protocol uses IP addresses to route data to a particular machine, the TCP and UDP protocols in turn
use ports to route data to a particular process on that machine.

The combination of the IP address and port number of the processes on each of the two machines is referred to as
a socket pair, and is enough information to uniquely identify the various network conversations.

Most services we will encounter use the TCP/IP protocol, though a few use the UDP/IP protocol. For now, just
know that each protocol has a distinct set of ports, so that TCP port 80 is not the same thing as UDP port 80. As

2 of 9 22-Sep-10 7:13 PM
Discussion — Red Hat Academy 2.0 https://osu.redhat.com/content/courses/rha230-50-trial1/tag_netserv/tag_l...

we continue, we will often speak as if all IP applications use the TCP/IP protocol, because the majority do.

Clients and Servers

Most networking protocols are designed around a "Client/Server" architecture. To some extent, the features of
networking clients and servers can be compared to human clients buying a candy bar from human servers at the
local store.

Servers have well known addresses. Just as a customer knows where to find a candy bar server (at
the store around the corner, behind the counter), networking clients know where to find networking
servers (at the host <hostname>www.yahoo.com</hostname>, listening to port 80). When requesting a
new networking socket, servers generally ask for a specific port (such as 80). In contrast, clients are
usually randomly assigned a port number.

Servers are highly available. Because servers don't know exactly when clients will come along, they
need to be always ready. Just as the candy bar server needs to always stand behind the counter to be ready
for new customers, a network server needs to be always running and listening for new connections. Often,
network servers run as daemons, which are processes which disassociate themselves from the terminal
which started them. Therefore, unlike most commands, daemons continue to run even after the session
from which they were started has been closed.

Clients initiate transactions. A client initiates the buying of a candy bar by walking up to the counter.
Generally, networking clients initiate transactions by requesting new connections with a networking
server.

More about Ports

As was mentioned above, servers and clients generally handle ports differently; servers request a specific port on
startup, while clients are randomly assigned one. Which port does a server request?

Well Known Ports and the /etc/services File

Specific services have been assigned port numbers, so that clients know where to find them. These ports are
cataloged in the file /etc/services, which lists hundreds of well known ports. The following table lists a few of
the well known ports that we will be encountering in this course.

Table 1. Well Known Ports

Service Protocol Port

ftp TCP 21
ssh TCP 22
smtp TCP 25
domain (DNS) UDP 53
http TCP 80
pop3 TCP 110
imap TCP 143
https TCP 443
microsoft-ds TCP 445

Because ports for specific services are well known, when requesting new connections, ftp clients know to request a
connection to port 21, while web browsers know to request port 80.

Privileged Ports (1-1023)

3 of 9 22-Sep-10 7:13 PM
Discussion — Red Hat Academy 2.0 https://osu.redhat.com/content/courses/rha230-50-trial1/tag_netserv/tag_l...

The ports numbered 1 - 1023 are termed privileged ports, and only processes running as root can bind to these
ports. On machines which might have many users, this feature helps ensure that the right service is bound to the
right port. For example, the user <username>elvis</username> could not start his own version of a webserver,
only the user <username>root</username> could.

When clients are assigned a port number, the assigned port number comes from well above the privileged port
range.

The netstat Command

The netstat command can be used to observe which ports are open on a machine. Unfortunately, without
arguments, the netstat command displays probably its least helpful information (something called Unix domain
sockets, which aren't involved with networking and thus not relevant to this course). In order to make netstat
useful for our purposes, obscure command line switches need to be added. One combination which has the
combined advantages of providing reasonable output and being easy (for English speakers) to remember is
netstat -tuna.

[student@station ~]$ netstat -tuna

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 :::22 :::* LISTEN
udp 0 0 0.0.0.0:631 0.0.0.0:*

The output shows all open ports on the system, including their protocol, local address, and "Foreign" address.
Additionally, for TCP ports, the TCP state of the socket is shown.

When being run by the root user, an additional -p command line switch will request that the name and process ID
("pid") of the process owning the port be displayed as well. Unfortunately, this is considered privileged
information, and is not available to standard users.

[student@station ~]$ su
Password:
[root@station ~]# netstat -tunap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 3681/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 3793/sendmail: acce
tcp 0 0 :::22 :::* LISTEN 3763/sshd
udp 0 0 0.0.0.0:631 0.0.0.0:* 3681/cupsd

The Local Address

The "Local Address" is the combination of IP address and port number to which the processes are bound. Many
machines can support more than one IP address, possibly because they have more than one Ethernet card. In
fact, most machines support at least two: one "real" address, which is assigned to their Ethernet card, and a
"loopback" address (usually 127.0.0.1), which is a virtual address that just loops back to the same machine.

When binding to a particular port, networking servers generally choose one of three strategies when specifying
which particular IP address they want to bind to.

Bind to the loopback address only. In the output above, the cupsd (TCP) and sendmail daemons
have taken this approach. While using the TCP/IP protocol, these daemons will only receive connections
from processes running on the same machine as they are.

Bind to explicitly stated addresses. If a machine has more than one IP address, a daemon could
explicitly state which IP addresses to which it should bind. None of the processes above have taken this
approach. [1]

4 of 9 22-Sep-10 7:13 PM
Discussion — Red Hat Academy 2.0 https://osu.redhat.com/content/courses/rha230-50-trial1/tag_netserv/tag_l...

Binding to any address. When binding to a port, a process can specify that it should bind to all
addresses. This is generally represented by an IP address of 0.0.0.0 or just "::". [2] In the above output,
sshd and the UDP cupsd port have taken this approach.

The Foreign Address

When a client has connected to a service, the IP address and port number of the client is displayed. In the output
above, no clients have yet connected, so the Foreign Address is not very interesting. That is about to change.

State

For the TCP protocol, TCP connections are said to be in certain states. Although there are officially 11 states, for us
only two of them are important: LISTEN and ESTABLISHED. A socket in the LISTENing state is a server which is
willing to accept new clients. A socket in the ESTABLISHED state is actively bound to a particular client. All of the
TCP sockets above are in the LISTENing state.

Observing a typical Network Service: sshd

The Secure Shell Service

In order to illustrate many of the concepts mentioned above, we turn to examine the Secure Shell (ssh) service
which should be running on your machine. By default, most Red Hat Enterprise Linux installations run the
"Secure Shell" service, which allows clients (using the ssh command) to open shells on ("log in to") remote
machines running the sshd server.

Network servers are processes, just like any other process on the machine, so simply using the ps command is
usually enough to confirm that the sshd daemon is running.

[student@station ~]$ ps aux | grep sshd

root 3763 0.0 0.3 3984 1548 ? Ss 06:38 0:00 /usr/sbin/sshd
student 5238 0.0 0.1 4512 656 pts/4 R+ 06:44 0:00 grep sshd

Sure enough, /usr/sbin/sshd is running.

In order to confirm that sshd has opened a network socket, we first determine the well known port for the ssh
protocol by grepping it out of the /etc/services file.

[student@station ~]$ grep ssh /etc/services

ssh 22/tcp # SSH Remote Login Protocol
ssh 22/udp # SSH Remote Login Protocol
x11-ssh-offset 6010/tcp # SSH X11 forwarding offset

Ignoring the x11-ssh-offset, we find that "plain ole ssh" has a well known port of 22. Using the netstat -tuna
command, we confirm that port 22 is open.

[student@station ~]$ netstat -tuna | grep :22

tcp 0 0 :::22 :::* LISTEN

If fortunate enough to be the root user, we can include -p to confirm that sshd is the process which has opened it.

[root@localhost ~]# netstat -tunap | grep :22

tcp 0 0 :::22 :::* LISTEN 3763/sshd

Handling Clients: Forking Servers

5 of 9 22-Sep-10 7:13 PM
Discussion — Red Hat Academy 2.0 https://osu.redhat.com/content/courses/rha230-50-trial1/tag_netserv/tag_l...

So far, we have only witnessed servers in the LISTENing state. Clients have yet to enter the picture. Focusing on
just the sshd service, the following diagram shows the sshd process, with a pid of 3763, bound to port 22,
patiently LISTENing for clients.

Figure 1. A Listening sshd Server

What happens when a client comes along? Let's assume that the user elvis on the host
<hostname>station10.example.com</hostname> successfully uses ssh to connect to the machine
<hostname>station.example.com</hostname>. How does the sshd daemon handle the connection?

[elvis@staiton10 ~]$ ssh student@station.example.com

student@station.example.com's password:
[student@station ~]$

Most Unix networking daemons implement a "forking server" model. When a client establishes a connection, the
sshd daemon "forks", or creates a new process which is a (nearly) exact duplicate of itself. The new child process
is dedicated to that one particular client, while the parent process drops the client and returns to just listening to
new clients, as illustrated below.

Figure 2. A Forked sshd Child Handling a Client

6 of 9 22-Sep-10 7:13 PM
Discussion — Red Hat Academy 2.0 https://osu.redhat.com/content/courses/rha230-50-trial1/tag_netserv/tag_l...

The details of this figure can be confirmed by running the netstat command (as root). Notice the client address
now reflects the IP address and port number of the remote ssh client (192.168.0.10, and 48515). (Complicating
matters, netstat is displaying the IPV4 address in IPV6 format. For our purposes, the leading ::ffff: can be
ignored.)

[root@localhost ~]# netstat -tunap | grep :22

tcp 0 0 :::22 :::* LISTEN 3763/sshd
tcp 0 0 ::ffff:192.168.0.1:22 ::ffff:192.168.0.10:48515 ESTABLISHED 6879/sshd: student

The fact that the the sshd daemon forked a new child can be confirmed with the ps command.

[student@station ~]$ ps aux | grep sshd

root 3763 0.0 0.3 3984 1552 ? Ss 06:38 0:00 /usr/sbin/sshd
root 6879 0.0 0.4 6816 2144 ? Ss 08:07 0:00 sshd: student [priv]
student 6883 0.0 0.4 6816 2208 ? S 08:07 0:00 sshd: student@pts/5
student 6984 0.0 0.1 3800 648 pts/3 R+ 08:11 0:00 grep sshd

In fact, the sshd daemon takes the non-standard approach of forking two two child processes, one which drops
root privileges, but we're only going to focus on the one which manages the client connection (pid 6879).

What would the situation look like if a ssh client now connected from the local machine using the 127.0.0.1
loopback address?

Figure 3. Connections From the Same Machine

7 of 9 22-Sep-10 7:13 PM
Discussion — Red Hat Academy 2.0 https://osu.redhat.com/content/courses/rha230-50-trial1/tag_netserv/tag_l...

Now the netstat command reflects four connections related to port 22.

[root@localhost ~]# netstat -tunap | grep :22

tcp 0 0 127.0.0.1:48820 127.0.0.1:22 ESTABLISHED 7831/ssh
tcp 0 0 :::22 :::* LISTEN 3763/sshd
tcp 0 0 ::ffff:192.168.0.1:22 ::ffff:192.168.0.1:48515 ESTABLISHED 6879/sshd: student
tcp 0 0 ::ffff:127.0.0.1:22 ::ffff:127.0.0.1:48820 ESTABLISHED 7832/sshd: student

The ssh client itself (pid 7831). Notice what's considered the "Local Address" and "Foreign Address" is
inverted for the client.

The original listening daemon (pid 3763).

The first forked child, handling the remote ssh client (pid 6879).

A newly forked child, handling the local ssh client (pid 7832)

Again, the ps command should display a sshd daemon for each client (pid 6879 and 7832), and a listening sshd
daemon (pid 3763). (And, as mentioned above, a sshd daemon for each connection which is not directly related to
networking). As a bonus, we get the local ssh client as well.

[root@localhost ~]# ps aux | grep ssh

root 3763 0.0 0.3 3984 1552 ? Ss 05:16 0:00 /usr/sbin/sshd
root 6879 0.0 0.4 6816 2144 ? Ss 06:11 0:00 sshd: student [priv]
student 6883 0.0 0.4 6816 2208 ? S 06:11 0:00 sshd: student@pts/6
student 7831 0.0 0.3 4752 1828 pts/4 S+ 06:11 0:00 ssh student@127.0.0.1
root 7832 0.0 0.4 6816 2144 ? Ss 06:11 0:00 sshd: student [priv]
student 7836 0.0 0.4 6816 2208 ? S 06:11 0:00 sshd: student@pts/7
root 6005 0.0 0.1 5320 656 pts/7 R+ 06:19 0:00 grep ssh

When the ssh clients close their respective connections, the forked sshd daemons which are handling their
connections die, and we would be left once again with our original sshd listening daemon, patiently waiting from
new clients to come along.

8 of 9 22-Sep-10 7:13 PM
Discussion — Red Hat Academy 2.0 https://osu.redhat.com/content/courses/rha230-50-trial1/tag_netserv/tag_l...

Summary

This lesson was meant to serve as a refresher for basic TCP/IP concepts, and their implementation on the Linux
operating system. We covered a lot of ground, so here's a quick summary of the main points.

1. The IP protocol, with it's associated IP addresses, serves to route data from one computer to another.

2. The TCP and UDP protocols, with their associated port numbers, serve to route data received via the IP
protocol to a particular process on a machine.

3. Networking applications are processes like any other process on the machine, and can be monitored with
the ps command.

4. The netstat -tuna command can be used to view a list of open ports on a given machine.

5. Most Linux networking servers use a "forking" model, where the original server process merely LISTENs
for new connections, while a new child is forked off to manage each client's ESTABLISHED connection.

9 of 9 22-Sep-10 7:13 PM