Motivation for Separation Kernels

Inzemamul Haque
Department of Computer Science and Automation
Indian Institute of Science
Bangalore-560012
Email: inzemamul.haque@csa.iisc.ernet.in
AbstractThis article tries to figure out the motivation behind
the use of separation kernels as secure systems. In this article the
author has tried to draw a complete picture in the mind of the
reader that how the security issues in different types of systems
led to the development of separation kernels.
Keywordskernel, Multics, operating systems, security, security
kernel, separation kernel, UNIX.

I.

I NTRODUCTION

In this article we try to find out the loopholes in the

security of various operating systems which led to different
types of secure kernels viz. security kernels, separation kernels.
In section II, we shall try to define a secure operating system.
And then in later sections we shall go through different
operating systems in the course of the history of development
of operating systems and analyze their security. In section III,
IV, V, and VI, we shall discuss Multics, Unix, security kernels
and separation kernels respectively.
II.

S ECURE O PERATING S YSTEM

A secure operating system is an operating system which

implements the reference monitor concept.
A reference monitor concept enforces a mandatory protection system and giving the following guarantees:

Complete mediation: The system should ensure that

all security sensitive operations are mediated through
the access control enforcement mechanism.

Tamperproof: The access control enforcement mechanism cannot be modified by untrusted processes.

Verifiable: The access control enforcement mechanism

and the protection system should be small enough to
be subject to test and analysis.
III.

H ISTORY

MIT, General Electric and Bell Labs started the Multics

project[3] in 1963. This project was not successful because
of high ambitions of the project. Bell laboratories pulled out
in 1969. Most of the fundamental operating system concepts
including security were developed in the course of this project.
Finally Multics was bigger, slower and less reliable than
expected. Multics development was terminated in 1985, but
it was in use until 2000.
In mid-1970s, most of the researchers figured out that having two many features (like generality, performance, efficiency
and security) on the same system was difficult because of the

size of the system becoming too large. Hence the vendors and
the researchers started to choose one of the two directions (i) generality and performance oriented systems with limited
security and (ii) security oriented systems with reasonable
performance. The first one resulted in the development of
UNIX like systems and second one resulted in secure kernels
like security kernels and separation kernels.
As stated earlier Bell Labs left the Multics project in 1969.
Ken Thompson and Dennis Ritchie(researchers from Bell Labs
who also worked in Multics project) used their experience
from Multics project and developed the new operating system
UNIX, which was going to rule the world for almost next
four decades. UNIX was developed to play a game on a
PDP-7 computer. Hence it was very small in the beginning,
however it grew in size in later decades because of its use
in academics and industries. UNIX was simpler and smaller
than Multics. Hence it was easier to program and was better
in performance as compared to Multics. UNIX adopted many
of security features of Multics, but the security goals of UNIX
were different from Multics as it was a personal project. We
shall discuss the security problems and a few vulnerabilities
of UNIX in next section.
In 1970s research was going on to build a secure operating
system. And then the Anderson report [1] came up with the
reference monitor concept. Implementation of reference monitor concept came to be known as Security Kernels. The first
security kernel was developed by MITRE in 1974 which had
less than 20 subroutines in less than 1000 source lines of code.
Other systems which came out as a result of addressing the
security limitations of Multics were Secure Communications
Processor (Scomp) [4] from Honeywell, the Gemini Secure
Operating System (GSOS or GEMSOS) [8] from Gemini, the
Secure Ada Target (SAT) etc. We shall discuss the problems
with security kernels in section V.
Then in 1981 John Rushby came up with the idea of
a separation kernel [7] which tried to address some of the
problems in security kernels. Separation kernels are discussed
in section VI.
IV.

UNIX

Earlier we talked about history of UNIX and found out that

security was not one of the goals of UNIX. Hence there will
be some loopholes.
First of all UNIX is a discretionary access control system.
A system in which the process which has the ownership of
the object can change the permissions for other processes is
called a discretionary access control system. Alternatively it

process opens a file /tmp/X for which it has the access

permission. The operating system will first check the
access permission for the process to open the file by
using access system call. And then execute the open
system call but between the authorization and opening
of the file, the user process changes the file X to a
symbolic link for another file for which the access is
denied. In this way the attacker can access a file for the
permission is denied. These kind of attacks are called
time-of-check-to-time-of-use (TOCCTOU) attacks.

can be said that the protection state is at the discretion of the

users and any untrusted processes that they may execute. Here
if all users are non-malicious then we can ensure security but
nowadays users run a variety of processes some of which may
be supplied through the attackers (may be some links in the
e-mail).
UNIX authorizes operations on files by mediating file open
for read write and execute permissions. But this is not always
possible because some objects are not represented as files.
Another problem is that by even having the read access for a
file, a user process can perform a variety of operations on the
file besides reading by using fcntl or ioctl system call. UNIX
does not mediate security sensitive objects such as network
communications.
UNIX has a command named chroot which is used to limit
a process to a subtree of a file system. An attacker can create
/etc/passwd and /etc/shadow files in the subtree and can add an
entry for root. The attacker can login as this root and escape the
chroot environment by using the root access to kernel memory.
A. UNIX Security Analysis
As described in section II, a secure operating system should
meet the requirements of complete mediation, tamperproof
and verifiable. Now we shall analyse UNIX under these
requirements.

Complete mediation - UNIX fails to control the

access to information. As discussed above, UNIX
permits modification to file without having a write
permission for the file by using fcntl or ioctl. We
also saw that UNIX does not mediate access to some
of the system resources, e.g. in case of objects such
as network communications, UNIX does not provide
authorization at all.
Tamperproof - User-level processes can access and
modify kernel beyond system calls. It can install
kernel modules to special file systems (e.g., /proc or
sysfs). User process can have direct access to kernel
memory via the device file /dev/kmem.
Verifiable - The large size of UNIX makes the formal
verification very difficult. The extensible nature of
the UNIX system (by having new device drivers and
kernel modules) makes it more difficult.

So finally we can say that UNIX fails to meet the requirements of complete mediation, tamperproof and verifiability.
Hence UNIX is not a secure system.
B. UNIX vulnerabilities
In this part we try to show the problems which can be
faced if the system design does not focus on the security as
one of the main goals.

V.

S ECURITY K ERNELS

Security kernels are the systems which implement the

reference monitor concept. These systems do not provide as
many features as provided by ordinary operating systems like
UNIX, windows etc. These systems are very simple in terms
of functionality. By definition security kernels are secure as
the secure operating system is the one which implements the
reference monitor concept. While mediation and tamperproof
were fundamental to the design of the security kernels but
verifiability became the focus in building the security kernels.
The problem with the security kernels was the use of
trusted processes. The concept of trusted processes can be
explained by the example presented below.
Suppose there is a printer which has to print the jobs of
different class of users. Assuming that we are using the BellLaPadula policy model [2], the printer should have the highest
classification because it has to print the jobs of all class of
users. But there are two problems

users cannot check the performance of their job as

according to Bell-LaPadula model, lower class of user
cannot read above its class.

when the printer finishes some job, it has to delete the

file from the spooler, but according to Bell-LaPadula
model a user with higher classification cannot write
to the objects of a user who is below him.

To solve these problems, the concept of a trusted process

was introduced. A trusted process is the one which can violate
the policy and trusted by the system.
After the inclusion of the trusted processes in the system
the trusted computing base becomes kernel with trusted processes. So it should be ensured that privileges granted to trusted
processes should not be abused by the processes. Therefore
now both the security kernel and trusted processes need to be
verified. But the problem is that there does not exist a formal
model for such a combination of kernel and trusted processes.
And also there are a lot of trusted processes which makes
it more difficult to come up with such model. As Landwehr
mentions

Rootkits - Rootkits are the softwares which hide the

existence of malicious programs in the system. It is
very difficult to find the rootkit in the system.

. . . in the final version of their model, Bell and

LaPadula did include trusted processes. What is
not included in their exposition is a technique for
establishing when a process may be trusted.[6]

Time-of-Check-to-Time-of-Use (TOCTTOU) - This

is a big vulnerability in UNIX. Let us consider the
following example to understand this attack. A user

In a single word the main problem was verification

of system in presence of trusted processes. To solve the
problem of trusted processes, Rushby came up with the idea

of separation kernel [7]. The next section describes the details

of a separation kernel.
VI.

S EPARATION K ERNELS

Separation kernel is a type of security kernel which partitions the system into different partitions which are isolated
with each other. There can be some explicit communication
channels as given by the policy but no implicit communication
channels. Different processes can run in different partitions.
Rushby solved the problem of printer spooler by taking the
printer spooler and the file system in different partitions with
some explicit communication channels. Here the security is
maintained by the isolation provided by the separation kernel.
ACKNOWLEDGMENT
Most of the material covered in this article, especially the
section III is taken from the book Operating System Security
[5] by Trent Jaeger. The issues in security kernel because of
trusted processes described in section V are mostly from the
paper by Rushby [7].
The author would like to thank Prof. Deepak DSouza,
Indian Institute of Science, Bangalore and Arnab Kundu,
CAIR, DRDO for their valuable discussions on this topic.
R EFERENCES
[1]

J. P. Anderson, Computer Security Technology Planning Study, Technical Report ESD-TR-73-51, The MITRE Corporation, Air Force Electronics Systems Division, Hanscom AFB, Badford, MA, 1972.
[2] D. E. Bell and L. J. LaPadula, Secure COmputer Systems: Unified
Exposition and Mul- tics Interpretation,
Technical Report ESD-TR75-306, MITRE corporation, Bedford, MA, March 1976.
[3] F. J. Corbato and V. A. Vyssotsky, Introduction and Overview of Multics
System,
In Proceedings of the 1965 AFIPS Fall Joint Computer
Conference, 1965.
[4] L. J. Fraim, SCOMP: A Solution to the Multilevel Security Problem,
IEEE Computer, 16(7):26-34, 1983.
[5] T. Jaeger, Operating System Security, Morgan & Claypool Publishers,
2008.
[6] C. E. Landwehr, Assertions for Verification of Multilevel Secure Military
Message Systems, ACM Software Engineering Notes. 5(3):46-47, July
1980.
[7] J. Rushby Design and Verification of Secure Systems, In Proceedings of
the Eighth ACM Symposium on Operating System Principles, pp. 12-21,
Dec 1981.
[8] W. R. Shockley, T. F. Tao, and M. F. Thompson, An Overview of
the GEMSOS class A1 Technology and Application Experience,
In
Proceedings of 11th National Computer Security Conference, pp.238245, October 1988.