Thin Client Architecture in Support of
Remote Radiology Learning
Florian F. Schmitzberger Justus Roos
Stanford University, Stanford University,
Department of Radiology Department of Radiology
Lucas MRS Imaging Center Stanford University Medical Center
Stanford, California 94305 Stanford, California 94305
001 650 248 3467 001 650 723 7647
schmitzberger@stanford.edu justus.roos@stanford.edu
Geoffrey D. Rubin
Stanford University,
Department of Radiology
Grant Building, S-072
Stanford, California 94305
001 650 723 7647
grubin@stanford.edu
ABSTRACT
We implemented a system for remote radiology learning which
provides immediate feedback to the learner. Using a thin remote
client, expert readers are asked to answer questions about
specified radiological findings. These scans are presented as real-
time 2D and 3D presentations which allow the user to freely
manipulate them using a thin Java client with all 3D rendering
performed on the server side. Answers are stored on the server
and are used to provide feedback to learners who are presented
with the same questions, using the remote client. Learners can
practice on real datasets while receiving immediate feedback on
their diagnosis and measurements. Novel concepts introduced are
(1) the use of server-side rendering in radiology learning, (2)
providing immediate and specific feedback to trainees, (3) the
ability to provide useful feedback when a definitive gold standard
does not exist and (4) a thin, highly compatible client that runs on
common, existing hardware which allows to have more people
participating in very complex radiological evaluations, even if
there are not at the same site.
Categories and Subject Descriptors
D.3.3 [Programming Languages]: Language Constructs and
Features – abstract data types, polymorphism, control structures.
General Terms
Experimentation, Human Factors
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
SAC’09, March 8-12, 2009, Honolulu, Hawaii, U.S.A.
Copyright 2009 ACM 978-1-60558-166-8/09/03…$5.00.
842
Sandy Napel
Stanford University,
Department of Radiology
James H. Clark Center, S323
Stanford, California 94305
001 650 725 8027
snapel@stanford.edu
David Paik
Stanford University,
Department of Radiology
Lucas MRS Imaging Center
Stanford, California
001 650 736 4183
david.paik@stanford.edu
Keywords
Radiology, Learning, Computed Tomography, Teleradiology,
Healthcare Applications
1. PROBLEM DOMAIN
Radiology relies on the interpretation of recorded images to infer
anatomy and pathology. In many cases however there is no
immediate feedback on the correctness of the observations which
is seen as a major problem for medical learning. Both the
availability as well as the timing of feedback is crucial for
learning in radiology [1]. Radiologists learn in the classroom as
well as in the hospital setting. During classroom learning they use
books (with images and explanations) and stacks of images
viewed in cine mode to study anatomy and its pathology. These
methods of presenting material however do not fully display all
the characteristics of the available data: radiological images such
as Computed Tomography (CT) or Magnetic Resonance (MR)
scans are three dimensional blocks of data that are visualized as
slices from different angles, with different viewing settings, as
well as 3D renderings (visualizations) with a plethora of settings.
In the past, 2D slices have been sufficient for clinical
interpretation, but today, for certain imaging modalities, 3D
visualization is becoming a crucial element for the interpretative
process [2]. Much of the diagnostic power of the radiologist stems
from the ability to select the correct settings for visualizing the
available data to come to the correct conclusions. Figures 1 and 2
show the Stanford Remote Radiology Learning Platform
displaying two common views of a CT image. While typically
four views are displayed for any dataset (axial, coronal and
sagittal slices as well as a 3D rendering) any number of views can
simultaneously be shown.
During practical training, radiologists utilize clinical PACS
(Picture Archiving and Communication System) workstations,
both in clinical practice as well as during learning in small groups.
This adds many new degrees of freedom since the visualizations difficult cases to which no clear gold standard exists. The user can
can now be manipulated at will. Feedback is provided by teachers evaluate the validity of his answers compared to others,
when discussing cases but this feedback is limited by the particularly a large panel of expert readers whose opinions and
availability of teachers. expertise would not otherwise be available to the trainee. A
histogram of responses is automatically generated and presented
Another fundamental problem in radiology is agreement on
with relevant statistics (mean, mode, standard deviation) included.
diagnosis and in many cases there is no clear gold standard or a This is particularly important for questions where there is
“correct answer” [3]. This leads to obvious problems for learners significant disagreement between expert readers themselves.
in the field, as there is a lack of certain confirmation or negation Appropriate statistics are included on disagreement between the
for their interpretations. expert readers. See figure 4 for example feedback given to a
question with strong disagreement between experts themselves.
2. AUTOMATING FEEDBACK
We have created the Stanford Remote Radiology Learning
Platform (SRRLP) to address all the points mentioned above. In
short, clients connect to the SRRLP server and are presented with
a series of radiological images such as CT scans they can
visualize as they would on an actual PACS or clinical
visualization station. Attached to each image is an easily
configurable questionnaire that can ask questions in any format
(multiple choice, free response, etc.) as well as record length
measurements inside the images and mark any locations in the
images. This allows any question to be asked about the scan such
as the identification of anatomical landmarks or assigning a
confidence threshold to the diagnosis of a pathological finding.
Figures 1 and 2 show views associated with a sample session and
figure 3 presents an example of the accompanying questionnaire
window.

Figure 2: Axial view of a lung CT scan with a length

measurement (blue) and a number of marked features (red,
green)

Figure 1: 3D Rendering of a lung CT scan with a marked
nodule
Once the learner has entered his or her answers they are saved and
can automatically be compared to both all answers previously
recorded from other learners as well as responses solicited from
experts in the field. Statistics of comparison with any or all groups
are presented to the student as well as any known solutions which
can be displayed directly in the images. The user can still
manipulate the images at will, change rendering parameters and
explore the scans with the solutions clearly marked. Additionally
he/she is presented with all known answers to the questionnaire
questions posed. These can both be binary (true/false), used for
simple questions such as identifying clear anatomy as well as
continuous (range of answers given by peers and experts) for
3. IMPLEMENTATION DETAILS
Our solution utilizes existing networking infrastructure and
locally available computers in order to allow multiple sessions to
run simultaneously. A central server streams, in real time, 2D and
3D images (using the Fovia Inc renderer) to one or more viewers
who utilize our Java-based client-software to receive these images
and send their responses to the trial questionnaire. See figure 5 for
a schematic of this process. Distribution of the client program is
simple as it can be deployed in many ways including Java Web
Start, the download of a single Java Archive (JAR) file or in the
future even as an applet on a website. Emphasis was put on
making the client easy to install and use, essentially providing an
interface close to clinical visualization workstations the user
would be used to.
The server has a twofold purpose, it generates both 2D slices and
3D renderings that are being sent to the clients to be displayed and
it receives response data from the clients. Also, it prevents having
to ship large datasets to the client, which may contain embedded
private health information which would be forbidden by HIPAA.
Volume rendering and network transmission of the rendered
images is done using the Fovia Inc. [4] rendering engine which
allows for an interactive display of radiological images to the user
over a network.

843

Figure 3: Questionnaire window inquiring about the currently
displayed image
Figure 4: Network Structure for Remote Radiology
Learning
Our solution implements a number of concurrently running
server-processes. Referring to figures 1 and 2, these renderings
are generated remotely on the server and are sent in real time to
the user who can manipulate them individually. While presets are
initially loaded, the user can be granted full control to set
rendering parameters. The Fovia, Inc. High Definition Volume
Rendering® engine allows the user to access a great array of
visualization tools, functionality a radiologist would use when
evaluating cases on a clinical PACS workstation. When designing
the session, more or less control can easily be given to the user.
The client program provides the session questionnaire as well as
all control structures for the user and a server-process is
844
responsible for receiving all data generated by the user in addition
to responding with the appropriate statistics on user performance
in relation to others. When designing study sessions, different
types of commonly encountered questions can be implemented.
Shown in figure 3 are multiple choice answers, length selection as
well as free response where features can be marked on images in
both 2D and 3D.
Figure 5: Feedback comparing a learner's performance (red)
to previously recorded expert answers (blue).
Employing this server/client architecture adds a great number of
benefits. With a non-client/server architecture setup the obvious
limitations were in computer hardware as only one user at a time
in one physical location could be participating in a session, for
each dedicated computer. In order to allow for good performance
in calculating renderings these computers need to be fast but do
run on consumer-level hardware. Our current local setup uses an
Apple MacPro with 2 x 2.8 GHz Quad-Core Intel Xeon
processors with 4GB DDR2 RAM. All rendering is done using the
CPU so no specialized graphics hardware is necessary. These
hardware requirements strictly limit the number of people that are
able to simultaneously take advantage of such a system when used
locally. When utilizing a distributed approach with multiple users
per server these limitations become much more manageable and
are now represented primarily by available processing power and
memory in the server(s) (which does all the rendering for all
users) and by the available networking infrastructure, rather than
the total number of powerful workstations available at any time.
Other limitations are network latency which results in a minimal
delay when utilizing the system from distant locations and the
variety of physical displays (monitors) used by the clients.
The issue of processing power becomes important when a large
group of clients connect simultaneously to the system. While the
Fovia rendering engine can support multiple clients comfortably
(depending on the size of the images as well as the intensity of the
tasks), in many scenarios it is only necessary to display
significantly smaller subvolumes to the users anyway. Rendering
these smaller subvolumes is computationally far less expensive
and allows for many more users simultaneously. Additionally,
memory-sharing is utilized so that datasets do not need to be
loaded up repeatedly when viewed by more users. It is important
to note that this does still allow the user to manipulate the
renderings and settings individually. In addition to this, preferred
frame-rates can be selected either by the user or set in advance to
improve the user-experience. Load balancing can be conducted to
limit frame-rates and increase the numbers of users able to use the
system effectively at any time.
Networking infrastructure is typically only a small limitation due
to commonly available bandwidths. User tests have shown that the
rendering engine can run with acceptable results on a network
connection with speeds greater than 4 MBit/s with a minimum of
768kbps for usability reasons. However it has to be assured that
the server has a large enough bandwidth to handle all network
traffic. This is typically not a problem for common high speed
Internet connections (>100MBit/s). As with any web-service,
usage should be temporally distributed as much as possible but
peak-times have to be taken into account. The trial-response
component on the server requires only a negligible amount of
bandwidth, utilizing the TCP/IP protocol.
Network latency does not significantly affect the user experience
if within reasonable limits. Current studies on remote
visualization in human-computer-interaction (HCI) have shown
that “in order to retain a sense of presence, the graphics frame rate
should be at least 10 frames per second and the latency should be
less than 0.1 seconds.” [5] A series of tests conducted are shown
in table 1, demonstrating that the server can comfortably support
multiple connections unless dealing with constant manipulation of
high resolution 3D renderings. Simple manipulation of slice-views
(the most common visualization) is computationally relatively
cheap and even with a full-sized dataset and the setup outlined
above, 5-10 users can be accommodated without compromising
speed and interaction. It should be noted that these measurements
deal with constant movement which is not typical usage and
therefore significantly less CPU usage would be expected for a
typical user.
Table 1: Frame-rates and CPU usage of the SRRLP
Full dataset (512x512x304 pixel, 153MB)
View Resolution Frame Per Second CPU usage
3D Rendering 656x496 15.2 ~90%
3D Rendering 1312x1008 8.8 ~90%
Axial Slice View 656x496 22.6 ~10%
Axial Slice View 1312x1008 40.3 ~20%
Subvolume (50x50x21 pixel, 212 kB)
3D Rendering 656x496 15 ~10%
3D Rendering 1312x1008 22.1 ~30%
Axial Slice View 656x496 39 ~5%
Axial Slice View 1312x1008 19.9 ~10%
845
The implementations of both the server programs as well as the
client program were done using Java 6. Transmission of
questionnaire response was implemented using Java's built-in
object serialization [6]. Graph visualization for statistics was done
using JFreeChart [7]. Cropping images was done using a program
written utilizing the DCMTK libraries for C++ [8] as well as the
DICOM.pm library for Perl [9]. Distribution of the client is simple
and is done using Java Web Start [10] or alternatively a single
small JAR (Java Archive) file can be delivered. Utilizing Java
allows this program to be fully cross-platform and any operating
system for which a Java Runtime Environment exists can run the
client.
4. TRIAL APPLICATION
While the application is fully functional when given only one set
of solutions against which learners are measured, its full potential
is only explored when larger numbers of experts have evaluated
the cases presented.
Creating sessions requires two components: image files of the
radiological scans as well as meta-information about the images.
Image information is stored in the standardized DICOM data
format [11] that supports all common imaging modalities such as
Computed Tomography (CT), Magnetic Resonance (MR) or
Positron Electron Tomography (PET), etc. Meta-information is
included in a number of XML files specific to the session. Four
pieces of information are necessary:
1) General session information with descriptions, list of images,
etc.
2) Rendering parameters for the standard views of the images.
3) Details on the questions to be asked such as question text, type
and options. Questions can be open ended such as “mark all lung
nodules identified in the scan” or definitive such as “locate the
superior vena cava”.
4) Answer information: Answers to questions are stored in a
predefined XML format so the process of gathering expert
responses is simplified since both experts and learners utilize the
same program with equal setups.
Since the correct or expert answers can be specified by utilizing
the program in a similar way to the learners, the creation of
learning sessions can therefore be done without modifying any
source-code but by the simple creation or modification of these
XML files. A future project will be a user-interface for the
creation of the session files.
The quality of a session heavily depends on the availability of
expert answers, particularly for questions to which no gold
standard or known correct answers exist. We are currently
implementing a trial dealing with lung nodule identification and
classification and will be soliciting responses from members of
the Fleischner Society for Thoracic Imaging and Diagnosis, using
our developed program. The responses from these experts (the
society publishes widely recognized guidelines for the detection
of lung nodules) are of great value to future learners as they
present the joint expertise of a number of experts in the field on
actual cases rather than abstract guidelines.
5. CONCLUSIONS AND DISCUSSION
The concepts of electronic learning in radiology have been
discussed since the advent of personal computing. As early as
1990 researchers in radiology have understood the importance of
giving feedback throughout the learning process. These first
projects used teaching scripts and static radiological images to
present learning points [12]. The focus since then has been on
devising and implementing project management concepts that
combined all information about a radiological image and provides
limited feedback to student performance [13-15]. The use of
visualization in healthcare training has been demonstrated before
as well [16].
The Stanford Remote Radiology Learning Platform implements
four novel concepts in this area:
1) the use of server-side rendering in radiology learning
2) providing immediate feedback to trainees
3) the ability to handle instances where a unanimous gold
standard may not exist
4) a thin, highly compatible client that runs on common,
existing hardware which allows to have much more
people participating in very complex radiological
evaluations, even if there are not at the same site.
The paradigm of combining meta-information, aside from
information acquired during the scanning process itself, allows for
a great number of applications. The framework we have created
enables the easy creation of remote radiology trials with
centralized data-collection. The collected data is then available to
provide feedback for radiology learning.
6. ACKNOWLEDGEMENTS
This work was supported by NIH grant CA109089 (“Improving
Radiologist Detection of Lung Nodules with CAD”). We thank
Fovia, Inc. for their generous support and the use of their High
Definition Volume Rendering ® engine (HDVR).
7. REFERENCES
[1] Wood, B. Feedback: A Key Feature of Medical
Training. Radiology, 215 (Apr. 2000), 17–19.
[2] Rubin G., Napel S., Leung A. Volumetric Analysis of
Volumetric Data: Achieving a Paradigm Shift.
Radiology, 200 (Aug. 1996), 312-317.
[3] Armato SG 3rd, McNitt-Gray MF, Reeves AP, Meyer
CR, McLennan G, Aberle DR, Kazerooni EA,
MacMahon H, van Beek EJ, Yankelevitz D, Hoffman
846
EA, Henschke CI, Roberts RY, Brown MS, Engelmann
RM, Pais RC, Piker CW, Qing D, Kocherginsky M,
Croft BY, Clarke LP. The Lung Image Database
Consortium (LIDC): an evaluation of radiologist
variability in the identification of lung nodules on CT
scans. Acad Radiol. 14(11) (2007 Nov), 1409-21.
[4] Fovia, Inc. Fovia High Definition Volume Rendering ®
(HDVR)
[5] Bryson, S., Data Manipulation in Virtual Reality, in The
Visualization Handbook, C.D. Hanson and C.R.
Johnson, Editors. 2005, Elsevier. p. 413-430.
[6] Sun Microsystems, Java Object Serialization
Specification, December 1998.
[7] Gilbert, David. 2004. JFreeChart.
http://www.jfree.org/jfreechart/
[8] Kuratorium OFFIS. DCMTK: DICOM-Toolkit. http://
dicom.offis.de/dcmtk.php.en.
[9] Crabb A. DICOM.pm: A DICOM Library in Perl. http://
dicomperl.sourceforge.net/
[10] Sun Microsystems. Java Web Start Technology. http://
java.sun.com/products/javawebstart
[11] Digital Imaging and Communications in Medicine
(DICOM). NEMA Publications PS 3.1-PS 3.12. The
National Electrical Manufacturers Association. Rosslyn,
VA, 1992, 1993, 1994, 1995.
[12] Goldberg HI, Fell S, Myers HJ, Taylor RC. A
computerassisted, interactive radiology learning
program. Invest Radiol., 25(8) (Aug. 1990), 947-51.
[13] Selcer B. Computer-assisted Interactive Radiology
Courseware. Journal of Veterinary Medical Education.
20/3 (1993)
[14] Flanders A. What Is the Future of Electronic Learning
in Radiology? RadioGraphics, 27. (2007), 559-561
[15] Sparacia G., Cannizzaro F., D’Alessandro D.,
D’Alessandro M., Caruso G., Lagalla R. Informatics in
Radiology: Initial Experiences in Radiology e-learning.
RadioGraphics. 27 (2007), 573–581.
[16] Nunes F., Costa R. The Virtual Reality challenges in the
health care area: a panoramic view. Symposium on
Applied Computing: Proceedings of the 2008 ACM
symposium on Applied computing. (2008), 1312-1316