IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 55, NO.

1, FEBRUARY 2008

217

An Operation-Server Based Data Acquisition System Architecture

Clyde C. W. Robson, Samuel Silverstein, and Christian Bohm, Member, IEEE
AbstractDevelopments in networked embedded system technologies and programmable logic are making it possible to design new, highly exible data acquisition system (DAS) architectures. We present a networked DAS architecture where the software resources required by the data acquisition (DAQ) units are acquired from a server dynamically, to be stored and executed in local memory. When the DAQ unit needs to call an operation not contained in its memory, the new operation is requested from the server, while resources occupied by operations that have not been used recently are released to make space for it. In this way it is possible to reduce the resource use of the embedded DAQ unit to only what is necessary for efcient operation in its major modes. Since new code is easily distributed to the DAQ units from the server, system upgrades are also simplied. Index TermsAdaptable, data acquisition architecture, distributed system, object oriented, recongurable.

I. INTRODUCTION

N a distributed data acquisition system (DAS) parts of the processing resources are moved closer to the front ends into what we may call local data acquisition (DAQ) controllers. These may contain embedded processors, often as a part of a eld programmable gate array (FPGA). One of the benets of using a distributed system is the possibility to allocate specic nodes to handle specic tasks. A multitier architecture also allows parts of the embedded code to be removed from some embedded nodes to be executed on more suitable nodes. The size of the embedded nodes can be further reduced in a dynamically recongurable system, if at any given time only the functions needed in the current running phase are loaded in memory. Each client issues requests for an operation to be performed by the system. An operation is a set of one or more commands, executed in a specic order. From the embedded controllers point of view, a command is a function, locally stored in memory or downloadable from the control server. We have developed a general, distributed system architecture with these features. We have also tested the architecture in a typical distributed DAQ system implementation. II. THE UNDERLYING SYSTEM

Fig. 1. A network topology for a typical system built around the described architecture. Each embedded DAQ controller receives data from several front-end system elements (FEE), which are connected to a switch over a 100 Mbit ethernet connection. The switch is, in turn, connected to the server via 1 Gbit Ethernet. The clients connect to the server over Internet.

Fig. 2. A logical view of the system. The server is the access point to the system for the clients. It transmits the commands to the DAQ controllers. It can also make use of external resources, as Matlab, for more complex commands. This increased complexity is transparent for the clients.

The implemented system (Fig. 1) comprises a number of easily interchangeable nodes [1]. By using a common, in-house application level protocol [2] all nodes can easily access each
Manuscript received May 11, 2007, revised October 28, 2007. C. C. W. Robson is with the Instrumentation Physics Group, Department of Physics, Stockholm University, 10691 Stockholm, Sweden (e-mail: clyde@physto.se). S. Silverstein and C. Bohm are with the Department of Physics, Stockholm University, 10691 Stockholm, Sweden. Digital Object Identier 10.1109/TNS.2007.913491

other and make use of the services that other nodes have to offer. A service request consists of a command and its parameters. The front-end system elements are connected to embedded, FPGA based nodes for control. These embedded nodes are transmitting data over a 1 Gbit Ethernet to a dedicated PC, running MySQL.1 They are controlled by the control server, which is the access point for local and remote clients that are controlling and monitoring the system. This control server is also responsible for distributing commands to the embedded controllers, it may also use external services as a part of a command given by the client. An example of such external service is a PC running Matlab.2 By solving the need for mathematical processing in this way, we have thus access to the full power of Matlab in our system (Fig. 2). III. CLIENTS FOR CONTROL Clients for control and monitoring can be written in any convenient language, as long as they use the same application level
is a database server from MySQL AB. is a numerical computing environment and programming language from The MathWorks.
2Matlab 1MySQL

0018-9499/$25.00 2008 IEEE

218

IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 55, NO. 1, FEBRUARY 2008

Fig. 3. In this UML class diagram of the embedded software we are given a statically view of the architecture. The resource class contains the replaceable commands that can be downloaded from the control server.

protocol. We are currently using LabVIEW3 for rapid initial development and Java4 for the production phase. Several clients can use the system concurrently. IV. CONTROL SERVER The control server is the access point for software clients accessing the system from anywhere in the world. By wrapping in the embedded part of the system and using the embedded controllers commands as a basic set of atomic commands, it can also expand the total number of commands and their complexity almost indenitely by forming operations. The control server can also add logic to the operation in order to increase its processing power. An operation can therefore be a sequence of atomic commands within the embedded controllers, extra processing in the control server and external services on other nodes. All this increased complexity is transparent for the clients, however. The control server is also used to store and distribute precompiled functions, the atomic commands, to the embedded controllers when so requested. The control server maintains a list, updated with each precompiled function that has been downloaded by each embedded controller. When a new version is available, a simple message is transmitted to the affected embedded controllers and they can then download the new version as a regular request to the control server. V. EMBEDDED CONTROLLERS FOR DAQ The embedded controllers in our system have been implemented on FPGA based boards using Virtex-45 FPGAs from Xilinx [3]. They offer a network based set of commands for control and data acquisition. They can easily be replaced in the
3LabVIEW is a platform and development environment for a visual programming language from National Instruments. 4Java is an object-oriented programming language developed by Sun Microsystems. 5Virtex-4 is an FPGA family from Xilinx.

system as long as they make use of the application level protocol. At the moment we are using software network protocol stacks for communication, but we are looking into hardwarebased solutions for communication to improve performance. VI. EMBEDDED SOFTWARE The controller architecture is designed to be highly adaptable. It is implemented in C for small memory footprint and good performance. The main body is used for communication and control, and all tasks are implemented as work functions (commands) (Fig. 3). This object-oriented approach [4] to the design makes it easy to work with. The controller can easily be adapted to new situations and tasks by simply replacing commands. Since this can be easily done during runtime, there is no need to restart or redeploy the full system. All atomic commands implement a standardized signature, making the nodes in the distributed system interchangeable. This is inspired by the Command pattern in object-oriented terminology. The commands are precompiled les in the Executable and Linking Format (ELF). They are stored in and distributed by the Control Server. If the same command is used for different architectures, MicroBlaze,6 PowerPC,7 etc., the server is keeping track of which command implementation is required for each embedded controller. This is carried out by mapping the ip number of each embedded controller to software versions, making automatic updates very easy in a publishsubscribe way. When a new version of a command is deployed in the control server, the server noties the embedded controllers that are currently using that command of the presence of the new version. They can then make a request for the new version. When an embedded controller receives a command from the control server, it checks whether the requested command is among the deployed commands. If so, the command is executed and if any data is produced, it is transferred to the database
is a soft core CPU from Xilinx. is a RISC microprocessor architecture from IBM that is used in Xilinx high end series FPGAs.
7PowerPC 6MicroBlaze

ROBSON et al.: AN OPERATION-SERVER BASED DATA ACQUISITION SYSTEM ARCHITECTURE

219

Fig. 4. This image depicts the execution of a command, initiated by the client. 1. The client requests a command to be executed by the system. 2. The control server transmits the command, if it is atomic, to the DAQ controller. If it is a macro, the rst atomic command in sequence is transmitted. 3. The DAQ controller is executing the given command(s). But if one or several of them is not recognized as being loaded into memory, a request for the missing command is sent to the control server. 4. The control server is sending the missing, precompiled command to the DAQ controller. 5. The DAQ controller is executing the command. If the result is data being produced, the data is sent to the DataBase over a dedicated 1 Gbit Ethernet connection.

The system has also successfully been tested on embedded systems, including MECA,8 [5] an in-house developed prototyping system based on the ETRAX LX-100 4+16 MCM with 16 MB DRAM and 4 MB SRAM. The ETRAX is a 100 MHz, 32-bit CPU with an architecture optimized for networking. The operating system is built on the Linux 2.4.26 kernel. We run the very same tests here; download and execute a precompiled command 100 000 times, between the MECA and one of the earlier mentioned PCs as control server. This resulted in total execution times of 5 ms per operation using TCP and 1.23 ms per operation using UDP, the control server was implemented in C. The same test, using UDP and a control server implemented in Java resulted in a total execution time of 1.31 ms per operation. VIII. CONCLUSION We have developed a highly exible distributed system architecture that is adaptable to many different data acquisition applications. In addition to client-driven data pull operations, the architecture can also support event driven asynchronous and synchronous data driven push operations by having the DAQ controllers activate the operations. Potentially interesting applications include monitoring, where a number of different clients could simultaneously pull data from any of the front-ends, to supervise the systems functionality. In applications where nodes have minimal resources, such as FPGA-based systems with small internal memories, only a portion of the full command set can be activated at one time. The full set of commands is then kept in the command server and served to the node as needed. The system naturally allows command updates and command debugging, but can also be extended to allow network maintenance where failing nodes are reset, reloaded and congured both with respect to rmware and software. This requires suitably constructed nodes with hardware watchdogs and system boot capabilities. REFERENCES
[1] C. C. W. Robson, A. Bousselham, and C. Bohm, Replaceable middleware communication modules for distributed data acquisition systems, presented at the IEEE-NSS, San Diego, CA, Oct. 29Nov. 5, 2006. [2] W. R. Steven, TCP/IP Illustrated, Volume 1: The Protocols, and TCP/IP Illustrated, Volume 2: The Implementation. Reading, MA: Addison-Wesley, 1994. [3] C. C. W. Robson, A. Bousselham, and C. Bohm, An FPGA-based general-purpose data acquisition controller, IEEE Trans. Nucl. Sci., vol. 53, no. 4, pp. 20922096, Aug. 2006. [4] B. P. Douglass, Real Time UML: Advances in the UML for Real-Time Systems, 3rd ed. Reading, MA: Addison-Wesley, 2004. [5] S. B. Silverstein, J. Rosenqvist, and C. Bohm, A simple Linux-based platform for rapid prototyping of experimental control systems, IEEE Trans. Nucl. Sci., vol. 53, no. 3, pp. 927929, Jun. 2006.
8MECA stands for a multipurpose embedded system for control and data acquisition from Axis Communications.

server. If not, a request is sent to the control server for the requested command. Despite the fact that only the text segment of the ELF formatted le is necessary for the embedded controller, in order to be able to load a new command, we choose to send the whole le. This is because of the fact that the text segment is placed with a different offset for different versions of the ELF format. In our architecture the embedded controller is an independent entity, the object oriented paradigm, and we do not want the control server to need to know about the internal structure of other nodes. The extra bytes in form of segments and headers in the ELF format are compensated by the benet of an architecture where each object is responsible for its own data. This makes the distributed system much easier to understand and maintain. When the command is received, it is then loaded and executed (Fig. 4). VII. RESULTS The extra overhead for loading externally stored commands strongly depends on the connection type and speed. Running tests where we were downloading and executing a simple command of a total of 496 B 100 000 times, using a test rig comprising two modern PCs (1.8 GHz Pentium Core Duo processors, 2 GB DRAM, Gbit Ethernet) running Mandriva Linux resulted in an execution time of 0.25 ms per operation. The transport protocol was TCP and the control server was implemented in C. Using UDP instead for the transport protocol, keeping everything else the same, resulted in a total execution speed of 0.1 ms per operation. But the implementation of the control server is also of signicance here. Running the same tests with a control server implemented in Java resulted in a total execution time of 0.16 ms per operation, we were using UDP here. We want to use Java for the implementation of the control server due to its complexity, but this is a work in progress.