High-Performance Image Acquisition & Processing Unit Fabricated using COTS Technologies

Shinichi Kimura, Akira Miyasaka, Ryu Funase,

Hirotaka Sawada, Nobuoml Sakamoto & Naold Mlyashita

ABSTRACT It is becoming imperative to have visual capabilities for

space activities. There are increasing opportunities to use for spacecraft sensing and control. To fill this need, we have developed a small, low-cost, high-per formance

visual Images coupled with image processing technologies

image acquisition and processing unit (HP -IMAP), which uses commercial off-the-shelftechnologies. In 2 01 0 the , structure. Herein, we describe the BP-IMAP and discuss Its qualification tests. INTRODUCTION

BP-IMAP was launched to monitor a deployable

It is becoming imperative to have visual capabilities for space activities. Because of the advances in space systems, spacecraft and their missions are becoming highly complex, and both the telemetry as well as the visual data is important for monitoring these systems [1-4]. Also, visual capabilities can be used for sensing star sensors [5-6].Visual information is also highly educational and entertaining. Small satellites, including university satellites, are becoming more popular, and visual capabilities are important in such cases for education. Therefore, a small, low-cost space camera is needed for various applications. Although devices for terrestrial applications are continually being improved, limited progress has been made on devices that need to be extremely reliable for use in harsh environments in space. However, if devices that are commonly used on Earth could also be used in space and still provide high performance, the development costs for these

Author's Current Address: s. Kimura, A. Miyasaka, R. Funase, H. Sawada, N. Sakamoto and N. Miyasbita. Manuscript received June 6,2009; revised July 9,2010. Review was bandled by H. Uu. 088518985/111 $26.00 2011 IEEE

devices could be greatly reduced. We have previously used and verified commercial off-the-shelf (COTS) technologies and demonstrated verification experiments for advanced devices and technologies during a few missions in space [8-10]. To enhance the possibility of using visual images in space missions, it is necessary to couple image capture technologies with image processing technologies. Spacecraft maintenance is generally limited, and image compression technologies are indispensable to transmit images for monitoring. Moreover, if we use autonomous shape recognition or motion detection using image processing technologies, a part of the monitoring process can be made autonomous and the downlink bandwidth can be used more efficiently [1]. If we use on-orbit image processing capabilities, then visual images can make effective sensors for human monitoring of spacecraft control of the image capturing device. Such devices can also be used as navigation sensors [5-6], and Joergensen [7] has already developed a low-cost star sensor that uses COTS technologies. Therefore, the development of a small, low-cost, general-purpose image acquisition and processing unit will enhance spacecraft autonomy and mission capabilities. Low-cost cameras based on cots technologies are starting to emerge; however generally, their image processing capabilities are still not satisfactory. For example, the only image compression offered by the cameras is simple, ready-made image compression that cannot be customized. Even though the cameras offer flexible image processing capabilities and the size of the camera unit is very small, an additional image processing unit is required for image compression and processing [8-10]. Recently, small satellites are gaining interest, and in such satellites, space is a precious resource. Small, low-cost cameras that offer image processing capabilities can dramatically increase the capabilities of these satellites. We have developed a very small, high-performance image processing unit that is based on COTS technologies. It has a

mEE A&E SYSTEMS MAGAZINE, MARCH 2011

19

500 MIPS calculation capability in a single, 50 mm x 50 mm printed circuit board, and it incorporates various types of interfaces using field-programmable gate array (FPGA) technology. By using this image processing unit, we have developed a small, low-cost camera for spacecraft that incorporates image processing capabilities. The camera is called the high-performance image acquisition and processing unit (HP-IMAP), and in 2010, it was launched to monitor a deployable structure (Figure 1). In this article, we describe the HP-IMAP and discuss its qualification tests.
Front Back

Fig. 2. Block Diagram of the HP-IMAP Electrical Structure

Fig. 1. High -Per formance Image Acquisition and Processing Unit (HP-IMAP)

HARDWARE
Electrical Architecture

The HP-IMAP camera consists of an imager board and a processor board. For the processor board microprocessor unit, we use a Xilinx CPU core FPGAVirtex-II Pro (Figure 2). Using the FPGA capability to achieve peripheral interface flexibly, we obtain 500-MIPS calculation capabilities and 64-Mbyte synchronous, dynamic, random access memory in a 50 mm x 50 mm board processor board. We also installed 8-Mbytes of flash memory for program and operational system storage (Figure 3). Furthermore, FPGA technologies enable the processor board to cover various protocols for not only the imager interface but also interfacing to satellite systems, and the optimum interface can be selected depending on the satellite system. We have already treated USART, CAN, i2c, SPI, and Ethernet interfaces. The specifications of the image processor board are summarized in Table 1. A megapixel CMOS imager O V9630 is installed at the center of the imager board (Figure 4). The imager board is connected to the processor board through the imager data lines and the i2c control-interface line. Because we developed several types of imager boards with the same electrical and mechanical interface to the processor board, the best imager can be used depending on the mission

Fig. 3 High Performance Image . Processing Board using FPGA

requirements. For example, a high-speed global shutter imager board can be used to capture fast motion, and a high megapixel imager board can be used for high-resolution applications. For high-resolution applications such as Earth observations, we can use an 8 megapixel imager. The HP-IMAP requires only a single 5- power supply V because it regulates several internal power supplies, and its overall energy consumption is 600 mW.
Optics and Mechanical Structure

In general, customized designs and development of optical systems are costly, so we hope to use COTS camera lenses to reduce the cost of the on-orbit camera. However, to use COTS camera lenses in orbit, we need to consider at least two problems: contamination and degradation caused by outgassing and mechanical durability. Small COTS cameras often use plastic for lenses and structural parts to reduce development cost and weight.

20

IEEE A&E SYSTEM MAGAZINE. MARCH 2011

Table 1. HP -IMAP Specifications

Size Weight Power Consumption

56x 70x 55 mm 264g 60 0 mW (peak)

Focal Length FNumber FOV Image Size Pixel Size Resolution

2. 0 mm 2. 0 1 0 70 x 940 12 80 * 1 02 4 (SXGA) 4. 2 J.1m x 4. 2 J.1m 0. 08 1. 4cm in 10 m Distance Fig. 4: COTS CMOS Imager Board

Sensitivity SI N Ratio Dynamic Range Maximum Frame Rate

1.0 V/Lux-sec 54 dB 60 dB 15 fps

CPU Core

Power PC 405 42 0-MIPS

Memory

SDRAM: 342-MB Flash Memory: 8-MB

Image Output

Thumbnail I Bilevel Serial (UART)

RAW/JPEG

Interface

plastic; therefore, such lenses should not outgas or degrade in an ultra-vacuum environment (Figure 5). Because several types of lenses are aligned within the focal length and the F factor is based on the same mechanical interface, the specifications of the optimal lens can be selected to fit the application. The camera will serve as the lens tube to make the camera mechanically durable using a simple structure. Because the focal length of the board-camera lens is short, using the camera side wall as a lens tube increases the side-wall thickness up to a maximum of 10 mm. The side of the lens cylinder is equipped with Ml3 pitch single-thread screws. The lens is attached to the camera wall using double nuts to prevent lens motion and maintain the focal length. Because the four side walls and the camera flange are made from a single aluminum block by scraping out the inside of the camera body, the lens hole in the camera side wall remains

Fig. 5. Camera Lens from Sakai Glass Engineering

Under ultra-vacuum conditions, however, the solvent contained in the plastic material may evaporate (i.e., outgas) and cause degradation of the material. The optical performance and transparency are depredated by outgassing, particularly for plastic lenses, and the solvent thus liberated is harmful to other equipment, especially other optical sensors, because it can contaminate the sensor optics. Sakai Glass Engineering supplies a series of board-camera lenses that are made from glass and metal and do not contain

securely fixed in position. In addition, the fact that the lens has a very small mass is an advantage for mechanical durability (e.g., lens position with respect to the camera board), which we verify through the qualification tests discussed below.
SOFTWARE

IEEE A&E SYSTEMS MAGAZINE. MARCH 2011

21

Fig. 6 V . ibration Test per formed at JAXAIISAS Sagamihara

Adopting an operating system for the built-in computer system greatly increases software productivity, even though it comes at the cost of some memory and computational resources. Using device drivers and function calls enable us to call complex functions using a simple statement, and if we use a popular operation system, we can use software resources in a cross-platform manner. Another consideration regarding software is that the use of COTS technology is an effective way to enhance the software performance while reducing cost. We choose the Linux operating system, which is one of the most popular free operating systems, for the image processing unit. Using Linux, we can take advantage of the large amount of free software resources that have been developed for it. For the HP-IMAP, we installed free software applications for JPEG compression and to compress binary data. In addition to developing the original applications and device drivers, we can utilize various other resources (e.g., documentation on the Linux operating system). These resources and free software allow us to reduce not only the development costs for the on-orbit system but also the work load and development time.
QU LIFICATIO TESTS A N

from 0 to 100 Hz as well as random vibrations were applied in each axis. The basic functioning of the HP-IMAP camera passed the qualification test. Specifically, the electronics such as the image processing board and imager were not affected by the vibrations, and the optical parameters such as focal length and optical center also remained unaffected. These results show that the COTS board and camera lenses can successfully survive launching conditions and the simple lens tube structure that uses the camera side wall is rigid enough to maintain optical precision through the launch conditions.
Ultra-V acuum Test

To verify the feasibility of the HP-IMAP camera for use on-orbit and in the launching environment, we performed three types of qualification tests.
V ibration Test

Adaptation to the vibration conditions that occur during launching is one of the most important features for a space system. For optical equipment, it is essential to maintain not only the mechanical soundness and function but also the optical precision. Because the HP-IMAP uses COTS optics, survivability under these vibration conditions is thoroughly tested. The vibration test was performed at the vibration test facility at the JAXA Sagamihara Campus (Fig. 6). The vibration conditions were adjusted to the qualification test level of the H-IIA launching vehicle. Sinusoidal vibrations

The ultra-vacuum condition is another important characteristic of the orbital environment. This condition affects equipments in two main ways: through outgassing and thermal control. As mentioned above, outgassing causes degradation of optical systems and contamination of other systems (particularly optical systems). Because the camera lens used in the HP-IMAP is glass mounted in a metal tube, outgassing was expected to be small (but this was nevertheless verified). The ultra-vacuum condition also affects the thermal control of electronic equipment. In terrestrial conditions, air serves as an effective conduit to remove heat generated by the electronic equipment, thus protecting the equipment from overheating. Under ultra-vacuum conditions, this heat-transfer channel does not exist, so heat is transferred only through radiative emission or material conduction. Because HP-IMAP was designed to effectively transmit heat through conduction, we expected that the HP-IMAP would survive the ultra-vacuum conditions and this was also verified. The ultra-vacuum test was performed at the ultra-vacuum test facility of the JAXA Sagamihara Campus (Figure 7). The HP-IMAP camera was installed in a vacuum chamber that was pumped down to 10-4 Pa. Even in ultra-vacuum conditions, the HP-IMAP functioned without problems. Unfortunately, the small size of the chamber prevented optical calibration from being performed under ultra-vacuum conditions, but we verified that the camera parameters did not

22

IEEE A&E SYSTEMS MAGAZINE, MARCH 2011

Fig. 7. Ultra-Vacuum Test

Fig. 9 Images Acquired during the TID Test, . under the Bright Conditions

The temperature varied from -300 to 600 and test pattern images were captured by the HP-IMAP. According to OV9630 specifications, the OV9630 can acquire stable images in a temperature environment between 00 to 500 In the negative temperature range, the HP-IMAP stopped operating because the OV9630 seems to have anomalies at low temperatures. Therefore, we decided to install a heater to support the low temperature operation. Because the energy consumption of the processor board is large, the heat generated by the processor board will support low temperature operations once the processor board is switched on. From 00 to 500 the HPIMAP is stable and acquires good quality images. However, above 500, thermal noise and contrast degradation are observed in the acquired images. Therefore, the operational temperature range of the HP-IMAP is from 00 to 500, and lower temperatures can be covered if a heater is installed.

Fig. 8 TID Test with 6OCO at the . Takasaki Advanced Radiation Research Institute

change before and after the ultra-vacuum test. During the ultra-vacuum conditions, no significant outgassing was observed. These results show that the HP-IMAP can perform in ultra-vacuum conditions without outgassing or contaminating itself. Because the ultra-vacuum test facility used for the HP-IMAP test lacks thermal control equipments, the thermal vacuum condition test could not be performed on the HP-IMAP. However, the thermal stability of HP-IMAP is qualified using the thermal qualification test facility.

Drocj 3-1iMU2ch

F ig. 1 0 Images Acquired during the TID Test, . under the Light Shielded Conditions

IEEE A&E SYSTEMS MAGAZINE, MARCH 2011

23

Radiation Test on Imager

To reduce development costs and achieve high performance, we use a COTS imager for the image acquisition and processing unit. Because COTS imagers are not expected to be used under high-radiation conditions, it was necessary to verify their radiation tolerance. Because the radiation tolerance ofVertex-II pro is already assessed by NASA ("Xilinx Virtex-II Pro PowerPC Proton Test Results, " I10 1705_V2Pro, http://radhome.gsfc.nasa.gov/radhomel RadDataBase/RAdDataBase.html), we decided to focus on the radiation stability of the COTS image sensor. 6O Gamma radiation tests using CO were performed at the Takasaki Nuclear Laboratory (Figure 8). We found that the image contrast decreased slightly when the devices were exposed to over 3.0 krad, and a 60 krad exposure caused fatal degradation of the image acquisition capabilities (Figure 9). In the same test, we captured dark images to assess degradation (Figure 10). We observed a slight degradation at 10 krad, and observed that 30 krad was fatal. Thus, we determined the image tolerance to a total ionizing dose (TID), which is 3.0 krad. These results suggest that the CMOS imager can survive
..

The single event performance of these devices was assessed under radiation from a single ion beam at Brookhaven National Laboratory (Figure 12). No single-event latchur. (SEL) was observed in these devices upon exposure to 8 Br (linear energy transfer of 37.73 MeYcm3/mg). Therefore, we determined that no special treatment is required for SEL events other than the usual, precautionary current limiter. The imager experienced a lot of noise due to the radiation, but no fatal single-event upset

JiI

Fig. 1 2. Single Event Test at the Brook Heaven National Laboratory

A

100

....V .

0
OIl

...

:ono

.....

<n

,.,.,

.,. .

Total Doselkradl CMOS

....

(SEU), such as resetting or malfunctioning, was observed upon exposure to 19F (linear energy transfer of3.38 MeYcm3/mg). In addition, the SEU rate was quite low, so device initialization is the easiest way to mitigate SEU.
CO NCLUSION

MP

Fla h Mem ry

Fig. 1 1. Energy Consum ption during the TID Tes t

six months in the most severe conditions of geo-transfer orbit. They also suggest that when we assess the TID tolerance, gathering typical measurements of electric current is insufficient; functionalities should also be assessed during radiation tests. Simultaneously, we recorded the current from the power supply, which is considered to be an indicator of the degradation level due to the TID and tested a processor and flash memory (Figure 11). The processor was not affected by the radiation and maintained its current level. In contrast, the power consumption of the flash memory increased rapidly and caused a malfunction, which is considered to be a typical degradation due to the ionizing radiation. However, for the CMOS imager, no change in power consumption was observed. This result suggests that the degradation process of the CMOS imager might differ from that of the logical elements.

Using COTS technologies, we have developed a small, low-cost camera that includes image processing capabilities for spacecraft called the HP-IMAP. The CMOS imager and optics, also assembled using COTS technologies, offer low-cost and high-performance image acquisition. Using COTS FPGA technologies and free open-source software, we developed low-cost high-performance image processing capabilities that are easy to implement. The HP-IMAP is qualified by vibration, ultra-vacuum, and radiation tests and has been proven robust enough to use in orbit. Newly-available imagers are rapidly improving with regard to the available resolution, while themselves decreasing in size rapidly, as seen in both cellular phones and digital cameras. The functionality of imagers is also improving and imagers can compress and/or process images by themselves. If we utilize these technologies in space, image acquisition in space will also improve rapidly. We hope to continue utilizing these technologies in space. The HP-IMAP technologies are being utilized in the IKAROS (Interplanetary Kite-craft Accelerated by Radiation

24

IEEE A&E SYSTEMS MAGAZINE, MARCH 2011

of the Sun) that launched May 21,2010. IKAROS successfully became the world's first solar power sail craft generation during its interplanetary cruise. HP-IMAP IKAROS and support its operations. We are going the space qualification results in the near future. employing both photon propulsion and thin-film solar power technologies are being used successfully to acquire images of

25"' International Symposium on Space Technology and Science,Ishikawa,Japan,o-3-05v.

[6] Shinichi Kimura,Yasufumi Nagai,Hiroshi Yamamoto, Hiroshi Kozawa,Sachiko Sugimoto,Shinobu Nakamura,

to report

Kazumi Masuda and Naohiko Abe,(2006),

Rendezvous Experiments on SmartSat-l,

Space Mission Challenges for Information Technology (SMC-IT 2006),Pasadena, CA,USA,pp. 374-379.

R EF ER ENC ES
[I] Shinichi Kimura,Makoto Takeuchi,Koichi Harima, Yutaro Fukase, Hitoshi Sato, Tetsuji Yoshida,Akihiro Miyasaka,Hiroyuki Noda, Visual Analysis in a Deployable Antenna Experiment using SUb-pixel Cross-correlation, Vol. 40, No. I,pp. 247-258. IEEE Transactions on Aerospace and Electronic Systems, Kei Sunakawa,and Masanori Homma, (2004),

[7] Eisenman,A., Joergensen,J.L. and Liebe,C.C.,(1996), compass,

Real sky performance of the prototype 0rsted advanc stellar Proceedings of Aerospace Applications Conference, Aspen, CO. USA,Vol. 2,pp. 103-113.

[8] Shinichi Kimura,Hitoshi Mineno,Hiroshi Yamamoto,Yasufumi Shin-ichiro Nishida Shinichi Nakasuka Shinichi Ukawa, Hidekazu Hashimoto,Nobuhiro Takahashi and Keisuke Yoshihara, (2004), Preliminary Experiments on Technologies for Satellite Orbital Maintenance Using Micro-LabSat I, Advanced Robotics,Vot. 18,No. 2,pp. 117-138. Nagai, Heihachiro Kamimura,Satomi Kawamoto,Fuyuhito Terui,

[2] M. Oda, (2008), REXJ. Demonstration of an Astronaut Support Robot (Astrobot) on the International Space Station Japanese Experiment Module (ISS/JEM:KIBO), Glasgow,UK, IAC-08-B3.6.7. in Proceedings of International Astronautical Congress,

[9] Shinichi Kimura,Hiroshi Yamamoto,Yasufumi Nagai, Maki Akioka, Hidekazu Hashimoto,Nobuhiro Takahashi,Matsuaki Kato and Keisuke Yoshihara,(2005), Single-Event Performance of a COTS-Based MPU under Flare and Non-Flare Conditions, Vol. 41,No. 2, pp. 599-607. IEEE Transactions on Aerospace and Electronic Systems,

[3] T. Williams and S. Tanygin,(1998), Vehicle,

Onorbit Engineering Tests of the Aercam Sprint Robotic Camera Adv. Astronaut. Scio., 99,2,1001-1020.

[4] L. Kerstein,F. Seinsiek and E. Oraf,(1997), Inspector and the International Space Station, Space Technology, Vol. 16,No. 2,pp. 83.

[10] Shinichi Kimura,Hitoshi Mineno,Yasufumi Nagai,

Heihachiro Kamimura, Satomi Kawamoto,Fuyuhito Terui, Hiroshi Yamamoto,Shin-Ichiro Nishida,Shinichi Nakasuka,

Shinichi Ukawa,Hidekazu Hashimoto,Nobuhiro Takahashi and Keisuke Yoshihara,(2003), Experiments on High Performance Image Processing Technologies on Micro-LabSat, 54"' International Astronautical Congress 2003 (lAC 2(03), Bremen,Germany,IAC-03-IAA.l1.2.04.

[5] Shinichi Kimura, Yasufumi Nagai,Hiroshi Yamamoto, Naohiko, Abe,(2006), Possibility of small satellite in on-orbit servicing,

Hiroshi Kozawa,Sachiko Sugimoto,Kazumi Masuda and

IEEE A&E SYSTEMS MAGAZINE,MARCH 2011

25