REAL-TIME IMAGE PROCESSING

BASED ON
TASK PAIR SCHEDULING CONCEPT

Guided By:
Mrs. Khan Mam

Subitted By:
Komal More (BE11F05F030)
Geeta Popalghat (BE11F05F039)
Gauravi Patil (BE11F05F061)

INTRODUCTION

To be able to plan movements in an unknown environment,

a sensor is needed which provides at least information about
the near-field of the robot . The most general approach is to
provide information about the robots surrounding.

real-time system and real-time image processing tasks for near-field area
reconstruction and distance estimation as a prerequisite for motion
planning in unstructured environment or an outdoor area.
Ensuring hard real-time with complex algorithms
on modern computer architectures is difficult to realize due to
unpredictable execution times of algorithms with data dependent
runtime. To solve this problem a scheduling scheme based on task
pairs can be used to avoid an unpredicted state after a deadline
violation.
The scheduling solution to support the AnyTime concept:the extension of
the Real-Time Application Interface (RTAI) and the real-time image
processing task based on the SURF algorithm.

THE REAL-TIME PROBLEM

The image processing procedure has to fulfill real-time requirements.
Classical real-time programming (with WCET) cannot be achieved with
complex image processing algorithms on modern computer architectures.
The reasons are:
I.The runtime is only predictable when the algorithms and data lead to strictly
deterministic program execution, as well as only executing this process
without interruption. This is not possible due to:
Data variation.
Concurrent processes contaminate the cache.
II. A predictability of the runtime is, at best, feasible for a complete search.
Iterative calculations are not predictable due to the fact that, depending on
the present data, the calculation time fluctuates.

RTAI-Real Time Application

Interface
RTAI is an extension to the Linux kernel, which enhances it

with hard real-time scheduling capabilities and primitives for

applications to use this.
RTAI provides hard real-time guarantees alongside the
standard Linux operating system by taking full control of
external events generated by the hardware.
It acts as a hypervisor between the hardware and Linux, and
intercepts all hardware interrupt handling.
Using the timer interrupts RTAI does its own scheduling of
real-time tasks and is able to provide hard timeliness
guarantees and RTAI has support for multiple CPUs as well

ANYTIME TASK CONCEPT FOR REALTIME IMAGE

PROCESSING
AnyTime Task Concept:
-The AnyTime-task concept is based on the TAFT (Time-Aware
Fault-Tolerant) scheduling algorithm .
-Each AnyTime-task is designed as a TaskPair (TP) with a
commo deadline
-A TP comprise a MainPart (MP) and an ExceptionPart (EP), to
deal with
the fault-tolerance
-The EP execution time is defined as Worst-Case Execution
Time (WCET), while the MPs execution time can be seen as
Expected-Case Execution Time (ECET).

AnyTime Scheduling:
-The AnyTime scheduler is a hard real-time scheduler in the sense
that it always guarantees the completion of either the MP or the EP
before the deadline.
-A utilization factor equal or less than 1 denotes a feasible
schedule on one CPU core in a system with n independent,
preempt able tasks with deadlines equal to their periods. This
behavior can be expressed by the following equation .

DESIGN AND IMPLEMENTATION

For implementing the scheduling strategy
described above the RTAI extension was used. The goal is to
implement the same functionality for user and kernel mode
applications as well.
A. AnyTime Design
For extending the RTAI scheduler to support two
level scheduling to fulfill Any Time requirements, the
following extensions need to be made: Adding necessary
variables to the Task Control Block (TCB).
Tasks need a method to indicate to the scheduler that
they need to be run as AnyTime Tasks, as opposed to other
tasks which may want to maintain the default behavior.
Adding the two-layer scheduling policy as previously
described into the RTAI scheduler. The periodic timer
function which is responsible to set the next timer interrupt
needs to become aware of the new TP timing parameters.
Adding support for multi core systems as well as
support for LXRT user land applications.

B. AnyTime Implementation
Towards an AnyTime scheduler, its needed to extend
the
TCB with the new timing parameters to handle the new
introduced AnyTime task construct
Load ballancing criteria: To support multi core systems
load balancing was implemented to distribute new
created
tasks on available CPU cores.
A new created task is bound on the CPU core
with the lowest utilization factor. Alternatively it is
possible to bind a task on a specific core.
Scheduler Modifications:
Each moment the current time passes the release time
of the head element of the LRT list, the scheduler
migrates this task as head element to the ready queue
of the current CPU representing the runable task with
the highest priority.

 C.

AnyTime LXRT Library

As listed as scheduler requirement (section V-A)

the implementation
should have support for kernel and user LXRT mode
applications as well. Hence a TCB variable tp_status must
be accessible from inside the application to distinguish
between
MP and EP execution flow inside the currently running
task. From inside kernel mode applications access is
granted
without any further adaptations. In LXRT user mode the
access
of TCB parameters are only possible through expensive
system
calls. To avoid this a more efficient way to access the
required
TCB values from within a LXRT task was developed.

ANYTIME-SURF
The AnyTime-SURF is an extension of the standard Speeded Up
Robust Features (SURF) algorithhard real-time environment. SURF is used
for an efficient detection and description of points of interest in images.
SURF transforms the image information into gradient-features that are
robust to changes in distance, illumination, rotation, scaling and noise. Each
point of interest, called key point is described by a unique descriptor.
A. Concept
The advantage of the AnyTime-SURF is that a first result is available after
a very short computational time.
The initial results are improved as long as computational time is available.
The AnyTime-SURF algorithm provides corresponding key
points in stereo vision images as results.
To obtain these key points, the AnyTime-SURF solves the following tasks:
1) Calculation of the key points in the left and right image.
2) Searching for corresponding key points.

 1) Task Structure: To allow a parallelized calculation,

AnyTime-SURF is divided into three independent tasks. Task

1 calculates the key points of the left image and task 2 does
the same for the right image. Matching between images is
performed within task .
2) SURF-Phase: Two changes to the standard SURF were
made to achieve real-time capabilities together with the
availability
of an early first result.
Following the different SURF phases and realized modifications
are presented:
1) Building a scale space (4 octaves).
2) Extrema determination.
3) Points of interest calculation.
4) Calculation of descripto In standard SURF, each phase is
calculated for all octaves
before the next phase is performed.
Disadvantage: The results are only available after the last phase
has
finished. This is different within AnyTime-SURF.

3) Sectioning MP and EP: As denoted in section IV, the calculation steps of the
algorithm must be separated into a Main- and an ExceptionPart . The purposes of the
ExceptionParts are:
Task 1 and task 2:
Setting flags for memory management.
Setting semaphores for process synchronization.
Saving the results.
Release RTAI heap memory.
Matching task:
Saving the results.
Delete key points and their descriptors of the current
matching.
Setting flags for memory management.
4) Memory Concept: Image size and the amount of prospective key points are not
known before the program
starts. Therefore it is impossible to allocate all memory in advance. Dynamic memory
is allocated on the RTAIheap and is performed in hard real-time mode.
B. Behavior when Missing a Deadline
If the deadline DLMP of a MainPart is violated, the Main-Part will be canceled. There
are different results for further processing, depending on the time of suspension.
Concerning task 1 and task 2: If the calculation of the first octave is finished before
DLMP is violated no useful intermediate results are available. All results are discarded
and the initial start state of the task is resumed.

MEASUREMENTS
To verify the performance of the Anytime-

SURF, a setof measurements have been

carried.

ALL MEASUREMENTSARE PERFORMED ON THE

FOLLOWING TEST SYSTEM

Dell Latitude E4300 (Intel Core2 Duo P9400;

4GB RAM) with an Ubuntu 9.04 32Bit OS

(custom 2.6.29.4 Kernel with Anytime RTAI 3.7.1
patch applied).
Only one core was used for the SURF algorithm.
To determine the test results several images of
different sizes have been used.
Measurements over 100 runs for each
imagewere taken to get average times for ECETs
of MainParts.

APPLICATIONS
There are more applications whose runtimes

are not predictable.

If algorithms have a data-dependent runtime,
it is impossible to model a WCET.
In such cases the Any Time concept can help
to ensure real-time.
If an algorithm may be realized using
incremental computation of results, the shown
scheme may also be used.

Another excellent candidate is sampling

based planning.
A certain amount of samples are needed to
get first results. More samples may be
processed to get a better solution.
After first results are available the process
may be stopped if the deadline
is reached. Also in this case, the best
solution may be collected by the EP.

CONCLUSION
This paper presented an implementation of an

Anytime algorithm based on the task pair concept

using a RTAI/Linux real-time operating system.
The developed solution is single core and
multicore capable and a CPU load balancing is
realized.
In case of time-outs the tasks provide results and
will be cleared by the system for a restart.
Other applications such as sampling based
planner may also be used in conjunction with this
scheduling scheme to ensure real-time operation.