A Case Study of Design Space Exploration
for Embedded Multimedia Applications on SoCs ∗
Isabelle Hurbain, Corinne Ancourt†, François Irigoin
École des mines de Paris
F-77300 Fontainebleau, FRANCE
Frédéric Pasquier
THOMSON R&D France
F-35576 Cesson-Sévigné, FRANCE
Abstract
Embedded real-time multimedia applications usually im-
ply data parallel processing. SIMD processors embedded
in SOCs are cost effective to exploit the underlying par-
allelism. However, programming applications for SIMD
targets requires data placement and operation scheduling
which are NP-complete problems.
In this paper we show how our tool (based on concurrent
constraint programming) can be used to explore the design
space of a kernel in H.264 standard (video compression).
Different cost functions are considered (e.g. execution time,
memory occupancy, chip cost ...) to derive different source
codes from the same functional specification. Future work
includes model refinement as well as full code generation
for rapid prototyping of such hardware and software inten-
sive systems.
1. Introduction
Embedded real-time multimedia applications usually im-
ply data parallel processing. Indeed image processing in-
volves vectors or matrices of pixels for instance. Moreover
they are often (statically) predictable and well-structured in
such a way that the potential parallelism can be extracted
at compile time. More than 80% of the execution time is
spent in loop nests that define and use arrays. Efficiently
performing these regular computations obviously is a key
issue.
SIMD (Single Instruction Multiple Data) architectures
are well suited for this multimedia application domain be-
∗ This
work is funded by the RNTL DREAM-UP project. URL:
http://www.telecom.gouv.fr/rntl/projet/Posters-PDF/RNTL-Poster-
DREAM-UP-Thomson.pdf
† Corresponding author: ancourt@cri.ensmp.fr
Michel Barreteau, Nicolas Museux
THALES Research & Technology
F-91767 Orsay,FRANCE
cause they naturally take advantage of stream-oriented par-
allelism. Even if architecturally speaking this technology
can be seen as an old-fashioned one, it remains easily scal-
able and can be exploited in addition to other techniques.
Such SIMD units are now embedded in SoCs and GPPs
(General Purpose Processors) e.g. AltiVec technology from
Motorola [4] and Streaming SIMD Extensions from In-
tel [1].
Programming such domain-specific applications for
these targets requires data placement and operation schedul-
ing which are NP-complete problems. But the constraint
technology [6] enables the efficient exploration of the com-
binatorial space of tiling and scheduling. A set of cost func-
tions lets the user obtain automatically a variety of solu-
tions, close to expert given ones, that bound the valid design
space.
This article briefly presents our APOTRES tool and its
underlying constraint technology [5, 11] and is focused on
a case study to show how the design space is explored. The
next section describes the application and the architecture.
Section 4 introduces constraint-based models such as tiling
and scheduling, which define the solution space. Section 5
presents our Concurrent Constraint Logic Programming ap-
proach. Then several optimization criteria are considered
such as machine size (7), memory capacity size (8), execu-
tion time with technological or real time constraints (9,10).
Results for the different cost functions are compared in Sec-
tion 11.
2. Application description
The new standard H.264 [13] provides advanced video
coding techniques and has been adopted by many organi-
zations and companies. It increases the compression ratio
by a factor two in comparison to MPEG-2 and thus reduces

the bit rate needed to transfer images of the same quality.
However, H.264 kernels require more computation levels
and their implementation should be adapted to various ar-
chitectures.
We use the fractional sample interpolation of H.264 [13]
as running example. Interpolation is used to determine in-
tensity values at non integer pixel positions to perform mo-
tion compensation. In Figure 1 the grey squares represent
existing pixels. Pixel i is the desired output. A vertical con-
volution produces pixels such as cc, dd, h, m, ee, ff. An
horizontal one produces j from these pixels. Finally, i is
derived from h and j.
Figure 1. Fractional sample interpolation
For our application we compute the 8×8 matrix of pixels
i with the following equations:
H[i, j] = conv(X[i − 2 : i + 3, j])
K[i, j] = conv(H[i, j − 2 : j + 3])
Y [i, j] = K[i, j] + H[i, j]
where X is the 13×13 input matrix, H a 8×13 intermediary
matrix, K a 8 × 8 intermediary matrix, Y the 8 × 8 output
matrix, and conv a 6-tap convolution filter.
2.1. Application code
The interpolation code is expressed in a pseudo-C lan-
guage in Figure 2. It is derived from the equations defining
H, K and Y, code is composed of parallel loops in single as-
signment form. Each loop nest implements a task that reads
one or more multidimensional data arrays and updates one
different array. The call function arguments represent all
the array elements read by the function. The loop nest tasks
are called T0 , T1 , T2 , T3 and T4 .
2
float X[0:12][0:12], Y[0:7][0:7], H[0:7][0:12],
K[0:7][0:7], OUTPUT[0:7][0:7],
COEF1[0:5],COEF2[0:5];
for(i0=0; i0<13; i0++)
for(j0=0; j0<13; j0++)
X[i0][j0] = input();
for(i1=0; i1<8; i1++)
for(j1=0; j1<13; j1++)
H[i][j] = conv(X[i1:i1+5][j1],COEF1[0:5]);
for(i2=0; i2<8; i2++)
for(j2=0; j2<8; j2++)
K[i2][j2] = conv(H[i2][j2:j2+5],COEF2[0:5]);
for(i3=0; i3<8; i3++)
for(j3=0; j3<8; j3++)
Y[i3][j3] = K[i3][j3] + H[i3][j3];
for(i4=0; i4<8; i4++)
for(j4=0; j4<8; j4++)
OUTPUT[i4][j4] = Y[i4][j4];
Figure 2. Interpolation code
3. The global approach
The goal is to find a mapping of the application onto the
machine that satisfies the programming model and architec-
tural constraints. The global optimization problem is de-
composed into many subproblems: memory minimization,
task tiling and scheduling, communication overlapping, re-
spect of dependencies, latency minimization,... each being a
fully optimization problem. Only a concurrent model mod-
ular approach can meet all our needs simultaneously.
Figure 3 illustrates our global approach. A Constraint
Logic Programming Model approach is used (Section 5).
The application, the architecture, the memory, the task
scheduling and tiling, the communication, the data-flow de-
pendencies and the latency are modelized with linear and
non-linear constraints. These models are introduced in Sec-
tions 4 and represent the core of APOTRES.
The tool takes as input the application and architectural
parameters. The processor number, the local and global
memory sizes, the pipeline depth are given in a specification
file. The information coming from the application is auto-
matically extracted using the PIPS compiler. The task time
durations are estimated using [8] and array elements refer-
enced by each task are evaluated by the array region [14].
These pieces of information are taken into account and
propagated together with deduced information from model
to model. During the resolution the models exchange range
information about their variables. The cost function se-
lected among the execution time, memory, architecture cost
and latency guides the resolution through a specific heuris-
tics. A first solution satisfying all the constraints is found.
 Constraint Solver

programming model Resolution

Memory liveness cost function
Scheduling
memory sizes memory Capacity
processor number size
pipeline depth Architecture data nb of events
volume dependence event duration
nb procs

computation Physical Heuristics

task duration Tiling
array volume Time
dataflow dependencies Application dependence
array elements dependence comm latency Solutions
duration

Communication I/O

comm events

Figure 3. The Concurrent Constraint Programming Approach

The solving process is automatic. with

There is never bad solution, since any solution produced
∀l, 0 ≤ L−1 l < 1 (2)
meets all the constraints. To obtain legal results, the tool
−1
designer has to approximate the architecture and the solu- ∀p, 0 ≤ P p < 1 (3)
tion design by proper models. The introduction of a new ∀i ∈ I, 0 ≤ i < b
architectural scheme implies the development of new set of det(L) 6= 0
constraints.
det(P ) 6= 0 and P diagonal by a permutation
The associated triplet (c, p, l) can be interpreted as follows:
4. Models at a logic time c, each processor p runs l iterations.
P and L define a tiling of the iteration domain. The
solver must find the numerical values of their elements.
As explained above the mapping of applications onto the
architecture uses different interacting models. The main
ones used concurrently by the solver are introduced in this
4.2. Scheduling
section.
Schedules are computed with respect to tilings. For each
loop nest, an affine function represents the schedule. A
4.1. Tiling schedule is legal if it respects the data flow dependence con-
straints. A schedule function is

For each loop, the tiling partitions the iteration set and
distributes it along three dimensions: (1) a cyclic temporal
dimension, (2) a “processor” dimension, (3) a local dimen-
sion which exploits the local memory.
The tiling is formally defined in [12] and summarized
here. Let I be the loop nest iteration set (with n loops)
contained in Zn defined by I = {0, . . . , b1 − 1} × · · · ×
{0, . . . , bn − 1} where 1 ≤ k ≤ n, bk ∈ N∗ . Let P and
L be n × n square diagonal integer matrices with non-null
determinant. Then for each point i of I, there exists one
and only one triplet (c, p, l) of points of I such as:
d(c) = α.c + β (4)
where c is the index of a computation block, α a line vector
of the same dimension as c, “.” the standard scalar product
and β an integer. The solver selects values for all α and β.
4.3. Data flow dependence
If a block c′ uses a value defined by a block c, c must
be executed first. This is called a data flow dependence. If
there is a data flow dependence between two computation
blocks c and c′ , then any legal schedule meets the constraint

i = LP c + Lp + l (1) d(c) < d(c′ ) (5)

3
4.4. Memory capacity
A capacitive memory model is used. As all processors
execute the same code, the required memory size is the
same for each processor. Each task is allocated a private
input buffer, but all tasks share a common output buffer.
As soon as they are computed, results are sent to all input
buffers of tasks using them. The output buffer is sized to fit
the output of any task and to support a flip-flop mechanism
used to overlap computations and communications.
The size of a task input buffer is the sum of the spaces
required for each argument. Its capacity is the minimum
obtained with four possible schemes.
1. The full array size;
2. The volume of the hypercube accessed by all cycles;
3. The volume of references accessed per cycle, multi-
plied by the maximal number of live iterations, which
is relevant when overlaping occurs. Liveness informa-
tion is carried by the dependencies and the schedule;
4. The number of referenced accesses per cycle multi-
plied by the maximal number of live iterations, useful
in case of non-contiguous accesses.
The memory constraints are linked to the partitioning, de-
pendence and scheduling parameters.
5. A Concurrent Constraint Logic Program-
ming Model (CCLP)
While some constraints can be translated into linear in-
equations and solved by classical linear programming al-
gorithms, others like resource constraints require non linear
expressions. Solving techniques for both constraints require
the combination of integer programming and search. We
use the Constraint Logic Programming approach.
CCLP handles linear and non linear constraints and
yields, through the concurrent propagation of constraints
over all models, solutions satisfying the global problem.
Figure 3 illustrates our models. The models are linked by
variables that appear concurrently in different models. For
example, Constraint 6 links tiling and architectural models.
The number of processors required by the tiling must be
smaller than the number of processors available for any task
k.
Y
n
k
P rocessorN umber ≥ maxk ( Pi,i ) (6)
i=1
As another example, the computation block index c ap-
pears:
4
• in the tiling model as block index of the tiled loop nest
(Equation 1);
• in the dependence model in the precedence relations
(Equation 5);
• in the scheduling model (Equation 4).
During the resolution process, the models communicate
their partial information, value intervals, about their vari-
ables to others.
The CCLP system builds a solution space on a model-
per-model basis. The global search looks for partial solu-
tion in the different concurrent models. Only relevant in-
formation is propagated between models. Several global
heuristics are used to improve the resolution, e.g. sched-
ule choices are driven by computing the shortest path in the
data-flow graph.
6. Case study
To show the characteristics of our tool, we present in the
next sections various mappings of the application obtained
for three different optimization criteria. These results are
successively taken into account to size the architecture:
• Firstly the cost criterion (Section 7) gives the cheap-
est circuit configuration able to execute the application
meeting or not real-time constraint;
• Secondly the memory minimization criterion (Sec-
tion 8) gives a lower bound for the local memory ca-
pacity per processor;
• Finally the execution time criterion (Sections 9,10) to-
gether with the constraints chosen from the previous
results provides solutions that fit both architectural and
real-time constraints.
We choose a target architecture having up to 16 pro-
cessors, each with a local memory of 128 or 256 bytes.
These parameters are entered as architectural constraints in
APOTRES.
A pipelined multiply-add is used and task durations esti-
mated by PIPS [9], the optimizing compiler used as a front-
end to APOTRES, are respectively 1, 6, 6, 1 and 1 cycles.
7. Cost minimization
The cost of SoCs is key for industrial exploitation. It de-
pends on the surface of a single processor, on the surface of
a memory unit, and on the number of processors and mem-
ory units required by the application.
 The cost of a processor outweighs that of the memory.
So, cost minimization induces solutions where the proces-
sor number is minimal.
Without the memory constraint, APOTRES selects a
one-processor target machine, as could be expected. Table 1
presents the solution using 203 memory bytes. It takes ad-
vantage of the target architecture and pipelines up to 8 or 13
local iterations per computation block.
Table 1. Tilings for cost minimization, 256b
Tasks T0 T1 T2 T3 T4
# processors 1 1 1 1 1 Figure 2. It uses six array elements produced by Task T0 .
# blocks 13 13 8 8 8
# local iterations 13 8 8 8 8 plies only one additional element of storage and all 16 pro-
If a constraint on the memory capacity per processor
such as #mem ≤ 128 is added, APOTRES chooses the Ta-
ble 2 mapping on four processors using 117 bytes each. The
reduction of the pipelined local iterations per computation
block decreases the data liveness and thus the memory used.
In particular, 56 elements of Array H have to be stored be-
tween the two orthogonal convolutions T1 and T2 instead of
104 for the first solution.
Table 2. Tilings for cost minimization, 128b
Tasks T0 T1 T2 T3 T4
# processors 1 4 4 4 4 candidates for our application. Two cases are successively
# blocks 13 13 8 16 16
# local iterations 13 2 2 1 1
APOTRES provides an as soon as possible schedule,
represented in Figure 4. From the α and β scheduling
parameters (Section 4.2), the schedules can be expressed
using regular expressions: (C0 C1 )5 (CALL )8 and
(C0 C1 )6 CALL (C0 C1 C3 C4 CALL )3 (C3 C4 C2 C3 C4 )4 C3 C4
In order to shorten the scheduling formulations, Ci cor-
responds to the computation block with li pipelined local
iterations of Task Ti : Ci = Tili , and CALL is used for
C0 C1 C2 C3 C4 .
The logical durations of the schedules in Figure 4 are
respectively 1355 and 519 cycles, while the memory capac-
ities per processor are 203 and 117 bytes.
8. Memory constraint minimization
In order to size the architecture we wish to know the min-
imum memory required to execute the application on the
target architecture having up to 16 processors.
The best solution found by APOTRES uses only 46 bytes
on the 16 processors. The tiling chosen is shown in Table 3.
5
Table 3. Tilings with memory minimization
Tasks T0 T1 T2 T3 T4
# processors 13 13 16 16 16
# blocks 7 4 4 4 4
# local iterations 2 2 1 1 1
To minimize data storage during the execution of data
flow dependent tasks, computation blocks have only one lo-
cal iteration, except Tasks T0 and T1 . Look at T1 code in
Pipelining two contiguous local iterations of T0 and T1 im-
cessors can be used to exploit the parallelism available in
tasks T2 , T3 and T4 .
The execution time is 117 cycles. Computations are
again scheduled according to an as soon as possible sched-
ule: (C0 )3 (C0 CALL )2 (C1 C2 C3 C4 )2
9. Execution time under memory constraint
Multimedia applications often must meet real-time con-
straints. Here we wish to set the execution time to a value
strictly less than the 500 cycles found in one previous so-
lution in Section 7. Furthermore, the results of Sections 7
and 8 make processors having from 64 to 128 bytes good
studied with memory sizes of 128 and 64 bytes.
To reduce the execution time, APOTRES has to maxi-
mize the number of processors. Table 4 presents a solution
that takes advantage of the available parallelism: processors
and software pipelines.
Table 4. Tilings for time optimization, 128b
Tasks T0 T1 T2 T3 T4
# processors 13 13 16 16 16
# blocks 1 1 1 1 1
# local iterations 13 8 4 4 4
The schedule is not interleaved: C0 C1 C2 C3 C4 . The ex-
ecution time, computed by the solver, is 98 cycles and the
memory capacity required per processor is 72 bytes.
When memory size is limited to 64 bytes, the applica-
tion cannot be executed on the machine without additional
partitioning. APOTRES finds the solution in Table 5 and its
related schedule: C0 13 C1 C2 C3 C4
This solution uses 110 cycles to execute the application
and requires 62 bytes per processor. It actually does not
differ from the previous one, except that Task T0 has been
tiled. Now, at each iteration, Task T0 writes only one array
 T4
T3
T2
T1
T0

Schedule for local memory < 256

T4
T3
T2
T1
T0

Schedule for local memory < 128

Figure 4. Schedules for cost minimization

Table 5. Tilings for time optimization, 64b Table 7. Tilings with time optimization, 4 proc
Tasks T0 T1 T2 T3 T4 Tasks T0 T1 T2 T3 T4
# processors 13 13 16 16 16 # processors 4 4 4 4 4
# blocks 13 1 1 1 1 # blocks 7 7 4 2 2
# local iterations 1 8 4 4 4 # local iterations 7 4 4 8 8

element instead of 13. The output buffer size is smaller and

that is enough to fit the tighten memory constraint.
10. Execution time under processor constraint
In order to compare the solutions efficiencies, the results
have to be computed for the same optimization criterion.
Here are briefly given the mappings of the application on
one and four processors using the execution time as function
cost.
On one processor, the memory capacity required is 241
bytes and the execution time is 1343 cycles.
Table 6. Tilings with time optimization, 1 proc
Tasks T0 T1 T2 T3 T4
# processors 1 1 1 1 1 12. Related work
# blocks 13 13 4 4 4
# local iterations 13 8 16 16 16
On four processors, the memory capacity required per
processor is 125 bytes and the execution time is 367 cycles.
11. Discussion
Table 8 summarizes the different solutions according to
the optimization function used and additional constraints,
hardware or real time requirements.
Typically, to minimize the silicon area of a SoC, the two
main components to take into account are the number of
processors and the memory size of each processor. These
two criteria yield the solution with 1 processor and 203
bytes of local memory. Unfortunately, this solution is exe-
cuted too slowly for a real-time embedded application such
as video decoding. So the number of cycles should also be
considered. Furthermore, decreasing the number of cycles
required to execute an application allows to decrease the
SoC frequency and thus to reduce the SoC surface by tight-
ening the layers. Tradeoffs should be made between speed
and cost. Solutions 4 and 6 are efficient. They satisfy both
the application and architectural constraints.
Using constraint programming to solve such mapping
and scheduling problems can be seen as an automatic ap-
proach. Manual approaches may be preferred by designers
who want to use their own design strategy and control the
whole process. Mapping in G EDAE[2] is performed manu-
ally by allocating tasks to hardware resources and schedul-
ing is based on dynamic heuristics. But architecture fea-
tures such as memory hierarchy or communication paths
are not shown explicitly and managed by the tool itself.
This restricts in practice the scope of usable architectures.
The SPEAR[10] Design Environment removes this limita-

6
 Table 8. Comparative table for the 7 solutions

Optimization Additional # procs Duration Local Total Efficiency Schedule

Function Constraints memory memory
1 Cost mem ≤ 256 1 1355 203 203 0.97 (C0 C1 )5 (CALL )8

2 Exec. Time #proc = 1 1 1343 241 241 1 (C0 C1 )5 (C0 C1 CALL )4

3 Cost mem ≤ 128 4 519 117 468 0.65 (C0 C1 )6 CALL (C0 C1 C3 C4 CALL )3

(C3 C4 C2 C3 C4 )4 C3 C4

4 Exec. Time #proc = 4 4 367 125 500 0.92 (C0 C1 )3 (C0 C1 C2 CALL )2

5 Memory mem ≤ 128 16 117 46 736 0.72 (C0 )3 (C0 CALL )2 (C1 C2 C3 C4 )2

6 Exec. Time mem ≤ 128 16 98 72 1152 0.86 C0 C1 C2 C3 C4

7 Exec. Time mem ≤ 64 16 110 62 992 0.76 C0 13 C1 C2 C3 C4

tion by making an open architectural dimension available to References

P TOLEMY II[3]. This enables rapid exploration of the de-
sign space. These tools make the architecture model and de-
sign choices explicit to guide the design space exploration.
Our work is unique because it takes into account simul-
taneously: the mapping of the complete application, with
scheduling and sizing using tiling, the application memory
requirement and the operational constraints.
13. Conclusion
This article shows how a multimedia application can
be rapidly mapped onto a SIMD architecture. Our map-
ping tool is able to explore the tiling and scheduling spaces
within the combinatorial space of solutions according to dif-
ferent criteria. It enables finding in a few minutes the best
trade-off depending on the embedded real-time constraints
and target cost.
APOTRES is connected to PIPS, a tool that automati-
cally analyzes and transforms codes written in Fortran. An-
other potential input is ANSI-C code whose functional re-
sults can be checked (by THOMSON). Hence our prototyp-
ing chain is nearly seamless in the sense that a multime-
dia code can be parallelized from any standard specification
(sometimes not parallel at all) translated “à la Fortran” or
from a C sequential code.
To make APOTRES more useful, some improvements
are planed in two areas: data communication and code gen-
eration (control, allocation, communication). Communica-
tions have to be modeled with respect to the communication
resources. APOTRES generates integer values which are in-
terpreted as mapping directives. We are currently studying
control code generation using CLooG [7], which generates
an efficient control C code from a description of iteration
domains and schedules.
[1] http://developer.intel.com/design/pentium4/manuals/index new.htm.
[2] http://www.gedae.com.
[3] http://www.ptolemy.eecs.berkeley.edu/ptolemyii.
[4] http://www.simdtech.org/altivec.
[5] C. Ancourt, D. Barthou, C. Guettier, F. Irigoin, B. Jeannet,
J. Jourdan, and J. Mattioli. Automatic data mapping of sig-
nal processing applications. IEEE International Conference
on Application Specific Systems, Architectures and Proces-
sors, pages 350–362, 1997.
[6] K. R. Apt. Principles of Constraint Programming. Cam-
bridge University press, 2003.
[7] C. Bastoul. Efficient code generation for automatic paral-
lelization and optimization. In ISPDC’2 IEEE International
Symposium on Parallel and Distributed Computing, pages
23–30, Ljubjana, october 2003.
[8] B.Creusillet and F.Irigoin. Interprocedural array region anal-
yses. Lecture Notes in Computer Science - Languages and
Compilers for Parallel Computing, pages 46–60, August
1995.
[9] F. Irigoin, P. Jouvelot, and R. Triolet. Semantical interpro-
cedural parallelization: an overview of the pips project. In
ACM International Conference on Supercomputing, ICS’91,
Cologne, Allemagne, June 1991.
[10] E. Lenormand and G. Edelin. An industrial perspective:
Pragmatic high-end signal processing environment at thales.
2003.
[11] J. Mattioli, N. Museux, J. Jourdan, P. Savéant, and
S. de Givry. A constraint optimization framework for map-
ping a digital signal processing application onto a parallel
architecture. In Principles and Practice of Constraint Pro-
gramming, 2000.
[12] N. Museux. Aide au placement d’applications de traitement
du signal sur machines paralleles multi-SPMD. PhD thesis,
Ecole des Mines de Paris, 2001.
[13] T. Wiegand, G. Sullivan, and A. Luthra. Itu-t rec. h.264 —
iso/iec 14496-10 avc - final draft. Technical report, Joint
Video Team (JVT) of ISO/IEC MPEG & ITU-T VCEG,
May 2003.
[14] L. Zhou. Statical and Dynamical analysis of Program Com-
plexity. PhD thesis, Universite P. et M. Curie, 1994.