Received October 13, 2017, accepted November 24, 2017, date of publication November 28, 2017,

date of current version December 22, 2017.

Digital Object Identifier 10.1109/ACCESS.2017.2778220

Reconfigurable VLSI Architecture for

Real-Time 2D-to-3D Conversion
WEIZHI XU 1,2 , SHOUYI YIN2 , ZHEN ZHANG2 , HAO DONG2 ,
RUI SHI2 , LEIBO LIU2 , AND SHAOJUN WEI2
1 School of Information Science and Engineering, Shandong Normal University, Jinan 250014, China
2 Institute of Microelectronics, Tsinghua University, Beijing 100084, China
Corresponding author: Shouyi Yin (yinsy@tsinghua.edu.cn)
This work was supported in part by the China Major S&T Project under Grant 2013ZX01033001-001-003, in part by NNSF of China
under Grant 61774094, Grant 61602285, and Grant 61602284, and in part by the Shandong Provincial Natural Science Foundation, China,
under Grant ZR2015FQ009.

ABSTRACT 2-D-to-3-D conversion is one way to make full use of 2-D contents to produce 3-D contents.
Real-time 2-D-to-3-D conversion is required for 3-D consumer electronic devices, which demands fast
processing speed especially for high-definition videos. In this paper, we propose a reconfigurable VLSI
architecture for real-time 2-D-to-3-D conversion. Two different depth-retrieval methods are implemented in
this architecture in order to support the choice of a best method or combination of different methods. The
proposed architecture can also support different resolutions (4K, 1080p, and 720p) and the original view
can be configured as either left view or right view. In order to overcome the problem of ‘‘Memory Wall’’,
we propose a data reuse method to reduce memory traffic for the proposed architecture so that the overall
performance is improved to realize real-time conversion. The experiment result shows that the implemented
2-D-to-3-D architecture can achieve state-of-the-art throughput (4K@30f/s).

INDEX TERMS 2D-to-3D conversion, data reuse, memory traffic, real-time, reconfigurable.

I. INTRODUCTION Different depth-retrieval methods may be effective in

Three-dimensional video is becoming more and more popular
and attracting more and more viewers to experience vivid
stereoscopic visual effects. More and more common con-
sumer electronics can play 3D video, such as 3D-TV, laptop,
tablet and so on. However, it is costly and time-consuming
to produce high quality 3D contents directly. To alleviate the
problem of 3D material shortage, converting 2D to 3D is one
way to make full use of the already existing 2D materials.
The difference between 2D contents and 3D contents is that
3D contents contain the depth information which represents
the relative distances between the objects. The generation
of comfortable and natural depth is important for 2D-to-3D
conversion. There are two steps for 2D-to-3D conversion.
First, a depth map with the depth values of all pixels is
extracted from a 2D picture. Second, depth-image-based ren-
dering (DIBR) is applied to generate new views according to
the original view and the depth map. Semiautomatic depth
map generation [1]–[4] is very time-consuming. Therefore,
algorithms for automatic depth map generation are proposed
to reduce the time overhead [5]–[12].
different scenarios, so it is necessary to support differ-
ent depth-retrieval methods in a system for selection of
a best method or optimal combination of different meth-
ods [10]. On the other hand, real-time 2D-to-3D conversion is
required for 3D consumer electronic devices, which demand
fast processing speed especially for high-definition videos.
‘‘Memory Wall’’ [13] is an obstacle for real-time conver-
sion, which rises from the performance gap between com-
putation speed and memory access speed. A few VLSI
architectures are proposed to accelerate real-time 2D-to-3D
conversion or view synthesis [14]–[17], but these architec-
tures did not consider the depth-retrieval part of 2D-to-3D
conversion. A CPU+GPU platform is proposed to achieve
1080p@30fps 2D-to-3D conversion in [18]. After the
optimization of the 2D-to-3D conversion algorithm,
1080p@36fps is achieved on an Intel Core i7 processor [19].
In this paper, we propose a reconfigurable VLSI archi-
tecture for real-time 2D-to-3D conversion. The proposed
architecture is a complete (including both depth generation
and DIBR) 2D-to-3D VLSI architecture for high-definition

2169-3536
2017 IEEE. Translations and content mining are permitted for academic research only.
26604 Personal use is also permitted, but republication/redistribution requires IEEE permission. VOLUME 5, 2017
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
W. Xu et al.: Reconfigurable VLSI Architecture for Real-Time 2D-to-3D Conversion

videos. The real-time performance of the 2D-to-3D archi-

tecture can reach 1080p@120fps or 4K@30fps. The pro-
posed architecture can support two different depth-retrieval
methods, edge-based method and color-edge-based method,
in order to support the choice of a best method or the
best combination of different methods [10]. The proposed
architecture can also support different input/output video
resolutions, such as 4K, 1080p and 720p. The inputted
original view can be outputted as left view or right view
according to the habits of the viewers. The hardware mod-
ules are reused by different configurations to reduce the
hardware overhead. In order to overcome the problem of
‘‘Memory Wall’’, we propose different levels of data reuse FIGURE 1. Workflow of Sobel edge detection.
to reduce memory traffic so that the overall performance is
improved to realize the real-time 2D-to-3D conversion. The
proposed data reuse method is implemented in the on-chip
memory design of the 2D-to-3D architecture. The contribu-
tions of this paper are as follows. (1) A real-time complete
2D-to-3D VLSI architecture is proposed and implemented.
(2) The proposed architecture is reconfigurable to support FIGURE 2. The templates for Sobel edge detection.
different depth-retrieval methods and different resolutions.
(3)The proposed data reuse method is useful to reduce the
memory traffic not only for 2D-to3D conversion but also for Y direction gradient (Gy) and the threshold gradient (Gth) are
other similar applications. similar to that of X direction gradient and two other templates
The rest of the paper is organized as follows. The 2D-to-3D in Fig. 2 are used. The three templates share the same input
algorithms supported by the proposed architecture are intro- pixels. The final gradient G is computed according to (2) and
duced in Section II. The proposed 2D-to-3D VLSI architec- compared with the threshold (Gth) to determine whether the
ture is presented in Section III. Experiment results are shown pixel is on an edge. The Sobel operation results in the output
and analyzed in Section IV. Section V is the conclusion. of binary values 1 (on the edge) and 0 (not on the edge).
Then the binary values 1 and 0 are transformed to values
II. 2D-TO-3D ALGORITHMS of 255 and 0 respectively and an initial depth map is generated
In this section, we present the two 2D-to-3D algorithms for the input image [20].
supported by the proposed architecture. The two algorithms
Gx = (−1) × f (x − 1, y − 1) + 0 × f (x, y − 1)
are both divided into two parts, depth-retrieval and DIBR.
For the two 2D-to-3D algorithms, the depth-retrieval parts are + 1 × f (x + 1, y − 1) + (−2) × f (x − 1, y)
different while the DIBR parts are the same. + 0 × f (x, y) + 2 × f (x + 1, y) + (−1)
× f (x − 1, y + 1) + 0 × f (x, y + 1)
A. TWO METHODS TO RETRIEVE DEPTH MAP
+ 1 × f (x + 1, y + 1) (1)
Two depth-retrieval methods, edge-based method and edge- q
color-based method, are implemented in the proposed archi- G = G2x + G2y (2)
tecture to generate the initial depth map. The two methods are
presented in this subsection. 2) EDGE-COLOR-BASED METHOD
The edge-color-based method considers two depth cues, edge
1) EDGE-BASED METHOD and color [18]. There are three steps to generate the initial
The edge-based method uses the edge information in the depth map.
image as the depth cue. Sobel operator is used for edge detec- First, we use Sobel edge detection to judge whether a pixel
tion. The Sobel edge detection computes the edge direction of belongs to an edge, which is the same as the edge-based
target pixel and checks whether the target pixel is on an edge. method. The outputs of this step are 1 (on the edge) and
The workflow of Sobel edge detection is described 0 (not on the edge), which are sent to the next step.
in Fig. 1. The computation for edge detection utilizes the first Second, using the edge information, a global depth for
two templates (matrices) in Fig. 2 to computes the gradients each row is obtained from the cumulative horizontal edge
of the target pixel in X direction (Gx) and Y direction (Gy) histogram, according to (3). Depthglobal is the global depth
respectively. This step needs eight reference pixels around the for a row, which means that all the pixels in one row have the
target pixel to multiply by the two templates. For example, same global depth. Edgecount is the number of edge pixels for
Gx is computed as in (1), where f(x,y) is the pixel in a the current row. Edgesum is the total number of edge pixels in
frame with the coordinate of x and y. The computations of the previous frame. The previous frame is used because the

VOLUME 5, 2017 26605

 W. Xu et al.: Reconfigurable VLSI Architecture for Real-Time 2D-to-3D Conversion
number of edge pixels in the current frame is not known until
the end of the frame for our hardware implementation and
the number of edge pixels is usually very close between two
adjacent frames. Therefore, Edgesum needs to be updated at
the end of each frame.
Edgecount
Depthglobal = × 255 (3)
Edgesum
At last, the combination of Y , Cb and Cr color channels
is used to refine the global depth to achieve a local depth
according to (4), where f1 , f2 and f3 are linear functions. The
local depths of all the pixels in the current frame compose the
initial depth map.
Depthlocal = Depthglobal × f1 (Cr ) × f2 (Cb ) × f3 (Y ) (4)
B. DIBR
Depth-image-based rendering (DIBR) is used to generate
the new view from the original 2D video sequence and the
extracted depth information (depth map). DIBR includes four
parts (Fig. 3), pre-processing of the initial depth map, convert-
ing depth to disparity, pixel shifting and hole filling [21], [22].
FIGURE 3. The workflow of DIBR.
First, as a pre-processing step, a Gaussian filter is used to
smooth the initial depth map and the final depth value for
each pixel is generated. The Gaussian filter employs a 5 × 5
template (Fig. 4), which needs 24 reference pixels around the
target pixel.
FIGURE 4. Template for Gaussian filter.
Second, the smoothed depth value is converted to the dis-
parity between the position of the pixel in the original view
and that in the new view, according to (5). D represents the
viewing distance from the display. tc is the interpupillary dis-
tance which is the human eye separation. Z is the normalized
depth value for each pixel.
D pixel belongs to an edge, which is shared by the two depth-
disparity = xR − xL = tc (1 − ) (5)
D−Z
Third, the pixels of the new view are generated according
to the disparity and the original view. However, there may
be some holes between pixels of the new view because some
of the pixels fall in the same position after pixel shifting.
Therefore, at last, hole-filling is applied to fill color into these
holes if there are still holes.
26606
III. PROPOSED ARCHITECTURE FOR
2D-TO-3D CONVERSION
In this section, we first present the overall architecture (Fig. 5)
for real-time 2D-to-3D conversion, which supports the choice
of a best method or optimal combination of different methods.
Then we explain each module of the proposed architecture in
detail.
FIGURE 5. Overall architecture for 2D-to-3D conversion.
A. OVERALL ARCHITECTURE
The 2D-to-3D architecture adopts a pixel-level pipeline.
The 2D sequence is inputted pixel by pixel and outputted
unchanged as the original view. The new view is generated
after the processing of two modules, depth map generation
and DIBR. The input buffer is designed as a line buffer to store
several lines of the original image in order to be accessed by
different modules of the proposed 2D-to-3D architecture at
the same time. The configuration module can be configured
to select a best depth generation algorithm for the depth
map generation module, which supports two different depth
map generation methods. The configuration module can also
be configured to support different resolutions, such as 4K,
1080p or 720p. The depth map generation module is used
to generate an initial depth map. After that, DIBR module
processes the initial depth map and generates a new view.
Depth map buffer stores several lines of the initial depth map,
which will be pre-processed by Gaussian filter in the DIBR
module.
B. RECONFIGURABLE DEPTH MAP GENERATION
The depth map generation module supports two depth-
retrieval methods, an edge-based method and an edge-color-
based method (Fig. 6). The Sobel module detects whether a
retrieval methods. The global depth module and local depth
module are only used for the edge-color-based method to
generate the global depth and the local depth respectively. The
value conversion module is used by the edge-based method to
convert the values 1 and 0 to 255 and 0 respectively to get an
initial depth. The initial depth values generated by the best
depth-retrieval method can be selected through configuration
VOLUME 5, 2017
W. Xu et al.: Reconfigurable VLSI Architecture for Real-Time 2D-to-3D Conversion
FIGURE 6. Architecture for the reconfigurable depth map generation. The
white module is reusable between two depth-retrieval methods while the
grey modules are not.
FIGURE 7. Architecture of Sobel module.
module (Algorithm selection) according to the content of the
image, and then stored to the depth map buffer.
1) SOBEL MODULE
The Sobel module (Fig. 7) checks whether the target pixel
is on the edge. It receives three pixels each cycle from the
input buffer. The three pixels are from adjacent three lines in
the input buffer. There is a register array for storing the input
pixels because nine pixels (3×3) are needed each cycle for
the computing units of Sobel module.
The fx module and the fy module computes the gradients
of target pixel in x direction and y direction respectively,
using the first two templates in Fig. 2. The threshold module
computes the threshold with the third template in Fig. 2 and
reads the same nine pixels (3×3) as fx module or fy module
from the register array. The fx module, the fy module and
the threshold module all use multiple multipliers to finish
the multiplications in parallel (Fig. 8). After that, an adder
tree is implemented to finish the additions and the gradi-
ents (Gx, Gy and Gth) are generated. This design ensures the
fastest processing speed for Sobel edge detection.
Final gradient module computes the combined gradient G
of x direction and y direction, according to (2). After that,
we can check whether the target pixel is on an edge by
comparing G with Gth.
VOLUME 5, 2017
FIGURE 8. Parallel multipliers with an adder tree for four modules, fx, fy,
threshold and Gassian filter.
FIGURE 9. Architecture of global depth module.
2) GLOBAL DEPTH MODULE
The architecture of the global depth module is shown
in Fig. 9. It generates the global depth for each row. In this
design, Row/Col Count module gives control commands to
determine the end of a line or a frame, according to input
signals from the Sobel module (Edge_ready) and the configu-
ration module (Width/Height of the frame). The edge count of
the current row is accumulated in the Edge_row module. The
Edge_frame module keeps both the accumulated edge count
in the current frame and the sum of edge count in the previous
frame. The global depth of a row is computed by Depth_comp
module when Row/Col Count module finds the end of this
row, using the edge count of the current row in Edge_row
and the edge count of the previous frame in Edge_frame
according to (3). Edgesum is supposed to be edge count of
the current frame, but we use previous frame to enable the
pipeline structure. Therefore, overflow should be avoided in
the Depth_comp module.
3) LOCAL DEPTH MODULE
When the global depth of a row is inputted to the local depth
module, as shown in Fig. 10, local depths of all pixels in
this row are computed one by one according to (4). A depth
register is used to store the global depth of the current row.
26607
 W. Xu et al.: Reconfigurable VLSI Architecture for Real-Time 2D-to-3D Conversion
FIGURE 10. Architecture of local depth module.
One 24-bit pixel from the input buffer is divided to three 8-bit
data, Cr_data, Cb_data and Y_data. Three linear functions are
computed in parallel by three modules, F(Cr), F(Cb) and F(Y)
respectively. The final multiplication of the four values in (4)
is performed in pairs for less delay, (DepthGlobal , f1 (Cr )) as a
pair and (f2 (Cb ) , f3 (Y )) as a pair. Therefore, there are three
multipliers in the Multiply module for (DepthGlobal ×f1 (Cr )),
(f2 (Cb ) × f3 (Y )) and the final local depth (Local_depth_data)
respectively.
FIGURE 11. DIBR Architecture.
C. DIBR ARCHITECTURE
The DIBR architecture includes four modules (Fig. 11),
Gaussian Filter, Disparity, Shift and Hole Fill. As a pre-
processing module, Gaussian Filter smoothes the initial depth
map from the depth map buffer and the final depth value for
each pixel is generated. The final depth value is converted to
the disparity by the disparity module. The pixels of the new
view are generated by the shift module using the disparity and
the original view. The hole fill module is applied to fill color
into the holes generated by the shift module.
1) GAUSSIAN FILTER
The Gaussian filter reads the depth values from the depth map
buffer and employs a 5 × 5 template (Fig. 4) to smooth the
depth map. A highly parallel architecture is implemented to
ensure real-time processing (Fig. 8). The design of Gaussian
filter module is very similar to that of fx in Sobel module
except that the scale of the Gaussian filter is larger. First,
a multiplier array with 25 multipliers is used to do the
25 multiplications in parallel. Then, an adder tree is used to
implement the sums of 25 values in the lowest latency.
26608
FIGURE 12. Architecture of Disparity, Shift and Hole fill.
2) DISPARITY, SHIFT AND HOLE FILL
The disparity module receives the smoothed depth value from
the Gaussian filter one by one (Fig. 12). The depth value is
first normalized and then transformed to the disparity accord-
ing to (5). The Shift module first transforms the disparity to
the address of the register array with a decoder. Then the
pixels of the original view from the input buffer are stored
into the register array according to the address. The L/R signal
from the Configurations module is used to choose whether
the conversion is from left view to right view or reverse. The
Hole fill module is used to fill color into the holes which are
generated after the pixel shifting. The pixel from the Shift
module is first judged whether it is a hole or not. If it is a
hole, the pixels near it are used to fill the hole. There is also
a register array to store the line of pixels for hole filling. The
output data from the hole fill module are the final data of the
new view.
D. MEMORY TRAFFIC REDUCTION FOR
REAL-TIME CONVERSION
We propose three levels of data reuse to reduce the memory
traffic for the real-time 2D-to-3D architecture. Exploiting
data reuse can improve the performance of memory access
and help overcome the ‘‘Memory Wall’’ to ensure the real-
time processing. The data reuse method is implemented in
the buffers design for Sobel module and Gaussian module
to reduce the both the off-chip and on-chip memory traffic
in the proposed architecture. Sobel and Gaussian are two
widely used methods in many applications, so the data reuse
method proposed in this paper can also be used in other
applications. Some basic concepts are explained in Table 1.
One frame or depth map is divided into many blocks (BK)
with the block size N×N. A block strip (BS) represents a row
VOLUME 5, 2017
W. Xu et al.: Reconfigurable VLSI Architecture for Real-Time 2D-to-3D Conversion

TABLE 1. Basic concepts.

FIGURE 14. Level BK+ data reuse in a frame or a depth map.

FIGURE 13. Level BK data reuse in a frame or a depth map.

FIGURE 15. Level BS data reuse in a frame or a depth map.

TABLE 2. Memory traffic and buffer size for different data reuse levels.

of blocks. W and H stand for the width and height of a frame

respectively.

1) DIFFERENT LEVELS OF DATA REUSE

For Sobel edge detection or Gaussian filter, a block-based
processing mode is used in a frame or in a depth map. This
processing mode contains at least two levels of data reuse,
block (BK) level and block strip (BS) level. For the BK
level data reuse (Fig. 13), an (N−1)×N buffer is needed for
reusable data between two neighboring blocks. For the BS
level data reuse (Fig. 15), a W×(N−1) buffer is used for
storing reusable data between two neighboring block strips.
With no data reuse (Table 2), the memory traffic is supposed
to be N×N×W×H pixels per frame because W×H blocks
with the size of N×N need to be loaded. In this case, no buffer
is needed. The memory traffic is reduced to N×W×H and
W×H for the two data reuse levels, BK and BS respectively.
However, a more efficient data reuse level usually requires a
larger buffer, (N−1) ×N for BK and (N−1) ×W for BS.
There is another data reuse level between BK and BS,
which is named as BK+ (Fig. 14). In each step of BK+,
FIGURE 16. Input buffer design with a Level BS data reuse.
N +n pixels are loaded into the buffer and there are n+1 target
lines or block strips which are processed at the same time.
The memory traffic of BK+ is (N+n)×W×H/(n+1) with the
buffer size (N−1) × (N+n) (Table 2). The different data reuse
levels gives a good trade-off between the memory traffic and
buffer size. According to Table 2, we can choose a proper
data reuse level for a given application scenario. For example,
we can choose Level BS if the performance is important while
the buffer size is sufficient.
2) IMPLEMENTATION OF DATA REUSE FOR SOBEL MODULE
According to the proposed data reuse methods, we design the
input buffer to implement the BS level data reuse (Fig. 16)

VOLUME 5, 2017 26609

 W. Xu et al.: Reconfigurable VLSI Architecture for Real-Time 2D-to-3D Conversion

FIGURE 17. Register array design for Sobel module.

FIGURE 18. Depth map buffer designed for Gaussian filter.

TABLE 3. Memory traffic and buffer size of different data reuse levels for
Sobel to proecess one frame.
shifted into the register array. Nine pixels are outputted to the
computing logic of Sobel module at each cycle. In this way,
the memory traffic between the input buffer and the register
array is reduced to 3×W×H, which is 1/3 of the case with no
data reuse.

3) DEPTH BUFFER IMPLEMENTATION FOR GAUSSIAN FILTER

The depth map in our design is produced by the depth map
generation module on chip instead of storing the depth map
in the off-chip memory. A depth map buffer is designed
to store the depth map on chip (Fig. 18). The depth buffer
avoids storing the depth data to the off-chip memory and
then loading them to the on-chip memory so it can reduce
and a register array to implement the BK level data reuse
a lot of off-chip memory traffic and can improve the overall
(Fig. 17) so that a two-level data reuse method is implemented
performance of 2D-to-3D. The structure of the depth map
for Sobel edge detection module.
buffer is similar to that of the input buffer. The difference is
The input buffer contains four LineStorages (Fig. 16),
that the 8-bit depth data are stored in the depth map buffer
each of which can store a line of pixels and works as a
instead of 24-bit pixels. Because the window size of Gaussian
shifting buffer. A shifting buffer means that the pixels in
filter is 5×5 in our design, four DepthStorages are used in
the buffer are shifted from the left to the right in the buffer.
the depth map buffer. Each DepthStorage contains a line of
The first LineStorage receives pixels one by one from the
depth data.
input. The other LineStorages receive pixels one by one from
In our implementation, Level BK data reuse is also
their neighboring LineStorages. The pixels from the top two
employed. A 4×5 register array is designed to reduce mem-
LineStorages are used by DIBR module and local depth
ory traffic from depth map buffer to Gaussian filter module
module respectively. The length of LineStorage is designed
on chip, which is similar to the shifting register array for the
to be 4096 pixels to support both the maximum resolution
Sobel module.
of 4K and other resolutions less than 4K.
For Sobel edge detection, the block size N equals 3. Two TABLE 4. Memory traffic and buffer size for different data reuse levels of
LineStorages are used for Sobel to store two lines of pixels. gaussian filter.
The input pixel (Pixel_in) is directly used as the third pixel
line of Sobel edge detection. In this way, Level BS data reuse
is realized in the input buffer for Sobel edge detection. The
number of pixels loaded from off-chip is W×H when using
the input buffer (SRAM) for the Sobel module, while the
number of pixels from off-chip is 9×W×H with no data reuse
(Table 3). The input buffer size is 2×W to implement the
Level BS data reuse for Sobel edge detection.
For Sobel edge detection, a shifting register array is also
designed for implementing Level BK data reuse (Fig. 17).
Six (2×3) registers form a two-column register array. In each The memory traffic and buffer size of different data reuse
cycle, three pixels are received from the input buffer and levels for Gaussian filter are listed in Table 4 with N=5

26610 VOLUME 5, 2017

W. Xu et al.: Reconfigurable VLSI Architecture for Real-Time 2D-to-3D Conversion
and n=2. If there is no depth buffer on chip (no data reuse),
the off-chip memory traffic is at least 25×W×H bytes. The
reason is that the depth data are needed to be loaded from
off-chip memory. In our implementation, all depth data are
stored and reused on chip and no depth data need to be
loaded from off-chip memory. Level BK data reuse is used
to reduce the memory traffic on chip between the depth
map buffer and the 4×5 register array for Gaussian filter.
If Level BK+ or Level BS is used on chip, the memory
traffic is reduced and buffer size is increased, compared with
Level BK.
IV. EXPERIMENTAL RESULTS
In this section, we first give the results (red-cyan images) of
the two 2D-to-3D conversion algorithms which are supported
by the proposed architecture. Then we describe the FPGA
platform for implementing the 2D-to-3D architecture. At last,
we give the synthesis results and comparison with other
architectures and other platforms like CPU and GPU.
FIGURE 19. The red-cyan images achieved by the edge-color-based
method.
FIGURE 20. The red-cyan images achieved by the edge-based method.
A. 2D-TO-3D RESULTS
We give the red-cyan images which are generated by the two
2D-to-3D methods supported by the proposed architecture
in Fig. 19 and Fig. 20 respectively. We can see that the
edge-color-based method usually achieves better results than
the edge-based method. However, the focus of this paper is
not on the algorithm improvement but on the architecture
innovation. The proposed architecture gives a chance for
the selection of a best method or combination of different
methods.
B. FPGA IMPLEMENTATION OF THE
PROPOSED ARCHITECTURE
We have implemented the proposed 2D-to-3D architec-
ture on a Xilinx Spartan-6 FPGA platform (Fig. 21). The
2D sequence is inputted from the personal computer (PC)
VOLUME 5, 2017
FIGURE 21. The 2D-to-3D platform with a Xilinx Spartan-6 FPGA chip,
which is used to implement and verify the proposed 2D-to-3D
architecture.
FIGURE 22. Photo of the real-time 2D-to-3D FPGA Platform.
through the HDMI interface (HDMI TX/RX) to the evalu-
ation board. The 2D-to-3D module is on the FPGA chip with
two HDMI interfaces. The output of the 2D-to-3D module
is displayed on a 3D-TV through an HDMI interface. The
HDMI chips and the FPGA chip are configured by the I2C
bus which is connected with the PC through USB interface.
Fig. 22 gives the photo of our FPGA platform.
C. ASIC IMPLEMENTATION OF THE
PROPOSED ARCHITECTURE
To the best of our knowledge, there is not a complete
2D-to-3D VLSI architecture in literature. Therefore, we
compare the implementation of our design with a VSRS
design [17] and a DIBR design [23] (Table 5).
We implemented the proposed 2D-to-3D architecture in
synthesizable Verilog HDL. Synthesis results using TSMC
65GP technology are given in Table 5. The gate count
(809.0K) and area (1.76mm×1.79mm) demonstrate that our
design costs a small hardware resource. Compared with
the other two implementations, our implementation gets the
state-of-the-art performance, 4K@30f/s. Fig. 23 gives the
layout of the chip. In the proposed design, we employ module
reuse between different configurations to reduce the chip
area. For example, two depth-retrieval methods can share the
Sobel module, the input buffer, the depth map buffer and the
DIBR module.
26611
 W. Xu et al.: Reconfigurable VLSI Architecture for Real-Time 2D-to-3D Conversion
TABLE 5. Implementation of the proposed 2D-to-3D architecture and
comparison with other implementations.
FIGURE 23. Layout of the 2D-to-3D chip.
TABLE 6. Performance comparison with FPGA, CPU and GPU.
2D-to-3D algorithm was also implemented on other plat-
forms, such as CPU [19], CPU+GPU [18], and FPGA [14].
Table 6 gives the performance comparison between different
platforms. In [19], the 2D-to-3D application is implemented
on a 4-core laptop system, with one Intel Core i7-2820QM
processor running at 2.3 GHz. Each core is equipped with
26612
a 32KB L1 data cache, a 32KB L1 instruction cache and a
256KB L2 cache. The four cores share an 8MB L3 unified
cache. In [18], the system is implemented on a notebook
computer. CPU of the notebook is a 1.60 GHz quad-core
CPU with 6M cache, featuring simultaneous multithreading.
The notebook also has a 1.375 GHz GPU with 7 stream
processors inside. Each stream processor consists of 16 cores.
As an FPGA implementation [14], DIBR is implemented on
a Xilinx Virtex-5 FPGA platform. We find that the proposed
design can achieve higher throughput than the other three
platforms.
V. CONCLUSION
In this paper, we propose a reconfigurable VLSI architecture
for real-time 2D-to-3D conversion. The proposed architecture
can support two depth-retrieval methods and different resolu-
tions. We implement a novel data reuse method to reduce the
memory traffic for real-time conversion. Experiment results
show that our design achieves state-of-the-art performance,
which is higher than the performance of implementations on
various platforms (FPGA, ASIC, CPU and GPU).
ACKNOWLEDGMENT
The authors would like to thank all the reviewers for
their valuable comments. The authors also want to thank
Xiudong Li for his contribution to this work.
REFERENCES
[1] Royal Philips Electronics. (2008). Whitepaper WOWvx BlueBox. [Online].
Available: http://www.business-sites.philips.com/global/en/gmm/images/
3d/3dcontentceationproducts/downloads/BlueBox?white?paper.pdf
[2] C. Wu, G. Er, X. Xie, T. Li, X. Cao, and Q. Dai, ‘‘A novel method for
semi-automatic 2D to 3D video conversion,’’ in Proc. 3DTV Conf. True
Vis. Capture, Transmiss. Display 3D Video, May 2008, pp. 65–68.
[3] H.-M. Wang, C.-H. Huang, and J.-F. Yang, ‘‘Depth maps interpolation from
existing pairs of keyframes and depth maps for 3D video generation,’’ in
Proc. IEEE Int. Symp. Circuits Syst., May/Jun. 2010, pp. 3248–3251.
[4] H.-M. Wang, C.-H. Huang, and J.-F. Yang, ‘‘Block-based depth maps
interpolation for efficient multiview content generation,’’ IEEE Trans.
Circuits Syst. Video Technol., vol. 21, no. 12, pp. 1847–1858, Dec. 2011.
[5] I. A. Ideses, L. P. Yaroslavsky, B. Fishbain, and R. Vistuch, ‘‘3D from
compressed 2D video,’’ Proc. SPIE, vol. 6490, pp. 64901C-1–64901C-8,
Mar. 2007.
[6] P. Favaro and S. Soatto, ‘‘A geometric approach to shape from defocus,’’
IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 3, pp. 406–417,
Mar. 2005.
[7] W. J. Tam, C. Vázquez, and F. Speranza, ‘‘Three-dimensional TV: A novel
method for generating surrogate depth maps using colour information,’’
Proc. SPIE, vol. 7237, pp. 72371A-1–72371A-9, Feb. 2009.
[8] J. Zhang, Y. Yang, and Q. Dai, ‘‘A novel 2D-to-3D scheme by visual
attention and occlusion analysis,’’ in Proc. 3DTV Conf., True Vis. Capture
Transmiss. Display 3D Video, May 2011, pp. 1–4.
[9] H. Dong et al., ‘‘An automatic depth map generation for 2D-to-3D conver-
sion,’’ in Proc. 18th IEEE Int. Symp. Consum. Electron. (ISCE), Jun. 2014,
pp. 1–2.
[10] T.-C. Chen et al., ‘‘Brain-inspired framework for fusion of multiple
depth cues,’’ IEEE Trans. Circuits Syst. Video Technol., vol. 23, no. 7,
pp. 1137–1149, Jul. 2013.
[11] Z. Zhang, S. Yin, L. Liu, and S. Wei, ‘‘Real-time time-consistent
2D-to-3D video conversion based on color histogram,’’ in Proc. IEEE Int.
Conf. Consum. Electron. (ICCE), Jan. 2015, pp. 179–180.
[12] H. Dong, S. Yin, G. Jiang, L. Liu, and S. Wei, ‘‘An Automatic Depth Map
Generation Method by Image Classification,’’ in Proc. IEEE Int. Conf.
Consum. Electron. (ICCE), Jan. 2015, pp. 179–180.
VOLUME 5, 2017
W. Xu et al.: Reconfigurable VLSI Architecture for Real-Time 2D-to-3D Conversion

[13] W. A. Wulf and S. A. McKee, ‘‘Hitting the memory wall: Implications ZHEN ZHANG received the B.S. degree in micro-
of the obvious,’’ ACM SIGARCH Comput. Archit. News, vol. 23, no. 1, electronics from Tsinghua University, China,
pp. 20–24, Mar. 1995. in 2010, where he is currently pursuing the Ph.D.
[14] Y.-K. Lai, Y.-C. Chung, and Y.-F. Lai, ‘‘Hardware implementation for real- degree. His current research interests include VLSI
time 3D rendering in 2D-to-3D conversion,’’ in Proc. IEEE Int. Symp. SoC design and high-level synthesis of CGRA.
Circuits Syst. (ISCAS), May 2013, pp. 893–896.
[15] Y.-C. Fan, B.-L. Lin, and S.-Y. Chou, ‘‘Hardware structure of 2D to 3D
image conversion system for digital archives,’’ in Proc. IEEE 17th Int.
Symp. Consumer Electron. (ISCE), Jun. 2013, pp. 111–112.
[16] S.-F. Hsiao, J.-W. Cheng, W.-L. Wang, and G.-F. Yeh, ‘‘Low latency design
of Depth-Image-Based Rendering using hybrid warping and hole-filling,’’
in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), May 2012, pp. 608–611.
[17] Y.-R. Horng, Y.-C. Tseng, and T.-S. Chang, ‘‘VLSI architecture for real-
time HD1080p view synthesis engine,’’ IEEE Trans. Circuits Syst. Video
Technol., vol. 21, no. 9, pp. 1329–1340, Sep. 2011. HAO DONG received the B.S. degree from the
[18] S.-F. Tsai, C.-C. Cheng, C.-T. Li, and L.-G. Chen, ‘‘A real-time 1080p 2D- School of Instrumentation Science and Optoelec-
to-3D video conversion system,’’ IEEE Trans. Consum. Electron., vol. 57, tronics Engineering, Beihang University, Beijing,
no. 2, pp. 915–922, May 2011. China, in 2012. He is currently pursuing the
[19] J. Feng, Y. Chen, E. Li, Y. Du, and Y. Zhang, ‘‘Accelerating 2D-to-3D video M.S. degree with the Institute of Microelectronics,
conversion on multi-core systems,’’ in Proc. 11th Int. Conf. Appl. Parallel Tsinghua University, Beijing. His main research
Sci. Comput. (PARA), Jun. 2012, pp. 410–421.
interests include computer vision and VLSI SoC
[20] W. J. Tam, A. S. Yee, J. Ferreira, S. Tariq, and F. Speranza, ‘‘Stereoscopic
design.
image rendering based on depth maps created from blur and edge informa-
tion,’’ Proc. SPIE, vol. 5664, pp. 104–115, Mar. 2005.
[21] W.-Y. Chen, Y.-L. Chang, S.-F. Lin, L.-F. Ding, and L.-G. Chen, ‘‘Effi-
cient depth image based rendering with edge dependent depth filter and
interpolation,’’ in Proc. IEEE Int. Conf. Multimedia Expo, Amsterdam,
The Netherlands, Jul. 2005, pp. 1314–1317.
[22] L. Zhang, C. Vázquez, G. Huchet, and W. J. Tam, ‘‘DIBR-based conversion RUI SHI received the B.S. degree from Tsinghua
from monoscopic to stereoscopic and multi-view video,’’ in 3D-TV System University, Beijing, China, in 2012, where he
With Depth-Image-Based Rendering. New York, NY, USA: Springer, 2013, is currently pursuing the M.S. degree with the
pp. 107–143. Institute of Microelectronics. His research inter-
[23] W.-Y. Chen, Y.-L. Chang, H.-K. Chiu, S.-Y. Chien, and L.-G. Chen, ‘‘Real- ests include mobile computing and computing
time depth image based rendering hardware accelerator for advanced
architecture.
three dimensional television system,’’ in Proc. IEEE Int. Conf. Multimedia
Expo (ICME), Jul. 2006, pp. 2069–2072.

WEIZHI XU received the Ph.D. degree in

computer architecture from the Institute of Com-
puting Technology, Chinese Academy of Sci-
ences, China, in 2013. He was a Post-Doctoral
Researcher with the Institute of Microelectron-
ics, Tsinghua University. He is currently with
Shandong Normal University as an Assistant
Professor. His research interests include high per-
formance computing and parallel processing.
LEIBO LIU received the B.S. degree in electronic
engineering from Tsinghua University, Beijing,
China, in 1999, and the Ph.D. degree from the
Institute of Microelectronics, Tsinghua University,
in 2004. He is currently serves as an Associate
Professor with the Institute of Microelectronics,
Tsinghua University. His research interests include
reconfigurable computing, mobile computing, and
VLSI DSP.

SHOUYI YIN received the B.S., M.S., and Ph.D.

degrees in electronic engineering from Tsinghua
University, China, in 2000, 2002, and 2005,
respectively. He was with the Imperial College
London as a Research Associate. He is currently
with the Institute of Microelectronics, Tsinghua
University, as an Associate Professor. He has
authored or co-authored over 40 refereed papers,
and served as TPC member or reviewers for the
international key conferences and leading journals.
His research interests include SoC design, reconfigurable computing, and
mobile computing.
SHAOJUN WEI was born in Beijing, China, in
1958. He received the Ph.D. degree from the
Faulté Polytechnique de Mons, Belgium, in 1991.
He is currently a Professor with the Institute of
Microelectronics, Tsinghua University. He is also
a Senior Member of the Chinese Institute of Elec-
tronics. His main research interests include VLSI
SoC design, EDA methodology, and ASIC design.

VOLUME 5, 2017 26613