OpenSPARC T1 on Xilinx

FPGAs Updates
Thomas Thatcher
thomas.thatcher@sun.com
OpenSPARC Engineering

Paul Hartke
Paul.Hartke@Xilinx.Com
Xilinx University Program

RAMP Retreat August 2008, Stanford

Agenda
Quick OpenSPARC Overview
Progress timeline
Current Status
>
>
>
>
>

OpenSPARC T1 1.6 Release

OpenSPARC Book
OpenSPARC FPGA Board
Multi-core T1 design
T1 core on BEE3

Roadmap
Q&A
www.opensparc.net

RAMP Retreat-Aug 2008

What is OpenSPARC?
Open-Sourced versions of Sun's Microprocessor
Products

> RTL, Verification Env,documentation, system software

> Available for download at www.opensparc.net

Two Processors Available

> OpenSPARC T1
> 8 cores, 4 hardware threads per core
> 1 floating-point unit external to core, shared by all cores
> 4 banks of L2 cache
> OpenSPARC T2
> 8 cores, 8 hardware threads per core
> Floating-point internal to core, one per core
> 8 banks of L2 cache

www.opensparc.net

RAMP Retreat-Aug 2008

OpenSPARC T1
SPARC V9 implementation
Eight cores, four threads each
32 simultaneous threads
All cores connect through a
134.4 GB/s crossbar switch
High BW 12-way associative 3
MB on-chip L2 cache
4 DDR2 channels (23 GB/s)
70W power
~300M transistors
www.opensparc.net

RAMP Retreat-Aug 2008

Sun/Xilinx Partnership: Big Goals

Proliferation of OpenSPARC technology
Proliferation of Xilinx FPGA technology
Make OpenSPARC FPGA friendly
> Create reference design with complete system functionality
> Boot Solaris/Linux on the reference design
> Open it up
> Seed ideas in the community

Enable multi-core research

www.opensparc.net

RAMP Retreat-Aug 2008

Timeline

July 06

Jan 07

June 07

Jan 08

Aug 08

OpenSPARC T1
Sun/Xilinx OpenSpARC
Stand-alone
OpenSolaris on
Collaboration T1 on ML411 program under
ML411 board
Begins
board
hypervisor
First ML505 Support

www.opensparc.net

RAMP Retreat-Aug 2008

Today

New Developments
OpenSPARC T1 1.6 Release
OpenSPARC Book
New OpenSPARC Development Kit

> ML505 board with XC5VLX110T FPGA

Multi-core Design
OpenSPARC T1 core running on BEE3 Board

www.opensparc.net

RAMP Retreat-Aug 2008

OpenSPARC T1 1.6 Release

Released May, 2008
Implementation of 4-thread T1 core on Virtex 5 FPGAs
> ML505-V5LX110T board
> EDK Project files (for EDK 9.2)
> Scripts to run complete RTL regression on hardware

Complete setup to boot Solaris

> Networking support, including telnet and ftp

Quick start ace files included

> Creates an out-of-the-box experience

> T1 core boots OpenSolaris in 30 minutes

www.opensparc.net

RAMP Retreat-Aug 2008

Hardware Block Diagram

MultiPort
Memory
Controller

FPGA Boundary
Cache-processor
interface (CPX)

SPARC T1 Core

CCX-FSL
Interface

External DDR2 Dimm

Xilinx Embedded
Developers
(EDK) Design

MCH-OPB MemCon

Microblaze Proc

Microblaze Debug UART

SPARC T1 UART

processorcache interface
(PCX)

Fast Simplex
Links interface
(FSL)

10/100 Ethernet

Developed and
Working

IBM Coreconnect
OPB Bus

www.opensparc.net

RAMP Retreat-Aug 2008

Software Setup
OpenSolaris is booted from a RAM disk Image
Memory Allocation:
>
>
>
>

1 MB used by Microblaze firmware

1 MB used for OpenSPARC Boot PROM image
80 MB for RAM disk image
Leaving 174 MB for OpenSPARC RAM

Microblaze firmware does address translation to map

SPARC addresses to board addresses.

www.opensparc.net

RAMP Retreat-Aug 2008

10

OpenSPARC Development Kit

A kit for OpenSPARC development now available
>
>
>
>

Board based on the ML505, but with an XC5VLX110T FPGA

Includes USB interface for FPGA programming
Tested with OpenSPARC T1 release 1.6 release design
Eliminates the need to buy a board and then upgrade the FPGA

Shipping now!
Kit Includes:
>
>
>
>

Board, with power supply and 256 MB DRAM

Platform USB download cable
Host to host SATA crossover cable
Compact flash card with OpenSPARC T1 1.6 ace files

www.opensparc.net

RAMP Retreat-Aug 2008

11

Kit Contents

www.opensparc.net

RAMP Retreat-Aug 2008

12

OpenSPARC Kit Donation Program

Sun will donate
OpenSPARC
Development Kits to
qualified universities
> Web address below:

See the web page for

more details
Also available from
directly from Digilent
> http://www.digilentinc.com

http://www.opensparc.net/edu/university-program.html
www.opensparc.net

RAMP Retreat-Aug 2008

13

OpenSPARC Internals Book

Covers both OpenSPARC T1 and T2
Includes
>
>
>
>
>

Architectural Overview
Development environments for OpenSPARC
Source (RTL) code overview
Configuring, extending, and verifying OpenSPARC
Porting operating systems to OpenSPARC

350 Pages
Available in both hardcopy (Amazon.com) and PDF
format
Sign-up sheet for early release PDF copy (by poster)
www.opensparc.net

RAMP Retreat-Aug 2008

14

Implementing a Multi-core design

We have created a multi-core system by interconnecting
two boards.
> Opens the door to multi-core designs on BEE3 board

Uses Xilinx Aurora link-layer protocol running over

RocketIO GTP serial tranceivers
> Connected through the SATA connectors on the board

Each GTP channel is 16 bits at 75 Mhz

> Connected to Microblaze through an FSL FIFO

www.opensparc.net

RAMP Retreat-Aug 2008

15

Multi-core System Block Diagram

FPGA Boundary

Xilinx CacheLink (XCL)

External DDR2 Dimm
Fast Simplex Links (FSL)

SPARC T1 Core

CCX-FSL
Interface

Xilinx Embedded
Developers
(EDK) Design

MemCon

Microblaze Proc

Microblaze Debug UART

SPARC T1 UART

Aurora
over GTP

Ethernet

Developed and
Working

FSL connected Aurora-over-GTP

module to connect to other board

www.opensparc.net

RAMP Retreat-Aug 2008

New

16

Dual-core system implementation

SATA Cable

Master Node
www.opensparc.net

RAMP Retreat-Aug 2008

17

Four-core system implementation

SATA Cable

SMA cables

SATA Cable
Master Node
www.opensparc.net

RAMP Retreat-Aug 2008

18

Initial Configuration
Master FPGA hosts entire OpenSPARC Address space.

> However, Each client MicroBlaze will run firmware code out of

its own memory

Both boards have the same bit file

> Avoids need to develop and implement separate bit files

> CPU ID set by DIP switches on the board

However, software will be different for each board

> Master software: services all memory requests

> Slave software: only routes memory requests to the other board

www.opensparc.net

RAMP Retreat-Aug 2008

19

OpenSPARC on the BEE3

BEE3 board uses Virtex 5 FPGAs (same family as
ML505)
Re-implemented Release 1.6 design on BEE3
>
>
>
>
>

Very easy to re-target design.

Updated design to EDK 10.1
Re-implemented on both XC5VLX110T and XC5VLX155T
Generated ace files for BEE3 board
Verified OpenSolaris Boot

Seamless and trouble-free porting experience

Validated BEE3 board infrastructure
Stop by to see our demo!
www.opensparc.net

RAMP Retreat-Aug 2008

20

OpenSPARC on the Bee3 Details

Single 4-thread OpenSPARC core
62.5MHz OpenSPARC; 125MHz
Microblaze
6-LUTs: 59,350 / 97,280 61%
36kbit BRAM: 147 / 212 69%
Ethernet in design but not tested
1.5 hour implementation time

MPMC/MIG

Microblaze

EDK project based on Bee3 EDK

reference design
Uses EDK MPMC4 and MIG
www.opensparc.net

RAMP Retreat-Aug 2008

4-thread
OpenSPARC core
21

Bee3 OpenSPARC Four-Core

System Diagram (Master-node Configuration)
305.00

QSH cross-over cable

105 .00

105 .00

25.00

25.00

Fujitsu 2x2
CX4

RJ45

15.00
30.00

Fujitsu 2x2
CX4

RJ45

35.00

20.00
40.00

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

1.8V

1.8V

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

1.0V
QSH-DP-040

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

QSH-DP-040

1.8V

1.8V

1.0V

DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3

150 .00

40.00

Ring Wiring
CCX link
www.opensparc.net

70.00

Master Node
RAMP Retreat-Aug 2008

107 .00

QSH cross-over cable

22

180.00

10.00
23.00

18.00
102.00

180.00

Ring Wiring
CCX link
TXL5V
F6F311

133 133

5VLXT
FF1136

78.00

QSH-DP-040

DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3

TXL5V
F6F311

CX4
133 133

Master Node

40x2

QSH-DP040

29.00

24 pin ATX PWR

CX4

CX4

QSH-DP-040

User3
5VLXT

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

40x2

380.00

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

QSH-DP040

PCI-E
8X
User4
72*
5VLXT

21.00

CX4

72*
PCI-E
8X

CX4

QSH-DP040

12V
8-pin

72*

40x2

12V
4-pin

PCI-E
8X

User2
5VLXT
PCI-E
8X

65.00

50 pin 2mm Header

CX4

72*

2.5V

Ring Wiring
CCX link

User1
5VLXT

JTAG

40x2

100.00

PCI-Express 8x

QSH-DP040

CX4

5VLXT
FF1136

PCI-Express 8x

CX4

60.00

PCI-Express 8x

133 133

PCI-Express 8x

133 133

Ring Wiring
CCX link

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3

Bee3 Multi-core Node Block Diagram

Xilinx CacheLink (XCL)

FPGA Boundary

External DDR2 Dimm

Fast Simplex Links (FSL)

SPARC T1 Core

CCX-FSL
Interface

Xilinx Embedded
Developers
(EDK) Design

MemCon

Microblaze Proc
Ring
Ring
Ring
Wiring
Wiring
Wiring

Microblaze Debug UART

SPARC T1 UART
Ethernet

Developed and
Working

FSL interconnect to other

FPGA(s)

New

Maintain compatability between

ml505_v5lx110t and Bee3 designs
www.opensparc.net

RAMP Retreat-Aug 2008

23

Bee3 OpenSPARC Four-Core

System Diagram (Full Mesh interconnect)
305.00

QSH cross-over board

105 .00

105 .00

25.00

25.00

Fujitsu 2x2
CX4

RJ45

15.00
30.00

Fujitsu 2x2
CX4

RJ45

35.00

20.00
40.00

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

1.8V

1.8V

1.0V
4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

150 .00

QSH-DP-040

DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3

QSH-DP-040

1.8V

1.8V

1.0V

5VLXT
FF1136

78.00

QSH-DP-040

DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3

133 133

QSH-DP040

TXL5V
F6F311

CX4
133 133

TXL5V
F6F311

40x2

40.00

10.00
23.00

Ring Wiring
CCX link
www.opensparc.net

70.00

107 .00

QSH cross-over board

RAMP Retreat-Aug 2008

24

180.00

Ring Wiring
CCX link

18.00
102.00

29.00

24 pin ATX PWR

CX4

CX4

QSH-DP-040

User3
5VLXT

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

40x2

380.00

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

QSH-DP040

PCI-E
8X
User4
72*
5VLXT

21.00

CX4

72*
PCI-E
8X

CX4

QSH-DP040

12V
8-pin

72*

40x2

12V
4-pin

PCI-E
8X

User2
5VLXT
PCI-E
8X

65.00

50 pin 2mm Header

CX4

72*

2.5V

Ring Wiring
CCX link

User1
5VLXT

JTAG

40x2

100.00

PCI-Express 8x

QSH-DP040

CX4

5VLXT
FF1136

PCI-Express 8x

CX4

60.00

PCI-Express 8x

133 133

PCI-Express 8x

133 133

Ring Wiring
CCX link

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3

4 GB DDR2-667 DRAM
4 GB DDR2-667 DRAM

DDR2 DIMM0
DDR2 DIMM1
DDR2 DIMM2
DDR2 DIMM3

180.00

4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M
Q S H -D P -0 4 0

4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M
Q S H -D P -0 4 0

Q S H -D P -0 4 0

TXL5V
F6F311
4 G B D D R 2 -667 D R A M
4 G B D D R 2 -667 D R A M
T XL5V
F6F311
Q S H -D P - 0 4 0

1 .8 V
1 .8 V
1 .0 V
1 .8 V
1 .8 V
1 .0 V
1 .8 V
1 .8 V
1 .0 V

4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M

T XL5V
F6F311

4 G B D D R 2 -667 D R A M
4 G B D D R 2 -667 D R A M

TXL5V
F6F311
4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M

Q S H -D P - 0 4 0

4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M

Q S H -D P -0 4 0

Q S H -D P -0 4 0

2 4 p in A T X P W R

1 .8 V

1 2V
8 -p in

1 .8 V

12 V
4 -p in

2 .5 V

2 4 p in A T X P W R

4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M

Q S H -D P -04 0

10.00

180.00

1 2V
8 -p i n

2 .5 V

180.00

4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M

23.00
18.00

5 0 p i n 2 m m H e a de r

4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M

1 2V
4 -p in

4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M

5 0 p in 2 m m H e a d e r

4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M

150.00

JTA G

P C I- E x p r e s s 8 x
P C I- E x p r e s s 8 x
P C I- E x p r e s s 8 x
P C I- E x p r e s s 8 x

JT A G

P C I-E x p re s s 8 x
P C I-E x p re s s 8 x
P C I-E x p re s s 8 x
P C I-E x p re s s 8 x

78.00

107.00
70.00
102.00

25
RAMP Retreat-Aug 2008
www.opensparc.net

10.00
40.00

40.00
23.00
18.00

40.00
20.00

29.00

CX4 cables
180.00
29.00

100.00

5VLXT
FF1136

65.00
60.00

35.00
30.00

21.00
21.00

5VLXT
FF1136
150.00

107.00
70.00
102.00

78.00

380.00
380.00

180.00

5VLXT
FF1136

30.00

65.00
60.00

20.00
35.00

RJ45

RJ45
15.00

RJ45
Fujitsu 2x2
CX4
15.00

105.00
105.00

305.00

Fujitsu 2x2
CX4
25.00
25.00

25.00
25.00
105.00
105.00

Fujitsu 2x2
CX4
Fujitsu 2x2
CX4
RJ45

100.00

5VLXT
FF1136
4 G B D D R 2 -6 6 7 D R A M
4 G B D D R 2 -6 6 7 D R A M

1 .0 V

Bee3 OpenSPARC Eight-Core, Two-Board

System Diagram
Use Aurora over CX4 cables to connect extended fourcore Bee3 board to base four-core Bee3 board.
> Maintain native OpenSPARC T1 4-way L2 architecture
> [Almost] full OpenSPARC T1 32-thread system!
305.00

40.00

Roadmap
OpenSPARC T1 release 1.7

> Setup to boot Ubuntu Linux

> Update of EDK project to EDK 10.1
> Improvements to memory controller
> Improvements to place and route
> Multi-core design

Future Work (possible projects)

> Connect T1 core directly to system

> Increase size of L1 caches
> Current size doesn't efficiently utilize the Block RAMs
> Should be able to quadruple size without increasing logic

www.opensparc.net

RAMP Retreat-Aug 2008

26

Summary
OpenSPARC: The tools you need to do multi-core
research!
>
>
>
>
>
>

Complete EDK project to implement a system

Implemented on both BEE3 board and OpenSPARC Kit
Complete verification environment
Complete software stack
OpenSolaris Boot demonstrated
Ubuntu Linux boot underway

www.opensparc.net

RAMP Retreat-Aug 2008

27

OpenSPARC momentum

Innovation
will happen everywhere
Innovation Happens Everywhere > 8400
downloads
www.opensparc.net

FCRC-RAMP-2007-San Diego

28

Team

Ismet Bayraktaroglu

Durgam Vahia

Thomas thatcher

Paul Hartke (Xilinx)

Not Pictured: Gopal Reddy

www.opensparc.net

RAMP Retreat-Aug 2008

29