ELIT/ULTIMAT

Overview
JTAC Slides
Agenda

 ELIT ASIC overview and building blocks

• ULTIMAT Overview
• High level Chip overview
• Ingress/Egress Pipeline
• VOQ
Product FEATURES
QFX10002-72Q QFX10002-36Q
 72 Ports of 40G (QSFP+)  36 Ports of 40G (QSFP+)
→ or 24 Ports of 100G (QSFP28) → or 12 Ports of 100G (QSFP28)
→ or 288 ports of 10G (QSFP+ Break- → or 144 ports of 10G (QSFP+ Break-
out cable) out cable)
 Four Power Supply Unit (PSU) FRUs  Two Power Supply Unit (PSU) FRUs
accessible from back accessible from back
 PSUs in power sharing mode,  PSUs in power sharing mode,
support full redundancy support full redundancy
 Typical Power Consumption 1200W  Typical Power Consumption 700W
Common Features
• Three Fan Tray FRUS accessible from back
• Front Side also includes:
• RS232 Console port RJ45
• Copper and Optical Management Ethernet ports (RJ45 & SFP)
• USB 2.0 Port
• PTP Timing Port (SFP)
SYSTEM DESIGN DETAILS
• Intel IVY Bridge 4 core 2.5GHz CPU
• 16GB DDR3 SDRAM – Two 8GB DIMMs
• 25GBx2 Internal SSD Storage
• Packet Forwarding Engine (PFE) Slice consists of one PE ASIC and three
2GB HMC memory chips, and drives 12 ports
• QFX10000-72Q has Six PFE Slices; QFX10000-36Q has Three
• QFX10000-72Q uses Two PF ASICs for switching fabric, whereas
QFX10000-36Q uses One
• Hardware design can support IEEE1588, in Boundary Clock as well as
Transparent Clock mode
 ELIT HW
MAIN BOARD
4TX,4R
X
4TX,4R
X
PE0
2x 4TX,4RX
2x3 16TX+16RX
4TX,4RX GRP0
QSFP
4TX,4R
4TX,4R
X
X
GRP1
ELIT: ELIT Lite:
4TX,4R
X
HMC
HMZ
HMZ
6 x PE chips 3 x PE chips
4TX,4R

2 x PF chips 1 x PF chip
X PE1
2x 4TX,4RX 16TX+16RX
GRP0
2x3 4TX,4RX
GRP1 PF0
QSFP

72 x 40G 36 x 40G
X
4TX,4R
X
4TX,4R
16TX+16RX
HMC
HMZ
HMZ

4TX,4R
X
4TX,4R 16TX+16RX
X PE2
16TX+16RX
2x 4TX,4RX
GRP0
2x3 4TX,4RX 16TX+16RX
GRP1
QSFP
X
4TX,4R
X
4TX,4R

10G Channelization
HMC
HMZ
HMZ

MEZZ CONN

4TX,4R
100G
Intel CPU
X
4TX,4R
X PE3
2x
4TX,4RX 16TX+16RX
2x3 GRP0
4TX,4RX 16TX+16RX
QSFP GRP1

PTP FPGA
16TX+16RX
X 16TX+16RX
4TX,4R
X
4TX,4R

HMC
HMZ
HMZ

4TX,4R
X
4TX,4R
2x X PE4
2x3 4TX,4RX
PF1
QSFP 4TX,4RX GRP0
GRP1 16TX+16RX
4TX ,4RX
X
4TX,4R

HMC
HMZ
HMZ

4TX,4R
X
4TX,4R
X PE5
2x 4TX,4RX
GRP0
2x3 4TX,4RX 16TX+16RX
GRP1
QSFP X
4TX,4R
X
4TX,4R
MEZZ BOARD
HMC
HMZ
HMZ

Source: Elit HW functional Spec

5 JUNIPER NETWORKS RESTRICTED & CONFIDENTIAL
 PE-PF
FPC - 4 PG in each PE consisting
Port 0 QSFP+
PE 3 ports.
- 12 ports in one PE
- 36Q will have 3 PE’s
Port 1 QSFP+
PG0..3
PG0 Avago-serdes + - 72Q will have 6 PE’s
MTIP/PG - Each PG can be configured
In 3 modes – 12x10G,3x40G,
Port 12 QSFP+ 1x100G
- Each PFE connect in X1P
PF Mode using SerDes@21G to P
- Avago serdes to connect PE
PE2 With PF.

PE3
HW Overview: MTIP on PE
• Four Port Groups at Network side (aka wanio in PE) on each PE

• Each PG has one Channelized MAC/PCS module

• Each PG can be configured to 12x10G, 3x40G, or 1x100G

• All ports connected to same PG needs to operate at same-speed.

• So channelization-dechannelization configuration enforces this rule. No

mixed-speed support per PG.

• Per ASIC errata, when a port-0 speed changes, the traffic can be disruptive
on other ports (1 and 2) in same PG.
 100GE Ports
QFX10002 100GE PORTS

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35

36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70

37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71
Elit and Elit-Lite PE and PF
• Paradise PE/PF ASIC.
• Integrated FPC contains 6 PEs (Elit) or 3 PEs (Elit-Lite) with each PE forming
a PFE.
• Elit Fabric consists of 2 PFs, Elit-Lite consists of 1 PF, each fabric plane
contains 1 PF.
• Each PFE connect in X1P mode using serdes@21G to PF
• Avago 28nm serdes
• CCL 2.0
• Each PF in Elit has 96 X1P ccl links to the PFEs.
• Elit – Each of the 6 PFE has 16 links to PF-0 and 16 links to PF-1
• Elit-Lite – Each of the 3 PFE has 32 links to PF-0
• PE uses CCL2 link to connect the fabric planes.
• Cell based forwarding
Software Architecture
• The modular software architecture of QFX10000 switches provides the
following specific benefits:
• Multicore Linux kernel (based on Windriver Yocto Linux)
• Higher control plane performance (running on four cores)
• Topology-independent in-service-software upgrades (TISSU)
• Hosting third-party apps in a virtual machine
• Zero touch provisioning (ZTP)
• Automation with Puppet, Chef, Ansible, and Python
 ULTIMAT

• Midplane-less chassis
• Ultimat Control Board (UCB) / Ultimat Routing Engine (URE)
– URE and UCB are within the same FRU (QFX10000-RE) in Ultimat
QFX10000 system architecture
Control Plane
 Individual I2C bus segments
from both Control Boards to all
FRUs in the system, used for
FRU identification and power
up/down
 Gen 2 PCI Express connectivity
from both Control Boards to all
Six SIBs in the system, used to
configure and control PF ASICs
on the SIBs
 10G Ethernet connectivity
between both CBs, all LC
CPUs, and all PE ASICs, used
for code download and IPC
 RS232 connectivity between
CB and all LCPUs, used for
CTY
Data Path
 Packet Forwarding is based on
Paradise PFE (PE) ASICs on
line cards
 WAN Interface on PE ASIC
drives the front panel ports on
Line Cards, either directly or
through retimers
 Each PE ASIC uses HMC
memory devices for data and
table storage.
 Each PE ASIC in the system
connects to the Switching
Fabric either directly or through
retimers
 PE ASICs on a line card are
initialized and controlled by the
local CPU
Switching Fabric
 Switching Fabric consists of up
to six SIB boards, each with
two Paradise Fabric (PF)
ASICs.
 Every PE ASIC in the system is
connected to every PF ASIC,
resulting in single-hop
connectivity from any PE to any
other PE
 Switch fabric is configured and
controlled by Master CB in the
system
UCB/URE Functionality:
• Handles system control functions
• Maintains hardware forwarding table
• Maintains routing protocol states
• Handles environmental monitoring
• Handles integrated Precision Time Protocol (PTP)
• UCB/URE Components summary:
– Intel IVY Bridge 4 core 2.5GHz CPU
– 32GB DDR3 SDRAM – Four 8GB DIMMs
– 50 GB Internal SSD Storage
– One 2.5” External SSD slot
– 10GB Ethernet Switch for Control Plane connectivity with Line Cards
– PCI Express Switch for Control Plane Connectivity with SIBs
– I2C bus segments from CB FPGA to all FRUs
– RS232 Console Port & USB Port
– RJ45 and SFP Management Ethernet
– Includes PTP logic, and SMB connectors for PTP
UCB/URE Components Details
• CPU (Intel Ivy Bridge Gladden)
– Four execution cores.
root@localhost:~# cat /proc/cpuinfo

• CPU Cache memory protection

– 32-KB data first-level cache (L1) for each core, parity protected.
– 32-KB instruction first-level cache (L1) for each core, ECC protected.
– 256-KB shared instruction/data second-level cache (L2) for each core, ECC protected.
– Up to 8-MB shared instruction/data third-level cache (L3) across all cores, ECC protected.

CPU Memory Interface (DDR3)

root@localhost:~# vmstat –s
32470716 total memory
14177184 used memory
13440860 active memory
188460 inactive memory
18293532 free memory

– One or two channels of DDR3 memory with a maximum of two UDIMMs per channel
– Support Single-channel modes and Dual-channel mode
– 72-bit wide channels, 64-bit data + 8 bit ECC
• Direct Media Interface (DMI) x4 (10G full duplex) to PCH
• PCIe Root Complex (Gen1 4x1) interface.
root@localhost:~# lspci
00:00.0 Host bridge: Intel Corporation 3rd Gen Core processor DRAM
Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
processor PCI Express Root Port (rev 09)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
processor PCI Express Root Port (rev 09)
00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
processor PCI Express Root Port (rev 09)
00:06.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
processor PCI Express Root Port (rev 09)
Platform Controller Hub (PCH) (Intel Cave Creek)
• PCIe Root Complex interface (10G full duplex) (connects to CPU)
• PCIe EndPoint interface (5GT/s) (connects to UCBC FPGA (Control FPGA), PTP FPGA,
GbE interface)
• Integrated GbE – different LAN for different purpose
– EM 0/SGMII0 – Marvell 88E1112 GbE PHY - Mgmt RJ45
– EM 1/SGMII1 – Mgmt SFP
– EM 2 – Host (192.168.1.x)
root@Ultimat-re0:RE:0% sysctl -a | grep hypervisor_ip
hw.re.hypervisor_ip: 192.168.1.1

- EM 3/SGMII4/bme0.0 – CPE Switch/FPC 128.0.0.0 (16+FPC slot – PFE on FPC’s)

- EM 4/SGMII3/bme1.0 – Other CB (one to one connection)
- EM 5/bme2.0 – Connects to Host for LCM, SPMB (SW) 128.0.0.0
- EM 6 – path between Junos VM & PFE front panel ports
- EM 7 – path between Junos VM & internal Switch front panel ports
Internal Network Connectivity on Chassis Platform
Control Board 0 Control Board 1

JUNOS VM
em6 em4 Secondary RE
192.168.1.X 128.0.0.48 192.168.2.X 128.0.0.1
em0 em1 em2 em5 em6 em7 em3

128.0.0.16 LC-1
br br br br br CB PFEM

128.0.0.17 LC-2
mgmt0 HOST Fab Guest PFEM
MGR VM
NIC

mgmt1 LCM Guest SR-IOV

VM
128.0.0.16+N
PFEM
LC-N
• root@localhost:~# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 128.0.0.48:58660 128.0.0:afs3-fileserver ESTABLISHED
tcp 0 0 192.168.1.1:42042 vjunos0:33081 ESTABLISHED
tcp 0 132 192.168.1.1:login vjunos0:705 ESTABLISHED
tcp 0 0 128.0.0.48:57422 128.0.0.1:58000 ESTABLISHED

-Low Pin Count (LPC) interface for RE FPGA

-UART interface (UART 0 for Console and UART 1 for PTP debug)
-GPIO (x67) for hardware control
-Serial ATA (SATA) to internal SSD and pluggable SSD slot (support hot- swappable) (1
port each)
-SPI interfaces for NAND Flash memory (stores BIOS)
ULTIMAT-8 (QFX10008)
• Ultimat-8 has 8 FPCs and 6 SIBs.
• The FPCs use PE asics (Paradise) for forwarding and the SIBs use PF asics for fabric
switching.
• There are two FPCs – 40G and 100G.
– 40G FPC – 3 PE
– 100G FPC – 6 PE
• Each SIB has 2 PF.
– 6 SIBs in total, so 6*2 = 12 PF in the system.
– We term all fabric links from one PF as a “plane”, so we have 12 planes in total in
Ultimat-8.
– Each PFE connect in X1P mode using serdes@25G to PF
– Avago 28nm serdes
QFX10000-36Q 40G line card
QFX10000-36Q LC Features

 Uses Three Paradise PFE (PE) ASICs

 Each PE ASIC has Three 2GB HMC Memory
devices attached to it
 Each PE ASIC connects to all PFs in the
system to form a fully distributed switching
fabric
 Local CPU subsystem consists of Freescale
P5020 Dual-Core CPU, 8GB Memory in the
form of two 4GB MiniDIMMs, and glue logic
 10GB Ethernet Switch for Control Plane
connectivity between two CBs, LCPU, and all
PEs
 40G Dual Retimers for 40G-only Ports
 100G Retimers for 100G/40G Ports
QFX10000-30C 100G line card QFX10000-30C LC Features

 Uses Six Paradise PFE (PE) ASICs

 Each PE ASIC has Two 2GB HMC Memory
devices attached to it
 Each PE ASIC connects to all PFs in the
system to form a fully distributed switching
fabric
 Local CPU subsystem consists of Freescale
P5020 Dual-Core CPU, 8GB Memory in the
form of two 4GB MiniDIMMs, and glue logic
 10GB Ethernet Switch for Control Plane
connectivity between two CBs, LCPU, and all
PEs
 100G Retimers used on port side as well as
fabric side for longer connectivity
Software Architecture
SOFTWARE ARCHITECTURE HIGHLIGHTS
• Decouple Platform & PFE Daemons from Junos
• Accelerate TTM for product variants (use Linux
drivers & abstractions for new platforms)
• Improve performance – Multicore CPU
• Improve BFD performance
• Mac learning (by taking two big processes
out of JUNOS)
• Leverage Linux drivers for CPU, Power Suppies,
fan trays
• Path to provide disaggregation of SW from
hardware
• Network becomes a platform for innovation by
augmenting JUNOS control plane
Process dependencies
• PFE, LCM, SPMB and JunOS VM can be spawned in any order
• There is a client-server relationship between JunOS VM & PFE/LCM/SPMB
• JunOS VM acts as the server and waits for clients to connect.
• PFE & LCM do not communicate with each other.
• They might share access to same hardware (as case may be)
• LCM is a stateless process and can be restarted.
• System is configured to restart LCM for a finite limit
• Cross the finite limit causes a reboot of the system
• PFE maintains LC state and can be restarted
• SPMB maintains SIB states and can be restarted(Warm restart supported)
TVP on Ultimat Chassis
Route Engine
Fabric Manager JDM
Junos VM
Platform Guest App
Daemon Switch
DST Manager Packet I/O

FANs, PS,
Sensors, LEDs SW NIC

Line Card
SW
Forwarding
PE
Daemon
DST
Sensors
Yocto Linux

Fabric Card
Sensors PF
 TVP Chassis Device Ownership
Component Run Location Devices
JUNOS Master & Standby RE FPGA(Mastership), Console
HostOS on RE Master & Standby RE USB, Mgmt ports, DISK, PCIe
Controllers, NIC, PCH
HostOS on LC LC Forwarding Daemon - PE
Forwarding Daemon LC Data port optics, LEDs, ASIC
LCM Master & Backup RE FPGA, I2c, Fan controller, PSU,
Temp Sensor on CBs & LCs, FPM,
Mid-plane, Power mgmt.

Fabric Manager Master & Backup RE SIBs (SPMB), PF

CB Switch Manager Master & Standby RE Ethernet Switch on RE & LCs
Platform Components

JunOS VM • Console
• Mgmt. Ports

Chassid

PFE Process

LCM process SPMB

CMLC
USD PE DRIVER

CPU I2C Chassis-D SIB 0-5 I2C

PHY PFE I2C • Board ID PROM
LED • Transceivers I2C
• FANs • NVRAM
• NVRAM • Temp Sensors • LogEPROM
• LogEPROM • PSU
• DIMMs
• Security EEPROM
System Internals
TVP Chassis: Redundancy
• Dual RE with h/w assist mastership
• Mastership Arbitration: Driven by Chassisd
• CB FPGA register space responsible for mastership will
be mapped to JUNOS VM
• GRES, Nonstop-routing and Nonstop-bridging support
• CB0 has higher Mastership priority when both CBs are
present
• Support for keep-alive failure trigger mastership
switchover
Chassisd Interactions
• Interactions in JunOS
– Kernel
– MGD
– Alarmd
– SNMP
– Mastership
• Platform Daemons(on Host):
– FRU management(LCM) – Environment MGMT
– Fabric management(SPMB) – Fabric State machine
• Forwarding Daemon:
– FPC/PIC state machine(CMLC)
– IFD/Port management
Platform Daemons
• LCM(RE)
– Owner of FPGA, I2C, LED, Temp sensor devices
– Handle Requests from Chassisd
• Power on FPCs, Offline/Online request
• Responsible for Environment policy
– Unsolicited Notifications
• FRU events, OIR
• Periodic FRU states, Statistics Reporting
– APIs.. Similar to Elit/OPUS
• Fabric Manager(SPMB)- RE
– Responsible for SIBs(detection & Power on)
– PF chip Init and Fabric link, plane mgmt.
Platform Daemons: Contd..
• CB Manager (RE)
– Control Ethernet switch on CB and each Line Card
– Initialize & program Switch
– Allow FPCs to download image from CB
• Packet Forwarding(FPCs)
– Responsible for PIC, Optics & Port LEDs
– Similar to Elit/OPUS TVP DC-PFE
• PTP Manager (RE)
• Shared resource in CB FPGA
– PCIE: LCM, SPMB & JUNOS VM (Chassisd/Kernel)
– I2C: LCM & SPMB
Paradise Block
Diagram
PE: High level pipeline
External memory controller

(lookup control
information)
grant
VOQ
Ingress packet processor
manager req

(headers)
Fabric req
Ingress buffer Fabri
schedule grant
manager c
Policer/ r
Port Filter intf
group (loopback) counter
Egress buffer Output Q
manager manager

(updated headers)

Egress packet processor packet

Key/notification
header/notification
commands
 A LOGICAL VIEW OF PE PKT PROCESSING
Ingress Ingress
Destination Next-hop
Header Interface Ingress Filter
Lookup
HMC Processor
Parsing Lookup
Ingress packet processor

IBUF IGP SLU FLT IRP VOQ

Port group

FABRIC
Egress
WAN Egress Egress Header
Header Egress Filter Descriptor Parsing
host Rewrite Fetch
Egress packet processor

EBUF ERW FLT EDF EGP OQM

DLU internal loop Ingress packet Egress packet

Ingress header Egress header

Source: PTX-NG overview

34 JUNIPER NETWORKS RESTRICTED & CONFIDENTIAL
PE main components and functions
Paradise PFE operation
Main function blocks in Paradise
 Port groups – Physical interfaces, PCS, MAC, and preclassifier. Responsible for handling
all media specific behaviors.
 Ingress buffer manager – onchip storage of packet and absorbs bursts
 Ingress packet processor – parsing, source lookup, filter, destination lookup, tunnel
termination, sampling, VoQ selection
 Filter – multiple tuple match against various set of rules
 Policer & counter – policer and maintain counters
 External memory controller – HMC controller
 Egress buffer manager – stores packet during egress queueing and processing
 Egress packet process – packet encapsulation, rewrite, filtering, policer, counter
 VOQ manager – maintain VOQ, reads packets from HMC and sends to Fabric
 Fabric scheduler – schedules transaction through the fabric
 Output queue manager – reassembles packets and send to egress process
 Fabric interface – interacts with fabric and handles fabric i/o protocols
Ingress pipeline
 IBM (Ingress buffer manager) – receives packet from wan interfaces. Creates
header notification and sends it to ingress processing blocks.
 IGP (ingress parser) – packet parsing and hash generation
 SLU (source lookup) – context setting and BA classification
 FLT (filter) – multiple tuple match and action
 DLU (destination lookup) – examine destination related header information and
determines packet’s ultimate transit interface
 IRP (ingress result processor) – performs action determine by ingress pipeline
such as sampling, recirculation, reorder, VoQ section
 PLCT (policer & counter) –ingress policing and counting
 IRW (ingress rewrite) – ingress packet rewrite, add vlans, strip bytes.
 VoQ manager – manages queue memory allocation and Fabric Flow control.
Egress pipeline
 EBM (Egress buffer manager) – short term storage for packet during
egress processing
 EGP (Egress parser) – breaks down the packet, extract packet header
information. It gets an metadata from ingress parser (IGP) in ingress PFE
to assist parsing
 EDF (Egress descriptor fetch) – responsible for getting descriptor that
determine new header content, revise packet length, policer/counter
trigger, header update instruction
 ERW (Egress rewrite) – responsible for performing any modification to the
packet
 ESMP – Egress sampling
 FLT (filter) – multiple tuple match
 PLCT (policer & counter) – responsible for egress policing and counting
VOQ Architecture
 NON-VOQ SWITCH

Fabric Output Off-chip Output Off-chip

Queues buffers Queues buffers
PFE-A (Ingress) PFE-X (Egress) Extra DRAM & extra latency
Both ingress and egress PFEs needed
many ms of delay bandwidth buffering

Power wastage
Input Output Packets are written to and read from off-chip
Ports PFE-B (Ingress) PFE-Y (Egress) Ports buffer twice

Head of line blocking

No traffic Fate sharing of ingress ports destined to
congested egress PFE

Fabric switch
HoL
blocking
JUNIPER NETWORKS RESTRICTED & CONFIDENTIAL
 VoQ – Better latency, power & congestion MGMT
Virtual Output
Queues
Off-chip buffers PFE-A (Ingress) PFE-X (Egress) On-chip
~40ms buffers
Buffering only at Ingress
(~10’s us)
All the VoQs, at ingress PFEs,
corresponding to the egress port,
are part of buffer Q for that port
(Note the color code)

Input Output Sched is controlled @

Ports PFE-B (Ingress) PFE-Y (Egress) Ports egress
The queueing hierarchy is
Not controlled by the port scheduler
impacted at the egress PFEs

No Head of Line Blocking

Fabric switch
FAB –
Requests and Grants for Packet groups
1. Packet
Effective Fabric Flow control
DRAM Packet Request and grant for every page – multiple
Buffer Group packets

2. Request for packet group

Ingress PFE 3. Grant for the packet group Egress PFE

4. Data cell(s) for 5. Reorder and

the packet group Assemble

No Saw-tooth effect
Fabric - Multiple cell sizes for better alligment
- Page need to fit into fixed chunks – little
padding per ~every huge-size page
VOQ ARCHITECTURE SUMMARY
The JUNOS Express chipset fabric design aimed for efficiency, scalability, and simplicity
 Virtual output queueing technology
 Non blocking
 Each packet written to and read from external memories only once
 Egress driven scheduling hierarchy
 No sustained congestion in the fabric
 Distributed congestion management with global buffer size control

44 Copyright © 2011 Juniper Networks, Inc. www.juniper.net

Juniper Proprietary and Confidential. Under NDA.
VOQ
TFXPC0(vty)# show shim jnh cpclass dev 0 47
SEQ: [202804a0] Final SIZE: 4 EGPRT_VAL: 1 PRT_TYP: 1 VPFE: 025 GRPID: 00
ACT: [0400004f] EgNHId: EG_NHID: 00004f <<Flabel
ACT: [140000a8] QoS/Smp: TC 5, << OQ is 5
ACT: [1c000031] Policer: plc-idx 0x31, qual-bit 0 <<<Policer
ACT: [1801ff23] Counter: cnt-idx 0x1ff23, qual-bit 0
ACT: [70400000] Offset: l3 offset, ovwr 0

TFXPC0(vty)# bringup jspec read pechip[0] register pgq cpu_voqd 5

bringup jspec read pechip[0] register pgq cpu_voqd 5

0x048e0014 pe.pgq.cpu_voqd[5] 00000000

aqid_valid[24:24] : 0x0
aqid[23:7] : 0x0
qsntx[6:0] : 0x0

45 Copyright © 2011 Juniper Networks, Inc. www.juniper.net

Juniper Proprietary and Confidential. Under NDA.
TERMINOLOGY
• IGP – Ingress parser
• SLU – Source lookup unit
• FLT – Filter block
• DLU – Destination lookup unit
• IRP – Ingress result processor
• CPCLASS - Control packet classification
• Flabel or Egress NH id (EG_NHID) – Reason code
• L2L header – Lookup to Lookup header
• PLCT – Policer
• TOE – Trinity offload engine
• DMA – Direct Memory access
• FP – Fastpath
• TTP – Trivial transport protocol
• HOSTIF0 – Host interface
• OQ – Output queue
• VPFE – Virtual packet fowarding engine
• Switch Processor Mezzanine Board (SPMB)
• ChipToChipLink (CCL) / Fabric Link
• Physical Coding Sublayer (PCS)
• Physical Medium Attachment (PMA)
• Media Independent Interface (MII)
 PE PFE SW MODULES
• CMQFX QFX Chassis manager
• PE Drivers PE chip drivers
• TOE Microcode TOE Microcode
• DLU Microcode DLU Microcode
• JIF ifl, vlan, vlan membership
• JFDB Route table management
• JNH, JENCAP Ingress Nexthop, Egress Nexthop
• JSAL Source Address management, learning.
• JFW, JPLCT Firewall, Policer, Counters
• JCHASH Hash Table management
• JALPHA, JBETA, JCV Filter, Route Lookup algorithms
• HALP-PKT Host path processing

47 JUNIPER NETWORKS RESTRICTED & CONFIDENTIAL

 PE FORWARDING TERMS

• Iport: logical port, per ifd. LAG members use parent’s lport
• Gport Id: global port, a system-wide unique number for port.
• L2domain: Layer 2 forwarding domain capable of switching, learning, flooding,
routable mac etc
• Gl2domain Id: global L2dmain id, a system-wide identifier
• L3vpn: Layer3 forwarding domain
• Flabel: Fabric label, an identifier for header rewrite information with in scope of
egress PE.
• Egress NHId: Same as flabel, see above.

48 JUNIPER NETWORKS RESTRICTED & CONFIDENTIAL

 PE HW BLOCKS
•MTIP More Than IP Port MAC block
•HMC Hybrid Memory Cube Off-chip memory for Route tables, counters and packet buffering
•PG Port Group Serdes, lane mapping etc
•IPW, EPW Packet Writer Packet from wan/fabric port to onchip memory
•IQM, OQM Queue Manager
•IBUF, EBUF Buffer Manager Onchip packet buffer memory manager
•IGP, EGP Packet Header Parser
•SLU Source Lookup Unit Ifl, Source Mac, My Mac, Vlan lookups etc
•FLT Filter
•DLU Destination Lookup Unit Programmable lookup engine
•IRP Ingress Result Processor Ingress Nexthop
•IRP_SMP Ingress Sampler
•IRW, ERW Packet Header Rewriter
•IPR, EPR Packet Reader
•PGQ Page Queue
•CM Congestion Manager
•RT, RS Request Table, Request Scheduler
•GT, GS Grant Table, Grant Scheduler
•DT Data Table
•PS Port Scheduler
•FO, FI Fabric Output/Input interface
49 JUNIPER NETWORKS RESTRICTED & CONFIDENTIAL
•Fabrord Fabric Reorder
•OQM Output Queue Manager
 RESOURCES:

•ELIT/ULTIMAT Engg TOI:

https://junipernetworks.sharepoint.com/teams/CSS/bangaloretac/switching/Shared%20Documents/Forms/AllItems.aspx
?RootFolder=%2Fteams%2FCSS%2Fbangaloretac%2Fswitching%2FShared%20Documents%2FTech%2DStuff%2FQ
FX10000&FolderCTID=0x0120004600A5B8121633409C3E65F34CFB4F8F&View=%7B30018369%2DEB3E%2D462A
%2DA9B1%2D4A50E0841016%7D

50 JUNIPER NETWORKS RESTRICTED & CONFIDENTIAL

Access SPMB/SIB and troubleshoot commands:
• root@Ultimat-re0:RE:0% vty spmb0
TOR platform (2499Mhz Pentium processor, 2047MB memory, 0KB
flash)
• SPMB0(Ultimat-re0 vty)# show ccl
asics show all ASICs registered with the CCL toolkit
bist show CCL bist information
channel show CCL channel information
channel-from-phy show CCL channel information based on physical link id
channel-map show CCL channel maps
errors show CCL errors for all ASICs
instance show ccl instance information
statistics show CCL statistics for all ASICs
tdm show ccl tdm and sort table from hardware and buffer
• SPMB0(Ultimat-re0 vty)# show pfchip rates summary
• SPMB0(Ultimat-re0 vty)# show pfchip <number 0-11> link_errors
Fabric CLI COMMANDS
Commands to check the fabric status
root@st-72q-2b-02> show chassis fabric sibs
Fabric management SIB state:
SIB #0 Online
FASIC #0 (plane 0) Active
FPC #0
PFE #0 : OK
PFE #1 : OK
PFE #2 : OK
PFE #3 : OK
PFE #4 : OK
PFE #5 : OK
FASIC #1 (plane 1) Active
FPC #0
PFE #0 : OK
PFE #1 : OK
PFE #2 : OK
PFE #3 : OK
PFE #4 : OK
PFE #5 : OK
Fabric CLI COMMANDS
Commands to check the fabric status
root@st-72q-2b-02> show chassis fabric fpcs local
Fabric management FPC state:
FPC #0
PFE #0
SIB0_FASIC0 (plane 0) Plane Enabled, Links OK
SIB0_FASIC1 (plane 1) Plane Enabled, Links OK
PFE #1
SIB0_FASIC0 (plane 0) Plane Enabled, Links OK
SIB0_FASIC1 (plane 1) Plane Enabled, Links OK
PFE #2
SIB0_FASIC0 (plane 0) Plane Enabled, Links OK
SIB0_FASIC1 (plane 1) Plane Enabled, Links OK
PFE #3
SIB0_FASIC0 (plane 0) Plane Enabled, Links OK
SIB0_FASIC1 (plane 1) Plane Enabled, Links OK
PFE #4
SIB0_FASIC0 (plane 0) Plane Enabled, Links OK
SIB0_FASIC1 (plane 1) Plane Enabled, Links OK
PFE #5
SIB0_FASIC0 (plane 0) Plane Enabled, Links OK
SIB0_FASIC1 (plane 1) Plane Enabled, Links OK
 PF DRIVER RE cli commands
• Available for Elit
• CLI commands on RE have been added to show “PFCHIP” information
• Sample commands are given below:
– show chassis pfchip total pfchip-id <pfchip-id>
– show chassis pfchip rates pfchip-id <pfchip-id>
– show chassis pfchip parity-errors pfchip-id <pfchip-id>
– show chassis pfchip link-errors pfchip-id <pfchip-id>
– show chassis pfchip vital-errors pfchip-id <pfchip-id>
– show chassis pfchip interrupt-errors pfchip-id <pfchip-id>
– “pfchip-id” will be 0 or 1 depending on the PF asic selected
FM LOGS
root> show log chassisd | grep FM
Sample logs:
Jan 8 22:46:22 FM: Initializing QFX10 fabric management
Jan 8 22:49:24 FM fm_qfx10_ev_sib_online_start: event id:0, slot:0, type:SIB_ONLINE stage:START
Jan 8 22:49:24 FM fm_qfx10_ev_sib_online_start: event- fpc_ack_mask:0x0, fpc_recv_mask:0x0 sib_ack_reqd:0x0 rcv:0x0
Jan 8 22:49:24 FM:fm_qfx10_ev_sib_online_start No FPCs are online
Jan 8 22:49:24 FM fm_qfx10_ev_sib_online_finish: event id:0, slot:0, type:SIB_ONLINE stage:START
Jan 8 22:49:24 FM fm_qfx10_ev_sib_online_finish: event- fpc_ack_mask:0x0, fpc_recv_mask:0x0 sib_ack_reqd:0x0 rcv:0x0
Jan 8 22:50:38 FM fm_qfx10_ev_fpc_online_start: event id:1, slot:0, type:FPC_ONLINE stage:START
Jan 8 22:50:38 FM fm_qfx10_ev_fpc_online_start: event- fpc_ack_mask:0x0, fpc_recv_mask:0x0 sib_ack_reqd:0x0 rcv:0x0
Jan 8 22:50:38 FM fm_qfx10_ev_fpc_online_hsr_start_stage: Sending Rx link train message to FPC 0
Jan 8 22:50:38 FM fm_qfx10_ev_fpc_online_hsr_start_stage: Sending link train message to SIBs
Jan 8 22:50:38 FM: Starting HSR train: plane:0, fpc:0 fe:0 ccl(inst,sc):(8,0)
Jan 8 22:50:38 FM: Starting HSR train: plane:0, fpc:0 fe:0 ccl(inst,sc):(8,1)
Jan 8 22:50:38 FM: Starting HSR train: plane:0, fpc:0 fe:0 ccl(inst,sc):(8,2)
Jan 8 22:50:49 fm_pd_elit_sib_send_link_update: FM: plane 0 fpc 0 fe 0 sc 64 Link OK
Jan 8 22:50:49 fm_pd_elit_sib_send_link_update: FM: plane 0 fpc 0 fe 0 sc 65 Link OK
Jan 8 22:50:49 fm_pd_elit_sib_send_link_update: FM: plane 0 fpc 0 fe 0 sc 66 Link OK
Jan 8 22:51:28 FM fm_qfx10_fpc_online_hsl_stage: Sending spr_ctl to SIBs
Jan 8 22:51:28 FM fm_qfx10_fpc_online_hsl_stage: Sending spr_ctl to FPC 0
Jan 8 22:51:28 FM fm_qfx10_ev_fpc_online_finish: FPC 0 online complete status 1
FM LOGS
root> show log messages (also can be seen on vty console)
Sample logs:
[Sun Jan 8 22:51:23 2012 LOG: Debug] cmqfx10_fm_rx_ctrl_vector: Received card add is_sib:0 Slot:0
[Sun Jan 8 22:51:23 2012 LOG: Info] CMQFX10: Starting HSR training on fe 0 plane 0 ccl_sc(logical) 0 ccl_id:0 PE0
[Sun Jan 8 22:51:23 2012 LOG: Info] CMQFX10: Starting HSR training on fe 0 plane 0 ccl_sc(logical) 1 ccl_id:0 PE0
[Sun Jan 8 22:51:23 2012 LOG: Info] CMQFX10: Starting HSR training on fe 0 plane 0 ccl_sc(logical) 2 ccl_id:0 PE0
[Sun Jan 8 22:51:27 2012 LOG: Err] cmqfx10_elit_ccl_rx_bond: Serdes DFE tuning complete for PE0 inst 1 sc 0
[Sun Jan 8 22:51:27 2012 LOG: Err] cmqfx10_elit_ccl_rx_bond: Serdes DFE tuning complete for PE0 inst 1 sc 1
[Sun Jan 8 22:51:27 2012 LOG: Err] cmqfx10_elit_ccl_rx_bond: Serdes DFE tuning complete for PE0 inst 1 sc 2
[Sun Jan 8 22:51:27 2012 LOG: Err] cmqfx10_elit_ccl_rx_bond: Frame alignment Successful for PE0 inst 1 sc 0
[Sun Jan 8 22:51:27 2012 LOG: Err] cmqfx10_elit_ccl_rx_bond: Frame alignment Successful for PE0 inst 1 sc 1
[Sun Jan 8 22:51:27 2012 LOG: Err] cmqfx10_elit_ccl_rx_bond: Frame alignment Successful for PE0 inst 1 sc 2
[Sun Jan 8 22:51:28 2012 LOG: Debug] CMQFX10FM:link train req complete
[Sun Jan 8 22:51:28 2012 LOG: Info] CMQFX10FM: Received spray ctrl message 0xff 0xf
[Sun Jan 8 22:51:28 2012 LOG: Info] set_ppfe_spray_mask: pfe 0 dest 0 en_pl:0x3 dis_pl:0xfffffffc admin:1
[Sun Jan 8 22:51:28 2012 LOG: Info] FM config spray: pfe:0 dest:0 actual_en_mask 0xfffffff3 actual_dis_mask 0x3
[Sun Jan 8 22:51:28 2012 LOG: Info] FM config spray: pfe:0 dest:0 actual mask to asic 0xfffffff3
[Sun Jan 8 22:51:28 2012 LOG: Info] cmqfx10_elit_fm_set_ppfe_spray_mask: First plane going up for {pfe0, destppfe0}
[Sun Jan 8 22:51:28 2012 LOG: Info] set_ppfe_spray_mask: pfe 0 dest 1 en_pl:0x3 dis_pl:0xfffffffc admin:1
[Sun Jan 8 22:51:28 2012 LOG: Info] FM config spray: pfe:0 dest:1 actual_en_mask 0xfffffff3 actual_dis_mask 0x3
[Sun Jan 8 22:51:28 2012 LOG: Info] FM config spray: pfe:0 dest:1 actual mask to asic 0xfffffff3
[Sun Jan 8 22:51:28 2012 LOG: Info] cmqfx10_elit_fm_set_ppfe_spray_mask: First plane going up for {pfe0, destppfe1
[Sun Jan 8 22:51:28 2012 LOG: Info] set_ppfe_spray_mask: pfe 0 dest 2 en_pl:0x3 dis_pl:0xfffffffc admin:1
[Sun Jan 8 22:51:28 2012 LOG: Info] FM config spray: pfe:0 dest:2 actual_en_mask 0xfffffff3 actual_dis_mask 0x3
[Sun Jan 8 22:51:28 2012 LOG: Info] FM config spray: pfe:0 dest:2 actual mask to asic 0xfffffff3
[Sun Jan 8 22:51:28 2012 LOG: Info] cmqfx10_elit_fm_set_ppfe_spray_mask: First plane going up for {pfe0, destppfe2}
Debugging commands for Optics
• show chassis hardware in RE
• show qsfp list in PFE
• show qsfp <index> diagnostics
Debugging commands for Wanio in PFE
(cheat-sheet)

• Show pepic 0 wanio  Summarizes all PE mapping per ifd

• Show pepic dfe-status <port-number>
• Show asic <pe-n-n(n)> avago-serdes <sbus-number> debug-info
• Show mtip-chmac summary  get Index
• Show mtip-chmac <index> registers -> Read twice
• Show mtip-chmac <index> stats
• Show mtip-chpcs summary
• Show mtip-chpcs <index> registers -> Read twice
Thank You !