kupdf.com_fna-installation-and-operation-guide-v804.pdf

Facebook Network Appliance Deployment | 1
© 2015-2016 Facebook. All rights reserved.

Cisco®, Cisco Nexus®, and the Cisco Logo are registered trademarks of Cisco Systems, Inc.
FreeBSD® and the FreeBSD Logo are registered trademarks of The FreeBSD Foundation.
IBM® is a trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other countries.
Juniper Networks® and Junos® are [registered] trademarks of Juniper Networks, Inc. in the United States and other countries.
Linux® is a registered trademark of Linus Torvalds in the United States.
Mac® and Mac OS® are registered trademarks of Apple Inc.
Hewlett-Packard® and HP® are registered trademarks of Hewlett-Packard Company.
Windows® is either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
Thank you for choosing to install the Facebook Network Appliance (FNA)!
Facebook Network Appliance is Facebook’s content caching program. FNA provides

Internet Service Providers (ISPs) with an efficient means of delivering static Facebook
content from within their existing network. After deploying an FNA kit, an ISP will offload a
significant amount of Facebook content from its backbone network and vastly improve the
Facebook user experience.
A single FNA kit consists of a Top-of-Rack (ToR) switch, four servers, 4-post rack mounting
hardware, transceivers and cables. The hardware is suitable for deployment in data centers,
colocation facilities, and outside plant environments (the industry standard 19-inch form
factor allows integration into most network environments).
This document describes how to install, configure, and operate FNA equipment.
This section lists the equipment required for building a single FNA cluster. Ensure that you
have received all of the equipment required for either your SKU1 or SKU2 order (Table 1
and 2, respectively).
The following table lists equipment that is included with a SKU1 order. If equipment is
missing, refer to the Support section for contact information to seek assistance.
Table 1: FNA SKU1 Provided Equipment
(1) Cisco Nexus 3132Q-X switch2 1RU
(4) HP ProLiant DL380 Generation 9 (Gen9) server3 2RU
(4) 4-Post rail kit N/A
(8) C13-C14 AC Power Cord 2 Meters
(4) Network Cable 3 Meters
(1) HP ProLiant DL380 Generation 9 (Gen9) server user manual N/A
(1) Cisco Nexus 3132Q-X switch user manual N/A
(2) SFP+ optical transceiver, LR or SR, (2) per SKU4 N/A

1See Appendix I: Hardware Dimensions and Weight for more dimension details.
2Hereafter referred to as Nexus switch.
3Hereafter referred to as ProLiant DL380 server.
4Caution: SFP+ optical transceivers are sensitive to static electricity.
The following table shows equipment that is included a SKU2 order. If equipment is missing, refer to the Support
section for contact information to seek assistance.
Table 2: FNA SKU2 Provided Equipment
(4) HP ProLiant DL380 Generation 8 (Gen9) server2 2RU1
(4) 4-Post rail kit N/A
(8) C13-C14 power cable 2 Meters
(4) Network cable 3 Meters
HP ProLiant DL380p Generation 8 (Gen8) server user

(4) N/A
manual
(1) Text configuration file N/A
(2) SFP+ optical transceiver, LR or SR, (2) per SKU3 N/A

1See Appendix I: Hardware Dimensions and Weight for more dimension details.
3Hereafter referred to as ProLiant DL380 server.
3Caution: SFP+ optical transceivers are sensitive to static electricity.
SKU2 does not include a Cisco Nexus switch. However, the order includes SFP+ optical transceivers for additional uplink
capacity. Use the existing switch at your FNA deployment.
The quantity of servers you receive depends on your current and forecasted Facebook traffic volume. Servers are shipped
in groups of four. The equipment will arrive at your facility in one shipment. IBM (FNA’s logistics partner) will contact you with
shipping information.
This section shows additional equipment required for the FNA installation that is not included with a SKU order.
Table 3: FNA SKU1 and SKU2 Equipment Required but Not Included
(2) Fiber Patch Cables from Cisco Nexus 3132Q-X switch to router1,2
1 The FNA accepts both SR and LR optical transceivers. Facebook provides the most optimal optical
transceiver type corresponding to the type you currently use to connect your routers.
2 Fiber
connectivity on the FNA side is the Lucent Connector (LC) type. Fiber connectivity on the router side
can be LC-LC, LC-SC, (Lucent Connector – Subscriber Connector), etc.
This section describes how to install equipment to build an FNA cluster. After installing the
equipment, you will need to configure network settings to ensure a peering session is
established. This information is provided in Section 4 Network Configuration.
Use the pre-installation checklist to confirm that your facility has the capacity and resources
required to install an FNA cluster:
Table 4: FNA Pre-Install Checklist
n x 10 Gigabit (10Gb) available ports on ISP switch or router1
n available SFP+ (or compatible) optics of same type - Long Range (LR) or Short Range
(SR)1
n available fiber patch cables (multi-mode or single-mode)1
n available input power connections2
Power connections are divided evenly between two different power sources
Sufficient power (1kW) for each Stock Keeping Unit (SKU)
Sufficient rack space is available3

1 Where n = 2 for every FNA SKU deployed.
2 Where n = 2 times the number of devices (e.g., SKU1 is (5) devices, so (10) input power connections are
required).
3 For initial deployment (SKU1), 9RU (nine rack unit) is required. For additional capacity (SKU2), 8RU is
required.
Install the servers into a compatible server rack.
Utilize the manufacturer instructions for the server model included in your order:
DL380P Gen8: Server User Guide (Edition: 6)
DL380 Gen9: Server User Guide (Edition: 5)
Connect C13-C14 AC power cables from each server to your power distribution units (PDUs).
If you install the servers in a 2-post rack, you must provide the proper 4-post to 2-post conversion hardware.
Install the switch in the rack.
Utilize the instructions in the Cisco Nexus 3000 Series Hardware Installation Guide
(April 22,2015).
Connect each server to the switch using the supplied network cables (see Figures 3 and 4).
Each server’s NIC and network cable connector have a green sticker to show the correct port
connection and orientation. Ensure that the green sticker on the network cable connector is facing
up and connect it to the port with the green sticker.
Begin with the server at the highest location in the rack (labeled fna001) and work your way to the
bottom-most server. Connect fna001 to port 32 on the switch, fna002 to port 31, and so on.
See Figure in Appendix F: Additional Installation Images for a reference on server connectivity
order.
The Cisco Nexus switch QSFP+ ports 9 ~ 12 are not used and should not be utilized for any FNA or
non-FNA equipment. Each Cisco Nexus switch supports up to twenty servers.
Connect the switch to your facility’s router.

Connect SFP+ ports 1 and 2. These ports are pre-configured for the Link Aggregation Control
Protocol (LACP). See Appendix C: Link Aggregation/Link Aggregation Control Protocol
(LACP) Configuration for more information.
The Cisco Nexus’ (4) SFP+ ports and QSFP+ port 1 cannot be utilized at the same time. These ports are internally
multiplexed. See Figure 26 for more information.
Power on the FNA hardware.
Press the power buttons on the servers and power on any interfacing systems
(i.e., PDUs, existing router, etc.).
Connect AC power cables to the switch’s power supplies.

The Cisco Nexus switch does not have power buttons. They receive power when connected to a PDU with adequate
power.
Verify proper server operation by checking the behavior of PSU, NIC, and Solid State Drive (SSD)
LEDs on each server. For LED definitions, refer to the LED Definitions section in Appendix B:
Troubleshooting.
Check the light levels. For deployments utilizing a Cisco Nexus switch, refer to Measuring the Optical
Level through IOS, Document ID: 16150.
Light levels need to be between -2dB and -7dB for proper operation. Light levels outside of this range may
be indicative of a connectivity issue. If you find the levels out of range, see Appendix B:
Troubleshooting.
This completes the FNA hardware installation procedure. Figure 8 depicts a completed installation for
one FNA cluster.
If you are adding additional equipment to your initial FNA installation, see section 6 System Upgrades.
This section describes the process required to configure FNA equipment to peer with
Facebook’s network.
The FNA kit is preconfigured with the IP addresses that were specified in the order form. If IP
addresses need to be changed at any time, refer to the latest version of the FNA USB Boot Disk Creation
and IP Reconfiguration Guide (available upon request).
Connect a VGA monitor and USB keyboard to one of the FNA servers. After powering on the servers, you
will see the FNA Main Screen in Figure 10:
Configure the point-to-point connection. This connection will be configured on the Link-Aggregation
Control Interface (LACP). See Appendix C: Link Aggregation/Link Aggregation Control Protocol
(LACP) Configuration for additional LACP configuration details.
Untag the Virtual LAN (VLAN) link-aggregation interface.
Set a static route to the allocated subnets via the link-aggregation interface.
Facebook requires that the FNA have internet connectivity before BGP peering can be established. The BGP peering
session does not need to be routing before the FNA has internet connectivity.
Configure BGP settings.
Use the IPv4 and IPv6 addresses originally supplied in the FNA order form.
Enable Graceful Restart (GR). See Appendix D: BGP Configuration Details for additional
configuration information.
While enabling GR is not required, it is recommended to enable it. If you have questions about GR for FNA deployments,
contact fna-ops@fb.com.
Enable External BGP (EBGP) Multi-hop.

The Autonomous System Number (ASN) that you will be peering with is 63293.
Configure your router’s BGP settings. See Appendix D: BGP Configuration Details for
configuration information.
Once the router’s BGP settings have been properly configured, a latency based routing test will be initiated to optimize
routing paths. It is required that all FNA servable netblocks be announced for the purpose of this test. This process takes about
twenty-four hours to complete. BGP establishment is not required during this time. See section 4.4 Latency Based Routing
for more information.
The BGP peering session is intended to be a router-to-server configuration and not a switch-to-switch configuration.
Multi-hop sessions are supported.
After the BGP settings have been saved, the FNA will begin downloading application data to prepare it for
operation.
The time to finish the download depends on the cluster’s connection speed.
Within about two business days, BGP peering will be established:

The connection will be initiated by one of the FNA servers.
The connection will be coming from the 16th IP address in each allocated subnet
(e.g. 192.168.1.16, fe80::10).
FNA will not advertise any routes/prefixes.
The remote software delivery session will begin only after all of the FNA hardware has been powered on, the point-to-point
connection has been established, and routing to the internet is enabled.
This completes the network configuration of the FNA.
Contact fbticket@us.ibm.com to notify that the installation has been completed.
FNA applies inbound filtering to reject the following address spaces on BGP peering sessions to ensure
successful BGP establishment:
 Prefixes smaller than /24 (255 IPs)
 Default route
 Bogon prefixes
 RFC 1918
 CGNAT
 Facebook’s IP prefixes
It is critical that every FNA servable netblock is provided to Facebook in order to optimize routing paths for your FNA
deployment.
FNA utilizes a latency based routing test to optimize traffic routing. It is critical that all servable netblocks be
announced to optimize traffic routes. Prefixes announced to FNA implies those prefixes are eligible to serve
Facebook traffic.
The latency based routing test evaluates the round-trip latency of your network. This is accomplished by sending
a small sample of static content from the netblock population you announced. Initially, only a small sample set is
sent to the FNA cluster. The sample volume will ramp up during this test. This makes it possible for Facebook to
observe and construct the most optimal routing paths for your FNA deployment.
The accuracy of the latency based routing service is dependent on the percentage of FNA servable netblocks
announced. Providing a large percentage of FNA servable netblocks allows the service to construct more optimal
(low-latency) routing paths. A small percentage of netblocks announced (i.e., a test netblock) will not allow the
test to function as intended.
Please complete this checklist after you have installed the FNA equipment to verify proper
installation.
All ProLiant DL380 servers in the cluster are powered on.
Each power supply of each ProLiant DL380 server indicates an ON, Steady Green
state.
The Life (L) and Status (S) LEDs for each SSD in each ProLiant DL380 server indicate
an On, Steady Green state.
All accompanying LEDs for the connected fiber uplink SFP+ ports are green.
All network cables are seated properly to each ProLiant DL380 server.
The port 2 (P2) Link LED (LNK) on each NIC in the cluster indicates an On, Steady
Green state.
Light level readings for all uplinks are between -2dB and -7dB
This section describes how to add additional capacity your original FNA cluster and provides a
suggested growth path reference.
This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG
may cause service interruption. If this is the case, it is recommended to drain network traffic first. See section 7 Maintenance and
Monitoring for instructions on how to drain network traffic.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the
hardware is available.
Rack the additional servers, connect the peripheral cables (power and network), and power on the
servers. Utilize section 3 Hardware Installation for reference.
Connect (2) additional 10G fiber uplink cables from SFP+ Port 3 (S3) and SFP+ Port 4 (S4) on the switch
to the same router (or switch) next to the initial two 10G fiber uplinks are connected (see Figure 11).
Update the Link Aggregation (LAG) configuration on the router (or switch) to accommodate the additional
interfaces.
All cluster augments are implemented with an increment of four ProLiant DL380 servers.
This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG may cause service
interruption. If this is the case, it is recommended to drain network traffic first. See section 7 Maintenance and Monitoring for
instructions on how to drain network traffic.
This upgrade process requires installing QSFP to SFP+ adapter module. Go to section 6.2 QSFP to SFP+ Adapter
Module Upgrade for more information about this part.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the
hardware is available.
Rack the servers, connect the peripheral cables (power and network), and power on the additional
servers. Utilize the Hardware Installation section for reference.
Connect the provided QSFP to SFP+ adapter module assembly to QSFP Port 2 (Q2) on the Cisco Nexus
switch. See Figure 12 for adapter module location.1
If you are upgrading from a (16) server to a (20) server cluster, connect the 2nd provided
QSFP to SFP adapter module to QSFP Port 3 (Q3) on the switch.
Connect (2) additional 10G fiber uplink cables for each group of (4) servers from the QSFP to SFP+
adapter module to the same router (or switch) where the initial uplinks are connect (S1 ~ S4).
Update the LAG configuration on the router (or switch) to accommodate the additional interfaces.
The provided QSFP to SFP+ assembly only supports LR single-mode connections.
Refer to Figure : System Architecture Diagram for a diagram of a (20) server FNA cluster.
When upgrading an FNA cluster beyond eight servers, a QSFP to SFP+ adapter needs to be installed for each
additional group of four servers. If upgrading to a 12, 16, or 20 server FNA cluster, specifiy that this part will be
needed.
 Manufacturer: Cisco
 Part Number: CVR-QSFP-SFP10G
This section provides a reference for best practices regarding augmentations. This growth path optimizes for
deployment failover scenarios by reducing the quantity of Single Points Of Failure (SPOF).
When building an FNA cluster beyond eight servers, it is best practice to install the additional servers in a new rack and maintain
equal size clusters (as depicted in growth path 3 and 4).
When performing either singular or recurring maintenance jobs, email fna-ops@fb.com to

report the planned maintenance. Provide the following information in the email:
 To: fna-ops@fb.com
 Subject: [FNA ISP] Maintenance Notification
 Body: Provide the following:
o Company name and ASN
o Cluster name (e.g., FBSB1-1)
o When the maintenance will be performed (start and end times)
o If the maintenenace will be recurring (specify the interval)
o Reason for maintenance
o Contact information
Before powering down the FNA cluster for maintenance, all network traffic must be properly drained. This
procedure describes how to properly drain traffic from the cluster:
This procedure requires withdrawing prefixes. It is important that the BGP peering session is
maintained while withdrawing prefixes.
Withdraw prefixes.
While maintaining the BGP peering session, begin withdrawing BGP prefixes.
Traffic will begin to drain.
Traffic will fully drain in less than one hour.
You should shut down the FNA only if it is absolutely required and traffic has been properly drained from the
system (see 7.2 Draining Network Traffic). This should be a rare occurrence. To shut down the FNA, follow the
procedure outlined below or contact fna-ops@fb.com for assistance:
Connect a monitor and keyboard to the server you want to power off.
Use the arrow keys navigate to the Shut Down text.
Press the [Enter] key. The FNA will power off.
Facebook regularly monitors internet reachability from various simulated endpoints. When reachability issues are
detected on these Virtual IPs (VIPs), Facebook may drain traffic from an FNA cluster. This measure ensures that
Facebook maintains a high quality of service for the Facebook platform.
The Terms of Service agreements that you signed when you installed the FNA in your facility
included specific language pertaining to the disposal of storage media. That language is provided
here for your convenience.
Upon signing the Terms of Service, the ISP agrees to:
 Only deal with and process Edge user data in compliance with, and subject to, the
instructions received from Edge and will not use or process the Edge User Data for any
other purpose whatsoever.
 Take custody of and secure, and support the destruction of, all server storage media (example: hard disk,
solid state disks, flash memory cards, etc.) that are uninstalled from the servers, and provide secure
handling, storage, packing, and shipping of the storage media per IBM provided instructions and at IBM's
expense to an IBM designated asset recovery center for destruction.
The Network Partner Portal is the central hub for ordering FNA kits, reviewing deployment
information and metrics, and troubleshooting issues you may encounter with your
deployment.
This section describes how to use various features within the Network Partner Portal.
URL: https://partners.facebook.com/network/
Logging into the NPP is as easy as logging into Facebook. If you have an existing Facebook account associated
with your work email address, the portal will use that account to log in. If you have a Facebook account
associated with your personal email address, you must associate your work email address with your Facebook
account.
If you don't have a Facebook account, you need to create one. Go to http://www.facebook.com and follow the
instructions for creating an account. To associate your work email address with your Facebook account, perform
the following steps:
Click the Menu icon ( ) located at the top right of a Facebook page. Click Settings.
Click Email.
Click Add another email or mobile number.
Enter your work email address in the New Email field. Click Add.
Re-enter your Facebook password when prompted.
A confirmation code will be sent to verify the email address. Click Confirm and enter the code. Click OK.
Click Save Changes.

The portal allows you to authorize others as administrators and set their permissions.
To add a new administrator, perform the following steps:
Click the Settings icon ( ) located at the top right of the portal page.
Enter the name and email of the user you wish to make an administrator.
Click Add.
An invite will be sent to the user's email address. Their permission status will remain "pending" until they
accept the invite.
Admin access grants access and control to all settings and data within the portal.
You can also set the user's permissions to "Read-Only Access". This gives the user complete access to all portal data, but
does not allow access to any settings or controls. To set a user's permission to read-only access, follow the steps in the
Modifying Permissions section and select Read-Only Access.
To modify an existing user's permissions, perform the following steps:
Click the Settings icon ( ).
Click the Edit link on the user's card.
Click the chevron icon ( ) below the user's email address. Select either Admin Access or Read-Only
Access.
Click Done.
To remove a user from the portal, perform the following steps:
Locate the user you would like to remove.
Click the Remove link on the user's card.
Enter your Facebook password if prompted.

FNA deployments need to connect to your network. To enable this connection, you need to provide your ISP's
Autonomous System Number (ASN). Facebook requires this data to estimate the number of servers your FNA
deployment will require. Facebook also uses this information during the provisioning process to properly route
traffic to the FNA.
To add your ASN, follow these steps:
Click Autonomous Systems located in the left menu.
Enter your ASN and AS Name.
Enter your ISP's ASN and AS Name.
Click Add.
To remove an ASN, follow these steps:
Click Autonomous Systems located in the left menu.
Click the Remove link next to the ASN entry you wish to remove.
You can enter multiple ASNs. All ASNs should be kept up-to-date.
If the ASN entry has a yellow clock icon ( ), it is currently being reviewed by Facebook.
Requesting an FNA kit is a multi-step process that takes several weeks to complete. Facebook evaluates
submissions for qualified ISPs that can utilize an FNA. If your application is approved to receive an FNA,
Facebook will send an FNA kit to the designated ISP address.
To request an FNA kit, follow these steps:
Click the Support link at the top of the page. The Support page will appear.
Click the Request a New Kit hyperlink.
Enter the deployment country where the FNA will be installed. Click Next.
Select the Type of Kit.
Review the information and click the Begin Order Form button. An online order form will appear in a new
window or tab. Use table 6 as a guideline for order form field entry:
Provide contact information for the primary This can be the engineering contact.
Shipping Contact contact that will receive FNA shipments.
Provide contact information for the primary This can be the engineering contact.
contact that will install or lead installation of
Installation Contact the FNA equipment.
Provide contact information for the primary

Engineering
engineering contact that can answer
Contact
technical questions about the network.
Provide the location where FNA equipment

Shipping Address
will be sent.
Site Installation Provide the location where FNA equipment

Address will be installed.
Special Shipping Provide any notable unique shipping This is an optional field.
Instructions instructions.
Rack Name Enter a name for the FNA rack. This is an optional field.
 Provide power source

requirements for the FNA
deployment (AC or DC).
Facility /
Deployment  Provide the type of optical
Technical transceiver your deployment would
Information require – either long range (LR) or
short range (SR)
 Provide the power outlet type your
deployment would require.
Networking and IP
 Provide the primary peer ASN.
Information  Provide any additional ASNs or
communities to be served.
Provide IPv4 allocation for caches, gateway

address and network, cache address and
 Gateway and Cache Peer-to-Peer
(P2P) IP addresses must be within
network, and Border Gateway Protocol
the same subnet.
IPv4 Settings (BGP) peer address.
 BGP address / network should not
overlap with IP allocation for
caches.
Provide IPv6 allocation for caches, gateway

address and network, cache address and
 Gateway and Cache Peer-to-Peer
(P2P) IP addresses must be within
network, and BGP peer address.
the same subnet.
IPv6 Settings
 BGP address / network should not
overlap with IP allocation for
caches.
Fill out and submit any required order forms located in the Required Forms section near the top of the
order form. Forms that require submission are denoted with the "X" icon ( ). Forms that are currently
pending review are denoted with the "exclamation" icon ( ).
Click Submit if you have completed the order form or Save Draft if you plan to complete the order form at
a later time.
If Facebook selects you to participate in the FNA program, International Business Machines (IBM) will contact the
shipping contact you indicated with further details and will make arrangements to deliver FNA equipment. IBM is
Facebook's logistics partner for FNA equipment.
Once you have installed your FNA equipment and it is brought online, access the Manage FNA to:
 Review Installed Kit Information
 View Traffic Information
 Review Contact, Exchange, and Facility Information
 View Peering Sessions
The installed kit information page allows you to view the number of servers per cluster, IPv4 and IPv6 network
addresses, advertised prefixes, and disk image. To view this page, follow these steps:
Click the Caching tab located at the top of the portal.
Click Manage FNA.
This page displays FNA information sorted by cluster name. You can also request a new kit by clicking
Request a New Kit in the top-right corner.
FNA traffic metrics are located in the Insights page. This page allows you to observe cache-fill and retransmit
information sorted by FNA cluster name. To view traffic information, follow these steps:
Click the Caching tab located at the top of the portal.
Click Insights.
The FNA Traffic information will appear. Each plot represents either "To Subscribers" or cache fill for each
FNA cluster installed.
To view traffic retransmission information, click the button at the top left of the graph. Click FNA
Retransmits. This page will show traffic retransmission as a percentage of the total FNA traffic.
The contact information you provided on the FNA order form can be found in the Peering page. To review this
information, follow these steps:
Click the Peering tab.
Click Peering Summary (this should load by default).
This page allows you to review your contacts, exchanges, and facilities information.
The contact information you provided on the FNA order form can be found in the Peering page. To review this
information, follow these steps:
Click the Peering tab.
Click Peering Sessions.
This page allows you to view peering session details, including ASN, location, peer type, link speed, router
address, peer port, and status.
If you have issues receiving equipment or installation and configuration issues, use Table 7 to
contact the most appropriate support team for assistance.
Table 7: Support and Contact Information
Email:
fna-ops@fb.com
 Network Support is
Facebook Required
Network  BGP Peering Issues
Appliance
 Assistance with Draining
Network Traffic
Email:
fbticket@us.ibm.com
 Hardware Support is  Notification of Hardware
Required Installation Completion
 Hardware is missing from
order. Provide the following:
IBM o Missing Item(s)
o ISP Name
o ISP Address
o Equipment
Received Date
o Your Contact
Information
In some cases, it may be necessary to return the equipment. It is recommended to keep the original packaging material for
two months.
What type of data is being stored on FNA?
An FNA cluster will cache the following types of data:
 Static Facebook content

 Static Instagram content
FNA does not cache WhatsApp or Oculus content.
How much traffic can the FNA cluster serve?
The amount of traffic that the FNA cluster may serve varies based on:
 The size of the cluster and the number of available machines

 The amount of prefixes being advertised / number of users
 The RTT latency to advertised prefixes
 Content type variance: profile pictures, large images, and video
Each server in the cluster is capable of delivering ~4Gbps of traffic. However, depending on the deployment
region, service type (e.g., mobile vs. fixed line), and variance in content type (e.g., images and video), the FNA
cluster may deliver less overall traffic (~3Gbps is expected).
Generally, a high variance in content type will yield lower traffic throughput and low content type variance will yield
higher traffic throughput.
The installation procedure indicates installing the supplied switch directly to our existing router. Our
facility aggregates to our existing switches before connecting to a router. How does this modify the
installation process?
FNA provides aggregate connections from the cache servers directly to the router. However, if you connect the
supplied FNA switch to an intermediary aggregation switch, this configuration is fine and the logical topology will
remain the same.
Is FNA Carrier-Grade NAT (CGNAT) compatible?
Yes. FNA is compatible with CGNAT for the RFC 1918 and RFC 6598 address spaces.
I’ve advertised a certain prefix but it doesn’t seem to be served by FNA. Why is this happening?
Advertising a prefix marks means it as eligible to be served from FNA, but does not guarantee it will be served by
FNA. There are a few reasons why FNA may not be serving a prefix which has been advertised to it:
 If it has been less than 24 hours since the FNA cluster has been online or since you have started
advertising the prefix (see Latency Based Routing). Facebook may not have collected enough latency
samples in order to decide whether the FNA cluster will serve the prefix.
 If Facebook is observing less latency to the prefix from a nearby Point of Presence (PoP), then the PoP
will be preferred for serving the prefix.
 If the system resources (e.g. CPU utilization) have been exhausted.
Which routes should I advertise towards Facebook and how does failover occur between FNA’s?
In most cases, the best option is to announce all customer prefixes. In cases where there may be congestion from
region to region, the best practice is to announce local routes only.
Why should I advertise all of my prefixes?
Facebook’s content routing platform measures Round Trip Time (RTT) latency between each Point of Presence
(PoP) and FNA deployment to all prefixes being advertised by a peer ISP. Based on this measurement and a
number of other factors (such as available system resources and link capacity), a client is routed to the most
optimal location for making content requests.
What is the FNA cluster naming convention?
FNA cluster naming is based on the International Air Transport Association (IATA) code, the site number, and the
cluster number. ‘F’ is prepended before the IATA code (which represents the nearest airport recognized by the
IATA); the site number and cluster number are enumerated values.
For example, FBKK5-3 represents the 3rd cluster in the 5th location in Bangkok (where ‘BKK’ represents the IATA
code for Suvarnabhumi Airport).
How does an ISP qualify to participate in the program?
The following is a summary of the qualification criteria to receive an FNA kit:
 The ISP is considered an ‘eyeball’ ISP (not a transit ISP).

 The installation site has at least 5Gbps of peak Facebook traffic in a region where a Facebook PoP is not
located.
 The ISP maintains the ability to deliver Facebook content from a single deplyment location in an efficient
manner.
 The ISP has the ability to allocate IPv6 address space for the FNA cluster.
 The installation site has the ability to provide the following (summarized) space, power, and network
requirements:
o Space: 9RU
o Power: Minimum of 1kW

o Network: 2x10G uplink connectivity, BGP peering session capability
 IPv4: Up to /29 and /26 address space
 IPv6: Up to /125 and /64 address space
 The installation site is located in a developing market / high-growth region.
This section provides troubleshooting resources for events that may be encountered during installation or
operation.
Each ProLiant DL380 server and Nexus switch has LEDs that can be used to diagnose the FNA for issues. These
LEDs also useful for verifying whether the FNA was installed correctly and for receiving quick technical support.
The server is powered off or the power supply has

Off ____________________
failed.
On, Steady Green ____________________ The power supply is on.
Activity
On, Blinking Green ____________________ There is data activity on the card.
(A)
The drive has sufficient life for programming and

On, Steady Green ____________________
erasing the memory.
The drive has approximately 10% or less life

Life (L) On, Steady Yellow ____________________ remaining for programming and erasing the
memory.
The drive no longer has program or erase cycles.

On, Steady Red ____________________
Data can be read, but writing data is at risk.
On, Steady Green ____________________ The drive is operating normally.
A network administrator is toggling the LED to

On, Blinking Green ____________________
locate the server.
Status Drive warning. One of the following has occurred:

(S)
On, Steady Yellow ____________________
 The maximum temperature threshold has
been reached.
 Another issue has occurred.
On, Blinking Red ____________________ A firmware fault code has been issued.
Fatal drive warning. One of the following has

occurred:
 Drive failure.
 Drive has exceeded its maximum
Status temperature threshold.
(S)
On, Steady Red ____________________  The RAID volume has failed.
 The RAID volume is not configured.
 The drive’s internal backup power supply has
failed.
 Another issue has occurred.
Activity
On, Blinking Green ____________________ There is network activity.
(ACT)
Link A connection exists between the server and the

On, Steady Green ____________________
(LNK) network.
Power
Supply On, Steady Green ____________________ The power supply is on.
Fan On, Steady Green ____________________ The fan is powered on.
On, Steady Green ____________________ The system is operating normally.
System The system has posted an environmental issue

On, Steady Orange ____________________
Status code.
The system has posted an environmental issue

On, Steady Red ____________________
code.
In cases where light levels are outside the normal range (between -2dB and -7dB), utilize Table 9 to resolve the
issue.
Table 9: Signal Level Troubleshooting
The signal is too strong. This could be Fix with your normal operating procedures for strong
Above -2 caused by many reasons. signals.
None (Normal Operating Range) None (Normal Operating Range)

-2 to -7
 A degraded fiber optic cable is  Disconnect, clean, and reconnect the fiber optic
installed cable(s) and/or SFP+ optical transceiver(s).
-8 to -20  A degraded SFP+ optical
If this does not resolve the issue, either replace
transceiver is installed
TX/RX hardware or solve with your normal fiber
 A fiber optic cable has a dirty replacement procedures.
connection
 A fiber optic cable is bent  Inspect the fiber optic cable(s) along its length.
 Network connectivity has not been
Gently roll the fiber if you observe bends or
kinks.
-21 to -40 established
 All cables are connected to their
 Check that the port connection is enabled
proper ports, but the port(s) is  Ensure the cables are connected to the proper
disabled port interface. See Figure 12 for port interfaces.
 All ports are enabled, but a cable is
connected to the incorrect port
The following are examples of LACP configurations for the most common Cisco IOS and Juniper Junos routers.
Apply the lines of code (below) that are appropriate to your specific router model.
The following configuration settings should be applied to a Juniper Junos router for FNA LACP compatibility:
ae1 {
aggregated-ether-options {
lacp {
active;
periodic fast;
}
}
unit 0 {
description FACEBOOK-FNA;
family inet {
address <address/mask>;
}
family inet6 {
address <address/mask>;
}
}
}
xe-0/0/0 {
description FACEBOOk-FNA;
gigether-options {
802.3ad ae1;
}
}
xe-1/0/0 {
description FACEBOOk-FNA;
gigether-options {
802.3ad ae1;
}
}
To confirm whether the LACP settings were properly applied, run the following commands:
show lacp interfaces xe-4/3/3
show interfaces ae3 | grep Physical
If the following output appears, the LACP settings were properly configured for the Juniper Junos router:
show lacp interfaces xe-4/3/3
Aggregated interface: ae4

LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-4/3/3 Actor No No Yes Yes Yes Yes Fast Active
xe-4/3/3 Partner No No Yes Yes Yes Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
xe-4/3/3 Current Fast periodic Collecting distributing
show interfaces ae4 | grep Physical
Physical interface: ae4, Enabled, Physical link is Up

The following configuration settings should be applied to a Cisco IOS-XR router for FNA LACP compatibility:
!
interface Bundle-Ether1
description FACEBOOK-FNA
ipv4 address <address/mask>
ipv6 address <address/mask>
!
interface TenGigE0/0/0/0
bundle id 1 mode active
lacp period short
!
interface TenGigE0/1/0/0
bundle id 1 mode active
lacp period short
!
To confirm whether the Cisco IOS-XR LACP was properly configured, run the following command:
show port-channel sum interface port-channel 911
If the following output appears, the LACP settings were properly configured on the Cisco IOS-XR router:
show port-channel sum interface port-channel 911

Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
911 Po911(RU) Eth LACP Eth18/9(P) Eth18/10(P) Eth18/11(P)
Eth18/12(P)
The following are examples of BGP peering code for the most common Cisco OOS and Juniper Junos routers.
These examples allow you to configure BGP peering sessions to send Facebook traffic to the FNA cluster.
Contact Facebook (fna-ops@fb.com) if you encounter problems when setting up BGP peering.
After the FNA cluster has network access, it will begin downloading application data that enables the FNA to operate. All
application data must be installed before the BPG session is established. The application data may take some time to
download.
The following router BGP configuration should be used for FNA deployments that utilize a Juniper Junos router:
protocols {
bgp {
group FACEBOOK-FNA-IPv4 {
neighbor <FNA IPv4 address> {
multihop;
family inet {
unicast;
}
export FACEBOOK-FNA-EXPORT;
peer-as 63293;
}
}
group FACEBOOK-FNA-IPv6 {
neighbor <FNA IPv6 address> {
multihop;
family inet6 {
unicast;
}
export FACEBOOK-FNA-EXPORT;
peer-as 63293;
}
}
}
}
policy-options {
policy-statement FACEBOOK-FNA-EXPORT {
term export {
from {
prefix-list FACEBOOK-FNA;
}
then accept;
}
term reject {
then reject;
}
While enabling GR is not required, it is recommended that it is enabled. If you have questions about GR for FNA
deployments, contact fna-ops@fb.com.
To enable BGP Graceful Restart on a Juniper Junos router, perform the following procedure:
Utilize Juniper’s Configuring Graceful Restart Options for BGP reference document.
Navigate to the appropriate hierarchy level and change the disable statement to enable.
Evaluate whether GR was successfully negotiated by running the following command:
show bgp neighbor <IP Address> | grep Options
Locate the applicable Junos device and evaluate whether GracefulResart is within the Options output:
show bgp neighbor <IP Address> | grep Options
Options: <Preference AdvertiseInactive GracefulRestart LogUpDown AddressFamily PrefixLimit Refresh>

Options: <MtuDiscovery>
If the Options output string contains GracefulRestart, GR was successfully negotiated.
By default, GR is disabled on Junos devices.
Enabling Graceful Restart on the Junos device does not imply that it is in use. Both sides of the session must support and
negotiate Graceful Restart.
The following router BGP configuration should be used for FNA deployments that utilize a Cisco IOX-XR routers:
!
router bgp x
!
address-family ipv4 unicast
!
!
neighbor <FNA IPv4 address>
remote-as 63293
route-policy FACEBOOK-FNA-EXPORT out
!
neighbor <FNA IPv6 address>
remote-as 63293
route-policy FACEBOOK-FNA-EXPORT out
!
While enabling GR is not required, it is recommended that it is enabled. If you have questions about GR for FNA
deployments, contact fna-ops@fb.com.
To enable BGP Graceful Restart on a Cisco NX-OS router, perform the following procedure:
Utilize Cisco’s Configuring a Graceful Restart procedure.
show ip bgp neighbors <IP Address> | grep "Graceful Restart"

If the output string ‘Graceful Restart capability: advertised received’ appears, GR was
successfully negotiated.
The following diagram demonstrates the route of a Facebook request to your ISP both before and after the FNA
deployment:
QSFP Port 1 is internally multiplexed with the (4) SFP+ uplink ports for both the Cisco Nexus 3132Q and 3132Q-X. The (4)
SFP+ ports and QSFP Port 1 cannot be utilized at the same time.
The following diagram shows the topology of a (20) server FNA installation:
This diagram depicts an FNA cluster with the maximum number of supported servers per switch. However, when building
an FNA cluster beyond eight servers, it is best practice to install the additional servers in a new rack and maintain equal size
clusters (as demonstrated in Section 6.2 Growth Path).
Table 10: FNA System Benchmarks
Expected Bandwidth Offload 75%
Maximum Sustained Throughput 4Gbps per cluster

Table 11: AC Power and Temperature Requirements
Input Current (by PSU) 8.9A @100V

4.3A @ 200V
Steady State Power 750W @ 100 V to 120V

750W @ 200V to 240V
Frequency 50-60Hz
Dual Path Power Required
Dual PDUs Required
Operating Temperature Range 10°C to 35°C
Typical Operating Power 210W with Twinax @ 100% load; (2) PSUs
240W with SR4 optics @ 100% load; (2) PSUs
Maximum Power 340W
Input Voltage by PSU 100V to 240V
Frequency 50-60Hz
Efficiency 89% to 91% @ 220V
Dual PDUs Required
Typical Heat Dissipation 717 BTU/hr (with Twinax at 100% load)

819 BTU/hr (with SR4 optics at 100% load)
Maximum Heat Dissipation 1160 BTU/hr
Cordset Rating 10A @ 250V
SKU1 ~ 800W (per SKU)

Table 12: DC Power and Temperature Requirements
Input Voltage Range (V DC) 36-72V DC
Nominal Input Voltage (V DC) 36/48/72V DC
Maximum Rated Output Wattage 750W @ 36V DC / 48V DC / 72V DC

Rating (W)=
23A @ 36V DC
Nominal Input Current (A) 17A @ 48V DC
11A @ 72V DC
Maximum Rated Input Wattage Rating 815W @ 36V DC

(Watts) 805W @ 48V DC
795W @ 72V DC
Maximum British Thermal Unit Rating 2780 BTU-Hr @ 36 V DC

(BTU-Hr) 2740 BTU-Hr @ 48 V DC
2720 BTU-Hr @ 72 V DC
Typical Operating Power 210W with Twinax @ 100% load; (2) PSUs
240W with SR4 optical transceiver @ 100% load; (2) PSUs
Maximum Power 340W
Input Voltage by PSU –40V DC to –72V DC
Maximum Current 33A
Power Supply Efficiency 88% @ -48VIN (Typical Power Draw)

85% @ -48VIN (Maximum Power Draw)
Dual PDUs Required
Typical Heat Dissipation 717 BTU/hr (with Twinax @ 100% load)

819 BTU/hr (with SR4 optical transceiver @ 100% load)
Maximum Heat Dissipation 1160 BTU/hr

Refer to the specifications for the equipment included in your FNA kit (this can vary according to when the order
was placed).
Table 13: Hardware Dimensions and Weight
Dimensions 8.74 x 44.55 x 69.85 cm (3.44 x 17.54 x 27.50 in)

(Height x Width x Depth)
Weight (Approximate)

Weight (Approximate)

Weight 9.3 kg (21.5 lb)

Weight (Approximate) 9.3 kg (21.5 lb)
Length 2 Meters
Plug SS10A
Connector HS10S
American Wire Gauge (AWG) 30
Length 3 Meters
Dimensions 8.5 x 13.4 x 56.5 mm

Weight 75 grams (2.64 ounce)

Bien plus que des documents.

kupdf.com_fna-installation-and-operation-guide-v804.pdf

Transféré par

Informations du document

Description :

Droits d'auteur :

Formats disponibles

kupdf.com_fna-installation-and-operation-guide-v804.pdf

Transféré par

Description :

Droits d'auteur :

Formats disponibles

Titres liés

Facebook Network Appliance Deployment | 1

© 2015-2016 Facebook. All rights reserved.

Linux® is a registered trademark of Linus Torvalds in the United States.

Mac® and Mac OS® are registered trademarks of Apple Inc.

Hewlett-Packard® and HP® are registered trademarks of Hewlett-Packard Company.

Facebook Network Appliance is Facebook’s content caching program. FNA provides

Table 1: FNA SKU1 Provided Equipment

(1) Cisco Nexus 3132Q-X switch2 1RU

(4) HP ProLiant DL380 Generation 9 (Gen9) server3 2RU

(4) 4-Post rail kit N/A

(8) C13-C14 AC Power Cord 2 Meters

(4) Network Cable 3 Meters

(1) HP ProLiant DL380 Generation 9 (Gen9) server user manual N/A

(1) Cisco Nexus 3132Q-X switch user manual N/A

(2) SFP+ optical transceiver, LR or SR, (2) per SKU4 N/A

Table 2: FNA SKU2 Provided Equipment

(4) HP ProLiant DL380 Generation 8 (Gen9) server2 2RU1

(4) 4-Post rail kit N/A

(8) C13-C14 power cable 2 Meters

(4) Network cable 3 Meters

HP ProLiant DL380p Generation 8 (Gen8) server user

(1) Text configuration file N/A

(2) SFP+ optical transceiver, LR or SR, (2) per SKU3 N/A

Table 4: FNA Pre-Install Checklist

n x 10 Gigabit (10Gb) available ports on ISP switch or router1

n available fiber patch cables (multi-mode or single-mode)1

n available input power connections2

Sufficient power (1kW) for each Stock Keeping Unit (SKU)

Sufficient rack space is available3

Install the servers into a compatible server rack.

Install the switch in the rack.

Connect the switch to your facility’s router.

Power on the FNA hardware.

Connect AC power cables to the switch’s power supplies.

Untag the Virtual LAN (VLAN) link-aggregation interface.

Configure BGP settings.

Enable External BGP (EBGP) Multi-hop.

Multi-hop sessions are supported.

Within about two business days, BGP peering will be established:

This completes the network configuration of the FNA.

Contact fbticket@us.ibm.com to notify that the installation has been completed.

 Prefixes smaller than /24 (255 IPs)

All ProLiant DL380 servers in the cluster are powered on.

The provided QSFP to SFP+ assembly only supports LR single-mode connections.

When performing either singular or recurring maintenance jobs, email fna-ops@fb.com to

Traffic will begin to drain.

Traffic will fully drain in less than one hour.

Use the arrow keys navigate to the Shut Down text.

Press the [Enter] key. The FNA will power off.

Upon signing the Terms of Service, the ISP agrees to:

Click Add another email or mobile number.

Re-enter your Facebook password when prompted.

Click Save Changes.

To add a new administrator, perform the following steps:

To modify an existing user's permissions, perform the following steps:

Click the Settings icon ( ).

Click the Edit link on the user's card.

To remove a user from the portal, perform the following steps:

Locate the user you would like to remove.

Click the Remove link on the user's card.

Enter your Facebook password if prompted.

To add your ASN, follow these steps:

Click the Settings icon ( ).

Click Autonomous Systems located in the left menu.

Enter your ASN and AS Name.

Enter your ISP's ASN and AS Name.

To remove an ASN, follow these steps: