How to monitor and analyze network capacity u... https://www.reddit.com/r/networking/comments...

LOG SIGN
Search r/networking
IN UP

 124   How to monitor and an CLOSE

Posted by u/Engineer-at-Optiver 6 months ago 


124
How to monitor and analyze network

capacity usage
I've recently been working on an interesting problem
that I wanted to share: how to monitor and analyze
network capacity usage. I was brought onto the
problem because we face a difficult challenge in
Optiver–our business is driven by financial markets,
which can unexpectedly become busy when, for
example, major news events occur. When these market
spikes happen, applications across our network
suddenly try to send data as quickly as possible to
financial exchanges across the world where we trade.
These synchronized flows can then collide, overloading
the capacity of our network equipment, leading to
packet drops and long delays as the entire system
needs to retransmit data. These delays can be very
costly – packet drops can cause delays of multiple
seconds in distributing our theoretical prices, far longer
than the timescale at which markets move today.

Image 1: Example of incoming packet rates on a

logarithmic scale to three machines (solid lines) vs
historical baselines (dashed lines) around the US market
open (15:30 Amsterdam) and an unexpected news event
(~17:05)

From an infrastructure perspective, there are two ways

we could try to solve this problem. On the one hand, we
could make a major investment in the capacity of our
infrastructure, buying new network switches with deep
buffers, additional network lines, etc. On the other
hand, it wasn't clear whether we were using our
capacity wisely–for a long time, many people in the
company had assumed our core infrastructure had
infinite capacity. The process of gathering the data to
check how we used our capacity turned out to be a very
interesting and exciting project.

1 of 3 8/14/19, 1:34 PM
How to monitor and analyze network capacity u... https://www.reddit.com/r/networking/comments...

LOG SIGN

IN UP

 124   How to monitor and an CLOSE

2 of 3 8/14/19, 1:34 PM
How to monitor and analyze network capacity u... https://www.reddit.com/r/networking/comments...

LOG SIGN

IN UP

 124   How to monitor and an CLOSE

3 of 3 8/14/19, 1:34 PM