Flannel vs Calico a battle of L2 vs L3 based networking.

We discuss today the networking in container world and primarily in context of K8s .

Flannel is an overlay network mechanism where as Calico is basically a pure L3 play.

Flannel works by using a vxlan device in conjunction with a software switch like linux bridge or ovs.

Container A when tries to reach container B on different host the traffic is pushed to the bridge on host
A via the VETH pair. The bridge then based on ARP tries to get the mac of container B. Since container B
is not on the host the traffic by bridge is forwarded at L2 to the vxlan device (software device) which
then allows flannel daemon software to capture those packets and then wrap then into a L3 packet for
transport over a physical network using UDP. Also vxlan tagging is added to the packet to isolate them
between tenants.

Flannel shown diagrammatically

In case of Calico, the approach is little different.

Calico injects a routing rule inside the container for gateway at this IP 169.254.1.1.

default via 169.254.1.1 dev eth0

169.254.1.1 dev eth0 scope link

What this means is that any traffic from the container first tries to go to the default gateway IP. Since
the default gateway IP is reachable at eth0 , the ARP request is sent to eth0 for determining the mac
address for gateway IP.

The trick here is the arp proxy configured at the veth device on host side.

This arp proxy responds back with its mac for the ARP request for 169.254.1.1.

Post this resolution the packets are sent to the veth device with source IP of container and destination
IP of target container. From here on the L3 routing takes effect which knows how to route for the
destination container IP. This route synchronization is provided via BGP protocol.

So here you can see in Calico solution, we got rid of software bridges as well as preserved the source IP.

Also the overlay complexity is out of the picture and it’s a pure L3 solution.