Home Data Center Project: Load balancing multiple ethernet ports

Published: 26 October 2019

The following is part of a series of posts called "Building a data center at home".

Living in the SF bay area, you can acquire 2nd hand data center equipment relatively cheap. The following are a series of posts detailing my deep dive in building a cluster with data-center equipment at home (often called a homelab) which consists of 48 CPU cores, 576Gb RAM, 33.6TB storage across 60 x 6Gb/s HDD's with a combined weight of just over 380lb/170kg within a budget of $3000.

So far in my home data center project, each of the servers that I’ve set up is only utilizing 1 ethernet port. All of the servers have only 1 NIC exposing 4 ethernet ports except for the X400 which has two motherboard ports and a PCI card with 2 ports.

This is done in Ubuntu Server 18.04 via netplan. To do this I edit the current configuration file in /etc/netplan and replace it with the following:

network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: no
    eno2:
      dhcp4: no
    eno3:
      dhcp4: no
    eno4:
      dhcp4: no
  bonds:
    bond0:
      dhcp4: no
      interfaces:
        - eno1
        - eno2
        - eno3
        - eno4
      parameters:
        mode: balance-alb
      addresses:
        - 10.0.0.210/24
      gateway4: 10.0.0.1
      nameservers:
          search: [otherdomain]
          addresses: [1.1.1.1]

This configuration disables dhcp for each of the ports, then adds each of them to a bond interface which uses mode 6 balance-alb which is adaptive load balancing. This mode allows for fault tolerance if any of the ports go down, and implements incoming load balancing due to special ARP replies from the NIC based on the load of each port. This configuration then also adds a static IP to the bond so that each all of the ports are assigned to a static IP (in this case 10.0.0.210).

For the X400, the following configuration is used so that each NIC has it’s own IP. I’m unsure if bonding across different NICs is possible, I believe it might be, but for now I’m happy to split them so that one NIC is dedicated to NFS in the Kubernetes cluster, and the other NIC is used for access and linking up the X400 with the Kubernetes cluster in much the same way the= other servers connect to one another.

network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0:
      dhcp4: no
    enp4s0:
      dhcp4: no
    enp8s0f0:
      dhcp4: no
    enp8s0f1:
      dhcp4: no
  bonds:
    bond0:
      dhcp4: no
      interfaces:
        - enp3s0
        - enp4s0
      parameters:
        mode: balance-alb
      addresses:
        - 10.0.0.220/24
      gateway4: 10.0.0.1
      nameservers:
          search: [otherdomain]
          addresses: [1.1.1.1]
    bond1:
      dhcp4: no
      interfaces:
        - enp8s0f0
        - enp8s0f1
      parameters:
        mode: balance-alb
      addresses:
        - 10.0.0.221/24
      gateway4: 10.0.0.1
      nameservers:
          search: [otherdomain]
          addresses: [1.1.1.1]

I haven’t yet tested the speed differences, but the fault tolerance works just fine. I’ve set a ping on interval a given server from one of the other servers in the project and then changed which ethernet ports where attached or not. THe only failure that occurred is when all ethernet ports where disconnected. Will need to test the speed in future though, but theoretically, the throughput for each server to another server in the cluster should be quadrupled.