HowTo: NIC Bonding in CentOS 5 and 6

by Justin Edmands

I have been bonding network cards at work left and right. Our first project was to get our tape backup server on this system. We immediately saw the effect of bonding the network cards by checking the throughput to the machine. Our EMC NetWorker backup software -and- our Cacti graphing server both reported roughly 160-180MB/s while writing to tape. On top of maxing out tape throughput, this decreased our backup window and provided switch redundancy. Ideally you would connect one port to a different switch in your racks to help balance the loads.

Do you really need multiple NICs?

[root@boron ~]# ifconfig
bond0     Link encap:Ethernet  HWaddr XX:XX:XX:XX:45:ED  
          inet addr:192.168.212.166  Bcast:192.168.213.255  Mask:255.255.254.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:23201084537 errors:134 dropped:0 overruns:0 frame:134
          TX packets:7857662111 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:33751562832054 (30.6 TiB)  TX bytes:971139704959 (904.4 GiB)

eth0      Link encap:Ethernet  HWaddr XX:XX:XX:XX:45:ED   
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:11743246375 errors:90 dropped:0 overruns:0 frame:90
          TX packets:2453797534 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:17287802062602 (15.7 TiB)  TX bytes:158836452607 (147.9 GiB)
          Interrupt:177 Memory:91a80000-91aa0000 

eth1      Link encap:Ethernet  HWaddr XX:XX:XX:XX:45:ED   
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:11457838162 errors:44 dropped:0 overruns:0 frame:44
          TX packets:5403864580 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:16463760769452 (14.9 TiB)  TX bytes:812303253030 (756.5 GiB)
          Interrupt:193 Memory:91980000-919a0000 

As you can see, the above machine for just 45 days of uptime has seen 30+TB of traffic. Since this is our backup server, we needed a way to handle to loads while it tries to write to tape. This shows that the load isn't 50/50, but pretty close at 15.7 TiB on eth0 and 14.9TiB on eth1

Step 1

Get two NICs recognized by your system as eth0 and eth1

put some ifconfig and system-config-TUI stuff here

Step 2

Ethernet Config

Edit your network-scripts for eth0 and eth1

[root@samba-server ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

[root@samba-server ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

Bonding Config

Edit your network-scripts for bond0

[root@samba-server ~]# vim /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.212.161
NETMASK=255.255.254.0
GATEWAY=192.168.212.1
NETWORK=192.168.212.0
USERCTL=no

Step 3

Configure your modprobe configuration files to understand the "bonding"

CentOS 5 Modprobe

[root@fileserver ~]# vim /etc/modprobe.conf
alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptspi
alias scsi_hostadapter2 ata_piix
alias eth0 e1000
alias eth1 e1000
alias bond0 bonding
options bond0 mode=2 miimon=100

CentOS 6 Modprobe

Modprobe for CentOS 6 has changed to /etc/modprobe.d/bonding.conf. The reason for this change is explained

[root@samba-server ~]# vim /etc/modprobe.d/bonding.conf
alias bond0 bonding
options bond0 mode=2 miimon=100

Step 4

Restart your network interfaces

[root@samba-server ~]# service network restart
[root@fmrif-samba ~]# service network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                       [  OK  ]
Bringing up loopback interface:                           [  OK  ]
Bringing up interface bond0:                               [  OK  ]

Complete!

Other options

You can configure your network cards to have a different type of bond. This is configured by choosing a different type of bond in your modprobe configuration file. Below is a list of the different bonding modes according to the Red Hat docs for RHEL6 :

balance-rr or 0

Sets a round-robin policy for fault tolerance and load balancing. Transmissions are received and sent out sequentially on each bonded slave interface beginning with the first one available.

active-backup or 1

Sets an active-backup policy for fault tolerance. Transmissions are received and sent out via the first available bonded slave interface. Another bonded slave interface is only used if the active bonded slave interface fails.

balance-xor or 2

Sets an XOR (exclusive-or) policy for fault tolerance and load balancing. Using this method, the interface matches up the incoming request's MAC address with the MAC address for one of the slave NICs. Once this link is established, transmissions are sent out sequentially beginning with the first available interface.

broadcast or 3

Sets a broadcast policy for fault tolerance. All transmissions are sent on all slave interfaces.

802.3ad or 4

Sets an IEEE 802.3ad dynamic link aggregation policy. Creates aggregation groups that share the same speed and duplex settings. Transmits and receives on all slaves in the active aggregator. Requires a switch that is 802.3ad compliant.

balance-tlb or 5

Sets a Transmit Load Balancing (TLB) policy for fault tolerance and load balancing. The outgoing traffic is distributed according to the current load on each slave interface. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed slave.

balance-alb or 6

Sets an Active Load Balancing (ALB) policy for fault tolerance and load balancing. Includes transmit and receive load balancing for IPV4 traffic. Receive load balancing is achieved through ARP negotiation.

Troubleshooting

Here is an example of an incorrect modprobe configuration. I stumbled upon this when attempting to follow my normal procedures for CentOS 5 on CentOS 6.

[root@samba-server ~]# service network restart
Shutting down interface eth0:                                 [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                              [  OK  ]
Bringing up interface bond0:  WARNING: /etc/modprobe.conf line 1: ignoring bad line starting with 'alias'
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
WARNING: /etc/modprobe.conf line 1: ignoring bad line starting with 'alias'
WARNING: Deprecated config file /etc/modprobe.conf, all config files belong into /etc/modprobe.d/.
Device bond0 does not seem to be present, delaying initialization.
                                                           [FAILED]

Double check your system-config-network settings

Sometimes NetworkManager can interfere with network related changes to a system. Ensure that NetworkManager is off before attempting this above procedure.

Make sure all three network interfaces show up

You'll need 2 Ethernet and 1 Bond to show up

Check the bond0 settings

This will be the IP of the server. Check all of your settings.

Check your eth0 and eth1

Nothing should be in the files for this. Just the descriptor of eth0 and eth1.