General idea

We set up a cluster consisting of a head node that handles load balancing and monitoring and several pool nodes, capable of running virtual machines.

Networking

We have 2 networks: for physical hosts and virtual hosts. Head node has two interface for both of the networks. We use bridge interfaces on the physical hosts with direct connection to the underlying network.

Physical host network: 192.168.1.0/24

Virtual host network: 192.168.122.0/24

Routing table

At the head node:

 Kernel IP routing table
 Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
 192.168.122.100 192.168.1.67    255.255.255.255 UGH   0      0        0 br0
 192.168.122.103 192.168.1.65    255.255.255.255 UGH   0      0        0 br0
 192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 br0
 0.0.0.0         192.168.1.199   0.0.0.0         UG    100    0        0 br0

Explanations:

• 192.168.122.100 is IP of a virtual machine running on a physical machine with IP 192.168.1.67.
• 192.168.122.103 is IP of a virtual machine running on a physical machine with IP 192.168.1.65.
• Head node has 2 interfaces: 192.168.1.68 and 192.168.122.3.
• VMs get IPs starting from 192.168.122.100 (convention).

At the pool node:

 Kernel IP routing table
 Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
 192.168.122.100 0.0.0.0         255.255.255.255 UH    0      0        0 br0 
 192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 br0
 192.168.122.0   192.168.1.68    255.255.255.0   UG    0      0        0 br0
 0.0.0.0         192.168.1.199   0.0.0.0         UG    100    0        0 br0

Explanations:

• 192.168.122.100 is IP of a VM running on the same machine.
• 192.168.122.0 is a virtual machine network, all requests there (if concrete IP is unknown) are forwarded to the head node (.1.68).

At the virtual machine:

 Kernel IP routing table
 Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
 192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
 0.0.0.0         192.168.122.3   0.0.0.0         UG    0      0        0 eth0

Explanations:

• 192.168.122.3 is IP of a head node. It serves as a default gateway for all the requests from the VM.

Monitoring

We use ganglia to monitor the machines. Ganglia gmetad is installed at the head node.

gmond.conf of the virtual machine should include the following line.

 udp_send_channel {
  host = 192.168.122.3
  port = 8650
  ttl = 3
 }

Starting/Stopping a new VM

We use libvirt over Qemu+KVM to run the virtual machines. Host OS is Ubuntu 9.04.

Network modification

Before starting a new VM:

• <vmIP> is IP of a newly created VM.
• <pIP> is IP of a pool machine that hosts the new VM.

On the head node:

 route add -host 192.168.122.<vmIP> gw 192.168.1.<pIP> br0

On the pool node:

 route add -host 192.168.122.<vmIP>  br0

After stopping a VM: same commands just use del instead of add.