Load-Balancing network traffic when it gets hot in Winnemucca (i.e. User defined load balancing)

OK--a tad bit facetious. But I just want to make a point that given a new hook in the Vyatta wan load balancing feature it's now possible for users to define custom health-checks. Before going further this post does assume that the user has some familiarity with the Vyatta router and configuration.

This new "knob" allows for user defined interface health tests. In the past there has been icmp echo support (i.e. ping), and more recently a time to live (ttl) expired test--both of which are supported directly via the implementation in the feature. And this got me thinking a while back that perhaps load balancing based on other criteria could be even more useful (i.e. user-defined criteria). This other criteria could be defined by the deployment (i.e. within a virtualized environment), or other network data or statistics (i.e. flows and rate).

What has now been added to the interface is the ability for the user to now define their own test target (or health-test) script. This means that you can define the success/failure criteria for each outbound interface. A few examples of the types of tests that can be performed are:

- test whether an http server is reachable
- disable interface after 6 pm
- enable additional interfaces based on traffic load
- disable interface based on virtualized host cpu loading



Where this excels is in allowing the deployed system to determine the criteria by which an interface is enabled or disabled. For example, virtualized environments can have specific (and unique) criteria that can now be tailored to. This is now a simple script away from being realized.

Now I'm going to set up and show an example of this in action. What I want to do is to set up a fail-over configuration with two interfaces eth0 and eth1. I want traffic to use eth0 if the temperature in Winnemucca (by the way I had the pleasure to stay one night in this little town somewhere between Reno and the desert) is hot (say 72 and above), eth1 otherwise. Granted this is a really silly example, but it does illustrate the degree of customization possible.

So, let's get started then. The first link that you'll need to set up is the little weather app that we are going to rely on to get the actual temperature in little ole' Winnemucca--follow the example in this posting below to get this up and running:

weather data example

(By the way this is a great blog.)

Now let's point our Vyatta wan load balancing configuration to our as yet undefined script:

The configuration is simple enough:

vyatta@vyatta# show load-balancing 
wan {
  interface-health eth0 {
    nexthop 10.3.0.1
      test 1 {
      test-script /tmp/weather.pl
      type user-defined
    }
  }
  interface-health eth1 {
  nexthop 10.3.0.1
  test 1 {
  test-script /tmp/weather.pl
  type user-defined
  }
}
rule 1 {
  inbound-interface eth2
  interface eth0 {
  }
  interface eth1 {
  }
}
}

Notice that there is a new test type "user-defined", which means that you get to define the success/failure criteria for that interface. The script that does the work we are going to call "weather.pl" and for now put it in a temporary location "/tmp/weather.pl". The remainder of the configuration is the same old stock simple configuration.

This script will be called roughly every 5 seconds for each interface by the load balance process, and for each interface we want to check what the temperature is in Winnemucca and if it's cool we are going to send traffic out eth1, hot will be directed towards eth0.

The key pieces of this script are:

A exit code of 0 is a success, meaning the interface is active, non-zero failure

The environment variable "WLB_SCRIPT_IFACE" indicates which interface the test is to be performed on.

The script below needs to generate the output is. What this script does is switch exit codes based on the temperature in Winnemuca, Nevada.

#!/usr/bin/perl                                                                                                             

use strict;
use warnings;
use POSIX;

#which interface we are performing test on                                                                                  
my $iface = $ENV{WLB_SCRIPT_IFACE};

my $hot = 72;

my @out = `weather -iKWMC -cWinnemucca -sNV -f -o --quiet`;

foreach my $line (@out) {
  if ($line =~ /^Temperature:/) {
    my @l = split(" ",$line);
    my $temp = $l[1];
                 
    $temp = ceil($temp);
    if (defined($iface) && $iface =~ /eth1/) {
      if ($temp >= $hot) {
        exit(1); #exit status code 1 (disable) if even   
      }
    }
    else {
      if ($temp < $hot) {
        exit(1); #exit status code 1 (disable) if odd 
      }
    }
  }
  exit(0); #exit status code 0 (enable) 
}

Which produces the following at 59 degrees (active eth0 and inactive eth1):

vyatta@vyatta# run show wan-load-balance 
Interface:  eth0
Status:  active
Last Status Change:  Fri Apr  1 18:41:01 2011
+Test:  user  Script: /tmp/weather.pl
Last Interface Success:  0s 
Last Interface Failure:  n/a                
# Interface Failure(s):  0

Interface:  eth1
Status:  failed
Last Status Change:  Fri Apr  1 18:41:01 2011
-Test:  user  Script: /tmp/weather.pl
Last Interface Success:  n/a                
Last Interface Failure:  0s 
# Interface Failure(s):  1

That's 59 degrees to be exact:

vyatta@vyatta# weather -iKWMC -cWinnemucca -sNV -f -o --quiet
Temperature: 59.0 F (15.0 C)

And when the temperature changes (now we are way up to 78 degrees--middle of the day):

vyatta@vyatta# run show wan-load-balance 
Interface:  eth0
Status:  failed
Last Status Change:  Fri Apr  1 23:07:24 2011
-Test:  user  Script: /tmp/weather.pl
Last Interface Success:  6s 
Last Interface Failure:  0s 
# Interface Failure(s):  1

Interface:  eth1
Status:  active
Last Status Change:  Fri Apr  1 23:07:24 2011
+Test:  user  Script: /tmp/weather.pl
Last Interface Success:  0s 
Last Interface Failure:  6s 
# Interface Failure(s):  0

Notice that we've swapped interfaces, eth1 is now the active interface and eth0 no longer forwards data. So, as you can see the simple script provides a load-balancing criteria (an arbritrary one that is). A more useful test would be one that measures the response of an http server, such as the code below:

use strict;
use warnings;
use POSIX;
use JSON;
use Data::Dumper;
use URI::Escape;


my $iface = $ENV{WLB_SCRIPT_IFACE};

my $code;
my $body;

#let's see if we can reach google out this interface
my @out = `curl -s -m 3 --interface $iface -i -X GET www.google.com`;
#now process output, for http status code and for response body
foreach my $out (@out) {
  if ($out =~ /^HTTP\/[\d.]+\s+(\d+)\s+.*$/) {
    $code = $1;
  }
  elsif ($out =~ /^\r/ || defined $body) {
    $body .= $out;
  }
}

#success is if http response code is 200 and body is returned
if (defined($code) && $code == 200 && defined($body)) {
  #WLB SUCCESS
  exit(0);
}

#WLB FAILURE
exit(1);

The above tests for whether the response contains a body and if the http response code is 200. If so the the interface is considered active. The test is performed out the respective interfaces and the configuration must provide a path (outside of load-balancing) for reachability to the http server.

That's it for now...