A little perl nugget: differences between two arrays

Something small, simple...

I have two perl arrays and I want to remove all elements in common between the two arrays, leaving only the elements in list_one that are unique to list_one.


For example pre-populate two arrays with the following values.

my @list_one = ();
push(@list_one,'a');
push(@list_one,'b');
push(@list_one,'c');

my @list_two = ();
push(@list_two,'b');

Setting up Perf for performance evaluation of your code

OK--some quick notes on setting up and running linux tools performance profiling (perf):

sudo apt-get install libelf-dev binutils-dev

wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.0.4.tar.gz

tar xvfz linux-3.0.4.tar.gz

cd linux-3.0.4/tools/perf/

make

scp /usr/lib/libelf.so.1 [to-target-system]
scp /usr/lib/libbfd-2.20.1-system.20100303.so [to-target-system]


# sudo /opt/bin/perf 

usage: perf [--version] [--help] COMMAND [ARGS]

The most commonly used perf commands are:
annotate        Read perf.data (created by perf record) and display annotated code
archive         Create archive with object files with build-ids found in perf.data file
bench           General framework for benchmark suites
buildid-cache   Manage build-id cache.
buildid-list    List the buildids in a perf.data file
diff            Read two perf.data files and display the differential profile
evlist          List the event names in a perf.data file
inject          Filter to augment the events stream with additional information
kmem            Tool to trace/measure kernel memory(slab) properties
kvm             Tool to trace/measure kvm guest os
list            List all symbolic event types
lock            Analyze lock events
probe           Define new dynamic tracepoints
record          Run a command and record its profile into perf.data
report          Read perf.data (created by perf record) and display the profile
sched           Tool to trace/measure scheduler properties (latencies)
script          Read perf.data (created by perf record) and display trace output
stat            Run a command and gather performance counter statistics
test            Runs sanity tests.
timechart       Tool to visualize total system behavior during a workload
top             System profiling tool.

See 'perf help COMMAND' for more information on a specific command.



to run (C specifies which CPU, and p specifies which process to attach to):

sudo /opt/bin/perf record  -p 4040 -C 5
sudo /opt/bin/perf report -i perf.data

AT&T UVerse fiasco

I hate to complain.

But this is just one I have to post about.

I made the decision to upgrade from AT&Ts DSL service to UVerse--to get a better data rate at a lower price. A reasonable decision--to be made unreasonable by AT&T.

First the UVerse modem was delivered one day after they cut off the old DSL service. OK--so that's a day without service. You'd think they could be a bit more effective at coordinating this, or just cut off service a day later. But I guess as I found out this was to set the stage for further incompetence on AT&T's part.

The next day, with equipment now wired up and showing some signs of life. UVerse still doesn't come up. Turns out the order was written incorrectly. Should have been written to include support for the POTs line, but was written without. I'm guessing that some tech only just needs to unplug the cable from socket A and into socket B. But this new addendum order takes another 24 hours to clear.

HP-15c is on it's way

Available in October according to one reseller.

More details can be found here:

Sounds like it will be limited to 10k. Hopefully the build quality is as good as the original (I have my doubts).

Enjoy.

Something to crow about: Gson Json code generator

Gee Whiz--this is a real time save. Gson is a google project to serialize and deserialize json data. Good enough. The problem is that to do this you really want to drop the deserialized output into a java object for further processing.

There are some generalized object containers out there, but these didn't really do what I was looking for. Guess what I wanted was something lite and easy--no go.

So, here's the whizbang part. This site takes your json, such as:

Network monitoring

I'm on a bit of R and R right now. Have to burn through the vacation time somehow. Earlier, this year I made a point of pushing at least one new blog entry per week. Seems like I missed this week for the first time in a while.

So, perhaps, just to make me feel "whole", here's a link to an article I wrote for Dr. Dobbs a few years back which describes a framework for network monitoring. Basically a cascading set of filters and processors (pipeline), which threading support in each filter manager:

The SecureScout Wi-Fi Security & Monitoring Framework

How to set up an email smtp client to work with smtp.gmail.com

There's a bunch of implementations out there that don't work (possibly outdated or just plain broken. So, that's why I'm publishing this today--so I have a handy reference when I need it again.

Anyways, this one DOES work using gmail as the smtp agent.

Enjoy.

String host = "smtp.gmail.com";
  int port = 587;
  String username = "your_gmail_login";
  String password = "your_password";
  String from = "support@belisarius2000.com";

  Properties props = new Properties();
  props.put("mail.smtp.starttls.enable", "true");
  props.put("mail.smtp.host", host);
  props.put("mail.smtp.user", username);
  props.put("mail.smtp.password", password);
  props.put("mail.smtp.port", "587");
  props.put("mail.smtp.auth", "true");


   Session session = Session.getInstance(props,new 
     GMailAuthenticator(username, password));

   Message message = new MimeMessage(session);
   message.setFrom(new InternetAddress
      ("support@belisarius2000.com"));
   message.setRecipients(Message.RecipientType.TO,
      InternetAddress.parse(email));
   message.setSubject("Hey there dude!");
   message.setText
      ("Here's the body of my email, blah blah blah");
 
   Transport transport = session.getTransport("smtp");
   transport.connect(host, port, username, password);
 
   Transport.send(message);

And this class too...

class GMailAuthenticator extends Authenticator {
  String user;
  String pw;
  public GMailAuthenticator (String username, String password)
  {
    super();
    this.user = username;
    this.pw = password;
  }
  public PasswordAuthentication getPasswordAuthentication()
  {
    return new PasswordAuthentication(user, pw);
  }
}

Steve Job's spaceship

Wow--just had a chance to watch the Cupertino city council presentation by Steve Jobs last night. Did he have them eating out of his hands or what? I had a couple of thoughts, obviously this guy really has a vision. And second these city council reps came off as a bunch of sycophants--what's the point of asking for an ipad for every resident of the Cupertino, or an Apple store for Cupertino. Kind of laughable.

But back to the spaceship. It seems like Apple is really on a roll here. It's a battle between the Big Three: Google, Apple and Facebook for new employees. So, this definitely ups the bidding. I wouldn't be surprised if in the basement there's booster rockets and cryogenic pods to blast off into space once this planet turns into a dustbowl (maybe from too many ipads?) a la Silent Running.

And the video

A C++ Boost client for the Vyatta REST api

Here's a Vyatta REST client I modifed from a stock Boost https client example. In some tests a while back I found that using a remote command line client implemented in C++/c was significantly faster than using a scripted Perl implementation, which makes sense when latency isn't the overwhelming factor in the latency of the response. And as I alluded to, with claims of performance there are other important criteria that come into play, such as latency, jitter, response time on the server, etc. Needless to say I my queue (somewhere) a pending post to quantify this performance difference using different client implementations.

You can see from the help that the executable requires the target, url (i.e. command), and optionally password/username to run.

Usage: vyatta_boost   [options]
  -h  host name or ip
  -c  rest api command path
  -m  HTTP method (i.e. GET, DELETE, POST, PUT)
  -u  username
  -p  password

And running this command:

Huffman and the STL

A while back I wrote a Huffman encoder using the C++ STL (Standard Template Library). It worked, but then a co-worker of mine took it upon himself to see how many lines of code he could squeeze out of my implementation. So, in the intervening years I always had this in the back of my head to revisit this and create a compact Huffman encoder with C++ and STL.

So, given a spare moment I wrote carved out a little chunk of code--probably not the absolute smallest in number of lines, but compact none the less. I'm sure a version in Perl would end up being far more compact, but then it would be written in Perl.

The code ends up looking simple and neat, which is one of things I love about using the STL.

See for yourself. The encoding calculation is below, with what is really just a scan for the frequency of occurrence of an ascii character.

vector ct_coll(256);
for (int i = 0; i < input.length(); ++i) {
  ct_coll[input[i]]._c = input[i];
  ++ct_coll[input[i]];
}
//now sort and build tree
multiset freq_coll;
copy(ct_coll.begin(),ct_coll.end(),inserter(freq_coll,freq_coll.begin()));
while (freq_coll.size() > 1) {
  freq_coll.insert(Data(new Data(*freq_coll.begin()),new Data(*++freq_coll.begin())));
  freq_coll.erase(freq_coll.begin());freq_coll.erase(freq_coll.begin());
}
//assign value
Data d = *freq_coll.begin();
assign_code(d);

How to see sounds with a Ruben's tube (Sound and Fire!)

DISCLAIMER: Nothing in this post is software/code related.

But who cares it's really cool none-the-less.

A bit of background first. I've been working on demonstrating the science of waves at my kids elementary school for the past 3 years now. At about 2 weeks out I start asking myself this question:

What experiment/trick can I perform to help the kids visualize a pressure wave?

We use a slinkys, use a spectrum analyizers, watch a video of the tacoma narrows bridge, etc.But what can I do that really is an attention grabber and helps cement the concept for these grade school kids? And can it rate high on the coolness factor?

So, the idea of building a Ruben's Tube looked light it might fit the bill here. A Ruben's tube basically is an experiement where sound affects fire--how cool is that? Pretty damn cool that's what I say. I asked my cohort (Ross) and wife (who helps to organize the whole affair) if this would fly... And to my complete surprise no one said NO.

So we went off and built us a Ruben's tube...

What exactly is a Ruben's tube? A Ruben's Tube is a length of tube filled with flammable gas (propane) with small regularly spaced holes. The tube is sealed at one end and capped at the other end with a speaker. The gas has nowhere to escape but through these little holes. And the sound (via the speaker) creates a standing wave (at the right frequency that is) that affects the gas pressure along the length of tube. This results is varying flame height based on the location of the hole and the degree of particle motion due to the standing wave. This would be a sine wave of flames along the length of the tube.

When a resonant frequency is pumped into the tube a standing wave will disturb the gas at the points where the motion in the p-wave is the greatest and suppress the gas leaving the tube at holes located at these points. Likewise holes near where the p-wave motion is less will then to escape at a greater velocity.

That mostly makes sense to me (having a bit of a background with acoustics myself).

This required a trip to a local scrapyard (for the tube), the nearby Lowes, and then expropriating a small speaker from Brian's speaker system. And a big ole tank of propane gas. After much drilling, and fitting and drilling we ended up with a 72 hole ruben's tube, about 4 feet in length.

Below we are in the middle of drilling the 72 hole array (that's Ross doing the work with Brian supervising).

The mostly assembled tube is here (sans propane tank connector):

With a closeup of the speaker end of the tube (tube and speaker are 2 inches in diameter):

So, the darn thing worked, but really really wants a sheltered place. Given that no such place exists anywhere in windscape of San Francisco, below video of the contraption working (briefly before the wind gets the better of it). I apologize for the wonky orientation of the video--next time I'll get that straight (the tube really is stationary in a horizontal position). The amplitude modulation of the waves shows up in the last few seconds of the video.

What we discovered what the tube could be shorted (i.e. fewer holes), or we need more pressure and we have to do this in an enclosed space (i.e. NO WIND).

Next up (when I get around to it) will be video of the Ruben's tube playing to "When the levee breaks".

Implementing a stand-in "loading" image in javascript

First of all Happy Bike to Work day (BTW DAY)!

This is something like my 12th BTW day and I love seeing the masses of bicyclists out on the road especially on this day.

Now on to the subject of this post...

This had me a bit flummoxed, until I nailed down the magic way to set this up. Basically what I wanted to be able to was insert a stock image in place of a slower loading image. This slower loading (well, technically stream) ends up taking seconds to bring up after the page has finished rendering.

What was happening without this was I was staring a big empty rectangle after loading of the page, and thought it would be much much nicer to put in a pretty, albeit temporary, image in place.

The solution involves setting two image tags in the same div section. One visible and one hidden. Something like this:

<div id='stream_temp' style='visibility:visible;'>
  <img height=450 width=600 src='img/camera.jpg' alt='videoFull' border='4' >
  </img>
</div>

<div id='stream' style='visibility:hidden;display:none;'>
  <img onload='displayStream()' height=450 width=600 
    src='my_stream&scrH=450&scrW=600'
    alt='videoFull' border='4' ></img>
</div>

Generating a random key

I want to keep this little block of code around as it might just come in handy in the future, not that it's terribly complex.

The key generator relies on /dev/urandom to help generate a random key. There's another file on the system call /dev/random. The difference between these two is that the command to open and read from /dev/random will block until enough system entropy has occurred on the system, whereas a call to open /dev/urandom will immediately succeed regardless of the entropy on the system. Meaning your key may not be secure at the point the value is read.

This should only really be of concern if the key is being generated somewhere close in time during the boot process.

Real live plotting in your browser

Here's how to generate nice and pretty live plots in your browser. Even turns out to not be too much of a hassle (by relying on the correct library that is). First you need to grab the following javascript files from here.

And reference the relevant plotting support libraries in the html code (as below):

<script type="text/javascript" language="javascript" src="js/jquery-1.2.3.js">
</script>
<script type="text/javascript" language="javascript" src="js/jquery.flot.js">
</script>

Then set up the location you want the plot to appear on the page via a div section:

<span style="padding:0px;"><div id="plotter" align="left">
</div></span>

VT100 in your browser

This is pretty cool. I was searching for this and guess what? It's possible to bring together an unholy marriage between the browser and the command line interface. It's actually so uncool that it's cool.

And it works pretty well too...

There's a VT100 emulator written in javascript (shellinabox), that is fairly well developed. There's even an implementation you can play with just to kick the tires a bit so to speak. It's even well developed enough that it seems completely usable with some limitations. Performance ain't too bad either. The primary limitation is obviously the delay in the request/response (or latency).

This implementation works by running a parallel shell process on the server and sending keystroke by keystroke back to the server for interpretation. There's some performance penalty with this approach, but it does promise to be a faithful representation of a linux cli session.

On the other hand I've had a chance to subvert this library even more by wiring it up to the Vyatta REST API--the nice thing about this is that commands can be sent when completed (thereby avoiding the keystoke by keystroke roundtrip). And guess what? It works pretty well too. Very nicely.

Now with this your imagination can run wild with the possibilities....

Woo-hoo! Maybe an upcoming re-release of the hp-15c?

I'm not usually into rumor mongering and the like, but there's a tantalizing tidbit just released here.

Set your alarm for June.


UPDATE (6/9/11):

Just saw that a supposedly email from the *inside* stated that the RUMORED hp15c won't be released on January 2012. In order to align with the 30th anniversary of the original release.


Hold tight.

A place in my heart for Reverse Polish Notation

No explanation really, but I'm still fascinated by the old RPN line of HP calculators. There's a bit of joy every time I find a use for one--which these days mostly consists of helping with my kids homework. And it's as much about the hardware as the software--a real RPN calculator beats any emulated alternative any day. Handsdown. And I guess that's why I throw an old HP-12c in with my gear each morning.

My first handheld was the HP-25c. To me that was my pda. Through the years I've managed to misplace about a half-dozen models along the way (always upgrading to the newer model).

There's a great site that still covers technical details in minutia (hp-museum) with a pretty active forum. And it's a must visit site if you are considering collecting these relics.

I now have the following tucked away (an hp 16c, 32s, 42s, 41c, and 12c).

Here's to the hope that HP would someday redo the HP-15c scientific (sign the petition here).

Load-Balancing network traffic when it gets hot in Winnemucca (i.e. User defined load balancing)

OK--a tad bit facetious. But I just want to make a point that given a new hook in the Vyatta wan load balancing feature it's now possible for users to define custom health-checks. Before going further this post does assume that the user has some familiarity with the Vyatta router and configuration.

This new "knob" allows for user defined interface health tests. In the past there has been icmp echo support (i.e. ping), and more recently a time to live (ttl) expired test--both of which are supported directly via the implementation in the feature. And this got me thinking a while back that perhaps load balancing based on other criteria could be even more useful (i.e. user-defined criteria). This other criteria could be defined by the deployment (i.e. within a virtualized environment), or other network data or statistics (i.e. flows and rate).

What has now been added to the interface is the ability for the user to now define their own test target (or health-test) script. This means that you can define the success/failure criteria for each outbound interface. A few examples of the types of tests that can be performed are:

- test whether an http server is reachable
- disable interface after 6 pm
- enable additional interfaces based on traffic load
- disable interface based on virtualized host cpu loading

Session support in the Vyatta REST API

While it may not be exactly faithful to RESTful principals, it is a good feature/option that further enhances security.

Authentication in the Vyatta REST API uses standard HTTP basic authentication (and has since day one). Vyatta uses https so the basic authentication parameters (i.e. username and password) are encapsulated within the https protocol to provide a secure channel. Since the auth is contained within each request the server is not required to maintain state (or track a session), as the username and password are presumably good forever.

The downside of this approach is that maintaining a username and password for access can be risky (see this post)

Supporting long polling in lighttpd with fastcgi

Long-polling (or comet) is a technique (or hack) designed to support asynchronous server to client messaging. In other words push notification from the browser. Given that the HTTP protocol is a client initiated protocol, there's no provision to support a non-client initiated event from the server to the client. That's where Long-polling comes in. Basically it involves a server that never closes the socket (in other words completes the response to the request), and a client that keeps a socket open (and continues to read from the socket).

This allows the client to continue to receive data from the server. From the HTTP perspective this is as if the response never ever finished. The big win, of course, is that the server is now able to push messages back to the client, creating a mechanism via HTTP that allows for asynchronous message pushing. But given that this is a bit of a hack the pushing of messages back to the client is outside the bounds of the HTTP protocol. Meaning that an "out-of-band" protocol specific to the application needs to be designed for this communication.

So, I've been looking at what it would take for lighttpd plus fastcgi to support this configuration. The good news is that it isn't a whole lot.

Simple scatter plots using R

Just some basic notes on getting scatter plots up and running. Without a lot of sweat. Occasionally I need to create plots of data. Each time the need arises I do the following: (1) google for the lowest learning curve plotting package, (2) find my way up the learning curve or bail and return to step (1). Then next time I have a plotting need (maybe 6 months later) it's back to step (1) again.

Now, with this post I will at least have a reference with notes to get started next time, skip step (1) and maybe even step (2) (and maybe you can benefit from this as well)...

Crazy date conversion in java for RSS feeds

Here's a solution I hated at the time--and still really hate.

This was a "solution" a few years back while processing RSS feeds in Java for their date/time format. The goal was to convert the various time formats found in random RSS feeds into the common epoch format.

Blondie and Philip Glass ??? WTF?

Totally not on the topic. But I can't let this one go.

A collaboration between Philip Glass and Blondie? Heart of Glass?

Now, the concept is intriguing to say to the least. An early 80's pop star who has not aged well (think Ozzy Osborne) and classical new music composer Philip Glass.

Well, I just heard the song today and it almost works, but was probably doomed from the start. Philip Glass sounds like he is rehashing some of the familiar early career material, and Blondie's voice is electronically enhanced a bit, but with less inflection--probably to match the Glass accompaniment.

Ends up not meshing together in a particular compelling way for me. But what a strange way to start the day listening to that...

Bandwidth estimator using a simple Kalman filter, Part 2 of 2

I realized that I didn't run through the simple Kalman filter algorithm in the last posting. So, let's do that now before the promised source code review. The Kalman filter I used is about as simple as it can get:

Weight calculation:

weight = bandwidth_variance / (bandwidth_variance + measurement_variance)

The bandwidth variance (the variability within any single measurement) and measurement variance are used to compute the weight.


Estimate calculation:

estimate = estimate + weight*(bandwidth_measurement - estimate)

Bandwidth estimator using a simple Kalman filter, Part 1 of 2

This here is a fairly compact C++ program I wrote that uses a simple technique to estimate a network circuits bandwidth. There's nothing new with the bandwidth estimation (there are more complex ones available), but I haven't seen this paired with an estimation algorithm (Kalman).

This post is going to be another one of the two parters since there was a bit of work to pull all this together. The first posting (this) covers the results and description of the technique, while the second will review the source code.

One of the problems with measuring bandwidth is that results are highly variable (i.e. noisy). This can be due to a number of reasons, but basically, packets travel to their target over several hops, each hop (or router, switch etc.) can introduce a variable amount of delay due to congestion, loading, system performance etc. Therefore, any single measurement of the bandwidth is unlikely to provide an accurate representation of the circuit bandwidth. In addition, bandwidth characteristics will change over time. Therefore, it's best to support a model that uses historical data to estimate the bandwidth without completely discounting large changes in the current measurement.

Simple C++ scope timing module

OK--I've been busy and tied down. And I've been trying to post an update on a weekly basis, so far so good this year. Except this is another short one. A longer posting is coming for next week though...

I've been using this little snippet of C++ code for ages, but it's nice to have the relatively simple source located at an easy to get location. Just uses the scope to determine the elapsed time in elapsed microseconds.

As you can see just requires a declaration that looks like:

Timer t("my time');

Which can be further simplified with a macro:

#define T Timer t(__PRETTY_FUNCTION__)

to:

T;

The implementation can (and has) been fancier. A static can be used to track nested or re-entry into blocks of code for easier consumption. That's great, however for multithreaded code a mutex needs to be thrown on the static which then perturbs the timing itself--so it's better to avoid unneeded complexity.

The header:

class Timer
{
public:
Timer(std::string id);
~Timer();

private:
struct timeval _start_time;
std::string _id;
};

The Source:

Timer::Timer(string id) : _id(id)
{
gettimeofday(&_start_time,NULL);
}

Timer::~Timer()
{
struct timeval end_time;
gettimeofday(&end_time,NULL);

unsigned long usecs = abs(_start_time.tv_usec - end_time.tv_usec);
unsigned long secs = end_time.tv_sec - _start_time.tv_sec;
if (_start_time.tv_usec >= end_time.tv_usec) {
secs = secs - 1;
usecs = 1000000 - usecs;
}

char buf[512];
sprintf(buf,"elapsed sec %d, usec %d",secs,usecs);
string msg = "timer(" + _id + "): " + buf;
string cmd = "echo '" + msg + "' >> /tmp/timer";
system(cmd.c_str());
}

Time printed is in elapsed time within the scope of the measurement. Obviously, this doesn't work well for scopes that are called multiple times, but is useful for outer scope timing. This can be easily modified to accumulate time though, at the expense of being thread unsafe.

Some time saving tips when setting up an Amazon EC2 network

I ran into several hiccups with my first foray into Amazon's EC2 and AMIs setting up a tiered server environment. As I hate making the same mistake twice (and love learning from others experiences) I put together a short list of "my" "lessons learned" when setting up an Amazon EC2 network for the first time. Which in retrospect seem somewhat obvious (but isn't that software development in general).

Growing a REST API, Part 2 of 2

OK, let's jump back into the fray here with part 2 (part 1 is here) of our discussion on the REST API for the Vyatta Appliance thingy. Now, we've already covered the API up to and including the Configuration mode. And found out that commands are relatively simple to construct. What's left you ask? The operational side of things and the weird red headed step-child interface: App mode.

Without further ado then...

Growing a REST API, Part I of 2

This REST API that I had a hand in (more like both hands in) for the Vyatta router (also known as the Vyatta Remote Access API) was a complete rewrite of a more traditional XML state-aware API (and which replaced an even earlier version).

In retrospect, given the progression of versions it was with REST that the simplest HTTP based API was designed. I'd even say that the work was well focused due to REST constraints applied to the design of the interface.

Web performance tool, simply and quickly: Pylot

Here's a tool I've used a few times, and each time it delivered what I wanted with minimal fuss. And in the world of development tools there's something to said about that. It's a simple straightforward http request/response performance/timing tool. But one that packs enough features to be useful in a variety of situations.

Communicating with the Kernel via Netlink, Part II

OK--continuing on from where we left off with part I. The previous post ran through key details of the overall layout of the objects, aspects of the Netlink interface and some source snippets from the listener and sender classes. This time I want to cover the event object and parsing of the netlink data object. Plus, source code is located at the end of this article.

Communicating with the Kernel via Netlink, Part I

This is a simple C++ class wrapper that interacts with the Linux Netlink interface. It's good all by itself or as a source code jumping off point, so to speak. The specific implementation was designed to work at the layer 2 and layer 3 OSI layer.

Just to give you an idea--here's an example of the output this program generates when started up and dumping the state of the interfaces:

results for lo(1)
  type: NEWLINK
  state: UP
  running: yes
  enabled: yes
  mtu: 16436
  mac: 00:00:00:00:00:00

results for eth0(2)
  type: NEWLINK
  state: UP
  running: yes
  enabled: yes
  mtu: 1500
  mac: 08:00:27:41:DE:23

results for lo(1)
  type: NEWADDR
  addr: 127.0.0.1
  broadcast: 0.0.0.0
  mask length: 8

results for eth0(2)
  type: NEWADDR
  addr: 10.3.0.185
  broadcast: 10.3.0.255
  mask length: 24

results for eth0(2)
  type: NEWADDR
  addr: 1.1.1.1
  broadcast: 0.0.0.0
  mask length: 20