Once upon a time, I would have said that its impossible to teach 5th graders to program in Java. Even the most basic hello world requires exposure to complex concepts: the print statement must be wrapped by a method with very specific modifiers and parameters, which is then wrapped in an class and compiled. Enter Greenfoot.

When helping to teach a class for the Northwestern CTD weekend program**, I was introduced to Greenfoot for teaching and learning Java. After my first day of class, I was so inspired by the educational possibilities of Greenfoot that I wrote a little Breakout clone to show the kids the next day what they could do with Greenfoot.

Rather than using the classic programming education sequence, from hello world to user input, string manipulation, file I/O, and so on, Greenfoot instructors typically start by letting the students play games in Greenfoot. Once the students understand the mechanics of the game, we step back to cover some of the theoretical bits of programming: data types, methods, objects vs classes, and so on. Then we go through the code of the game they played to see how the theory comes together to make the game work. And finally, we help the students to change the games to behave as they wish. Its gamification of education at its core.

Structuring the class in this way results in a very different environment. Instead of having to force-feed material to students, they’re actively engaged. The primary driver of students’ learning is now their intrinsic curiosity and creativity, rather than the fear of tests hovering overhead. Its a way to escape education’s death valley (at least for programming classes).

Now, don’t take this the wrong way. I still have a lot to learn about teaching Java, particularly to young learners. Covering the theoretical material is still particularly painful, but I believe its because we don’t know the best way to help students learn it yet. I’m confident that we can find the right techniques though. For example, I particularly enjoy trying to teach objects vs classes by drawing on the students’ own experience and intuition.

One thing that I’ve noticed, though, is that teachers aren’t properly trained for this kind of environment. The students come in with different backgrounds and exposure to programming, learn at different rates, and — driven by their own creativity — all want to make their games do something different. This makes for a hectic and sometimes challenging environment in which to operate.

In such a student-driven learning environment, its backwards to attempt to maintain the same “command and control” style of education to which teachers are accustomed. Although I’ve written about teachers transitioning from lecturers to facilitators before, I don’t really know the best way to implement it. But I know that its not telling the students to stop exploring, to be scared of editing code/changing anything (in the case of teaching programming), and to sit and listen. In my experience, the excitement of tinkering leads to student inquiry which leads to teacher assistance which leads to more exploring. What a lovely cycle! Now, the question remains, with a student:teacher ratio fixed by budgetary limitations, how can we actually accomplish this? If I ever figure it out, I’ll let you know.

** For those of you who don’t know, I’ve taught off-and-on through Northwestern’s Center for Talent Development since I moved to Chicago. My first job in the Chicagoland area was as a Residential Teaching Assistant for the Summer Program and since then I’ved helped through the Accelerated Weekend Experience (AWE) program.

I just started learning how to write my own Cocoapod yesterday. There’s a great tutorial to get you started. However, I ran into an issue when trying to lint my new spec. It looked like this:

$ pod spec lint MyAwesomeLibrary.podspec
 
 -> MyAwesomeLibrary (0.0.1)
    - ERROR | [iOS] [xcodebuild]  2013-05-29 23:03:08.370 xcodebuild[91499:3f03] error: Error Domain=NSPOSIXErrorDomain Code=2 "Non-zero exit code 255 returned from shell command: /usr/bin/gcc-4.2 -v -E -dM -arch armv7 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS6.1.sdk -x objective-c -c /dev/null 2>&1" UserInfo=0x4001c4e60 {NSLocalizedDescription=Non-zero exit code 255 returned from shell command: /usr/bin/gcc-4.2 -v -E -dM -arch armv7 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS6.1.sdk -x objective-c -c /dev/null 2>&1, NSLocalizedFailureReason=No such file or directory}
    - ERROR | [iOS] [xcodebuild]  2013-05-29 23:03:08.457 xcodebuild[91499:3f03] error: Error Domain=NSPOSIXErrorDomain Code=2 "Non-zero exit code 1 returned from shell command: /usr/bin/gcc-4.2 -v -E -dM -arch armv7s -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS6.1.sdk -x objective-c -c /dev/null 2>&1" UserInfo=0x4018946a0 {NSLocalizedDescription=Non-zero exit code 1 returned from shell command: /usr/bin/gcc-4.2 -v -E -dM -arch armv7s -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS6.1.sdk -x objective-c -c /dev/null 2>&1, NSLocalizedFailureReason=No such file or directory}
 
Analyzed 1 podspec.
 
[!] The spec did not pass validation.

When I manually ran the command in the above error message, I saw this error:

$ /usr/bin/gcc-4.2 -v -E -dM -arch armv7 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS6.1.sdk -x objective-c -c /dev/null 2>&1
gcc-4.2: error trying to exec '/usr/bin/arm-apple-darwin11-gcc-4.2.1': execvp: No such file or directory

I don’t have the file “/usr/bin/arm-apple-darwin11-gcc-4.2.1″. If you see this too, you should check whether you have a preferred C Compiler. This is configured via the CC environment variable. Mine was set to gcc-4.2.

$ echo $CC
gcc-4.2

To let cocoapods use the default compiler instead, just run the linter as

$ CC= pod spec lint MyAwesomeLibrary.podspec
 
 -> MyAwesomeLibrary (0.0.1)
 
Analyzed 1 podspec.
 
MyAwesomeLibrary.podspec passed validation.

Woohoo! It passed!

Now that we know how to use multiple SSL keystores in Java, how do we configure Apache HttpClient (embedded in Apache Camel) to use them? This is useful if you want to load additional keystores in addition to the “factory” installed ones, for example. There’s no obvious way to do this using HttpClient or Camel. If you look at any of the documentation online, you’ll either see configuration via the JSSE configuration utility, like so:

KeyStoreParameters ksp = new KeyStoreParameters();
ksp.setResource("/users/home/server/keystore.jks");
ksp.setPassword("keystorePassword");
 
KeyManagersParameters kmp = new KeyManagersParameters();
kmp.setKeyStore(ksp);
kmp.setKeyPassword("keyPassword");
 
SSLContextParameters scp = new SSLContextParameters();
scp.setKeyManagers(kmp);
 
ProtocolSocketFactory factory =
    new SSLContextParametersSecureProtocolSocketFactory(scp);
 
Protocol.registerProtocol("https", new Protocol("https", factory, 443));

Or you’ll see direct configuration of Apache HttpClient using HttpClientConfigurer or something like this:

Protocol authhttps = new Protocol("https", new AuthSSLProtocolSocketFactory(
  new URL("file:my.keystore"), "mypassword",
  new URL("file:my.truststore"), "mypassword"), 443);
 
Protocol.registerProtocol("https", authhttps);

However, we’re not reading from the keystore from a file, nor do we only want to use a single keystore. So all of these convenient wrappers that Apache Camel and Apache HttpClient are actually becoming an inconvenience to us.

This becomes even more frustrating when you realize you can’t even find many classes used in the examples, including the AuthSSLProtocolSocketFactory, on your classpath (because of Camel bundling older versions of HttpClient). Fortunately, if you hunt through the javadocs and sources long enough, you’ll learn as I did that it always boils down to an SSLContext or, more specifically, an SSLSocketFactory.

The protocol must be registered using Protocol.registerProtocol, so we need an instance of a ProtocolSocketFactory. Digging through the source shows that the promising-sounding SSLProtocolSocketFactory always uses the default SSLSocketFactory.  However, it turns out that the Camel-provided SSLContextParametersSecureProtocolSocketFactory can accept a SSLSocketFactory as a constructor argument instead of the SSLContextParameters shown in all the examples. That’s the key!

  @Provides @Singleton
  ProtocolSocketFactory provideSslProtocolSocketFactory(SSLContext sslContext) {
    return new SSLContextParametersSecureProtocolSocketFactory(sslContext.getSocketFactory());
  }

Now we just need to register this factory for use by the HTTPS protocol as in the examples.

  Protocol.registerProtocol("https",
      new Protocol("https", factory, 443));

Its simple once we know what we need, but time-consuming to hunt and peck through all the docs and sources to find it. Hopefully this has helped you save some time!

We can now make HTTPS requests through Camel using multiple (custom) keystores.

As a sidenote, compare this with Ning’s AsyncHttpClient, which exposes a setSSLContext method directly in the AsyncHttpClientConfig.Builder(). That’s almost too easy by comparison.

This is the last part in the Intro to Streams2 series. If you haven’t read Part 1 yet, you should. The examples here build on the ones presented there.

Part 2 is about design patterns for streams. Design patterns help us write clean code, like this.

function publish(data) {
  data
    .pipe(config.formatter())
    .pipe(config.connector())
    .on('error', function(err) {
      self.emit('error', err);
    })
    .on('success', function() {
      self.emit('success');
    });
}

You could imagine that config.formatter() and config.connector() create already-configured objects. Something like this:

  var config = {
    formatter: function() {
      return new CsvFormatter(fieldOrder, fieldSeparator || ',');
    },
    connector: function() {
      return new FtpConnector(host, port || 21, user, password, filename);
    }
  };

Since we might want to change the configuration to return a JSON formatter that connects to Amazon S3 instead, we want all formatting and connector logic to be entirely encapsulated within their services.

Too bad many components won’t work this way out of the box. So what do we do? Design patterns to the rescue!

Adapter Pattern

Piping streams together is fun! Which means that node modules that handle streams but can’t be inserted in a pipe chain are no fun. How do we fix this? We write wrapper streams that adapt the module to work the way we want.

In this case, the culprit is node-ftp. This would be a great FTP module if only you could pipe a file to an FTP server. Instead, you have to call

ftp.put(readableStream, filename, callback)

Let’s start by rewriting the FTP connector, adapting it so that we can pipe to it.

var stream = require('stream'),
  FTP = require('ftp');
 
/*
 * Stream a file to FTP.
 */
function FtpConnector(host, port, user, password, filename, ftp) {
  var self = this;
 
  ftp = ftp || new FTP();
 
  stream.PassThrough.call(self);
 
  ftp.on('ready', function() {
    ftp.put(self, filename, function(err) {
      ftp.end();
      if (err) { self.emit('error', err); return; }
      self.emit('success');
    });
  });
 
  ftp.on('error', function(err) {
    self.emit('error', err);
  });
 
  ftp.connect({ host: host, port: port, user: user, password: password });
}
 
util.inherits(FtpConnector, stream.PassThrough);

We create the stream as a subclass of a PassThrough stream. Then in the constructor we create a new FTP connection and pass self into its put method. This effectively passes all stream input through directly into the FTP upload. Pretty sweet use of the PassThrough stream, if I do say so myself.

Composite Pattern

Ok, on to the CSV.

My data source emits Javascript objects, so I need to transform the object into an ordered row for the CSV stringifier. But I don’t want to add another pipe in the publish chain to do this transformation. What if I want to change the formatter to an XML formatter? (haha) The details of the formatter shouldn’t be exposed to the publisher. So I’d like to wrap up multiple transformers/pipes into a single stream module.

Here’s how we can do this, building on the CsvPrepStream transformer introduced in Part 1.

var stream = require('stream'),
  CSV = require('csv'),
/*
 * Formats a batch of data as a CSV with fields in the given order.
 */
function CsvFormatter(fields, delimiter, prepper) {
 var self = this;
 
 prepper = prepper || new CsvPrepStream(fields);
 
 stream.PassThrough.call(self, { objectMode: true });
 
 self.pipe = function (dest, pipeOpts) {
   /*
    * override pipe to first stream through CsvPrepStream for converting datapoint objects
    * to ordered arrays and then delegate to node-csv to convert each record to a csv row
    */
   var super_pipe = stream.PassThrough.prototype.pipe,
     csv = CSV().to.options({ delimiter:delimiter, header:true, columns:fields });
   return super_pipe.call(self, prepper).pipe(csv).pipe(dest, pipeOpts);
 };
}
 
util.inherits(CsvFormatter, stream.PassThrough);

We’ve extended the PassThrough stream like we did with the CsvPrepStream. This time, however, we’re overriding its pipe method so that it first sends the incoming stream data through the CsvPrepStream, and then through node-csv, before piping it out to its final destination. Note that we’re letting node-csv write out the header as the first line by passing header: true, columns: fields as its options. So now we have a single stream module (CsvFormatter) that accepts Javascript objects and outputs CSV rows with the fields taken from the objects and put in the requested order.

 Conclusion

In part 2, we’ve looked at how classic design patterns can be used to solve problems with node streams. Starting with the idea of a sexy publish function that streams data through a CSV formatter and out to an FTP server, we applied the adaptor pattern and then composer pattern to make this pipe chain possible. And, in the process, we discovered two new uses of the PassThrough stream. Sweet!

What design patterns do you use for streams in node.js?

For communication between internal services at BrightTag, we use self-signed certs on both the client and server. Simple and cheap (free!). Most of the time, these services only communicate over HTTPS with other internal services, so its been fine to use our own keystore; we didn’t need access to the “factory” certificates anyway. However, I ran into a case last week where I needed to be able to talk to both internal and external services and realized there’s no simple way to use multiple keystores in Java. We wanted to use both the standard JVM keystore and our custom keystore. The cleanest solution I found was to write my own CompositeKeyManager and CompositeTrustManagers. Creating a new keystore with both the standard JVM certs and our custom certs was also considered, but ultimately we didn’t want the responsibility of updating the standard certs in a bundled keystore.

Let’s dive right in. HTTP clients need an SSLContext or SSLSocketFactory (which can be retrieved from SSLContext#getSocketFactory). To initialize an SSLContext from your own keystore, you need an array of KeyManagers and an array of TrustManagers. A KeyManagerFactory and a TrustManagerFactory are used to extract the KeyManagers and TrustManagers from your keystore. The standard code looks something like this:

KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(keyStore, password);
TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
tmf.init(keyStore);
SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), null);

Okay, so I can just merge the KeyManager arrays  from each keystore, right?

Wrong, thanks to the fine-print associated with the  SSLContext initializer:

Only the first instance of a particular key and/or trust manager implementation type in the array is used. (For example, only the first javax.net.ssl.X509KeyManager in the array will be used.

In my case, both the JVM and our certs are X509, so this approach doesn’t work. Given that X509KeyManager is the only KeyManager implementation that ships with the JDK, I suspect this is the case for more or less everybody.

So what to do now?

As always, I turned to stackoverflow for an answer. This question addresses my problem and provides code showing a custom KeyManager implementation. Thanks to Raz for following up with an answer to his question, and everyone who chimed in to help him.

However, this solution wasn’t quite satisfactory to me. The MultiStoreKeyManager explicitly checks the custom KeyManager and then falls back to the jvm KeyManager if an operation fails. I actually want to check jvm certs first; the best solution should be able to handle either case. Additionally, the answer fails to provide a working TrustManager, so it doesn’t completely solve my multiple-keystore-with-SSL problem.

My approach was to apply the composite pattern a little more directly to create a CompositeX509KeyManager and CompositeX509TrustManager. They both take in a list of their delegates in order of preference. This adds support for any number of keystores in an arbitrary order. Whichever keystore (er, KeyManager/TrustManager) comes first in the injected list will be preferred to those coming later. My solutions use Guava because I heart Guava, but you should be able to replace them with another library or your own implementation of the few methods if you prefer.

Without further ado, ladies and gentlemen, I present to you the CompositeX509KeyManager.

package com.mycompany.ssl;
 
import java.net.Socket;
import java.security.Principal;
import java.security.PrivateKey;
import java.security.cert.X509Certificate;
import java.util.List;
 
import javax.annotation.Nullable;
import javax.net.ssl.X509KeyManager;
 
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterables;
 
/**
 * Represents an ordered list of {@link X509KeyManager}s with most-preferred managers first.
 *
 * This is necessary because of the fine-print on {@link SSLContext#init}:
 *     Only the first instance of a particular key and/or trust manager implementation type in the
 *     array is used. (For example, only the first javax.net.ssl.X509KeyManager in the array will be used.)
 *
 * @author codyaray
 * @since 4/22/2013
 * @see http://stackoverflow.com/questions/1793979/registering-multiple-keystores-in-jvm
 */
public class CompositeX509KeyManager implements X509KeyManager {
 
  private final List keyManagers;
 
  /**
   * Creates a new {@link CompositeX509KeyManager}.
   *
   * @param keyManagers the X509 key managers, ordered with the most-preferred managers first.
   */
  public CompositeX509KeyManager(List keyManagers) {
    this.keyManagers = ImmutableList.copyOf(keyManagers);
  }
 
  /**
   * Chooses the first non-null client alias returned from the delegate
   * {@link X509TrustManagers}, or {@code null} if there are no matches.
   */
  @Override
  public @Nullable String chooseClientAlias(String[] keyType, Principal[] issuers, Socket socket) {
    for (X509KeyManager keyManager : keyManagers) {
      String alias = keyManager.chooseClientAlias(keyType, issuers, socket);
      if (alias != null) {
        return alias;
      }
    }
    return null;
  }
 
  /**
   * Chooses the first non-null server alias returned from the delegate
   * {@link X509TrustManagers}, or {@code null} if there are no matches.
   */
  @Override
  public @Nullable String chooseServerAlias(String keyType, Principal[] issuers, Socket socket) {
    for (X509KeyManager keyManager : keyManagers) {
      String alias = keyManager.chooseServerAlias(keyType, issuers, socket);
      if (alias != null) {
        return alias;
      }
    }
    return null;
  }
 
  /**
   * Returns the first non-null private key associated with the
   * given alias, or {@code null} if the alias can't be found.
   */
  @Override
  public @Nullable PrivateKey getPrivateKey(String alias) {
    for (X509KeyManager keyManager : keyManagers) {
      PrivateKey privateKey = keyManager.getPrivateKey(alias);
      if (privateKey != null) {
        return privateKey;
      }
    }
    return null;
  }
 
  /**
   * Returns the first non-null certificate chain associated with the
   * given alias, or {@code null} if the alias can't be found.
   */
  @Override
  public @Nullable X509Certificate[] getCertificateChain(String alias) {
    for (X509KeyManager keyManager : keyManagers) {
      X509Certificate[] chain = keyManager.getCertificateChain(alias);
      if (chain != null && chain.length > 0) {
        return chain;
      }
    }
    return null;
  }
 
  /**
   * Get all matching aliases for authenticating the client side of a
   * secure socket, or {@code null} if there are no matches.
   */
  @Override
  public @Nullable String[] getClientAliases(String keyType, Principal[] issuers) {
    ImmutableList.Builder aliases = ImmutableList.builder();
    for (X509KeyManager keyManager : keyManagers) {
      aliases.add(keyManager.getClientAliases(keyType, issuers));
    }
    return emptyToNull(Iterables.toArray(aliases.build(), String.class));
  }
 
  /**
   * Get all matching aliases for authenticating the server side of a
   * secure socket, or {@code null} if there are no matches.
   */
  @Override
  public @Nullable String[] getServerAliases(String keyType, Principal[] issuers) {
    ImmutableList.Builder aliases = ImmutableList.builder();
    for (X509KeyManager keyManager : keyManagers) {
      aliases.add(keyManager.getServerAliases(keyType, issuers));
    }
    return emptyToNull(Iterables.toArray(aliases.build(), String.class));
  }
 
  private @Nullable  T[] emptyToNull(T[] arr) {
    return (arr.length == 0) ? null : arr;
  }
 
}

And the accompanying CompositeX509TrustManager:

package com.mycompany.ssl;
 
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.List;
 
import javax.net.ssl.X509TrustManager;
 
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterables;
 
/**
 * Represents an ordered list of {@link X509TrustManager}s with additive trust. If any one of the
 * composed managers trusts a certificate chain, then it is trusted by the composite manager.
 *
 * This is necessary because of the fine-print on {@link SSLContext#init}:
 *     Only the first instance of a particular key and/or trust manager implementation type in the
 *     array is used. (For example, only the first javax.net.ssl.X509KeyManager in the array will be used.)
 *
 * @author codyaray
 * @since 4/22/2013
 * @see http://stackoverflow.com/questions/1793979/registering-multiple-keystores-in-jvm
 */
public class CompositeX509TrustManager implements X509TrustManager {
 
  private final List trustManagers;
 
  public CompositeX509TrustManager(List trustManagers) {
    this.trustManagers = ImmutableList.copyOf(trustManagers);
  }
 
  @Override
  public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {
    for (X509TrustManager trustManager : trustManagers) {
      try {
        trustManager.checkClientTrusted(chain, authType);
        return; // someone trusts them. success!
      } catch (CertificateException e) {
        // maybe someone else will trust them
      }
    }
    throw new CertificateException("None of the TrustManagers trust this certificate chain");
  }
 
  @Override
  public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {
    for (X509TrustManager trustManager : trustManagers) {
      try {
        trustManager.checkServerTrusted(chain, authType);
        return; // someone trusts them. success!
      } catch (CertificateException e) {
        // maybe someone else will trust them
      }
    }
    throw new CertificateException("None of the TrustManagers trust this certificate chain");
  }
 
  @Override
  public X509Certificate[] getAcceptedIssuers() {
    ImmutableList.Builder certificates = ImmutableList.builder();
    for (X509TrustManager trustManager : trustManagers) {
      certificates.add(trustManager.getAcceptedIssuers());
    }
    return Iterables.toArray(certificates.build(), X509Certificate.class);
  }
 
}

For the standard case of one keystore + jvm keystore, you can wire it up like this. I’m using Guava again, but in a Guicey wrapper this time:

@Provides @Singleton
SSLContext provideSSLContext(KeyStore keystore, char[] password) {
  String defaultAlgorithm = KeyManagerFactory.getDefaultAlgorithm();
  X509KeyManager customKeyManager = getKeyManager("SunX509", keystore, password);
  X509KeyManager jvmKeyManager = getKeyManager(defaultAlgorithm, null, null);
  X509TrustManager customTrustManager = getTrustManager("SunX509", keystore);
  X509TrustManager jvmTrustManager = getTrustManager(defaultAlgorithm, null);
 
  KeyManager[] keyManagers = { new CompositeX509KeyManager(ImmutableList.of(jvmKeyManager, customKeyManager)) };
  TrustManager[] trustManagers = { new CompositeX509TrustManager(ImmutableList.of(jvmTrustManager, customTrustManager)) };
 
  SSLContext context = SSLContext.getInstance("SSL");
  context.init(keyManagers, trustManagers, null);
  return context;
}
 
private X509KeyManager getKeyManager(String algorithm, KeyStore keystore, char[] password) {
  KeyManagerFactory factory = KeyManagerFactory.getInstance(algorithm);
  factory.init(keystore, password);
  return Iterables.getFirst(Iterables.filter(
      Arrays.asList(factory.getKeyManagers()), X509KeyManager.class), null);
}
 
private X509TrustManager getTrustManager(String algorithm, KeyStore keystore) {
  TrustManagerFactory factory = TrustManagerFactory.getInstance(algorithm);
  factory.init(keystore);
  return Iterables.getFirst(Iterables.filter(
      Arrays.asList(factory.getTrustManagers()), X509TrustManager.class), null); 
}

Yep, Java is that verbose. But at least its clean, reusable, flexible, and future-proof.

I think this meets Raz’s initial request for “a solution that can dynamically register multiple keystores in addition to the default keystore/certs in jre into jvm.” What do you think?

Do you have a cleaner approach to mixing internal and external certs for SSL in Java?

There’s a saying among the core Node contributors: if you’re not using streams, you’re doing Node wrong*. So I spent last week learning the new Node streaming API (“streams2″) while building a small node app. While the documentation is pretty good, there’s not a lot of examples that are both realistic and easily digestible.

This two-part tutorial aims to build on the DailyJS’s Five Minute Guide to Streams2, providing real use cases for each of the different types of new streams (part 1) and the application of more complex design patterns to streams (part 2). If you haven’t read DailyJS’s tutorial yet, go ahead and read it now. Really, go.

You’re back? Good. You’ll notice that I use a different style of inheritance. For now, consider it personal preference and feel free to rewrite using your preferred style.

Readable Stream

Let’s start by creating a Readable stream that emits Javascript objects. This was the first custom stream I wrote, since I was buffering data in memory and then wanted to stream it through a CSV formatter and out through FTP. We’ll create a streams2 Readable stream that accepts a Javascript array and emits one element of that array at a time.

var util = require('util'),
  stream = require('stream');
 
function ArrayStream(data) {
  var self = this;
 
  stream.Readable.call(self, { objectMode: true });
 
  self._read = function (size) {
    data.forEach(function(element) {
      self.push(element);
    });
    self.push(null);
  };
}
 
util.inherits(ArrayStream, stream.Readable);
 
module.exports = ArrayStream;

Notice that we pass objectMode=true as an option to the stream, which tells it we’re handling Javascript objects rather than buffers or strings. The size argument to _read is advisory, so we’ve safely ignored it. We push each element out, one at a time. Finally, you signal the end of a Readable stream by pushing null.

A jasmine-node test for the ArrayStream class will demonstrate ArrayStream usage.

var stream = require('stream'),
  ArrayStream = require("../lib/streams/array");
 
describe("Streams: ArrayStream", function () {
  it("should stream one element at a time from the array", function (finish) {
    var array = [ { a: 1, b: 2, c: 3 }, { x: 8, y: 9, z: 10 } ],
      source = new ArrayStream(array),
      actual = [];
 
    source
      .on('data', function (data) {
        actual.push(data);
      })
      .on('end', function () {
        expect(actual).toEqual(array);
        finish();
      });
  });
 
}); // ArrayStream

As you can see, ArrayStream emits ‘data’ and ‘end’ events. In fact, ArrayStream can do everything a Readable stream can do, including being piped.

Transform Stream

Okay, so now we have a stream of objects. What if we want to CSVify the objects? We need a Transform stream to convert each object into an array representing a single ordered CSV record. This is what a streaming CSV stringifier needs as input.

var util = require('util'),
  stream = require('stream');
 
/*
 * Converts a datapoint into a CSV row (ordered array of fields).
 */
function CsvPrepStream(fields) {
 var self = this;
 
 stream.Transform.call(self, { objectMode: true });
 
 self._transform = function (datapoint, encoding, callback) {
   var record = [];
   fields.forEach(function(field) {
     record.push(datapoint[field]);
   });
   self.push(record);
   callback();
 };
}
 
util.inherits(CsvPrepStream, stream.Transform);
 
exports.CsvPrepStream = CsvPrepStream;

Note the use of objectMode=true again. This transform stream takes in a datapoint object, loops over the fields to be present in the CSV file (i.e., keys in the datapoint object), and pushes the values of the fields into a record array. We then push the transformed record out and signal completion by calling callback(). If there was an error, we could pass it to the callback as callback(err).

PassThrough Stream

Now we’ll see one of the most frequent uses of the new PassThrough stream: testing! Let’s write a jasmine-node test for CsvPrepStream; this test will demonstrate usage of PassThrough streams in addition to the functionality of CsvPrepStream.

var CsvPrepStream = require('../lib/formatters/csv').CsvPrepStream,
  stream = require('stream');
 
describe("Formatter: CsvPrepStream", function () {
 
  var data = [{
    key1: "foo",
    key2: "bar",
    key3: "baz"
  }, {
    key2: "zar",
    key1: "boo",
    key3: "zas"
  }];
 
  var fields = ["key1", "key2", "key3"];
 
  it("should emit records with correctly ordered fields", function (finish) {
    var prepper = new CsvPrepStream(fields),
      source = stream.PassThrough({ objectMode: true }),
      actual = [];
 
    source.pipe(prepper);
    source.write(data[0]);
    source.write(data[1]);
    source.end();
 
    prepper
      .on('data', function (data) {
        actual.push(data);
      })
      .on('end', function () {
        expect(actual).toEqual([ ["foo", "bar", "baz"], ["boo", "zar", "zas"] ]);
        finish();
      });
  });
 
}); // Formatter: CsvPrepStream

As you can see, PassThrough streams make great sources of data in testing! Just write some data into it in chunks, then end the stream. It’s that easy. If you’re writing anything other than strings or buffers, just remember that it must have objectMode=true.

Note that in a real test you’d want to test all possible scenarios, such as fields in the array that aren’t present as keys in the object and keys in the object that aren’t present in the array. This merely serves as an example.

Writable Stream

The final stream we’re going to cover today is the Writable stream. Once the CsvFormatter was written, we needed a way to capture the entire CSV contents as a string for testing it. Thus, the StringBufferStream was born.

var util = require('util'),
  stream = require('stream');
 
function StringBufferStream() {
  var self = this;
 
  self.string = "";
 
  stream.Writable.call(self);
 
  self._write = function(chunk, encoding, callback) {
    self.string += chunk.toString();
    callback();
  };
}
 
util.inherits(StringBufferStream, stream.Writable);
 
module.exports = StringBufferStream;

And for completeness sake, let’s go ahead and write a jasmine test for our StringBufferStream.

var StringBufferStream = require('../libs/streams/string-buffer'),
  stream = require('stream');
 
describe("Streams: StringBufferStream", function () {
 
  it("should store the chunks in an internal string buffer", function (finish) {
    var source = stream.PassThrough(),
      buffer = new StringBufferStream();
 
    source.pipe(buffer);
    source.write("boo yeah, ");
    source.write("mi amigos");
    source.end();
 
    buffer.on('finish', function () {
      expect(buffer.string).toEqual("boo yeah, mi amigos");
      finish();
    });
  });
 
}); // StringBufferStream

Conclusion

In this tutorial, we have seen real use cases for almost** all of the “streams2″ stream: Readable, Writable, Transform, and PassThrough. We’ve also seen examples of how to pass Javascript objects through your stream pipeline and how to test your custom streams with jasmine-node. In Part 2 of this write-up, we’ll cover some more advanced uses cases and apply classic design patterns to streams to solve some tricky problems. Stay tuned!

* At least that’s what I heard from a colleague who attended the last NodeConf.
** Once I find a good use case for the Duplex stream, I’ll write something up for it too.

Shell scripts aren’t known for being the cleanest, most elegant part of your technology. But I argue that, as a vital piece of your stack, they should be afforded the same thought and care as other parts of your codebase.

Sure, shell scripts like bash have some inherent limitations, but they shouldn’t prevent you from following standard software engineering principles.

For example, bash functions can’t return values, but functionality should still be broken into small reusable chunks each having a single responsibility. Just call the function in a subshell and have it echo the result. Or use a standardized return value. Plus, you can easily take advantage of the return codes of invoked programs.

Consider this simple script for generating a random alphanumeric identifier that’s unique across a group of files within a folder (e.g., xml files grouped by folder).

#!/usr/bin/env bash
# Generates a unique random fixed-length alphanumeric identifier
# Usage: ./bin/generate-id [vendor|product] [length]
#
# Example: generate a unique id
#        ./bin/generate-id
#
# Example: generate a unique eight-character product id
#        ./bin/generate-id product 8
 
path=${1:-vendor product}
length=${2:-7}
 
function generate_id {
  # https://coderwall.com/p/4zux3a
  echo $(cat /dev/urandom | LC_CTYPE=C tr -dc "a-zA-Z0-9" | head -c $length)
}
 
function id_exists {
  grep -qr $1 $2
}
 
id=$(generate_id)
while id_exists $id $path ; do
  id=$(generate_id)
done
 
echo $id

Looks good to this software engineer.

I spent the start of 2013 working on my first iOS SDK. In no particular order, here’s some of what I’ve learned about Objective-C.

• The hash of a collection is equal to the number of elements in that collection
• Singletons (er, “shared singletons“) are an accepted pattern in Objective-C
• Categories are cool. They’re kind of like Ruby’s mixins.
• Class clusters defy expectations. You can’t cleanly override a method in a class cluster with a category.
• The community seems to be suffering from Apple’s control. Too many mentions of “I can’t tell you because of NDA” on StackOverflow
• Apple’s docs are pretty good… if you can find the right ones
• There’s no easy way to build a “fat archive” and static framework unless you start with this project template, which I didn’t.
• appledoc should be the de facto documentation engine now, but isn’t.
• Unit testing networking code is much harder than it should be.
• I miss Guava. Simple things like Objects.toStringHelper, Objects.hashCode, and EqualsTester.
• I miss the builder pattern. Objective-C loves telescoping constructors initializers. Likewise, I miss Guice.
• Many of my favorite libraries for unit testing have been ported to Objective-C.
• The various abstractions for multithreaded programming are at war. GCD (the latest and greatest) vs NSOperationQueue (the most abstract) vs RunLoops. (Does this belong in this category? NSURLConnection can specify a runloop OR a queue, so I think so.)
• Some (but not all) of CoreFoundation is open source. A lot of NS* classes are the same as their CF counterparts via “toll-free bridging“
• Variable argument lists don’t know the number of arguments without being told. Most people use a nil-terminated list. A format string (implicit count) or explicitly passing the count also works.
• There is no NullPointerException; sending a message to nil is fine (no exception raised). This can really bite you in the ass.
• There are best practices. Read them before you start your next (first?) iOS or Objective-C project. I found this after I figured them out in bits and pieces.

Since I was simultaneously writing an Android SDK, expect a similar post about what I learned about Android in the process.

If you’ve been following Cassandra’s development, you’ll know that its made huge progress in the last two years. My company first start using Cassandra v0.8.x, and the latest stable version is now v1.2.3.

One of the major changes was adding support for composite columns. In DataStax words, “composite comparators can be thought of simply as a comparator composed of several other types of comparators.” Since introduction of composite columns in v1.0, there’s been a trend to slowly fade out super columns in favor of composite columns. Cassandra itself is refactoring supercolumns to use composites under-the-hood in v2.0.

So when my company upgraded to v1.0, I wrote a little tool to migrate our data from using super columns to use composite columns instead. This was about a year ago, but I wanted to announce I’ve open sourced it on Github. Better late than never, right?

Grab the source and try it yourself **.

** Disclaimer: I waive all responsibility for your use of this code. Think twice before using it on a large production dataset.

Ever step off the scale and wonder if a weight increase was due to gained fat or muscle?

A couple weeks ago, I was frustrated that my scale was showing the same weight, yet I could see my progress in the mirror and on the mat. That’s when I had a DEXA scan, and I’ve since learned how few people know about it.

If you’ve read any of my other case studies, you’ll know that I use numbers to make a lot of decisions in my life. I love DEXA because of the awesomely detailed and accurate information it can provide. A couple days ago, we learned why smart scales don’t cut it for measuring body fat; not coincidentally, data from a DEXA scan was used to make the point.

Now that I’ve built it up, I bet your wondering what kind of information you can get from DEXA.

• whole body breakdown into lean pounds, fat pounds, and bone mineral content (BMC)
• lean pounds, fat pounds, and BMC for each leg and each arm, your android (abdominal) and gynoid (hips/thighs) areas
• lean mass symmetry, fat mass distribution, bone density, and more

Forewarning, your body-fat percentage will likely be higher than you expect. I was in shock on my first consultation. DEXA includes all the fatty tissues in your body, even those in your brain (around 8 pounds). If you weigh 160 pounds, that’s an instant 5% increase in body fat.

What’s this data look like? Well, I’m glad you asked.

Without further ado, here are my DEXA results as an example.

Personal note: I lost 18 pounds of fat and gained 10 pounds of muscle! That’s a 28 pound re-composition and a decrease of 9% body fat! Woo!

Ok, now you’re excited. How can I get my own DEXA scan?

If you’re in the Chicagoland area, I highly recommend DexaFit, where I have my scans done. Otherwise, if you can’t find a local center online, call your local orthopedic doctor or hospital. Ask about bone densitometry—DEXA scans are used to diagnose osteoporosis around the country. Its actually what they were designed to do.