Ok, so if you followed the previous link and checked out our app, you’ll see that it’s somewhat pretty (for a ripoff), and barely functional! Man we rock… but really, the competition wasn’t just about winning for our team as much as it was about fiddling around with some new JavaScript hawtsawce. So here’s what I learned:

Pick your 48 hour project Wisely

Yeah, so our idea was to actually build a NodeJS backend which could scan the directories input by the user, read the id3 tag info from the files within those directories (as a start, we could support ogg and flac and other format metadata schemes later), and then put the metadata into a CouchDB.

Then the client could connect to your server, grab the catalog from NodeJS, and play songs that NodeJS would convert on the fly into the proper format for your browser depending on capability (probably through a child process exec to some other encoder — haven’t figured that part out yet).

Unfortunately, we hit a snag on step 1 of this process: reading the data from an id3 tag, which is binary data stored at the beginning on an mp3 file. Node can read streams of data, like so:

  var s = fs.createReadStream(file, {start: 1, end: 2});

  s.on('data', function(data){
    console.log(data.toString());
  });

This is actually the code I was using to check whether a particular file begins with “ID3,” which would thus signify ID3 version 2 data. All the fun technical details behind ID3 can be read about here: http://www.id3.org/Developer_Information

But this is about when things started to break down. After this, and I’m literally talking about the first 10 bytes here, became totally beyond my grasp for a 48 hour competition. WTF does this mean, for example:

The ID3v2 tag size is stored as a 32 bit synchsafe integer (section
6.2), making a total of 28 effective bits (representing up to 256MB).

I mean, I somewhat get what that’s saying. If you just read section 6.2, you can clear as day see…

All this led me to one conclusion after a few hours: Choose your project wisely. Dylan Bathurst, who I’m officially dubbing my “Node Wingman,” is a Front end guy. I’m a front end guy. We probably should not have picked a project that not only required us to pick up server-side programming, but also get all Computer Science major trying to decipher binary data (I have a degree in Sociology — super helpful for that kind of thing). It seems like a simple game using WebSockets might have been a better call… but oh well, just pick a project that helps you learn Node and plays to your strengths!

Plan a bit in advance…

We did a bit of preliminary planning, but we really didn’t get into the nitty gritty of our roles and responsibilities until the competition began. Not that this isn’t a totally fun way to approach a 48 hour competition, and if you think Animal from the Muppets is the coolest guy ever, then just jumping headfirst into the competition might be the best thing for you.

But, it seemed like some of the best ideas had been well thought out a in advance, including splitting up roles for each team member so they could focus on their tasks. With only 2 days of actual design and programming time, getting the prep out of the way so you can focus on code is clutch.

The community rocks

Yeah, I’m talking to you. I hadn’t spent much time in IRC before that weekend, but I was able to ask questions and get prompt, helpful responses. Plugin creators took their time to explain basic stuff to me, and it was even easy to hit up the creator of NodeJS. Plus, the vibe was just electric. Man, it was just fun being in there with everyone.

I also gotta say that I was really impressed with the Joyent setup overall. Thanks for providing such an awesome platform on which to host the competition!

Even if you don’t know anything about NodeJS, you should still totally participate

I wasn’t completely ignorant of Node, but I’m by no means an expert, and I haven’t built any serious/real apps on it. Heck, I hadn’t even built much of a toy app with it. This was my first real attempt at anything of substance. And even though we didn’t get a fully-functional app going, I learned more in two days than I have in a long time.

I would encourage anyone who has even the smallest inclination to give Node a shot to join this competition next time it comes around. If you haven’t learned Node because you don’t really know what to do with it, this competition ends that excuse! Just dive in. The water is warm — I promise!

The future is lookin pretty good…

And you know what was so cool about this competition? Knowledge wasn’t the best thing I gained. I walked away with a sense of optimism I haven’t felt in a while. When you get into the daily grind, a lot of the knowledge we accumulate seems to be incremental rather than revolutionary. NodeJS is a revolution.

Now, of course, there are a couple of secrets I’ve gotta save for myself until next year to give myself a bit of a competitive edge (heyo!), but I gotta end this post thanking everyone who put the competition together, Joyent again for hosting, the awesome people who were hangin out in IRC, and my buddy Dylan for putting the app together with me! It was a blast, and you’d better believe we’re gonna win it next year, so be ready to bring it!

After cloning and examining a lot of his repos (Use the Source!), I found out in an endnote that TJ had published a book entitled Advanced JavaScript from DevMag. It’s a short book, but one that opened my eyes in a surprising number of ways.

Overview

The books Table of Contents are as follows:

• Ch 1: Closures, Scope and Context
• Ch 2: Prototypal Inheritance
• Ch 3: Advanced Meta-programming techniques
• Ch 4: Behavior Driven Development
• Ch 5: Creating a jQuery Clone
• Ch 6: Tools of the trade

The meat of this review will focus on Chapters 3-5, which made me rethink the way I develop JavaScript. No offense to the author(s), but the contents in Chapters 1 and 2 have been better explained in other texts, notably JavaScript: The Good Parts by Douglas Crockford, and Chapter 6 should have probably just been an Appendix.

But back to the good stuff: Chapter 3 goes through the creation of a JavaScript clone of some of the most basic functionality found in Sinatra, namely creating routes and executing them when matched. While the concept of DRY is often promoted as The Thing To Do, this chapter clearly explained some of the ways to make your programs smarter so that you can in fact avoid repetition! Very nice.

The revolution will not be televised, it will be tracked on GitHub

The revolution started for me in Chapter 4. I’d read about some agile programming techniques, but I’d never heard of BDD, or Behavior Driven Development. Chapter 4 talks about how to implement BDD with JavaScript using JSpec, an awesome unit testing framework, and one that totally changed how I think about my projects.

An aside on BDD

Essentially, BDD is an “outside-in” methodology, meaning that you start by identifying outcomes with your clients (internal or external), and then map out the features that will achieve such outcomes. This often happens in a story format.

Overall, it’s fantastic because it allows your clients to stay on the same page as you, the developer. You define concrete outcomes with your clients, and which steps needed to be taken for each feature to work. This means that both you and your clients have a solid understand of what needs to be done, and what “done” truly means. To read more about BDD specifically, check out a great explanation of BDD from the inventor, Dan North.

Using JSpec to implement BDD

With regard to chapter 4, JSpec helps us implement BDD from the developers perspective. Once the outcomes and features have been hashed out into specific steps, or “units,” then the developer can begin doing his/her job! Now, BDD is an extension of test driven development, and therefore implies that we should begin not by writing our implementation, but by writing Unit Tests to ensure that the functionality defined within our story is properly executed! By working this way, developers get the myriad of benefits that come from proper unit testing, such as avoiding regressions (breaking your code).

However, it had never really clicked in my mind how to really do convenient / good unit testing with JavaScript, and that’s what Chapters 4 and 5 go into in detail. There are a ton of libraries out there, but none of them encourage Unit Testing as part of the workflow the way JSpec does, because most of them are just simple unit testing frameworks written in JavaScript. But JSpec is so much more!

Jspec doesn’t just include a unit testing JavaScript file, you download a whole Ruby gem and get amazing command-line utilities for creating new projects with testing and documentation all set up, and then very convenient methods for running the unit tests in all your browsers! For example, if you execute the following command:

jspec init mySpecTest

You’ll get a new folder called mySpecTest with a directory full of goodies, such as a convenient folder structure for creating consistent projects, and all you need to run the JSpec unit tests. With the specs from your client in hand, you could then quickly begin writing your unit tests to achieve the desired functionality! Then all you have to do is write your JavaScript to pass all of your tests, and you have a very discrete piece of functionality “done!”

Wrap Up

The book does a great job of going through how to use JSpec in a number of ways, and how to build a cool, limited jQuery clone using the principles of BDD. Overall, as I said, this has literally changed the way I think about writing JavaScript in the future. This will certainly be the approach I take (I had never really written rigorous tests for my JS before), but one that I can immediately see will provide a ton of benefits to the quality of my code.

If you’re like me, and you thought “Unit testing is pretty cool, but I really have no idea how to properly implement it,” you will walk away from this book knowing exactly how to use unit tests, and you’ll also probably be bursting with ideas about how you can improve your code and your development methodologies.

“Advanced JavaScript” rules, and for $4 it is well worth the price. Go buy it.

Addendum/Nitpicks

Now, while the book totally kicks ass, I did have some minor problems with it. I really think the book needs a good editor. There are some frankly just kind of sloppy grammatical and spelling errors throughout the book, and the writing in general could use a good solid critique to make it a bit better. Also, there’s a lot of source code presented in the book, but the book didn’t come with all of the source in separate files. It had a few files, but it was missing some of the JSpec files that I would have found most interesting.

But when I could start hooking up my designs to really cool functionality, such as dmMonocle, my jQuery plugin for viewing images, I loved the combination that was possible. I could not only make beautiful designs, I could make beautiful web applications. My code could react to the actions of users — or in JavaScript lingo, the events of users.

Server side JS?

I’ve since learned to branch out, and have an affection for Python as well at this point! But I’m still in that weird designer/front end dev gray area, so I don’t get as much Python time as I’d like. However, with the advent of Node.js, I can now do some pretty awesome server side stuff in my favorite language. That’s right, you heard me: JavaScript is going server side — and even further in the future! Here’s the code I wrote to implement a very simple web server with Node.JS:

var fs = require('fs'),
    sys = require('sys'),
    http = require('http');

// Simple Server Function
// Pulls the url of the request and responds with the appropriate HMTL file
// If no HTML file found in the request url, returns a 404 error. Boosh!

http.createServer(function (req, res) {

    var HTML;

    if(req.url.search(/\.html$/) > -1) { // Searches to see if the URL entered by the user ends in .html
        fs.readFile('../' + req.url, encoding='utf8', function(err, data) { // If it does, it tries to read the file
            if (err) { // Oh noes! The file doesn't exist! 404-butter time!
                _404(err);
            } else {
                HTML = data;
                res.writeHead(200, {'Content-Type': 'text/html'});
                res.write(HTML);
                res.close();
            }
        });
    } else { // Hey, they didn't enter a valid URL! You know what that means -- 404 BUTTER TIME!
       _404();
    }

    function _404(err) {
        if (err) sys.log(err);
        fs.readFile('../404.html', encoding='utf8', function(err, data) {
           HTML = data;
           res.writeHead(200, {'Content-Type': 'text/html'});
           res.write(HTML);
           res.close();
        });
    }

}).listen(8080); // Listen on port 8080
sys.puts("Brian's HTML Server running at http://127.0.0.1:8080/"); // Output a string to STDOUT (the command line in most cases)

Lame, but WTF is require()?

Now, this server isn’t very durable. For example, the html files have to be one directory down from the node server — not awesome, and it serves no CSS. But it was just a simple test script I wrote in 5 minutes to see what could be done with Node, and was a heck of a lot of fun. It also introduced me to the CommonJS project, and their implementation of modules. First of all, the CommonJS project aims to produce a common spec for server-side and application-building JavaScript — i.e. JavaScript’s not just for browsers anymore!

Now on to modules: at the top of the code, you’ll see I’m declaring 3 variables, all of which call a require() function. That require function is an example of the CommonJS module implementation, a way for JavaScript to pull in the contents from another file in a similar fashion to Python’s import or Ruby’s require.

The way of the future?

Pretty cool stuff for a few minutes of compilation time, reading through the Node.JS API (which only takes about 45 minutes even if you’re slow like me), and then getting a little dirty — and by that, I mean not dirty at all.

There’s some real potential here, and I’m going to take some time to play around with Express. It’s a Node.JS web framework inspired by a Sinatra, a Ruby web framework. Sinatra is another intriguing piece of software, and combined with HAML and SASS, two great technologies for rapidly developing HTML and CSS, it’s a very nice package indeed. Express seems to have adopted those parts of Sinatra, and should be just a grand old time to tinker with.

Flame war! (Intro)

I was actually a sociology major in college, so I kind of enjoy the rigor of trying to test things well. This is not a rigorous test — but I did learn some fun stuff!

Over at NETTUTS there was a discussion about the best way to update the DOM using jQuery. The author suggested this method, and was promptly called an idiot:

var updatedText = "Value1
Value2";
$("#container").html(updatedText);

I actually used a variation on this method when writing dmMonocle, my jQuery plugin which makes viewing images within library digital collections, such as the Southern Nevada Boomtowns Collection, a bit easier. What dmMonocle does is generate a series of tiles (divs) and then injects them all together at once, just as the author suggested above. Others claimed that the built-in jQuery methods (.html() & .append()) were more readable, and should therefore be used.

Test me once, shame on you (Methods)

In these cases, I like to let the numbers speak for themselves, and so I created a series of tests using JSFiddle that test the performance of three methods. I posted a couple of simple tests to NETTUTS this morning, but I thought it would be fun to get more in-depth and play around with loops of console profiles and timers to test performance (you know, how every sane person likes to spend their time)!

I wanted to time and profile three different functions:

• Build a string of spans and inject it once into a container using the .html() method
• Update each one of the spans individually with $(#span).html("new html")
• Building the spans within the contain using the .appendTo() method.

As the bad joke of a subheading should tell you, we don’t simply want to run these timers and profiles once, we want to run them 100 times (more would probably be better). To run them once would be a grave mistake. Don’t you remember doing 10 trial runs in science class to eliminate error? Did you think the cheeky lab report workflow of this post was a mistake?

I’m also running these tests on a Summer 2009 Macbook Pro, and would love to hear about results from other machines.

Show me the code! (Data)

• Timer Tests
• Profile Tests

As I said, these tests run each method 100 times. Scratch that, Firefox ate it when I tried that =P I’ll just run em a bunch manually and record numbers the good ol’ fashioned way. I’ll work on best practices for averaging results in a future post

Chrome

My new browser of choice:

1. First Test Average: 2-3ms
2. Second Test Average: 6-8ms
3. Third Test Average: 8-10ms

Firefox 3.6

Tried and true (but slooooooow):

1. First Test Average: 11-12ms
2. Second Test Average: 21-23ms
3. Third Test Average: 37-39ms

I mean look at that. 3-5x slower than Chrome 4 on simple operations.

Safari

Safari was… creepy with how consistent it was. I think I may be doing something wrong

1. First Test Average: 3ms
2. Second Test Average: 6ms
3. Third Test Average: 16ms

Internet Explorer

I can only run IE on a virtual machine at the moment, and I’m not sure how reliable that is. Give me some data!

Blah Blah Blah, what’s the bottom line (Results und Discussion)

Bottom line is that, according to my tests, using native JavaScript strings to build the html and injecting it once is roughly twice as fast as the updating method, and 3-5x times faster than append method (which is the most similar to the original string function). These results are due to the fact that the string method skips over calling not only the jQuery constructor $() on each iteration, but also the jQuery methods .html() and .append(), which themselves call multiple functions (as demonstrated by the profiling tests).

Based on those facts, when I need to inject any significant amount of code, I’m going to try to do as much building as I can in regular old JavaScript, avoiding the DOM and unnecessary jQuery methods as much as possible. Look, jQuery is great and really fast; I mean, I’m literally talking about 2-5x slower than 2 miliseconds). But, I’ll basically take any speed advantage I can get, and in this instance there is a clear winner. I personally don’t think the JavaScript is too much uglier or less maintainable than the jQuery fashion, but that is really a matter of taste. When it comes to the data, there’s no debate.