hello.

Boston will always have a special place in my heart. I went to school there, and in a lot of ways grew up there (in that way Millennials say they’re still “growing up” into their late 20’s).

So when I heard that Box organizes a Boston recruiting trip every fall, I was desperate to get involved. The only problem? The trip requires interviewing experience, and I had none.

So I started interviewing… a lot. I interviewed college students. I interviewed senior candidates. I interviewed everyone, and slowly began to recognize what made a good interview vs. a bad one. I even began to recognize all the things that I had been doing wrong as an interviewee. What the hell?!? I was actually learning something, when all I originally wanted was a free flight to Boston.

This is a talk about technical interviewing, and everything I know about doing it well. The talk is geared towards college students, but almost all of it will be relevant to anyone interested in the process. It includes the expected tips and tricks, but with a main focus on what happens before and after the interview… and how it matters much, much more than you think.

Slides

Video

Unfortunately, I wasn’t able to get a video of the talk (I really wish I had, because the Q&A afterwards was awesome). But, if you are a college student or professor and want me to come give this talk to your class, reach out. Seriously, I’d love to visit your school and give it. No catch.

Recently I’ve noticed a lack of resources on advanced Node.js topics. There are plenty of guides and tutorials for getting started, but very little is written on maintainable design or scalable architecture. So I created The Node.js Handbook, a series meant to address this gap by sharing tribal knowledge and best practices. You can read more here.

There are a million different ways to design a JavaScript module. Standard patterns like the singleton and custom type are widely adopted, and provide a dependable feature-set. Some other patterns, however, push the limits of what a module can (and should) actually be. The first group is often encouraged, while the second is denounced without further thought. This post will attempt to explore that second group.

Before jumping in, I want to explicitly point out that almost all of the concepts explained below should be avoided in production. These patterns have the potential to cause nightmares for you or your team down the road, with hidden bugs and unexpected side-effects. But they exist for a reason, and when used properly (read: very carefully) they can solve real problems that other, safer patterns can’t. But just… you know… with those terrible, dangerous side effects.

Monkey Patches

JavaScript is a dynamic language, which – when paired with its prototype-based nature – gives the developer free range to modify objects and classes across entire applications. So when you one day find yourself building a pig-latin generator and wishing that JavaScript strings handled this conversion themselves, you can do something like this:

1
2
String.prototype.pigLatin = function() { /* ... */ }
'Is this actually a good idea?'.pigLatin()    // 'Is-ay is-thay actually-ay an ood-gay idea-ay?'

Modifying already-existing methods can be a little trickier. You can simply overwrite them, but if you want to leverage the original function you’ll need to save it first. Using a more practical example than the one above, you may want to attach data to every every template that gets rendered in an Express application:

1
2
3
4
5
6
7
// Save the original render function to use later
res._render = res.render;
// Wrap the render function to process args before rendering
res.render = function(view, options, callback) {
  options.global = { /* ... */ };
  this._render(view, options, callback);
}

This practice is called monkey patching, and it is generally considered to be a terrible idea. Monkey patches pollute your application’s shared environment. They can collide with other patches, and be impossible to debug even when working properly. The pattern is a powerful hack, but luckily its adoption and use is generally limited.

But desperate times can call for desperate measures, and sometimes a monkey patch is necessary. If the situation allows it, building your patch as a separate module will help keep the hack quarantined and decoupled from the rest of your application. Organizing your monkey patches in one place can also make it easier to find when/if debugging is needed.

The first thing you’ll want to do is make as many assertions about the environment as possible. Assert that the method you’re adding/modifying hasn’t been added/modified yet. Check that its version is correct. Check that everything exists exactly as you expect. Check all of this first, and throw an error if any of it doesn’t look right. While this might sound over-the-top now, it could save you days of debugging later if you fail to catch some subtle collision.

You should also consider exporting your monkey patch as a singleton, with a single apply() method that executes the code. Applying the patch explicitly (instead of as a side effect of loading it) will make your module’s purpose clearer. It will also allow you to pass arguments to your monkey patch, which might be helpful or even necessary depending on your use case.

1
2
3
4
5
6
7
8
9
// some-monkey-patch/index.js
module.exports = {
  apply: function() {
    /* check environment/arguments & apply patch */
  }
}

// later...
require('some-monkey-patch').apply();

Polyfills

Polyfills are most commonly found on the client-side, where different browsers have different levels of feature support. Instead of forcing your application down to support the lowest-common denominator (looking at you, IE) you can use a polyfill to add new features to old browsers and standardize across platforms.

As a server-side developer, you might think that you’re safe from this problem. But with Node’s long v0.12 development cycle, even Node.js developers will find new features that aren’t fully available to them yet. For example, async-listeners were added in v0.11.9, but you’ll have to wait until v0.12.0 before you’ll see them in a stable build.

Or… you could consider using an async-listener polyfill.

1
2
// load polyfill if native support is unavailable
if (!process.addAsyncListener) require('async-listener');

The polyfill is still a monkey patch at heart, but it can be much safer to apply in practice. Instead of modifying anything and everything, polyfills are limited to implementing an already-defined feature. The presence of a spec makes polyfills easier to accept, but all the same warnings and guidelines for monkey patching still apply. Understand the code you’re adding, watch out for collisions (specs can always change), and make sure you assert as much as possible about your environment before applying the patch.

JSON Modules

JSON is the data format of choice for Node.js, and native JSON support makes it easy to load and then interact with static data files as if they were actually JavaScript modules. The original http-status-codes-json module, for example, was entirely represented by a static JSON file. And because of Node’s JSON support, the module became an interactive dictionary of HTTP status codes.

1
2
3
4
5
6
7
8
9
// http_status_codes.json
{
  "100": "Continue",
  "200": "OK",
  /* ... */

// later...
var httpStatusCodes = require('http-status-codes-json');
console.log(httpStatusCodes[res.statusCode]); // 'NOT FOUND'

This feature can be powerful, but don’t refactor your code just yet. Modules are loaded synchronously, which means nothing else can run while the file is loaded and parsed. And once parsed, the result is saved and persisted in your module cache for the rest of your applications lifetime. Unless you intend to actually interact with the object as a module, stick to fs.readFile() and/or JSON.parse(), and save yourself the performance hit and added complexity.

Compile-to-JS Modules

Node supports JSON right out of the box, but require() will throw an error if you try loading anything else. However, if you roll up your sleeves and start poking around, you’ll find that Node can be made to support any number of file types, as long as you provide the parsers.

Here’s how it works: Node holds a collection of “file extensions” internally, which are responsible for loading, parsing, and exporting a valid representation of a given file. The native JSON extension, for example, reads the file via fs.readFileSync(), parses the results via JSON.parse(), and then attaches the final object to module.exports. While these parsers are private to Node’s Module type, they are exposed to developers via the require() function.

CoffeeScript is probably the most popular compile-to-js language, but to properly use it with Node you’ll need to compile it down to JavaScript after every change. Using the technique described above, fans could instead build CoffeeScript support right into Node.js, handling this extra step automatically:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
module.exports = {
  apply: function() {
    // Load your new CoffeeScript extension into Node.js
    require.extensions['coffee'] = function coffeescriptLoader(module, filename) {
      // Read the contents from the '.coffee' file
      var fileContent = fs.readFileSync(filename, 'utf8');
      // Compile it into JavaScript so that V8 can understand it
      var jsContent = coffeescript.compile(fileContent);
      // Pass the contents to be compiled like a normal JavaScript module
      module._compile(jsContent, filename);
    };
  }
}

// Later...
require('require-coffee').apply();

Note: This feature was deprecated once everyone realized that processing your code into JS and JSON before run-time is almost always the better way to go. Parsing directly during runtime can make bugs harder to find, since you can’t see the actual JS/JSON that gets generated.

MP3… Modules?

CoffeeScript was built with JavaScript in mind, so requiring a CoffeeScript module makes a lot of sense. But since Node.js leaves the file representation up to the developer, you can really require any file type you want. In this last section, lets see how this would work with something completely different, like an MP3.

It would be too easy to just load and return the file contents as an MP3 module, so lets go one step further. In addition to getting the MP3 file contents, the file extension should also generate song metadata (such as title and artist) via the audio-metadata module.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
var audioMetaData = require('audio-metadata');

// A custom type to represent the mp3 file and its metadata
function MP3(file) {
  // Attach the file contents
  this.content = file;
  // Process and attach the audio id3 tags
  this.metadata = audioMetaData.id3v2(fileContent);
}

// Attach your new MP3 extension
require.extensions['mp3'] = function mp3Loader(module, filename) {
  // Read the contents from the '.mp3' file
  var fileContent = fs.readFileSync(filename);
  // Export a new MP3 instance to represent the module
  module.exports = new MP3(fileContents);
};

// Later...
var song = require('/media/i-believe-in-a-thing-called-love.mp3');
console.log(song.metadata.artist + ': ' + song.metadata.title); // 'The Darkness: I Believe in a Thing Called Love'

Depending on the use case, this extension could be built to add even more functionality like streaming, playing, and otherwise interacting with the song, all automatically supported at load time.

Conclusion

This post isn’t meant to endorse or approve of any of the above patterns, but it isn’t a blanket denouncement either. What makes these modules dangerous is the same thing that makes them so powerful: they don’t follow the normal rules. Polyfills can update your feature set without actually updating the framework, while File Extensions change the idea of what a Node.js module can actually be. Understanding how any of this is possible will help you make smarter decisions when it comes to module design, and allow you to spot potential problems before they happen.

And one day, when you find yourself in a jam, one of these patterns might just help you out.

Module.js is one of the most important modules in Node.js core, yet its existance remains almost completely undocumented.

In this talk, I expand on my previous blog post by walking through the module and explaining how loading, compiling, and caching all work. If you’re interested in exploring Node.js core but don’t know where to start — or if you’re just curious about its inner workings — then this is the talk for you.

Slides

Video


Recently I’ve noticed a lack of resources on advanced Node.js topics. There are plenty of guides and tutorials for getting started, but very little is written on maintainable design or scalable architecture. This post is a part of The Node.js Handbook, a series created to address this gap by sharing tribal knowledge and best practices. You can read more here.

Update 07/28/14 – I just gave a talk on this at BayNode, which includes a walkthrough of all the code discussed in this post. If that’s more your style, check it out.

Almost any Node.js developer can tell you what the require() function does, but how many of us actually know how it works? We use it every day to load libraries and modules, but its behavior otherwise is a mystery.

Curious, I dug into Node core to find out what was happening under the hood. But instead of finding a single function, I ended up at the heart of Node’s module system: module.js. The file contains a surprisingly powerful yet relatively unknown core module that controls the loading, compiling, and caching of every file used. require(), it turned out, was just the tip of the iceberg.

module.js

1
2
3
4
5
function Module(id, parent) {
  this.id = id;
  this.exports = {};
  this.parent = parent;
  // ...

The Module type found in module.js has two main roles inside of Node.js. First, it provides a foundation for all Node.js modules to build off of. Each file is given a new instance of this base module on load, which persists even after the file has run. This is why we are able attach properties to module.exports and return them later as needed.

The module’s second big job is to handle Node’s module loading mechanism. The stand-alone require function that we use is actually an abstraction over module.require, which is itself just a simple wrapper around Module._load. This load method handles the actual loading of each file, and is where we’ll begin our journey.

Module._load

1
2
3
4
5
6
7
8
9
10
Module._load = function(request, parent, isMain) {
  // 1. Check Module._cache for the cached module. 
  // 2. Create a new Module instance if cache is empty.
  // 3. Save it to the cache.
  // 4. Call module.load() with your the given filename.
  //    This will call module.compile() after reading the file contents.
  // 5. If there was an error loading/parsing the file, 
  //    delete the bad module from the cache
  // 6. return module.exports
};

Module._load is responsible for loading new modules and managing the module cache. Caching each module on load reduces the number of redundant file reads and can speed up your application significantly. In addition, sharing module instances allows for singleton-like modules that can keep state across a project.

If a module doesn’t exist in the cache, Module._load will create a new base module for that file. It will then tell the module to read in the new file’s contents before sending them to module._compile.[1]

If you notice step #6 above, you’ll see that module.exports is returned to the user. This is why you use exports and module.exports when defining your public interface, since that’s exactly what Module._load and then require will return. I was surprised that there wasn’t more magic going on here, but if anything that’s for the better.

module._compile

1
2
3
4
5
6
7
Module.prototype._compile = function(content, filename) {
  // 1. Create the standalone require function that calls module.require.
  // 2. Attach other helper methods to require.
  // 3. Wraps the JS code in a function that provides our require,
  //    module, etc. variables locally to the module scope.
  // 4. Run that function
};

This is where the real magic happens. First, a special standalone require function is created for that module. THIS is the require function that we are all familiar with. While the function itself is just a wrapper around Module.require, it also contains some lesser-known helper properties and methods for us to use:

  • require(): Loads an external module
  • require.resolve(): Resolves a module name to its absolute path
  • require.main: The main module
  • require.cache: All cached modules
  • require.extensions: Available compilation methods for each valid file type, based on its extension

Once require is ready, the entire loaded source code is wrapped in a new function, which takes in require, module, exports, and all other exposed variables as arguments. This creates a new functional scope just for that module so that there is no pollution of the rest of the Node.js environment.

1
2
3
(function (exports, require, module, __filename, __dirname) {
  // YOUR CODE INJECTED HERE!
});

Finally, the function wrapping the module is run. The entire Module._compile method is executed synchronously, so the original call to Module._load just waits for this code to run before finishing up and returning module.exports back to the user.

Conclusion

And so we’ve reached the end of the require code path, and in doing so have come full circle by creating the very require function that we had begun investigating in the first place.

If you’ve made it all this way, then you’re ready for the final secret: require('module'). That’s right, the module system itself can be loaded VIA the module system. INCEPTION. This may sound strange, but it lets userland modules interact with the loading system without digging into Node.js core. Popular modules like mockery and rewire are built off of this.[2]

If you want to learn more, check out the module.js source code for yourself. There is plenty more there to keep you busy and blow your mind. Bonus points for the first person who can tell me what ‘NODE_MODULE_CONTEXTS’ is and why it was added.


[1] The module._compile method is only used for running JavaScript files. JSON files are simply parsed and returned via JSON.parse()

[2] However, both of these modules are built on private Module methods, like Module._resolveLookupPaths and Module._findPath. You could argue that this isn’t much better…

Recently I’ve noticed a lack of resources on advanced Node.js topics. There are plenty of guides and tutorials for getting started, but very little is written on maintainable design or scalable architecture. This post is a part of The Node.js Handbook, a series created to address this gap by sharing best practices. You can read more here.

Testing can be a tricky topic no matter what language we’re in. Javascript’s flexibility makes it easy to get started, but can leave us tearing our hair out days later. How do we handle an API callback? How do we deal with require? Without a proper setup, whether TDD is dead or not will end up meaning very little.

This post will explain the tools needed to overcome the challenges of testing with Node.js. Together, they form an essential testing suite that will cover almost any project. The setup isn’t the most complex or feature-rich, but you could almost say that’s on purpose. If that sounds counter-intuitive… read on.

Introduction: Zero Points for Clever Tests

Before introducing the tools, it’s important to emphasis the reason we write tests in the first place: confidence. We write tests to inspire confidence that everything is working as expected. If something breaks we want to be sure we’ll catch it, and quickly understand what went wrong. Every line of every single test file should be written for this purpose.

The problem is that modern frameworks have gotten incredibly clever. This is ultimately a good thing, but it means we’ll need to be careful: this extra power is easily gained at the expense of clarity. Our tests may run faster or have more reusable code, but does that make us more or less confident in what is actually being tested? Always remember: There are no points for clever tests.

Test clarity should be valued above all else. If our framework obfuscates this in the name of efficiency or cleverness, then it is doing us a disservice.

The Essential Toolkit

With that out of the way, lets introduce the four types of tools needed for successful Node.js testing:

  • A Testing Framework (Mocha, Vows, Intern)
  • An Assertion Library (Chai, Assert)
  • Stubs (Sinon)
  • Module Control (Mockery, Rewire)

A Testing Framework

The first and most important thing we’ll need is a testing framework. A framework will be our bedrock, providing a clear and scalable structure for our tests. We have a ton of options here, each with a different feature set and design. No matter which framework you go for, make sure you chose one that supports our mission: writing clear, maintainable tests.

For Node.js, Mocha is the gold standard. It has been around forever, and is well tested and maintained. Its customization options are extensive, which makes it incredibly flexible as well. While the framework is far from sexy, its setup/teardown pattern encourages explicit, understandable, and easy-to-follow tests.

1
2
3
4
5
6
7
8
9
10
  before(function(){
    // before() is the first thing we run before all your tests. Do one-time setup here.
  });
  it('does x when y', function(){
    // Now... Test!
  });
  after(function() {
      // after() is run after all your tests have completed. Do teardown here.
  });
});

An Assertion Library

With a new testing framework in place, we’re ready to write some tests. The easiest way to do that is with an assertion library.

1
assert(object.isValid, 'tests that this property is true, and throws an error if it is false');

There are a ton of different libraries and syntax styles available for us to use. TDD, BDD, assert(), should()… the list goes on. BDD has been gaining popularity recently thanks to its natural-language structure, but it should all come down to what feels best to you. Chai is a great library for experimenting because it supports most of the popular assertion styles. But if you’re a dependency minimalist, Node.js comes bundled with a simple assertion library as well.

1
2
expect(42).to.equal(42); // BDD Assertion Style
assert.equal(42, 42);    // TDD Assertion Style

Stubs

Unfortunately, assertions alone will only get us so far. When testing more complex functions, we’ll need a way to influence behavior and test code under explicit conditions. While it’s important to always stay true to the original behavior, sometimes we need to be certain that some function will return true, or that an API call will yield with an expected value. Sinon allows us to do this easily.

1
2
3
4
5
6
7
var callback = sinon.stub();
callback.withArgs(42).returns(1);
callback.withArgs(1).throws("TypeError");

callback();   // No return value, no exception
callback(42); // Returns 1
callback(1);  // Throws TypeError

Sinon includes a collection of other useful tools for testing, such as fake timers and argument matchers. In addition to Stubs, there are also Spies (a smaller subset of stub features for measuring function calls) and Mocks (for setting expectations on behavior) to experiment with. An entire book could be written on all of Sinon’s features, but you can always start simple and experiment as you go.

Module Control

We’re almost ready to start writing tests, but there’s still one last problem in our way: require(). Because most calls to require happen privately, we have no way to stub, assert, or otherwise access external modules. To really control our tests, we’ll need the option to override require to return modules under our control.

There are a few different ways to accomplish this, depending on how much power is needed. Mockery gives us control of the module cache, and lets us replace entries with modules of our own. It’s a cautious library, and will warn developers when it thinks they’ve done something unintentional, like overwriting or forgetting to replace certain mocks. Just be sure to disable & deregister mocks after the tests have run.

1
2
3
4
5
6
7
8
9
10
11
before(function() {
  mockery.enable();
  // Allow some ...
  mockery.registerAllowable('async');
  // ... Control others
  mockery.registerMock('../some-other-module', stubbedModule);
});
after(function() {
  mockery.deregisterAll();
  mockery.disable();
});

Rewire is another popular tool that is much more powerful than Mockery. We can get and set private variables within the file, inject new code, replace old functions, and otherwise modify the original file however we’d like. This may sound like a better deal, but with all this power comes the cliched responsibility. Just because we can check/set a private variable doesn’t mean we should. These additional abilities move our tested code farther away from the original, and can easily get in the way of writing good tests.

Bringing it all Together

To see these tools all working together check out a working example on GitHub. While I singled out a few favorite libraries above, there are plenty of good alternatives in each of the categories listed. Think I missed something important? Let me know in the comments, or on Twitter at @FredKSchott.