Formidable Labs

Our Focus

Whether on PAAS, IAAS, or bare metal.

• Design for Failure: Keep your Node.js apps up
• Avoidance: Get yourself out of the failover business
• Isolate: One failure at a time
• Analyze: Debug and diagnose problems quickly

1. Design for Failure

Fail and recover at multiple levels.

Let's look at failure from a system perspective.

Single Node.js Worker

Never ignore errors.

Have a strong bias for killing the worker.

• Handle: uncaughtException, Domains
• Listen: foo.on("error")

Multiple Node.js Workers

/*global process:false */
var recluster = require("recluster");
var cluster = recluster("./server.js");
cluster.run();

// Hot reload: kill -s SIGUSR2 CLUSTER_PID
process.on("SIGUSR2", function() {
  console.log("Got SIGUSR2, reloading cluster...");
  cluster.reload();
});

Server

• Use monit or alternatives
• Restart the Node.js master

2. Avoid Failures

Get out of the business of failover when you don't have to do it yourself.

Resources to Not Support

Don't rely on system / service resources you don't need to.

• Disk: NAS, disks, SSDs.
• Datastores: DB, cloud services.
• ... Load Balancers, DNS, etc.

How To Avoid

• Use SAAS wherever possible! (DB, LBs, storage).
• Or PAAS for some Node.js apps.
• Design Stateless, fungible servers (no disk risks).

Resources to Support

Look to resources you must depend on:

• CPU/Load: Run out of this and it's over.
• HTTP: Each different host you hit.
• Datastores: Connections? Different Hosts?
• ... also, memory, I/O, etc. and combinations thereof

Some Anecdotes

Node.js apps can be bad neighbors.

• DB (auto-suggest) vs. HTTP (vendor translations)
• DB (CRUD app) vs. CPU/Load (co-located PHP app)
• Read vs. Write DB operations.

How To Isolate

• Create "micro-services" that stand on their own.
• Monitor for cross-pressure and respond. (Next section!)

Log, Monitor, Mine

Decisions, Goals

Things to look for in Node.js apps...