Introduction to Node.js

Whoami

Giovanni LelaCTO@LinkMecofounder@MeanMilan@lambrojoshttp://github.com/lambrojos

Logistics

● Text editor● node.js >= 0.10.*● npm

● nodeschool.io

Brief history of server side js

Pre-nodejs● Netscape enterprise server (1994)● IIS (1996)Not very successful

Enter node.js

● Ryan Dahl creates node.js to easily give sites push capabilities

● 2009 - Birth of node.js● Joyent takes control

Hype, dissatisfaction, forks

In the following years node gains substantial traction.But in 2014 project stagnation leads to a fork - io.jsWhat now?

Node.js foundation

Node is currently sponsored by the node.js foundation, with some big names being listed in the sponsors.

The node.js foundation itself is hosted by the linux foundation.

Technical decisions are made by the Tecnical Steering Commitee

Why node.js matters

It can handle a lot of concurrent I/O operations.This is because of non-blocking I/O and javascript own nature.

Blocking I/O analogy

Suppose you own a fast-food restaurant:● the cashier takes the order than waits for the food

to be ready before doing anything else

● if you want to serve more customers you need more cashiers

● this is “classical” multi-threaded parallel execution

Non blocking I/O analogy

- As soon as you place your order, it’s sent off for someone to fulfill while the cashier is still taking your payment.

- When you are done paying, you have to step aside because the cashier is already looking to service the next customer.

- When your order is complete, you will be called back

In real life

- the node server receives a request for /about.html

- while node waits for the filesystem to get the file, the server thread services the next

- when the file is ready a callback function call is inserted in the server thread message queue - can be seen as a todo list

Everything runs in parallel, except your code

source: http://localhost:9000/images/event-loop.jpg

Event loops

There are event loop implementations for many languages in platform. But in node.js the event loop is the platform

One last diagram

What not to do in the event loop

Everything that is CPU- intensive:● image / video processing● heavy number crunching● compressing files● actually everything involving complex

algorithms(Hower you can write native extensions)

What do - remote APIs

● REST/JSON Api● quite fun to use node with non relational DB● lots of stable and mature frameworks:

○ Express○ Hapi○ Sails.js○ Loopback○ ...

Javascript everywhere

● also your frontend can be in javascript● the barriers between the front and the back

end look thinner - Isomorphism● Shared rendering● Shared models

Real time web apps

That what it was born for● Frameworks:

○ Primus○ Socket IO○ Meteor

● It can also be integrated inside an existing web stack

IoT applications

● Hardware ○ Arduino○ Rasperry PI○ ..

● Frameworks○ johnny five (robotics)○ node-red (visual integration)○ ...

Desktop applications

Electron by github is a framework for building desktop apps. Applications built on Electron include:● Atom ● Slack clients● Visual studio code● Kitematic (docker)

Differences from the browser

● No DOM ● No browser globals such as window, document● No browser inconsistencies - you just have the

v8● ES5 supported - no ugly hacks and polyfills● ES6 getting supported

Something you don’t have in the browser

● control over the process● streams● dependency management● lots of callbacks

Process

It is a global variable it allow us to interct with the OS● process.exit(code)● process.env.[env variable name]● process.argv - read command line

parameters $ node argv.js one two three four myArgs: [ 'one', 'two', 'three', 'four' ]

Node streams

● They emit events which represent I/O operations

● They can be readable, writable or both (duplex)

Node streams: example

var readable = getReadableStreamSomehow();

readable.on('data', function(chunk) {

console.log('got %d bytes of data', chunk.length);});

readable.on('end', function() {

console.log('there will be no more data.');});

Combining streams w/ pipe

var fs = require('fs');var readableStream = fs.createReadStream('file1.txt');var writableStream = fs.createWriteStream('file2.txt');readableStream.pipe(writableStream);

Modules

Modules are managed with common.jsexports.getGreetings = function() {

return 'Hello World';};

//example.jsvar hello = require('./hello.js');var greetings = hello.getGreetings();

Module management: NPM

Common js modules are installed with npm● npm install● npm uninstall● npm updateaModules can be installed globally (applications), or locally to your applications (dependencies).

Dependency management

Your application dependencies and their version are listed in the package.json files.It can store other informations such as version, author, scripts to be executed during the application lifecycle.

Node-style callbacks

a.k.a. Error-first callbacks a.k.a. Nodebacks

function callback (err, data) { /* ... */ }

Beware of callback hell!

Node.js as a Web server app platform

Node’s asynchronous naturemakes it an ideal platform for web server side development:● High I/O throughput● Efficient and easy management of concurrent

requests● Shares language and some tooling with the client

side● LOTS of open source libraries and modules

Frameworks?

Node’s own http module is a bit low level.

We want to be able to avoid boilerplate while retaining flexibilityMany, many choices

Hapi

● stable● well documented● well maintened● modular● is a good mix between flexibility and structure● good plugin ecosystem

Who uses Hapi?

How did it start?

Hapi short for HTTP API, or so they sayDeveloped in Walmart Labs by a guy Named Eran HammerEran Hammer also authored OAuth

What does it do?

● Routes● Caching● Sessions● Logging● Authentication● Plugins

Philosophy

● Configuration over code● Three Rs:

○ Reusability - plugins, handlers○ Reduce errors - strong validation, 100% test coverage○ Reflection - the applications knows it’s own structure .

this makes generating documentation easier

Show me the code - routes

var Hapi = require('hapi');var server = new Hapi.Server('localhost', 8000);server.route({ method: 'GET', path: '/hello/{name}', handler: function (request, reply) { reply('hello world'); }});server.start();

Config objects

The route object alse accepts a config object which controls every aspect of the request’s lifecycle such as validation, authentication, caching Its properties may depend on the plugins installed

Config example

{config: {

handler: handlers.mapUsername,description: 'Get todo',notes: 'Returns a todo item by the id passed in the path',tags: ['api'],validate: {

params: {username: Joi.number()

.required()

.description('the id for the todo item'),

}} [...]

Adding functionality with plugins

Almost every feature in hapi is isolated in a plugin.Even core features are registred as a plugin:● static content (inert)● validation (joi)● errors (Boom)● monitoring (Good)● … MANY more

Using plugins

server.register([ Inert, Vision, { register: HapiSwagger, options: swaggerOptions }],

function (err) { server.start(function(){ }); });

Writing plugins

exports.register = function (server, options, next) { server.route({ method: 'GET', path: '/test', handler: function (request, reply) { reply('test passed'); } });

next();};

So everything must be a Hapi plugin?

No, of course, you can use any module as long as its framework agnostic.

Most used plugins - Inert

● Download files with reply.file(path)● Can expose files or whole directoriesserver.route({ method: 'GET', path: '/{param*}', //multi segment parameter handler: { directory: { path: 'public', listing: true } }});

Template rendering - Vision

var handler = function (request, reply) { reply.view('basic/index,html', {

title: getTitle() ... });

};

//basic/index.html<html> <head>

<title>{{title}}</title>..

Template helpers

Providing template helpers is easy// this is the server.view config obj{

views: {helpersPath: 'helpers’ // path to the helpers dir

}};

// helpers/helper.js - only one function per file! module.exports = function(context){

// context contains the same data available to the templatereturn whatever(context.myvar);

}

Template helper view

<html><head><title>Help</title></head>

<body> {{myhelper}} <!-- helper filename --></body>

</html>

Validate with Joi

● Joi is a generic validation framework● It works by createing schemas and validating

objects against it● It just validates the object format it does not

peform things such as database validation● It is so powerful that documentation can be

generated from Joi schemas!

Joiful example

var schema = {

username: Joi.string().alphanum().min(3).max(30).with('birthyear').required(),

birthyear: Joi.number().integer().min(1900).max(2013)};

var err = Joi.validate(data, schema, config);console.log(err ? err : 'Valid!');

Hapi with Joi

server.route({

path: '/user/{id}',method: 'GET',handler: (request, reply) => {

reply(getUser(id))},config: {

validate: {params: {

id: joi.number().integer()}

}}

Conditional validation

Joi.object({propertyRequired: Joi.boolean().required(),property: Joi.string().when('isGuest', {

is: true, then: Joi.required()

}),property2: Joi.string().alphanum()

}).options({ allowUnknown: true }).without('property', 'property2')

Hapi, Joi and Swagger

Joi validations can be used to generate a swagger documentation pageThis is an excellent example of reflection in action

Errors - Boom

Boom makes easy to generate HTTP errors Boom.unauthorized([message], [scheme], [attributes])

Boom.forbidden([message], [data])

Boom.notFound([message], [data])

Boom.methodNotAllowed([message], [data])

Boom.notAcceptable([message], [data])

Boom.proxyAuthRequired([message], [data])

Boom.clientTimeout([message], [data])

Boom.conflict([message], [data])

Cookies aka server.state

server.state('session', { //cookie name path: '/', encoding: 'base64json', ttl: 10, domain: 'localhost'});

reply('success').state(

'session', //cookie name'session' //cookie value

)

Authorization strategies

Strategies are provided by plugins, and can be plugged in routes:

server.auth.strategy('simple','basic',

{ validateFunc: validate });

Auth validation function

function (request, username, password, callback) {

var isValid = username === user.name && password === user.password;

return callback(null, isValid, { name: user.name });

};

Time to code!

sudo npm install -g makemehapi

npm install hapi

Why debugging is important

● Javascript is an extremely flexibile and expressive language

● Maybe a little bit too much

● We need strong tooling support

The console object

● This is our first line of defense

● Was born inside browser developer tools

● Node provides us with a preconfigured console, exposed as global

stdin stderr

Other console methods

● dir: explore objects● trace: print current stack trace● time/timend: basic performance profiling

Static analysis

● A lot of errors can be discovered just by analyzed the source code text.

● The most two common analysis are typechecking and linting

Linting

● suspiciouse code is flagged by the linter to be reviewed by the programmer

● you can configure what should be suspicious

● can also be used to enforce coding styles (useful in teams)

ESLint

● Actually parses your code, instead of just analyzing text

● Pluggable

● Integrated with most code editors

ESLint example config

{

"env" : {

"node" : true //this determines which globals to expect

},

"rules" : {

"quotes": [2, "double"],

"no-use-before-define": 2

}

}

Logging

● An emitted stream of distinct messages from a running application

● Ordered in time

● Many applications, from debugging to performance analysis to security

Structured logs

07/Mar/2004:16:06:51 -"GET request from /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1" 200 4523

vs

{“date”: “2004-04-04”, “method”: “GET”, “uri”:”/twiki/bin/rdiff/TWiki/NewUserTemplate”: header: {

/../

}}

Structured loggers

Structured loggers log objects, not just strings.

We don’t have to restructure strings while analyzing data

Much easier for parsing

Stack traces

It’s the record of the function calls until a certain point (usually an error).

console.log(e.stack);

Error: Something unexpected has occurred.

at main (c:\Users\Me\Documents\MyApp\app.js:9:15)

at Object. (c:\Users\Me\Documents\MyApp\app.js:17:1)

at Module._compile (module.js:460:26)

at Object.Module._extensions..js (module.js:478:10)

at Module.load (module.js:355:32)

at Function.Module._load (module.js:310:12)

at Function.Module.runMain (module.js:501:10)

at startup (node.js:129:16)

at node.js:814:3

The problem with async stacktraces

setTimeout(function first() {

setTimeout(function second() {

setTimeout(function third() {

a.b.c;

}, 1);

}, 1)

}, 1)

long stack traces

ReferenceError: a is not definedat third [as _onTimeout] (/home/zio/stuff/nodeschool/bugclinic/provastack.

js:4:13)at Timer.listOnTimeout (timers.js:92:15)

This is because we can see only the stack of the message currently processedby the event loop

With advanced black magic, we are eventually able to get the full stack

Automated Testing

● Automated testing is incredibly cost effective

○ You get a much better understing of your code○ You get much safer code○ You are protected from regression○ Tests are the most formal kind of documentation○ You save of a LOT of time by letting the machine create

the test environment and executing the test for you○ .. this could go on for hours

Runners and assertions

A test environment is usually composed by:- a test runner- an assertion library

A test is usually composed by 2+1 phase- setup- assertion- teardown (optional)

Tape

We are going to use Tape- assertion and running in the same library- both very simple- outputs TAP strings

Node core debugging

If for some reason you want to get node’s internal debug message it’s easy, just prepend an env variable.

#prepare to get a lot of output

NODE_DEBUG=cluster,net,http,fs,tls,module,timers nod node myapp

The debug module

Yo u can use a similar approach in your code with the debug module:

var debug = require('debug')('http'), http=require('http'), name='My App';

debug('booting %s', name);

http.createServer(function(req, res){

debug(req.method + ' ' + req.url);

res.end('hello\n');

}).listen(3000, function(){

debug('listening');

});

Running in debug mode

Tracing vs debugging

Tracing is very specialized kind of debugging:

● It is never a functional requirement● It is meant for the developer , not the system administrator● It is commonly used for detecting bottlenecks, memory

leaks and other problem arising during continuous operations.

Interactive debugging: repl

The mighty Read-Evaluate-Print-Loop

Useful to test small bits of code or playing around with libraries.

Too long to reproduce actual application in it.. wouldn’t be cool if you could expose a repl in certain part of your app?

Good news, you totally can!

Replify

var replify = require('replify')

, app = require('http').createServer()

replify('realtime-101', app);

The node debugger

You can start an app with the “debug” flag:node debug myapp.js

Then the app will be in debug mode and you can:- set breakpoints in code (debugger;)- navigate- execute REPLs at leisure

NPM

Npm is not only a package / dependancy manager.

● It is a HUGE repository● It’s generic automation tool● A place to offer / find a job● and also a company

The site

On the site you can browse packages, search them by keyword and get a good lot of metrics:● number of downloads● github statistics● even try them out! well played NPMthis is extremely important due to impressive amount of javascript packages out therehttps://www.npmjs.com/package/lodash

Publishing on npm

● npm adduser● npm publish to publish the package.● Go to http://npmjs.com/package/<package>.

You can also create private modules.

Versioning

The whole npm ecosystem uses semantic versioning (semver for friends). A semantic version number is composed by three integers:

MAJOR.MINOR.PATCH

Semantic versioning

MAJOR is incremented when incompatible API changes are made

MINOR is incremented when backwards compatibile feature are added

PATCH is incremented when a bug is fixed

Semver operators

● >, <, >=, <= ● || - OR● * - gets the most recent - dangerous● ~ update only patch - ~0.5.0 >=0.5.0 < 0.6.0● ^ upgrade only minor - ^ 0.5.0 >= 0.5.0 < 1.0.0

package.json

It can be seen as the “manifest” of an applicationContains many metadata used for:● uniquely identify a package● establish authorship● lifecycle management

○ pre - post install scripts○ main file ○ ...

Dissecting a package.json file

http://browsenpm.org/package.json

Time to code

git clone -b exercise https://github.com/lambrojos/nodeintro.git

or

https://github.com/lambrojos/nodeintro/archive/exercise.zip

npm install

Nodemon is a friend

(sudo) npm install -g nodemon

nodemon server.js (you will get errors)

solutions are in the master branch

Troubleshooting

If you get compile errors try adding the --no-optional flag to npm install