Learning Node.js
For projects and for some experiments I want to get a better way to do server push. With the new HTML 5 becoming the standard and therefore WebSockets becoming available in all mayor browsers, it seems that WebSockets is the way forward. A very nice implementation of WebSockets that is backwards compatible with older browsers is available through Socket.io. This is a Node.js library. Node.js brings JavaScript to the server. If you are as sceptic about this as I was, stay with me, you will like it after trying.
Within this blog post I want to share some of the things I learned while creating a sample application. The sample shows twitter integration for authentication, server push in a chat application that also shows the other persons that are online. The most important node.js libraries I use in the sample are: expressjs, socket.io, node-oauth, connect, jade and sass
In my previous post on gridshore I wrote about Node.js. This blog post is a continuation of the blog post First steps with Node.js on gridshore. Some of the things I write are extension to what I have written in that post. So if things go to fast, check that blog as well.
The sample
The sample I am creating is just to show of some features of Node.js. It is not really a real life application. Still it is fun to play around with. The sample runs on my very small server at home. So please be gentile with it.
The sample mainly focusses on a chat application. You can login using your twitter account. If you do not want to use you twitter account, you can also set your name manually. New chat messages are send to all online people. When logging in you also receive a number of past messages. You also see a list of online persons. The sample works best using a browser that really supports WebSockets (Chrome or Safari).
Introducing Node.js
Node.js is all about efficiently handling request. The memory consumption is very low. Node.js uses JavaScript on the server. Events are the core of the system together with callbacks. Every call is a-synchronous and therefore non-blocking.
Node.js can be used with a package manager called npm. This makes it a lot easier to provide modules to your program. You can do it without npm as well. More on npm in a later section.
Building or installing
Installation of node is not hard of you do not need the latest and greatest and you are not developing using windows. A lot fo OS package managers provide packages for node. On Ubuntu you have apt-get and on the mac you have macports that both provide standard packages. If their is not standard package you are not out of luck as well. Building from source is not very hard. I wrote a blog post about installing Node.js on a synology NAS system. Building Node.js on the Mac is easy as well. Check the installation page as provided by Node.js to see how it works. In short you have to perform the following steps:
> git clone https://github.com/joyent/node.git > cd node > ./configure > sudo make > sudo make install
After that you need to install npm
> sudo curl http://npmjs.org/install.sh | sh
Now you can check your installation by doing node –version and npm list. Did you see all the modules that are already available though npm?
Setting up the project
Let us move on to the code for the project. I have created a tag to facilitate this blog. That way future changes will not interfere this blog.
https://github.com/jettro/nodejstryout/tree/vBlog-Learning-nodejs
The sample is structured in a way that we can easily change implementations. At the moment there is a start for two implementations. One using the standard Socket.io and one with the Nowjs library. This way socket.io specific configuration is handled in the app-socketio.js module. The following code block shows how the generic app modules is included.
var Socketio = require('./socketio');
var socketio = new Socketio();
var App = require('./app');
var app = new App();
app.start(socketio);
Line one shows how my own socketio module is included. Beware, this is not the actual Socket.io module, but just a module that contains all socket.io code. As you can see we load the modules using require(‘./<the name>’). If we want to load modules installed with npm we would omit the path.
In the next section we will have a look at using modules to group functionality.
Introducing modules
The following shows the basics for creating a module. We define the function that initializes the object. Than we expose or export the object and we define public methods using the Socketio.prototype.<name_of_method>.
function Socketio() {
}
Socketio.prototype.index = function(req, res) {
}
Socketio.prototype.init = function(app) {
}
module.exports = Socketio;
Time to talk server side request handling. In the following section I’ll introduce the express framework.
Loading a configuration file
Every application needs configuration. Allthough the deployment model using git is very easy, we still need to provide some environment specific configuration. This is done using code taken from this library. The following code block shows loading the config file that is written in a json format:
var fs = require('fs');
var properties_file = process.cwd() + '/config/config.json';
var properties = {};
try {
properties = JSON.parse(fs.readFileSync(properties_file, 'utf8'));
} catch (err){
console.log(err);
}
for (attrname in properties) {
properties[attrname] = properties[attrname];
}
module.exports = properties;
Request handling with express
I am coming form spring mvc land. Therefore the mvc model is very intuitive to me. Express allows me to use a similar model to work with. We start with configuring express, than we configure the request handling. The first block shows the overall configuration of express.
var pub = __dirname + '/public';
var properties = require('./properties');
var Site = require('./site');
var express = require('express')
, app = express.createServer()
, blog = require('./blog');
var site = new Site(properties);
app.configure(function() {
app.set('view engine', 'jade');
app.set('views', __dirname + '/views');
app.set('view options', { layout: 'layout' });
app.use(express.compiler({ src: pub, enable: ['sass'] }));
app.use(express.methodOverride());
app.use(express.static(pub));
app.use(express.logger());
app.use(express.bodyParser());
app.use(express.cookieParser());
app.use(express.session({ secret: "keyboard cat" }));
});
We tell express where to find the views, the view engine to use and the default layout. The public folder for static files. Configure a logger, body parser, cookies and the session. More on the session will follow in a later section. Another part to notice is the compiler for sass. More on this will also follow.
The next block shows the request mapping, we can map get and post requests to a handler. Handlers are implemented in the modules as introduced before.
app.get('/', site.index);
app.get('/authenticate', site.authenticate);
app.get('/authenticated', site.authenticated);
app.get('/blog', blog.index);
app.get('/blog/new', blog.blogShowForm);
app.post('/blog/new', blog.blogPostForm);
app.get('/chat', chat.index);
I want to give one example by providing the actual implementation of site.index. You saw in the previous code block how the index object was created. The next code block shows the implementation of one method.
Site.prototype.index = function(req, res) {
var loginName = '';
if (req.session.oauth && req.session.user) {
loginName = req.session.user.name;
}
res.render('index', {locals: {loginName:loginName}});
};
The last line is the most important to understand what happens in the view. We ask the index file and provide the parameter loginName to it. The next section shows how this is used with jade.
Building the user interface
Making a user interface is a lot of work. Any help in making it easier to create web pages is welcome. For this sample I used jade and sass. Jade made it easier to use templates and less verbose html. With sass you can do the same thing for stylesheets.
Templates using jade
For the views we use jade. You can use a template for all you pages and override it for specific case. You can also omit using template. Jade comes with some very basic structures like if-then-else, for-each and it can use provided parameters. I do not want to spend to much time on jade. The code speaks for itself. I created a separate layout for the chat pages. One thing I do want to mention is the importance of indents. They are used by the framework, so be careful with the code formatters.
A few hints for jade:
- Use a – when doing code like it-then-else, but place it in the indent like you would place the normal text. (see socketio/layout.jade)
- Use #{…} for writing out parameters (see socketio/layout.jade)
- USer form(action=’…’) to submit a form (see socketio/index.jade)
?
Styling using sass
Sass is a condensed way of writing stylesheets. You use a different language that is compiled into css. I like the idea but I am not sure if this is the right approach. Using a tool like intellij looses some of its power with code completion. Other than that, it is very intuitive. Check some of the code below. Read the online documentation to learn more about sass.
#information
:position absolute
:left 10
:top 180px
:width 300px
input.text
:width 180px
textarea
:width 180px
#message
:width 250px
:float right
div
:border 1px solid black
:margin 2px
#chat
:position absolute
:top 180px
:left 310px
div
:border-bottom 1px solid black
Doing server push
The initial goal for me to start playing with Node.js was the Socket.io library. I was very curious how the WebSocket support was and how it worked. In this section I want to explore the options that socket.io has in combination with the libraries I have already mentioned in the blog.
Introduce socket.io
The cool part about socket.io is not the WebSocket support. THe coolest thing is that they are backwards compatible to older browsers. Of course this has implications for your infrastructure architecture. If you want to do massive server push, using the older mechanisms like long-polling are less optimal. Still they do work.
In my application you can find most of the socket.io code for the server in the socketio.js file. Of course we need some javascript on the client as well. This can be found in public > javascript > local-socket-io.js
First we initialize the connection. The next code block shows the most important part at the server.
var io = require('socket.io');
var buffer = [];
var clients = [];
Socketio.prototype.init = function(app) {
io = io.listen(app);
io.on('connection', function(client) {
client.send({ buffer: buffer });
}
}
In the first lines we load the socket.io module, than we create the init method that is called by the app.js file. Within this method we initialize the server side by registering the socket.io module as a listener with the application. Than we register an event listener that wants to be notified in case a client is connected. For now we only send the buffer with received messages to the client. The client than needs to provide the name of the person that connected. This can be obtained from twitter or manually entered in a form. The next code block shows the most important parts of the client for the connection.
var socket = new io.Socket(null, {rememberTransport: false});
socket.connect();
socket.on('connect', function() {
message({ message: ['System', 'Connected']});
});
socket.on('disconnect', function() {
message({ message: ['System', 'Disconnected']});
});
socket.on('message', function(obj) {
message(obj);
});
Within the code we use the function message a lot. We will focus on that function later on. The important thing to notice is that we register event handlers for connect, disconnect and message events. Handling the messages in permorfed in the message function. So let us move on to that function.
function message(obj) {
if ('message' in obj) {
$("#message").prepend("<div>" + obj.message[0] + " : " + obj.message[1] + "</div>");
} else if ('chat' in obj) {
$("#chat").prepend("<div>" + obj.chat[0] + " : " + obj.chat[1] + "</div>");
} else if ('users' in obj) {
$(".user").remove();
for (var i = 0; i < obj.users.length; i++) {
$("#users").prepend("<div class='user'>" + obj.users[i] + "</div>")
}
} else if ('buffer' in obj) {
for (var j in obj.buffer) message(obj.buffer[j]);
} else if ('announcement' in obj) {
$("#message").prepend("<div>" + obj.announcement + "</div>");
} else {
$("#message").prepend("<div>Unknown message type: " + obj + "</div>");
}
}
The received message is a json object. We check the object for the presence of a property. Based on that property we determine the where the message is printed. We have a list of users, a list of messages and a list of chat messages. The actual printing to the lists is handled using some jQuery magic.
Before we can send chat messages, we want the persons to enter their name. This is what the next section is about.
Maintaining a list of connected persons
The next code block shows the sending of the name from the client perspective. First we tell the client that we have changed the name. Then we send the new name to the server using a JSON format. Notice that we use the first element as newName. After that we do some jQuery magic again to disable the input field and the button.
function setName() {
message({message: ['You','Name is set to: ' + $("#visitorName").val()]});
socket.send({newName: $("#visitorName").val()});
$("#visitorName").attr('readonly', true);
$("#setNameButton").hide();
}
How does this work on the server. Not that different from the client actually. We register an event listener for the message event and check what is in the message. Based on the contents we decide what to do. The next code block shows the server side of sending and receiving messages.
client.on('message', function(message) {
if ('newName' in message) {
console.log("Received a new name: " + message.newName);
clients[client.sessionId] = message.newName;
client.broadcast({ announcement: clients[client.sessionId] + ' connected' });
sendClients(client);
return;
}
var msg = { chat: [clients[client.sessionId], message.message] };
buffer.push(msg);
if (buffer.length > 15) buffer.shift();
client.broadcast(msg);
});
At the moment we expect two messages to be received on the server. We can get newName messages or chat messages. We store received names in an associative array with the clientId of the session as a key. Than we broadcast to all available clients that we have a new person that provided his name. We also send them a list with all available persons using the sendClients method. This method obtains all items from the array and sends them using the broadcast functionality. Within the next section we discuss the sending of chat messages.
Sending chat messages
Sending the message is of course handled by the client. Receiving the message can be seen in the previous code block. You can see we do not need the client to provide the name of the sender. We store that on the server using the clientId obtained from the session. The message itself is added to the buffer, only the last 15 messages are kept. Than the messages is broadcasted to all other connected clients. The next code block shows the client code for sending the message. Notice that we write the message to be send to the messages at the client first. Than we send the new chat message to the server.
function sendMessage() {
message({chat: ['You',$("#newMessage").val()]});
socket.send({message: $("#newMessage").val()});
$("#newMessage").val("");
}
Doing twitter authentication
Using the form you can enter the name to be used in the chat manually. There is another way, you can use twitter authentication. The top right link shows you name if you are logged in using twitter or a link to authenticate using twitter. Twitter makes use of oauth. Node.js has a nice oauth module that you can install using npm. If you want to learn more on oauth, check one of my other blogposts. has nothing to do with node.js, but the idea is exactly the same.
After clicking the link you get redirected to twitter, you enter you credentials and you will see the following screen. After approving access, you get redirected back to the website.
The next code block shows you the complete code that is required to do the authentication using twitter.
Site.prototype.authenticated = function(req, res, next) {
if (req.session.oauth) {
req.session.oauth.verifier = req.query.oauth_verifier;
var oauth = req.session.oauth;
oa.getOAuthAccessToken(oauth.token, oauth.token_secret, oauth.verifier,
function(error, oauth_access_token, oauth_access_token_secret, results) {
if (error) new Error(error);
req.session.user = {name:results.screen_name};
res.redirect("/");
}
);
} else {
next(new Error('No OAuth information stored in the session. How did you get here?'));
}
};
Site.prototype.authenticate = function(req, res) {
var sys = require('sys');
var OAuth = require('oauth').OAuth;
oa = new OAuth("https://api.twitter.com/oauth/request_token",
"https://api.twitter.com/oauth/access_token",
consumer_key, consumer_secret,
"1.0A", host + "/authenticated", "HMAC-SHA1");
req.session.oauth = oa;
oa.getOAuthRequestToken(function(error, oauth_token, oauth_token_secret, results) {
if (error) new Error(error.data);
else {
req.session.oauth.token = oauth_token;
req.session.oauth.token_secret = oauth_token_secret;
res.redirect('https://api.twitter.com/oauth/authenticate?oauth_token=' + oauth_token);
}
});
};
Authentication takes place in the authenticate method. That method prepares the request to the twitter api using the oauth library. The oauth object, the result of the authentication, is stored in the session. After successful authentication the redirect url is called. In our case that is the authenticated url. In the authenticated step some verification takes place and if everything goes well we redirect to the index of the site.
Additional stuff
That is it for the big topics. In this section I just want to mention a few things that I had a look at during the creation of the sample.
Easy reload of server
The run library scans your sources for changes. If something changes, it will restart the server. That way you do not have to manually stop and start the server to test your changes. Very useful when debugging some issues.
https://github.com/DTrejo/run.js
Running node as a service
When running node.js locally, just ctrl-c to stop it is enough. When running on a server you want more control. You want to run you app as a daemon. There is a node.js project available that runs you application as a daemon and restarts the server if it for some reason becomes unavailable. You can find more information about it here.
https://github.com/indexzero/forever
Sessions with connect
The express library uses the connect library. I already showed the configuration of express for the session. By default the in memory session store is used. It is also possible to use redis to store the session information.
var RedisStore = require('connect-redis');
connect.session({secret: 'keyboard cat', store: new RedisStore});
That is it, now session information is stored in redis. I do not want to go into redis at the moment, but it is not hard to learn. I liked the interactive tutorial to get an idea about what redis is.
The Future
Some improvements I want to make to the sample are:
- Use the Nowjs library to create the same sample. They make client/server interaction very easy
- Improve the way the loginName is added to every request. Express is based on layers. It should be easy to create a layer that adds the loginName to every response if the session is available. I want to do some research in that area.
?
Another thing I want to do in the nearby future is integrate Node.js with the Axon-Trader sample. Would be nice to use Node.js for the front end and integrate with the Axon framework using Rabbitmq and maybe the mongodb for the query database.
Concluding
I hope you liked this enormous blog post. Hope there was something for you to learn. If you have questions, feel free to ask.
