Wednesday, 10 October 2012

NodeJS - WebApp builder

Why build a WebApp? JavaScript and CSS do not need compilation!
Well, if you're using Less/Sass/Stylus to generate CSS and/or if your WebApp is bigger than just "a few" js files (e.g. 5 libraries, 10 jQuery plugins, 15 modules) - the time has come to get a build script!

In this tutorial I'd like to show you how to write and customize a simple build script that will combine and minify your JavaScript files and compile less files to css.

Step 0: What do I need?

Step 1: Set-Up the environment


Download here:
I will be working on Windows 7, but platform does not matter and you can apply this tutorial to any environment.
I will skip the installation process as it is very easy (just click "next" until you see "finish" :-)).
Now let's test it! Open command line/terminal/console/etc and type:

node -v

and hit Enter. This should show you your NodeJS version.

Windows tip: to quickly open Command Window, press: Win + R, type: cmd and hit Enter

Node Package Manager (NPM)

Once you install NodeJS - you also get NPM (Node Package Manager) - a command line tool for (obviously) managing node packages.
Let's test it then. In command line type:

npm -version

and hit Enter. This will show you your NPM version.

Node Modules

Modules are basically small, usually, one-purpose-tools written in JavaScript.
You have 2 options when installing modules: local and global (more:


npm install -g <module name> 

This command will install a module in global user folder. On Windows 7 it is: C:\Users\<user name>\AppData\Roaming\npm\node_modules.
This is good when you want to use modules as commands, e.g. instead:

node lessc style.less

you can write:

lessc style.less

But for your script you need to use local install, as global does not allow to use ("require") modules in scripts.
In other words - if you install modules as global, your script would not "see" them (you can have global modules and as many local as you wish. There are no conflicts between these two types).


npm install <module name> 

This command will install a module in the current folder (it will create "node_modules" folder and put modules there).

Installing Node Modules

First let's create a "Project" that will contain your project files (css, less, js, html, etc.). Here's the sample structure:

To keep things clean, we have another folder inside "Project", called "_builder" - this is where you will put your script and install node modules.
Now open command line/terminal and navigate to the "_builder" folder.

Windows tip: to quickly open Command Window in a particular folder: hold Shift and right-click the folder (in Explorer) and select "Open command window here" from the menu.


npm install less

This will install your first module - less compiler.
Inside your "_builder" you should now have "node_modules" folder containing less module.

Let's now install the rest:

npm install uglify-js

This is a JavaScript minifier.

npm install colors

This one will help you display messages to the console with some nice colors.

npm install commander

This will allow you to add parameters to your builder (e.g. node build.js -c -x --log).

That's it! You're ready to write a builder script!

Step 2: Script Config

In this step we're going to create a "config" file, which would be parsed by our script and used as a guide (which files go where and what to do with them).
This will be a simple JSON file, so it's human-readable and easy to parse in JavaScript.
Here it is:

    "js" : [
        { "input": [
         ],                                           "output": "../js/lib/widgets.min.js" },
        { "input": "../js/jquery-plugins-src",        "output": "../js/lib/jquery-plugins.min.js" },
        { "input": "../js/modules-src/module1.js",    "output": "../js/modules/module1.min.js" },
        { "input": "../js/modules-src/module2.js",    "output": "../js/modules/module2.min.js" },
        { "input": "../js/modules-src/module3.js",    "output": "../js/modules/module3.min.js" },
        { "input": "../js/app.js",                    "output": "../js/app.min.js" }

    "lessImportPaths" : [ "../less" ],
    "less" : [
        { "input": "../less/common.less",             "output": "../css/common.css" },
        { "input": "../less/widgets",                 "output": "../css/widgets.css" },
        { "input": "../less/modules/module1.less",    "output": "../css/module1.css" },
        { "input": "../less/modules/module2.less",    "output": "../css/module2.css" },
        { "input": "../less/modules/module3.less",    "output": "../css/module3.css" }
  • There are two main "nodes": js and less.
  • Every node is an array if "items".
  • Every item is a JavaScript object containing:
    • "input" - a relative (to the script) path and filename or folder name (a string or an array of strings/names)
    • "output" - a relative path and filename - the file that will be created from "input" files/folders
  • There's also "lessImportPaths" which is used by less parser - this is an array of folders (paths) where your "@imports" should be (keep reading).


  • Remember to wrap names in double-quotes, i.e. "less":[ ... ] and NOT: less:[ ... ]). It's still valid JS object without quotes, but not a valid JSON!
  • You can use "/" (slash) for paths (even on Windows) or "\" (backslash), but the latter must be escaped, i.e. "\\" instead of "\".
  • You should always make sure that your config is a valid JSON file (otherwise it will not parse correctly). There are plenty of online json validators (e.g.

Our script will be intelligent to understand whether "input" is a folder - it will then read all js/less files inside that folder.

Step 3: Build script

The idea is to create a two-fold functionality:

  • dev version - that will only merge files - without changing the content, to allow easier debugging.
  • production version - that will merge and minify/obfuscate files.

OK! Let's start with the easy bits.
NodeJS is a JavaScript engine, so let's write js:


Global Variables

First let's define some global variables:

var FS = require('fs'),                                                        // import FileSystem module
    COLORS = require('colors'),                                                // colors - to colorize command line output
    UGLIFYJS = require('uglify-js'),                                           // js minifier/obfuscator
    LESS = require('less'),                                                    // less compiler
    CLI = require('commander'),                                                // handles command line parameters (e.g. node builder.js -a -b -c)

    COMPRESS = false,                                                          // whether to compress files
    CONF = null;                                                               // parsed config object
  • Using "require" - we are importing node modules (the ones that we have just installed).

Error handling

In case we need to display some errors - a function that will receive an error object "e" and display more information about it (file, line, column, etc.):

var showError = function(e){
    console.error('\n' + COLORS.bold(;
    console.error('File: ' + e.filename);
    console.error('Line: ' + COLORS.bold(COLORS.white(e.line)) + 
                ', Column: ' + COLORS.bold(COLORS.white(e.column || e.col))
    //console.error(e.extract);                                                // this will display extract from the file with error
    return false;
  • console.error - similarly to firebug/chrome dev. tools/etc. - displays error text to the console. In this case - the console is the command line window.
  • COLORS module - allows us to change the color of the text.

File Extension

We are going to need a function that will return the extension from the file name. Simple as that:

var getExt = function (filename){ 
    var ext = (filename||'').split('.'); 
    return ext[ext.length-1]; 

Merge files

We need a function that will concatenate files. Here we are going to use the built-in FileSystem module (variable FS), imported earlier:

var concat = function(inp){
 var files ={ 
  return FS.readFileSync(path, 'utf-8'); 
 return files.length ? files.join('\n') : console.error('Files are empty!'.red);
  • inp parameter is an array of files (paths + filenames), e.g. inp = [ '../js/app.js', '../js/module1.js' ]
  • then we are looping through the inp array (using map function) and reading files' contents to another array - files.
  • FS.readFileSync - will read and return file contents.
  • concat function will merge all input files (with new line char - '\n') and return one big chunk of code (string).
  • .red - at the end of error message is just another way to use COLORS module.

Get files in folder

We also need a function that will return all files (file names) from a given folder:

var getFilesInFolder = function(folder, ext){
    var files = [], items = FS.readdirSync(folder), fiStat;{                                                  // loop through files
        if (getExt(item) !== ext) return;                                      // incorrect extension - move to next item
        item = folder + '/' + item;                                            // get real path to a file
        fiStat = FS.statSync(item);                                            // get file information
        if (!fiStat.isFile()) return;                                          // not a file - move to next item
        files.push(item);                                                      // push file to array
    return files;
  • folder parameter is a path to a folder.
  • ext parameter is an extension of files that we'll be looking for (in this case: "js", but we will use the same function for "less" files as well).
  • map will loop through the files, verifying that they are files and have the correct extension.
  • statSync function will return some file/folder information (like size, creation date, etc.).
  • isFile() will tell us whether an item is a file. If it's not - continue with the loop.

Get input files

This function will return an array of all file names from input (replacing folder names with names of all files inside them)

var getInputFiles = function(input, ext){
    var inputFiles = [];
    if (typeof input === 'string') input = [ input ];                          // if input is a string -> convert it to array{                                                   // loop through input files array
        var stat;
        try { stat = FS.statSync(inp); }                                       // get information about a single file or folder
        catch(e){return console.error('Resource not found: '+inp));}// show error if path is incorrect
        if (stat.isFile()) inputFiles.push(inp);                               // if it's a file - just add it to array
        else if (stat.isDirectory()){                                          // if it's a folder:
            var filesInFolder = getFilesInFolder(input, ext);                  // retrieve names of all files from this folder to array
            inputFiles = inputFiles.concat(filesInFolder);                     // merge (concat) this array with "inputFiles"
    return inputFiles;

Read Config

Last, but not least, we need a function to read our config file:

var readConfig = function(fname){
    fname = fname || 'build.json';                                             // default name of the config would be build.json
    if (!FS.existsSync(fname)) return false;                                   // config file not found
    return JSON.parse(FS.readFileSync(fname, 'utf-8'));                        // read file and parse as JS Object



First - a function to minify js code, using uglify-js module:

var uglify = function(files, output){
    var compiler = UGLIFYJS.uglify, outputContent;
    try { 
        outputContent = UGLIFYJS.parser.parse(files);                          // parse a js file
    catch(ex){ return showError(ex); }                                         // display parse error
    outputContent = compiler.ast_mangle(outputContent);                        // mangle names
    outputContent = compiler.ast_squeeze(outputContent);                       // compress & optimize code
    outputContent = compiler.gen_code(outputContent);                          // generate compressed code
    return outputContent;

Pretty straightforward, I hope. Comments should explain everything.

Build JS

And here's the main function that will handle your JS files:

var buildJS = function(){{                                                // loop through js items from the config file
        var inputFiles = getInputFiles(item.input, 'js');                      // retrieve input files (with js extension) for a single item
        if (inputFiles.length) inputFiles = concat(inputFiles);                // merge all input files (for 1 item)
        if (COMPRESS) inputFiles = uglify(inputFiles);                         // compress files if needed
        if (inputFiles) FS.writeFile(item.output, inputFiles, 'utf-8');        // save result to the "output" file

True power (and speed) of node lies in the asynchronicity


Compile LESS

We are not going to merge less files before compiling them, as this would make debugging more difficult (we wouldn't know in which file there is an error).

Also - less compiler has only the "async" version of the function which is better (faster) from the performance point of view, but makes it a bit harder to use on multiple items (you need to keep track when an item is converted).

var compileLess = function(files, output){
    var cssFiles = [], totalFiles = files.length;                              // totalFiles - number of files to process{
        var lessContent = FS.readFileSync(file, 'utf-8'),                      // read a single less file
            parser = new(LESS.Parser)({                                        // define less parser
                 paths: CONF.lessImportPaths.concat([]),                       // add paths for less "@import"
                 filename: file                                                // add current file name (if there is a parse error - you will know in which file)
        parser.parse(lessContent, function (e, tree){                          // PARSE!
            if (e) return showError(e);                                        // if parse error - display it and exit this function
            var cssContent ='';
            try {
                if (COMPRESS === true){ 
                    //cssContent = tree.toCSS({ yuicompress: true });          // yui comression (single line per file)
                    cssContent = tree.toCSS({ compress: true });               // normal compression (single line per class)
                else {
                    cssContent = tree.toCSS();                                 // don't use compression
            catch(ex){ return showError(ex); }                                 // display error
            cssFiles.push(cssContent);                                         // push file to cssFiles (output array)
            if (cssFiles.length == totalFiles) {                               // if all files processed
                FS.writeFile(output, cssFiles.join('\n'), 'utf-8');            // merge css content and save it as "output" file
        }); // parser end        
    }); // map end

Build LESS

And now - the main less function:

var buildLess = function() {{                                              // loop through config's "less" items
        var files = getInputFiles(item.input, 'less');                         // retrieve input files for item
        compileLess(files, item.output);                                       // compile less

Main Program

Finally the main part of the script that will "do everything"!

CLI                                                                            // Handle command line parameters and arguments
    .version('Application Builder 1.0')                                        // this will display application name and version (when you run script with "-h" parameter)
    .usage('[options] <config file>')                                          // display "usage" line (when you run script with "-h" parameter)
 // script parameters
    .option('-c, --compress', 'compress files')
CLI.on('--help', function(){
    console.log('  <config file>            '+                                 // display additional line when script was run with "-h" parameter
        'path and name of the build configuration file '+
        '(default to: build.json) \n'); 

CLI.parse(process.argv);                                                       // parse commands

if (CLI.compress) COMPRESS = true;                                             // "-c" parameter is present - also compress files

CONF = readConfig(CLI.args[0]);                                                // read config file (if not given - default name will be used)
if (!CONF) return console.error('Config file not found'.red);                  // config not found - display error and exit script

buildJS();                                                                     // BUILD JS
buildLess();                                                                   // BUILD LESS

Step 4: Testing

Now all you need to do is put all this code into one file. Call it builder.js and place it in the _builder folder, just next to config.json.

You are now ready to test your script! Open command line window/terminal, navigate to the _builder folder and type:

node builder.js -h

This should display the help text for your builder: version, option parameters, etc.
If you don't see it - make sure your code and folder structure matches (or download source files for this tutorial).

Finally - to build our WebApp type:

node builder.js

or with "-c" parameter, to also compress files:

node builder.js -c

Step 5: Bonus

Building WebApp in node.js is fast. Building WebApp in node.js is fast. The true power (and speed) of node.js, however, lies in its asynchronicity.
Async functions require a bit different approach to programming - similarly to AJAX requests - they do not return values they can only execute another function (a callback) when they finish.

In the demo archive you will also find an enhanced version of the builder, which makes full use of the async functions and also adds some more colors to the console, e.g.:



I have tried several different methods for building my scripts (e.g. wro4j Runner, less.js for windows and a couple of GUI-based solutions) but none of these can compare to node.js if we're talking about speed and flexibility. Sure, this might not be the easiest way, and it might require a bit more time, but believe me, it's worth it! This is only a tiny bit of what node.js can do, but I hope it might be a good starting point, as it was for me.

Wednesday, 6 July 2011

Get me a week

Here's another snippet that I've recently added to my js toolbox.

For one of my projects I needed to do some date calculations in js, namely - get the week number from a given date or get the number of weeks in a given year.

As it transpired - it's not that simple as you might thing:
- there can be 52 or 53 weeks in a year - which is completely not related to leap years
- January 1st is not always in the first week of the year
- December 31st is not always in the last week of the year
and there are many more of those assumptions/definitions that you'd never think of (see here).

Sounds pretty complex, doesn't it? Let's start than!

Probably the most important rule for us is:
- the first week of the year is the week with the year's first Thursday in it!

So the first thing we need to do is to get the first day of the year (let's take the current year - 2011):

var d = new Date(); // this is our test date - today
var jan1 = new Date(2011, 0, 1); // first day of the year

Than - get the weekday number from the test date (make Sunday number 7):
var dWeekday = d.getDay() || 7;

Get the nearest Thursday: current date + 4 - current day number
var nearestThu = d.getDate() + 4 - dWeekday;

Set the test date to the nearest Thursday:

Get the difference in days between the test date and Jan 1st:
var differenceInDays = ((d-jan1)/86400000)+1;

Calculate how many weeks are from Jan 1st to the nearest Thursday:
var weeksFromJan1 = Math.ceil(differenceInDays/7);

And that's it!
Let's wrap it with the function, or even better - as a Date object extension:

Date.prototype.getFullWeek = function(){
var jan1, w, d = new Date(this);
jan1 = new Date(d.getFullYear(),0,1);
w = Math.ceil((((d-jan1)/86400000)+1)/7);
return {y: d.getFullYear(), w: w };

This will return the week number and also the year, as it might not be the same as the year of the test date, e.g.: 2010-01-03 is week 53 of 2009.

To make it complete, we may add another function to just return the week number:

//Returns ISO 8601 week number
Date.prototype.getWeek = function(){ return this.getFullWeek().w; };

And - reusing what we already have - it's pretty easy to get the number of weeks in a year:

function getWeeksFromYear(y){ return new Date(y,11,28).getFullWeek().w; };

We assume here, that the December 28th is always in the last week of the year.

That's it.
You can use it like this:

var testDate = new Date(2010, 0, 1);
var weekNumber = testDate.getFullWeek();
// return: { y: 2009, w: 53 }

I hope this will help someone :-)

If you have a better idea how to calculate this or you use a different approach, please let me know in comments.

references: wikipedia, w3schools

Saturday, 4 June 2011

Split array to variables in JavaScript

PHP developers have their explode function:

   $str = '1,2,3';
   list($a, $b, $c) = explode(',', $str);
   echo $a.', '.$b.', '.$c;

This can assign extracted chunks of the string directly to variables.
JavaScript approach everybody is using was:

   var str = '1,2,3',

       arr = str.split(',');
   console.log(arr[0] + ', ' + arr[1] + ', ' + arr[2]);

Sure, you can assign a splitted array elements to variables:

   var arr = '1,2,3'.split(','),
       a = arr[0],
       b = arr[1],
       c = arr[2];
   console.log(a + ', ' + b + ', ' + c);

But is there a better, clearer way? Yes!

var a, b, c,   
                     // first we need to declare the variables
   [a, b, c] = '1,2,3'.split(',');  // than split array directly to them
console.log(a + ', ' + b + ', ' + c);

That's it! Easy and clean.
Let us know in your comments what do you think!
Or maybe you have a better/different solution or idea?

Saturday, 9 April 2011

JavaScript async function queuing

The problem

I've been working on a big webapp recently. One thing that I needed is to perform several ajax requests. The problem was that those requests had to be "queued", that is the second one should start after the first one was finished.
Why not use synchronious ajax then? A few reasons:
  • application cannot be blocked waiting for responses
  • there were a few other requests in the same time, not dependant on the queued ones
  • synchronious ajax is bad, because it is bad.

I've done some digging, but nothing I found was satisfactory: short, simple and easy to apply.

So I've created something on my own!

The solution

So what's the solution? It's simple: just nest second request inside first request's callback, than third request inside second one's callback, etc.
Pretty messy isn't it. But I've done exactly this, in a bit cleaner manner
  • First - we need to create a queue of functions
  • Second - a function that will run one queue element at a time
  • Third - put everything into a nice container :-)
1. Starting from the end - here's the container declaration.
It's just a simple js object - our scope:
var Loader = { queue: [] };
2. Next - a function to add functions to the queue.
This function takes 3 arguments:
a. fn - function that will be added to the queue
b. scope - a scope for this function (if not given - defaults to "window")
c. params - arguments for the function in the queue (if not given - defaults to an empty array)
This function simply adds an element (which also is a function) to the queue.
Each element is a function call with our scope and arguments wrapped with another function for convenience.
Loader.push = function(fn, scope, params){
 scope = scope || window;
 params = params || [];
 this.queue.push(function(){ fn.apply(scope, params); });
3. A function which will run next element in the queue.
This function checks if there are elements in the queue, removes the first one and runs it. = function(){
 if (!this.queue.length) return;
 var fn = this.queue.shift();;


Ok. We've got the solution, let's test!
For that I'm gonna use jQuery, to save myself some manual XHR coding :-)
Here are 3 functions with some ajax. The only thing we need here is to add at the end, i.e. after the request is completed:

var fun1 = function(){
 $.get('ajax/test.html', function(data){; });

var fun2 = function(){
 $.get('ajax/test.html', function(data){; });

var fun3 = function(){
 $.get('ajax/test.html', function(data){; });

And now: let's add them to the queue and run it:


Wrap Up

Just to wrap it up, the full code for the Loader in 1 simple object, with some short-cuts:

var Loader = { queue: []
 ,push: function(fn, scope, params){ 
  this.queue.push(function(){ fn.apply(scope||window, params||[]); }); 
 ,run: function(){ 
  if (this.queue.length) this.queue.shift().call(); 

Further reading

Sunday, 10 October 2010

GWatchman Updated!

I've just released a new version of my GWatchman.
With version there are lots of new features and improvements!
Most important is localization support based on simple xml files.
Now you can translate the application to your mother tongue yourself!
Another cool thing is checking for updates. You don't have to re-visit the website again to check for newer version. Just use the tray menu or check Check for updates on start in Settings ("General" tab).

There is also a small thing but I think it's very handy: when there are unread items in your mail/reader app icon in systray will get a little star over it to mark that. So now you don't have to click the icon to check for news every time. Look at the icon and if it has a star on it just open the browser :-)
Of course if you set the auto-checking to 3 hours you might still see the star icon even if you've just cleared your inbox. In that case just click the icon to re-check and it will update its status.

Icon will also animate when checking for new items and display little red exclamation mark when there's something wrong (server not responding or your login/password is incorrect).

Also many minor bugs were fixed and general stability of the application should improve.

Check it out here: GWatchman homepage
Or download here: GWatchman