Working with async module in Node.js - Part 1 (async.waterfall)

Background

Javascript, as we know, runs on a single thread and to prevent blocking operations like a network call or a disk I/O we use asynchronous callbacks. This essentially means tasks run in the background and we get a callback when the operation is done. If you wish to understand more of how Javascript works please watch below video -

So as you must have known by now there is a lot of asynchronous stuff that happens. Sometimes we need to order these to suit our business logic. Consider a simple example -

1. Download an mp4 file from the server
2. Convert it into a gif locally
3. Upload the gif back to the server
4. Delete the mp4 from the server

Now in this example, all 4 steps are an asynchronous operation. Also, we cannot move to the next step until the previous step is finished. 

The Callback way

We can use callbacks for this. Something like below -

/**
 * Program to demonstrate asyn nodejs module
 * @author : athalur
 */

var startDemo = function () {
    console.log("Starting Demo");
    download(function () {
        process(function () {
            upload(function () {
                del(function () {
                    console.log("Ending Demo");
                })
            })
        })
    });
}

var download = function (callback) {
    console.log("Starting download");
    delay();
    console.log("Finishing download");
    callback();
}

var process = function (callback) {
    console.log("Starting process");
    delay();
    console.log("Finishing process");
    callback();
}

var upload = function (callback) {
    console.log("Starting upload");
    delay();
    console.log("Finishing upload");
    callback();
}

var del = function (callback) {
    console.log("Starting del");
    delay();
    console.log("Finishing del");
    callback();
}

var delay = function () {
    var i, j;
    for (i = 0; i < 100000; i++) {
        for (j = 0; j < 10000; j++) {
            //do nothing
        }
    }
}

startDemo();

This print below output -

As you can see this is a mess that is created by cascading callbacks. Also if at any point there is an error then we need to send it back to the callback as well and each step would have an if-else check to handle it. Let us see how easy it is with async module.

The async way

First, you need to install async nodejs module -

• https://www.npmjs.com/package/async

To do so run following command -

• npm install async

Now using async our program becomes -

/**
 * Program to demonstrate asyn nodejs module
 * @author : athakur
 */

const async = require("async");

var startDemo = function () {
    console.log("Starting Demo");
    async.waterfall([download,
        process,
        upload,
        del],
        function (err, data) {
            if(err) {
                console.log("There was an error in the demo : " + err);
            }else {
                console.log("Demo complete successfully");
            }
        });
}

NOTE:  I have not included the actual methods again to avoid repetition.

And the output is -

Notice how cleaner our code has become. Async takes care of all the callbacks. It also provides a mechanism to send data from one step to another. If you call the callback with data it will be available in next step.

If you change our download and process method slightly like below -

var download = function (callback) {
    console.log("Starting download");
    delay();
    console.log("Finishing download");
    callback(null, "Downloaded file URL");
}

var process = function (data, callback) {
    console.log("Starting process");
    console.log("In process method. Data from download: " + data);
    delay();
    console.log("Finishing process");
    callback();
}

and re-execute we will get

Also, it provides a cleaner way of error handling. Let's say our download fails. Chain download method as below -

var download = function (callback) {
    console.log("Starting download");
    delay();
    console.log("Finishing download");
    callback("Error in download", "Downloaded file URL");
}

This essentially means our download failed - 1st argument in the callback. In this scenario, the next steps in the waterfall model will not be executed and you would get a callback method that you have provided at the end of the waterfall array. On executing above you will get -

That's all for async module - waterfall. In next post, I will show you how we can use an async module for lopping over an array of data -

• Working with asyn module in Node.js - Part 2 (async.eachSeries)

Related Links

• Philip Roberts: What the heck is the event loop anyway? | JSConf EU
• https://www.npmjs.com/package/async
• Working with asyn module in Node.js - Part 2 (async.eachSeries)(OSFG)