How to delete private EC2 AMI from AWS

Background

Most of the times you would want to customize the EC2 image that you want to run. It would help you quickly spin up a new instance or put it behind an auto scaling group. To do this you can create an image out of your current running EC2 instance. But this is an incremental process. So you may want to create a new image after making some changes to EC2 instance spun up from the current image. Once the new image is created you can delete the previous AMI/image. In this post, I will tell you what are the steps to do so.

How to delete private EC2 AMI from AWS

To delete a private AMI follow the steps below -

1. Open the Amazon EC2 console  (https://console.aws.amazon.com/ec2/).
2. Select the region from the drop-down at the top based on which region your Alocatedocaled.
3. In the navigation pane, click AMIs.
4. Select the AMI, click Actions, and then click Deregister. When prompted for confirmation, click Continue.
• NOTE: It may take a few minutes before the console removes the AMI from the list. Choose Refresh to refresh the status.
5. In the navigation pane, click Snapshots.
6. Select the snapshot, click Actions, and then click Delete. When prompted for confirmation, click Yes, Delete.
7. Terminate any EC2 instances that might be running with old AMI.
8. Delete any EBS volumes for those EC2 instances if "delete on termination" is not set.

When you create an AMI it creates a snapshot for each volume associated with that image. So let's say your EC2 instance has 2 EBS volumes C drive and D drive of 30 GB and 50GB respectively in size then you would see 2 snapshots for them under snapshots section. Your AMI size will be the sum of individual snapshot size (i.e 30GB + 50GB = 80GB). To successfully delete the AMI you need to deregister the AMI and then delete the snapshots manually. And finally, when both are done terminate the EC2 instances that you might have running with old AMI.

Source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/deregister-ami.html

NOTES:

1. When you deregister an Amazon EBS-backed AMI, it doesn't affect the snapshot that was created for the root volume of the instance during the AMI creation process. You'll continue to incur storage costs for this snapshot. Therefore, if you are finished with the snapshot, you should delete it.
2. You can deregister an AMI when you have finished using it. After you deregister an AMI, you can't use it to launch new instances.
3. When you deregister an AMI, it doesn't affect any instances that you've already launched from the AMI. You'll continue to incur usage costs for these instances. Therefore, if you are finished with these instances, you should terminate them.
4. AWS will not let you delete a snapshot associated with an AMI before you deregister the AMI.
5. Delete EBS volumes (unless they are set to delete on termination, in which case, they would be removed on terminating EC2 instance). This isn't necessary for S3 backed instances

Related Links

• Deregistering Your Linux AMI
• How to create an AMI in AWS EC2? (Youtube)

How to send an email using the AWS SES service?

Background

In many business use cases, you need to notify your users over an email. AWS provides its own service to do so. You can use AWS SES (Simple email service) to send out emails to your users. In this post, we will see how we can configure AWS SES service and send out emails.

If you want to skip all the text post below you can refer to the video I have created to cover the same thing -

Setting up configurations in AWS SES service

First and foremost go to SES service in your AWS console. There are limited regions were AWS SES service is available. Please choose one of those regions -

Next thing that you need to do is verify your email address - the address from which you want to send out emails. For this go to "Email addresses" section in the left panel. Now click on "Verify a New Email Address" button. Enter your email address and click "Verify this email address". Once done you should get an email to confirm and validate your email address. 

Click on the confirmation link. Once done you should see your email address as verified in SES "Email addresses" section.

Next thing that we would need to setup is email template. The email template is the content that you went send in your email. This would include your unique template name, subject, body etc. Also, this can be parameterized which means you can have placeholders in the template that can be replaced at the runtime. A sample template would be -

{
    "Template": {
        "TemplateName": "testtemplate",
        "SubjectPart": "Test email for username {{userName}}",
        "TextPart": "Test email body!",
        "HtmlPart": "<html>\r\n\r\n<head>\r\n    <title>Test Title<\/title>\r\n<\/head>\r\n\r\n<body>\r\n    <h1> Username : {{userName}}<\/h1>\r\n<\/body>\r\n\r\n<\/html>"


    }
}

HtmlPart is JSON escaped version of your actual HTML content. TemplateName is unique template name for your template that you would reference later from your code to send the mail. SubjectPart is the email subject. For more details refer -

• https://docs.aws.amazon.com/ses/latest/APIReference/API_Template.html

Notice how we have added {{userName}} in the subject and text. This is a placeholder that will be replaced at runtime. We will see how in some time. Once your template is set up it is time to add it to AWS SES. To do this you need to call following API -

• aws ses create-template --cli-input-json fileb://test_template.json --region us-east-1 --profile osfg

If you have already created the template and want to update it you can execute -

• aws ses update-template --cli-input-json fileb://test_template.json --region us-east-1 --profile osfg

NOTE: You need to have AWS CLI configured on your machine and setup with your AWS profile. I have multiple profiles on my machine which is why you see --profile argument. if you just have one then you do not have to mention it.

Once you have executed this API you can see your template under "Email Templates" section of SES service.

Now that you have your email verified and template setup you can send an email. Following is the node.js code to do it -

const aws = require('aws-sdk');
aws.config.loadFromPath('./config.json');
const ses = new aws.SES();

/**
 * author: athakur
 * aws ses create-template --cli-input-json fileb://test_template.json --region us-east-1 --profile osfg
 * aws ses update-template --cli-input-json fileb://test_template.json --region us-east-1 --profile osfg
 * @param {*} callback 
 */
var sendMail = function (callback) {
    var params = {};

    var destination = {
        "ToAddresses": ["opensourceforgeeks@gmail.com"]
    };
    var templateData = {};
    templateData.userName = "athakur";

    params.Source = "opensourceforgeeks@gmail.com";
    params.Destination = destination;
    params.Template = "testtemplate";
    params.TemplateData = JSON.stringify(templateData)



    ses.sendTemplatedEmail(params, function (email_err, email_data) {
        if (email_err) {
            console.error("Failed to send the email : " + email_err);
            callback(email_err, email_data)
        } else {
            console.info("Successfully sent the email : " + JSON.stringify(email_data));
            callback(null, email_data);
        }
    })
}


sendMail(function(err, data){
    if(err){
        console.log("send mail failed");
    }
    else {
        console.log("Send mail succedded");
    }
})

In the above code, you can see that we provide the ToAddresses which is an array field. So you can give multiple addresses here. I have used the same email address as the one we have used in the source (from email address - the one we whitelisted and verified in the first step of setup). Also, notice the template name that we have provided - it is the same name we had configures in the json template file. Finally, notice the placeholder value we have provided - for userName. This will be replaced by "athakur" in the template and the email will be sent.

You can find this code snippet in my GitHub gist-

• https://gist.github.com/aniket91/799e3c8a6132b79f6e35b0c8a99f7742

And you should get an email. In my case, it looks as follows -

NOTE: If you are not able to see the mail check your spam folder. Since it is sent by AWS SES service it might land up there. Mark it as non-spam and you should be good going forward.

Related Links

• Regions and Amazon SES
• https://gist.github.com/aniket91/799e3c8a6132b79f6e35b0c8a99f7742

Working with async module in Node.js - Part 2 (async.eachSeries)

Background

This is a continuation of my previous post on async module in Node.js -

• Working with async module in Node.js - Part 1 (async.waterfall)

In the last post, we saw how neatly we can write code using async.waterfall. In this post, I will show you a similar trick with async.eachSeries method. 

Without async

Let's consider the following scenario.

We get the type of events and we need to run them in a loop in order. Following is a sample code to do that -

/**
 * Program to demonstrate asyn nodejs module
 * @author : athalur
 */

const async = require("async");

var startDemo = function () {
    console.log("Starting Demo");
    var events = ["Download", "Process", "Upload", "Del"];
    events.forEach(event => {
        process(event, function () {
            console.log("Got callback for : " + event);
        });
    });
    console.log("Ending Demo");
}

var process = function (processType, callback) {
    var processTime = 0;
    switch (processType) {
        case "Download":
            processTime = 2000;
            break;
        case "Process":
            processTime = 1000;
            break;
        case "Upload":
            processTime = 4000;
            break;
        case "Del":
            processTime = 100;
            break;
    }
    setTimeout(function () {
        console.log("Finished : " + processType);
        callback();
    }, processTime);
}

startDemo();

And the output is -

Wait what happened here? We looped our events array in order -

1. Download
2. Process
3. Upload
4. Delete

But that did not clearly happen. Mostly because each process takes different time to finish. In a real-world scenario, it would mean it make API calls or disk IO, the time of which we cannot predict. Let's see how async comes to our rescue.

Change the main code as follows -

var startDemo = function () {
    console.log("Starting Demo");
    var events = ["Download", "Process", "Upload", "Del"];
    async.eachSeries(events, function (event, callback) {
        process(event, callback);
    }, function(err){
        if(!err){
            console.log("Ending Demo");
        }
    });
}

and rerun the code.

Now you can see all of them executed in a series.



NOTE1: async.forEach runs through the array in parallel, meaning it will run the function for each item in the array immediately, and then when all of them execute the callback (2nd argument), the callback function will be called (3rd argument).

NOTE2: async.eachSeries runs through the array in series, meaning it will run the function for each item in the array and wait for it to execute the callback (2nd argument) before going to next item and finally when all are done the callback function will be called (3rd argument).

Related Links

• Working with asyn module in Node.js - Part 1 (async.waterfall)(OSFG)

Working with async module in Node.js - Part 1 (async.waterfall)

Background

Javascript, as we know, runs on a single thread and to prevent blocking operations like a network call or a disk I/O we use asynchronous callbacks. This essentially means tasks run in the background and we get a callback when the operation is done. If you wish to understand more of how Javascript works please watch below video -

So as you must have known by now there is a lot of asynchronous stuff that happens. Sometimes we need to order these to suit our business logic. Consider a simple example -

1. Download an mp4 file from the server
2. Convert it into a gif locally
3. Upload the gif back to the server
4. Delete the mp4 from the server

Now in this example, all 4 steps are an asynchronous operation. Also, we cannot move to the next step until the previous step is finished. 

The Callback way

We can use callbacks for this. Something like below -

/**
 * Program to demonstrate asyn nodejs module
 * @author : athalur
 */

var startDemo = function () {
    console.log("Starting Demo");
    download(function () {
        process(function () {
            upload(function () {
                del(function () {
                    console.log("Ending Demo");
                })
            })
        })
    });
}

var download = function (callback) {
    console.log("Starting download");
    delay();
    console.log("Finishing download");
    callback();
}

var process = function (callback) {
    console.log("Starting process");
    delay();
    console.log("Finishing process");
    callback();
}

var upload = function (callback) {
    console.log("Starting upload");
    delay();
    console.log("Finishing upload");
    callback();
}

var del = function (callback) {
    console.log("Starting del");
    delay();
    console.log("Finishing del");
    callback();
}

var delay = function () {
    var i, j;
    for (i = 0; i < 100000; i++) {
        for (j = 0; j < 10000; j++) {
            //do nothing
        }
    }
}

startDemo();

This print below output -

As you can see this is a mess that is created by cascading callbacks. Also if at any point there is an error then we need to send it back to the callback as well and each step would have an if-else check to handle it. Let us see how easy it is with async module.

The async way

First, you need to install async nodejs module -

• https://www.npmjs.com/package/async

To do so run following command -

• npm install async

Now using async our program becomes -

/**
 * Program to demonstrate asyn nodejs module
 * @author : athakur
 */

const async = require("async");

var startDemo = function () {
    console.log("Starting Demo");
    async.waterfall([download,
        process,
        upload,
        del],
        function (err, data) {
            if(err) {
                console.log("There was an error in the demo : " + err);
            }else {
                console.log("Demo complete successfully");
            }
        });
}

NOTE:  I have not included the actual methods again to avoid repetition.

And the output is -

Notice how cleaner our code has become. Async takes care of all the callbacks. It also provides a mechanism to send data from one step to another. If you call the callback with data it will be available in next step.

If you change our download and process method slightly like below -

var download = function (callback) {
    console.log("Starting download");
    delay();
    console.log("Finishing download");
    callback(null, "Downloaded file URL");
}

var process = function (data, callback) {
    console.log("Starting process");
    console.log("In process method. Data from download: " + data);
    delay();
    console.log("Finishing process");
    callback();
}

and re-execute we will get

Also, it provides a cleaner way of error handling. Let's say our download fails. Chain download method as below -

var download = function (callback) {
    console.log("Starting download");
    delay();
    console.log("Finishing download");
    callback("Error in download", "Downloaded file URL");
}

This essentially means our download failed - 1st argument in the callback. In this scenario, the next steps in the waterfall model will not be executed and you would get a callback method that you have provided at the end of the waterfall array. On executing above you will get -

That's all for async module - waterfall. In next post, I will show you how we can use an async module for lopping over an array of data -

• Working with asyn module in Node.js - Part 2 (async.eachSeries)

Related Links

• Philip Roberts: What the heck is the event loop anyway? | JSConf EU
• https://www.npmjs.com/package/async
• Working with asyn module in Node.js - Part 2 (async.eachSeries)(OSFG)

How to make HTTP/HTTPS request in Node.js

Background

Many times you need to make an external API call from your Node.js application. A simple example would be calling an API gateway from you Node.js based Lambda in your AWS environment. In this post, I will show you two ways to do this -

1. The standard http/https library
2. The request library

Using the standard http/https library

 Let's see how we can use the standard http library to make an API request.

To use standard http or https library you can simply import the module using -

const https = require("https");
const http = require("http");

Now you can use these to make your http or https calls. A sample is provided below -

/**
 * Node.js code to demonstrate https calls.
 * @author athakur
 */
const https = require("https");

var startDemo = function () {
    console.log("starting demo code");
    executeHttps(function (err, data) {
        if (err) {
            console.log("Error in running demo code");
        }
        else {
            console.log("Successfully ending demo code");
        }

    });
}


var executeHttps = function (callback) {
    var options = {
        hostname: "opensourceforgeeks.blogspot.com",
        port: 443,
        path: "/p/about-me.html",
        method: 'GET',
        headers: {
            'Content-Type': 'text/html'
        }
    };

    var req = https.request(options, function (res) {
        console.log("Status for API call : " + res.statusCode);
        console.log("Headers for API call : " + JSON.stringify(res.headers));
        res.setEncoding('utf8');

        var body = '';

        res.on('data', function (chunk) {
            body = body + chunk;
        });

        res.on('end', function () {
            console.log("Body for API call : " + body.length);
            if (res.statusCode != 200) {
                console.log("API call failed with response code " + res.statusCode);
                callback("API call failed with response code " + res.statusCode, null)
            } else {
                console.log("Got response : " + body.length);
                callback(null, body);
            }
        });
    });

    req.on('error', function (e) {
        console.log("problem with API call : " + e.message);
        callback(e, null);
    });

    req.end();
}


startDemo();

You can get this code on my Github gist as well - https://gist.github.com/aniket91/2f6e92a005eb2a62fcc1ddd39aac6dc2

To execute just run (Assuming your file name is test.js) -

• node test.js

You can similarly do it for http as well. For http you need to use -

• const https = require("http"); 
• change port to 80 in options
• call http.request instead of https.request

NOTE: Notice how we are building the body on 'data' event listener and then processing the request on 'end' event. I have seen developers processing data on 'data' event listener only which is not correct. It will break if your response is huge and comes in chunks.

Similarly, you can execute POST method. Change options to -

    var options = {
        hostname: "opensourceforgeeks.blogspot.com",
        port: 80,
        path: "/p/about-me.html",
        method: 'POST',
        headers: {
            'Content-Type': 'text/html',
            'Content-Length': Buffer.byteLength(post_data)
        }
    };

and then before you close request using req.end(); add

• req.write(post_data);

Now that we have seen how http/https modules work in nodejs let's see how request module works.

Using the request library

Request module is more user-friendly to use.


To begin with, you need to install request module dependency since it is not a standard library that comes with nodejs. To install execute the following command -

• npm install request

 You should see a folder called node_modules getting created in your directory with the request and other dependent modules getting installed.

You can import request module using -

• const request = require('request');

Then you can use it as follows -

/**
 * Node.js code to demonstrate https calls.
 * @author athakur
 */
const request = require('request');

var startDemo = function () {
    console.log("starting demo code");
    executeRequest(function (err, data) {
        if (err) {
            console.log("Error in running demo code");
        }
        else {
            console.log("Successfully ending demo code");
        }

    });
}


var executeRequest = function(callback){
    var headers = {}
    headers['Content-type'] = 'text/html'
    //console.log('Payload for refresh_token: ', querString.stringify(payload))
    request({
        url: 'https://opensourceforgeeks.blogspot.com//p/about-me.html',
        method: 'GET',
        headers: headers
    }, function (err, response, body) {
        if (err) {
            console.error('API failed : ', err)
            callback(err)
        } else {
            console.log("Statuscode: " + response.statusCode);
            console.log("Got response : " + body.length);
            callback(null, body);

        }
    })
}

And the output is -

You can execute POST call as well by changing method type to POST. Eg -

    request({
        url: 'https://opensourceforgeeks.blogspot.com//p/about-me.html',
        method: 'POST',
        body: payload,
        headers: headers
    }, function (err, response, body) {
        if (err) {
            console.error('API failed : ', err)
            callback(err)
        } else {
            console.log("Statuscode: " + response.statusCode);
            console.log("Got response : " + body.length);
            callback(null, body);

        }
    });

Hope this helps! Let me know if you have any questions. Thanks.

Related Links 

• How to install Node and npm to run node.js programs in Linux(OSFG)
• https://gist.github.com/aniket91/2f6e92a005eb2a62fcc1ddd39aac6dc2
• https://www.npmjs.com/package/request
• https://nodejs.org/api/https.html