Simulating environment variables in NodeJs using dotenv package

Background

When you write code there are certain variables that may differ as per the environments like dev, qa prod etc. This might have sensitive data as API keys, passwords etc. You would definitely do not want to put it in your code directly since this will be added to some code repository like git and others may have access to the same. 

General practice that is followed is to use environment variables that can be defined at the environment level and can be read and used in the code. For eg. consider Elastic beanstalk or a Lambda in AWS world. You would define environment variables for the environment and use that in code. If it's your our physical box you might define the environment variables at the OS level or maybe at tomcat level if you are using tomcat as the container. Environment variables work fine in all such cases. 

But how would you do the same in local. In this post I will show how we can simulate environment variables in a NodeJs process with a package called dotenv.

This post expects you to know basics of NodeJs and have NodeJs and npm (Node package manager) installed on your machine. If you have not done that then please refer my earlier post -

• How to install Node and npm to run node.js programs in Linux

Simulating environment variables in NodeJs using dotenv package

First you need to install dotenv package using npm. To do so go to the directory where you would have your NodeJs file and execute following command -

• npm install dotenv

If you get some warning you can ignore it for now. You should see a directory called node_modules getting created in the same directory where you executed this command. This folder will have the package dotenv that we just installed.


Now that we have package installed let's see how we can simulate an environment variable. For this simply create a file name .env in the same directory and add the environment variable you expect to read in code in it. For this demo I will use 3 environment variables -

• ENVIRONMENT=local
• USERNAME=athakur
• PASSWORD=athakur

Now create NodeJS file in the same directory. Let's call it test.js. The directory structure is as follows -

Add following content in test.js -

'use strict';
const dotenv = require('dotenv');
dotenv.config();

const env = process.env.ENVIRONMENT
const username = process.env.USERNAME
const password = process.env.PASSWORD

console.log("Env : " + env);
console.log("Username : " + username);
console.log("Password : " + password);

Save the file and execute it as -

• node test.js

You should see following output on the screen -

Env : local
Username : athakur
Password : athakur

And that's it. You can add any number of environment variables in the ".env" file and read it in your node js code as process.env.variable_name.

NOTE :  .env file would be hidden in Ubuntu since Ubuntu hides all files that start with a dot (.). You can just press Ctrl + H to view hidden files or do a "ls -la" in console. More details -

• How to show hidden files and folder in Ubuntu

To read more about this package  you can read -

• https://www.npmjs.com/package/dotenv

Related Links

• How to install Node and npm to run node.js programs in Linux(OSFG)
• How to show hidden files and folder in Ubuntu(OSFG)
• https://www.npmjs.com/package/dotenv

How to show hidden files and folder in Ubuntu

Background

Some files are folders are hidden in Ubuntu. These are the ones that start with a "." in the beginning. Eg -

• ~/.bashrc
• ~/.vimrc etc

In this post, I will show you how you can make these files visible.

How to hide files and folder in Ubuntu?

The Files file manager gives you the ability to hide and unhide files at your discretion. When a file is hidden, it is not displayed by the file manager, but it is still there in its folder.

To hide a file, rename it with a "." at the beginning of its name. For example, to hide a file named example.txt, you should rename it to .example.txt.

You can hide folders in the same way that you can hide files. Hide a folder by placing a "." at the beginning of the folder’s name.

How to show hidden files and folder in Ubuntu CLI

To see the hidden files in the command line interface (CLI) you can just use -

• ls -la

To not see the hidden files you can just use -

• ls -l

 How to show hidden files and folder in Ubuntu Files explorer

To hidden files in the Files explorer, you can go to -

• View -> Show hidden files

or you can simply press

• Ctrl + H 

You can use the same shortcut or select the same setting again to toggle between showing and hiding hidden files in your Files explorer.

To make this permanent you can go to -

• Edit -> Preferences

and turn on the setting to show hidden files.

Related Links

• Hide a file

How to restore a corrupted or deleted partition with TestDisk and Ubuntu Live

Background

I erased one of my partitions recently which was mounted on my /home path in Ubuntu Linux. However I was able to restore the partition back and life was back to normal. 

I was trying to install Windows and the installer (from USB) was forcing UEFI mode instead of Legacy. So the NTFS partition did not work out and Windows could not be installed since partition was of type MBR instead of GPT (which is required by UEFI mode). So when I tried to make it GPT it started erasing entire disk instead of the disk partition I had selected. I stopped the process immediately but my partitions were gone and it was one disk without any partitions. As I mentioned earlier I was able to restore my previous partitions and data was intact.

In this post I will show you how we can do this.

Setup

You need to have a bootable USB with Ubuntu or gparted. Gparted has both tools -

• gparted and
• testdisk

installed so it is a much simpler option. But if you already have a bootable USB with ubuntu then you can use the same like I did.

Boot your machine from this USB drive.

NOTE : If you do not have a bootable USB you can create one using unetbootin.

• Make a bootable flash drive from an ISO image on Ubuntu

 

How to restore a corrupted or deleted partition with TestDisk and Ubuntu Live

After you boot from Ubuntu live USB go to "Software and Updates" and under Downloadable from internet select the entry with "Universe".

Now run -

• sudo apt-get update
• sudo apt-get install gparted
• sudo apt-get install testdisk

Open gparted to see your disks and partitions. If the partition is missing you should see an unallocated partition. Now run testdisk -

• sudo testdisk

 And follow next steps -

1. Select "No Log" option.
2. Select the disk drive you want to recover, e.g. /dev/sdc.
3. Select your partition table type. Usually it's Intel.
4. Select "Analyse" and then "Quick Search".
5. Your drive will be analysed and you will see a list of all found partitions.  Press Enter.
6. On the next screen you have the option to either perform a second Deeper Search, or Write the current partition table to disk. If the quick search was successful, choose Write.
7. Finally reboot your machine to see the reflected changes
8. You can resuse the gaprted to see that the partition is restored post reboot

Screenshots

That's it. Your partitions should be restored.

Related Links

• Make a bootable flash drive from an ISO image on Ubuntu(OSFG)
• How can I install Windows after I've installed Ubuntu?(SO)

Creating web application with Spark Java framework

Background

Spark is  a Java framework that let's you create web application. In this post we will see how we can write a basic web application using Java Spark framework.  Do not confuse this with Apache Spark which is a big data framework.  If you want to quickly bring up a local server to test something out Spark Java let's you do it in the simplest way possible. You do not need application server. It embeds Jetty server inside it.

Setup

Add following dependencies in your pom.xml for gradle build.

        <dependency>
            <groupId>com.sparkjava</groupId>
            <artifactId>spark-core</artifactId>
            <version>2.7.1</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.21</version>
        </dependency>

spark-core is the spark framework whereas slf4j-simple is for logging. Once above setup is done we can proceed to actual implement our rest application.

Getting Started with Java Spark

Following is a simple Spark code that starts a server and returns "Hello World!" in the response -

import spark.Request;
import spark.Response;
import spark.Route;
import spark.Spark;

/**
 * 
 * @author athakur
 *
 */
public class HelloWorldWithSpark {

    private static final Logger logger = LoggerFactory.getLogger(HelloWorldWithSpark.class);

    public static void main(String args[]) {
        Spark.get("/", new Route() {

            public Object handle(Request request, Response response) throws Exception {
                logger.debug("Received request!");
                return "Hello World!";
            }
        });
        ;
    }

}

Just run above Java code. It should start a jetty server and start listening for incoming requests. Default port that server listens on is 4567. So after running above code go to the browser and access following url  -

•  http://localhost:4567/

You should see "Hello World!" in the response.

Spark exposes static methods that let you define the URLs or routes you want to do some processing on and return some response. In above example we are listening on path "/" which is the root path and returning "Hello World!".


Same code in Java 8 perspective using functional programming/lambda would be -

public class HelloWorldWithSpark {

    private static final Logger logger = LoggerFactory.getLogger(HelloWorldWithSpark.class);

    public static void main(String args[]) {
        Spark.get("/", (req,res) -> {
            return "Hello World!";
        });
    }

}

NOTE : Here we are using GET verb but you can use any like POST, PUT etc.


You can easily create REST APIs from this. Sample example given below -

public class HelloWorldWithSpark {

    private static final Logger logger = LoggerFactory.getLogger(HelloWorldWithSpark.class);

    public static void main(String args[]) {
        
        Spark.get("/employee/:id", (req,res) -> {
            logger.debug("Got request to get employee with id : {}", req.params(":id"));
            return "Retrieved Employee No " + req.params(":id");
        });
        
        Spark.post("/employee/:id", (req,res) -> {
            logger.debug("Got request to add employee with id : {}", req.params(":id"));
            return "Added Employee No " + req.params(":id");
        });
    }

}

That was simple. Wasn't it? You want to deploy it in production like an actual web application in form of war you need to follow a bit different steps -

• http://sparkjava.com/documentation#embedded-web-server 

Related Links

• Spark Java documentation
• http://sparkjava.com/documentation#embedded-web-server

Difference between a forward proxy and a reverse proxy server

Background

Most of the companies out there have a proxy in between their corporate traffic and internet. This could be for multiple reasons - network security being one of them. In my previous post I showed how to set up a squid proxy -

• How to set up a squid Proxy with basic username and password authentication in Ubuntu

That was basically a forward proxy. There are other type of proxies called reverse proxies. In this post we will see difference between them and how they work.

Proxy in lay man terms mean someone acting on behalf of someone else. This is the main principle behind forward and reverse proxy.

Forward Proxy :

Working :

Forward proxy sits between client machines and an origin server. Client machines make a request to the forward proxy with target as the origin server. Forward proxy then makes a request to the origin server, gets the response and sends it back to the clients. Clients in this case need to be explicitly configured to use this kind of forward proxy.

So to summarize a forward proxy retrieves data from another website (origin server) on behalf of the clients.

Example : 

 Consider three computers - A, B and C. Now A want to request a website hosted on computer C. In normal case it would directly be

• A -> C

where computer A directly asks C for the website. However in case of Forward proxy there is an intermediate computer B. Computer A makes request to this computer B instead of directly making request to C. Computer B now makes a request to C gets the website and returns it back to the A. So the path would be

• A -> B -> C.

When :

There can be multiple cases in which a forward proxy might be useful. Some are -

• Client machines (Computer A in above case) are behind some firewall and have no access to internet and thereby no access to origin server.
• A company wants to block some of the malicious sites. They do this on the forward proxy and make sure all client make request via this proxy.
• A forward proxy also has feature to cache requests so that the response time is minimum.

Reverse Proxy :

Working : 

Forward proxy was used to shield the client machines where as a reverse proxy is used to shield a origin server. So client machines make call to the reverse proxy as if they are the origin servers. Reverse proxy now makes a call to the actual origin server and returns the response back to the client.

Example :
Let's consider a similar example of 3 computers - A,B and C. Again in a normal scenario A would directly request website from C.

• A -> C

In case of reverse proxy there is a computer B which hides C behind it. A makes call to B instead and B fetches the website from C and returns it back to A. So the path is again -

•  A -> B -> C

When: 

• Provide internet users access to a server that is behind the firewall.
• Load balance backend servers.
• Typical CDN deployment. Proxy server would tell the client the nearest CDN server.

 Difference between a proxy and a reverse proxy server

 If you see the example above in case of both forward and reverse proxy path is always -

• A -> B -> C

In case of forward proxy B shields machine A by fetching content by C itself and sending back to A. Where as in case of reverse proxy B shields C by fetching the data from C and sending it back to A.

In case of forward proxy, C would think that B is the machine sending it request where there could be multiple A's behind B. Similarly  in case of reverse proxy A would think that it is sending request to C but it would actually be a request to B and B would in turn talk to multiple C and send back the request to A.

NOTE: It's just nomenclature. As you take a basic reverse proxy setup and start bolting on more pieces like authentication, rate limiting, dynamic config updates, and service discovery, people are more likely to call that an API gateway (https://www.nginx.com/blog/building-microservices-using-an-api-gateway/).

Related Links

• Difference between proxy server and reverse proxy server(SO) 
• https://en.wikipedia.org/wiki/Reverse_proxy
• https://en.wikipedia.org/wiki/Proxy_server#Forward_proxy
• http://httpd.apache.org/docs/2.4/mod/mod_proxy.html#forwardreverse
• https://www.nginx.com/blog/building-microservices-using-an-api-gateway/