Open Source For Geeks: ToolsAndUtilities

Showing posts with label ToolsAndUtilities. Show all posts

Saturday, 16 May 2020

How to add code Syntax highlighting to your blogger blog?

Background

If you own a technical blog or a website you generally need to add code to illustrate your examples. In such cases highlighting the code becomes essential. You would have seen the code syntax highlighting in this blog itself.

In this post, I will show you how you can achieve this.

How to add code Syntax highlighting to your blogger blog?

For code syntax highlighting we will use SyntaxHighlighter. I will specifically tell you how to add this to your blogger blog.

Open your blogger blog dashboard
Go to Theme
Click on 3 dots beside "My theme" and click on "Edit HTML"

In the panel which opens and shows some HTML code search and go to the line with </head> tag. This is where your head tag ends. We need to add some include CSS and js files here along with some custom javascript.

Inside the head tag (Just before </head> add following code)

<!-- Syntax Highlighter START -->
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shCore.css" rel="stylesheet" type="text/css"></link>
<link href="http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css" rel="stylesheet" type="text/css"></link>
<script src="http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js" type="text/javascript">
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushAS3.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushColdFusion.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCSharp.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCss.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushDelphi.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushDiff.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushErlang.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushGroovy.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJScript.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJavaFX.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPerl.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPhp.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPlain.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPowerShell.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPython.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushRuby.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushScala.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushVb.js' type='text/javascript'/>
<script src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushXml.js' type='text/javascript'/>
<script language='javascript' type='text/javascript'>
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all();
</script>

Once done save the file and reload blog.

Now if you want any highlighting you can use the corresponding class in <pre> tag. Eg for Java you can do
```
<pre class="brush:java">This will be highlighted</pre>
```
Instead of Java you can have other languages as well. Choices are: cpp, c, c++, c#, c-sharp, csharp, css, delphi, pascal, java, js, jscript, javascript, php, py, python, rb, ruby, rails, ror, sql, vb, vb.net, xml, html, xhtml, xslt. You can see the latest list of supported languages.
You can only add js files for brushes you need (See optimization below)

Optimizations

This is the good part! We would not call ourselves programmers if we did not have an optimization part :)

You can see above there are a bunch of js files added in the head tag. You might not need all and each page load with load these external JS code which can slow loading if your blog. So Add only those JS files which you need. In fact, if you see the screenshot above I have used just the Java brush JS. I just use the same for all types of codes.
If you do not want the code highlighting to work for your homepage (Just for the the posts you write), you can add all above code inside the following tags:
```
<b:if cond='data:blog.pageType == "item"'>
</b:if>
```
Lastly, you would have also noticed the link base path for JS and CSS files are different in my code that what I originally provided. That's because I have used the CDN path(https://cdnjs.com/libraries/SyntaxHighlighter). This is done primarily for 2 things:

First, it allows highlighting to work even on https. By default with the above code loading your blog site with https protocol will not show highlighting. That's because your include scripts are HTTP and not supported for https.
Secondly, if the HTTP links are down you are screwed. CDN caches the scripts and cs files. So you can always rely on it (rely is is a strong word but it's better than those HTTP links :) )

Configuration

Another thing you might have noticed is the change of theme file I have used. The original set of code I proposed uses a default theme shThemeDefault.css but I have changed this to use shThemeEmacs.css. You can use whichever theme you like - Just include the corresponding theme CSS file (and remove the default one). Some of the available themes are: shThemeRDark, shThemeMidnight, shThemeMDUltra, shThemeFadeToGrey, shThemeEmacs, shThemeEclipse, shThemeDjango, shThemeDefault, shCoreRDark, shCoreMidnight, shCoreMDUltra, shCoreFadeToGrey, shCoreEmacs, shCoreEclipse, shCoreDjango, shCoreDefault

I already mentioned you should only include and use the JS files corresponding to language brushed you intent to use. This will reduce your page load time. You can also use the "b:if" tag I mentioned above so that these scripts load for your blog posts.

You can already see this blog using all of these customizations. Feel free to comment if you need any help. Thanks.

Saturday, 27 July 2019

How to install IntelliJ Idea plugin from local disk

Background

In the last post, we saw a basic tutorial on how to create a custom plugin for IntelliJ IDE's. In this post, I will show you how you can install a plugin you have on your local disk.

How to install IntelliJ Idea plugin from local disk

To install plugin from local disk, go to setting in IDE(Ctrl+Alt+S) -> Plugins. Next click on the gear icon and select "Install plugin from disk".

Select the zip file of your plugin you have stored locally and select Ok. The plugin should get installed. You may have to restart the IDE for change to take effect.

Now you can see the plugin in the installed tab of your plugins section of settings.

Directories used by the IDE to store plugins

If you are wondering where are plugins instaled then the path is config\plugins under your IDEA directory. For me it is:

C:\Users\anike\.IdeaIC2019.1\config\plugins

It should put your plugin jar in above dir.

To see more details on this path refer - Directories used by the IDE to store settings, caches, plugins and logs

Creating an Intellij plugin

Background

Intellij IDEA is one of the famous IDEs(Integrated development environment) used for Java development. Intellij has a variant of IDE's that they provide like -

Pycharm - For Python
Webstorm - For web development
IDEA - For Java

etc. In this post, I will show how you can write your own plugin for any of these IDEs. To develop a plugin you need Intellij IDEA IDE. You can use this to create a plugin for any other variant of IDE. In fact, the framework for IDE remains the same, so you can create a plugin that can potentially work in all IDEs.

To start with download IntelliJ IDEA. I am using version 2019.1.3(Community edition).

Idea

In this plugin, we are going to add a simple action functionality that takes in the selected text and searches on Stack overflow site. This action will be visible when you right-click on the editor panel of your IDE. Let's see how to do that,

Creating an IntelliJ plugin

Open your IDEA and create a new project. File-> New -> Project -> Intellij platform plugin

Once done, click on next, enter your project name and submit. This should create a new project for you. One of the important files is Project\resources\META-INF\plugin.xml. This gives information about your plugin. Think of it as the manifest file of Android project (If you have worked on Android apps before). Got me location is C:\Users\anike\IdeaProjects\StackOverflowSearch\resources\META-INF\plugin.xml

and my project name is StackOverflowSearch.

NOTE: You can choose Groovy as well to develop your plugin. I have selected default, which uses Java.

In the source folder create a class called StackoverflowSearch. This is going to be our action class. Make this class extend com.intellij.openapi.actionSystem.AnAction. AnAction is an abstract class provide by Intellij SDK framework. Once you extend it, you will have to implement the abstract methods in it

    @Override
    public void actionPerformed(@NotNull AnActionEvent anActionEvent) {
        
    }

Then you can add the following code to complete your simple action -

import com.intellij.ide.BrowserUtil;
import com.intellij.openapi.actionSystem.AnAction;
import com.intellij.openapi.actionSystem.AnActionEvent;
import com.intellij.openapi.actionSystem.CommonDataKeys;
import com.intellij.openapi.editor.CaretModel;
import com.intellij.openapi.editor.Editor;
import com.intellij.psi.PsiFile;
import org.jetbrains.annotations.NotNull;


public class StackoverflowSearch extends AnAction {
    @Override
    public void actionPerformed(@NotNull AnActionEvent anActionEvent) {
        PsiFile file = anActionEvent.getData(CommonDataKeys.PSI_FILE);
        Editor editor = anActionEvent.getRequiredData(CommonDataKeys.EDITOR);
        CaretModel caretModel = editor.getCaretModel();
        String selectedText = caretModel.getCurrentCaret().getSelectedText();
        BrowserUtil.open("https://stackoverflow.com/search?q=" + selectedText);
    }
}

This essentially gets the selected text from your editor and open a browser URL with that query parameter. If you do not understand much with this code, don't worry just go to the official documentation and see what each class means. For eg. PSIFile - it s file representation in Intellij framework world. You can see more details here.

Once done, you will have to list this action in the plugin.xml file we saw above. In this file, you should see the <actions> tag. Add below content inside it.

    <action
            id="Action.Stackoverflow.Search"
            class="StackoverflowSearch"
            text="Search Text on Stack Overflow"
            description="Search Text on Stack Overflow">
      <add-to-group group-id="EditorPopupMenu" anchor="last"/>
    </action>

Once done you are all set to go. The only important thing to note above is the add-to-group group-id field. EditorPopupMenu means this action will be shown in Editor when you right-click. You could have other possible values to show it in Console or top menubar.

My complete plugin.xml looks like below -

<idea-plugin>
  <id>com.your.company.unique.plugin.id</id>
  <name>Stackoverflow Search Plugin</name>
  <version>1.0</version>
  <vendor email="opensourceforgeeks@gmail.com" url="http://opensourceforgeeks.blogspot.com/">OSFG</vendor>

  <description><![CDATA[
      Simple plugin to open selexted text in Stack overflow site
    ]]></description>

  <change-notes><![CDATA[
      Simple plugin to open selexted text in Stack overflow site
    ]]>
  </change-notes>

  <!-- please see http://www.jetbrains.org/intellij/sdk/docs/basics/getting_started/build_number_ranges.html for description -->
  <idea-version since-build="173.0"/>

  <!-- please see http://www.jetbrains.org/intellij/sdk/docs/basics/getting_started/plugin_compatibility.html
       on how to target different products -->
  <!-- uncomment to enable plugin in all products
  <depends>com.intellij.modules.lang</depends>
  -->

  <extensions defaultExtensionNs="com.intellij">
    <!-- Add your extensions here -->
  </extensions>

  <actions>
    <!-- Add your actions here -->
    <action
            id="Action.Stackoverflow.Search"
            class="StackoverflowSearch"
            text="Search Text on Stack Overflow"
            description="Search Text on Stack Overflow">
      <add-to-group group-id="EditorPopupMenu" anchor="last"/>
    </action>
  </actions>

</idea-plugin>

Now you can simply run your project,

Run configuration should automatically be created when you click on run. It should be similar to the following -

NOTE: Notice that the JRE is Intellij Idea SDK.

This should start a new IDE instance with your plugin activate. You can verify your plugin in installed by going to settings(Ctrl_Alt+S) -> Plugins -> Installed

Now you can select a text, right-click and see the "Search Text on Stackoverflow" action. Click that and it should open the Stack overflow site with your selected text as a search parameter,

Distributing the plugin

To distribute the plugin, simply right click your plugin project and select - "Prepare plugin module for deployment". This should export a zip file which can be distributed. You should see a message like below when you have selected the above option.

To know how to install plugin from local disk refer - How to install IntelliJ Idea plugin from local disk

You can also put it into a plugin repository for others to use instead of distributing zip file. For more details on the plugin, repo see here. I will be adding more details on how to create an Intellij plugin. So stay tuned.

Tuesday, 27 February 2018

Create a gif from a video using ffmpeg with good quality

Background

In the last post, we saw to Resizing videos and images using ffmpeg. In this post, we will see how we can convert a video into a gif with good quality.

Create a gif from a video using FFmpeg

You can use the following shell script to convert your video to Gif -

#!/bin/bash
#----------------------------------------------------
#  Author : athakur
#  Version : 1.0
#  Create Date : 27/02/2018
#  Update Date : 27/02/2018
#  Description : Create gif from a video
#    Sample usage : ./creategif.sh input.mp4 output.gif
# ffmpeg static build can be downloaded from https://johnvansickle.com/ffmpeg/
#----------------------------------------------------
echo "Converting $1 to $2"

if [ -z "$1" ] || [ -z "$2" ]
  then
  echo "Incorrect arguments supplied. Format - ./creategif.sh input.mp4 output.gif"
  exit
fi

palette="/tmp/palette.png"

filters="fps=10,scale=1024:-1:flags=lanczos"

ffmpeg -v warning -i "$1" -vf "$filters,palettegen" -y "$palette"
ffmpeg -v warning -i "$1" -i $palette -lavfi "$filters [x]; [x][1:v] paletteuse" -y "$2"

echo "Completed gif creation"

You can find this code in my GitHub gists section-

https://gist.github.com/aniket91/c08ad48b1968dbcfa48cdb72188e5bba

To run you can simply execute following command -

./creategif.sh input.mp4 output.gif

Input need not be an mp4. It can be an .avi or a .webm too.

Sample video and gif are as follows -

Video :

Gif :

That's it your high-quality gif is ready.

You can do more with FFmpeg like scaling and cropping of video before converting to gig. Eg.

./ffmpeg -loglevel warning -y -i input.mp4 -i palette.png -filter_complex "crop=800:500:0:0[x];[x][1:v]paletteuse=dither=bayer:bayer_scale=3[tl];[tl]fps=10,scale=1024:-1:flags=lanczos"  target.gif

Friday, 23 February 2018

Resizing videos and images using ffmpeg

Background

If you are a developer and a geek and need to work on the images/videos formatting in terms of cropping, scaling or resizing then you can use FFmpeg for that. Static build for FFmpeg is available on -

https://johnvansickle.com/ffmpeg/

which you can download and store it on your local machine to use.

I have even added this to my Linux PATH so that I can access it from anywhere. If you are not aware how to do that refer -

How to permanently set $PATH on Linux?

Once you have added FFmpeg in your PATH you can simply run ffmpeg to see the help content -

Specifically, if you are working on making Android or iOS apps or chrome plugin and you have an icon of standard size and you want to resize it to fit other aspect ratios supported by the corresponding platforms then FFmpeg really comes in handy.

Rescaling images with FFmpeg

I have a simple icon with size 256*256 to be used as replay button.

I want it in 128*128 size. You can do this with the following command -

ffmpeg -i icon-256.png -vf scale="128:128" icon-128.png

And if you want 48*48 you can similarly do -

ffmpeg -i icon-256.png -vf scale="48:48" icon-48.png

If you want to want to retain the aspect ration you can do -

ffmpeg -i icon-256.png -vf scale="128:-1" icon-128.png

You can do similar resizing for a video instead of an image -

ffmpeg -i input.avi -vf scale="320:240" output.avi

If you want your image to be based on the actual image size then you can do that as well. For eg. You want the image to be double size of what the size it actually is -

ffmpeg -i icon-256.png -vf scale="iw*2:ih*2" icon-double.png

Since this is double this would be 512*512 since original was 256*256.
Similarly, if you want half you can do -

ffmpeg -i icon-256.png -vf scale="iw/2:ih/2" icon-half.png

Since this is half this would be 128*128 since original was 256*256.

NOTE :

iw : input width
ih : input height

Thursday, 1 February 2018

How to restore a corrupted or deleted partition with TestDisk and Ubuntu Live

Background

I erased one of my partitions recently which was mounted on my /home path in Ubuntu Linux. However I was able to restore the partition back and life was back to normal.

I was trying to install Windows and the installer (from USB) was forcing UEFI mode instead of Legacy. So the NTFS partition did not work out and Windows could not be installed since partition was of type MBR instead of GPT (which is required by UEFI mode). So when I tried to make it GPT it started erasing entire disk instead of the disk partition I had selected. I stopped the process immediately but my partitions were gone and it was one disk without any partitions. As I mentioned earlier I was able to restore my previous partitions and data was intact.

In this post I will show you how we can do this.

Setup

You need to have a bootable USB with Ubuntu or gparted. Gparted has both tools -

gparted and
testdisk

installed so it is a much simpler option. But if you already have a bootable USB with ubuntu then you can use the same like I did.

Boot your machine from this USB drive.

NOTE : If you do not have a bootable USB you can create one using unetbootin.

Make a bootable flash drive from an ISO image on Ubuntu

How to restore a corrupted or deleted partition with TestDisk and Ubuntu Live

After you boot from Ubuntu live USB go to "Software and Updates" and under Downloadable from internet select the entry with "Universe".

Now run -

sudo apt-get update
sudo apt-get install gparted
sudo apt-get install testdisk

Open gparted to see your disks and partitions. If the partition is missing you should see an unallocated partition. Now run testdisk -

sudo testdisk

And follow next steps -

Select "No Log" option.
Select the disk drive you want to recover, e.g. /dev/sdc.
Select your partition table type. Usually it's Intel.
Select "Analyse" and then "Quick Search".
Your drive will be analysed and you will see a list of all found partitions. Press Enter.
On the next screen you have the option to either perform a second Deeper Search, or Write the current partition table to disk. If the quick search was successful, choose Write.
Finally reboot your machine to see the reflected changes
You can resuse the gaprted to see that the partition is restored post reboot

Screenshots

That's it. Your partitions should be restored.

Sunday, 28 January 2018

Creating web application with Spark Java framework

Background

Spark is a Java framework that let's you create web application. In this post we will see how we can write a basic web application using Java Spark framework. Do not confuse this with Apache Spark which is a big data framework. If you want to quickly bring up a local server to test something out Spark Java let's you do it in the simplest way possible. You do not need application server. It embeds Jetty server inside it.

Setup

Add following dependencies in your pom.xml for gradle build.

        <dependency>
            <groupId>com.sparkjava</groupId>
            <artifactId>spark-core</artifactId>
            <version>2.7.1</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.21</version>
        </dependency>

spark-core is the spark framework whereas slf4j-simple is for logging. Once above setup is done we can proceed to actual implement our rest application.

Getting Started with Java Spark

Following is a simple Spark code that starts a server and returns "Hello World!" in the response -

import spark.Request;
import spark.Response;
import spark.Route;
import spark.Spark;

/**
 * 
 * @author athakur
 *
 */
public class HelloWorldWithSpark {

    private static final Logger logger = LoggerFactory.getLogger(HelloWorldWithSpark.class);

    public static void main(String args[]) {
        Spark.get("/", new Route() {

            public Object handle(Request request, Response response) throws Exception {
                logger.debug("Received request!");
                return "Hello World!";
            }
        });
        ;
    }

}

Just run above Java code. It should start a jetty server and start listening for incoming requests. Default port that server listens on is 4567. So after running above code go to the browser and access following url -

http://localhost:4567/

You should see "Hello World!" in the response.

Spark exposes static methods that let you define the URLs or routes you want to do some processing on and return some response. In above example we are listening on path "/" which is the root path and returning "Hello World!".

Same code in Java 8 perspective using functional programming/lambda would be -

public class HelloWorldWithSpark {

    private static final Logger logger = LoggerFactory.getLogger(HelloWorldWithSpark.class);

    public static void main(String args[]) {
        Spark.get("/", (req,res) -> {
            return "Hello World!";
        });
    }

}

NOTE : Here we are using GET verb but you can use any like POST, PUT etc.

You can easily create REST APIs from this. Sample example given below -

public class HelloWorldWithSpark {

    private static final Logger logger = LoggerFactory.getLogger(HelloWorldWithSpark.class);

    public static void main(String args[]) {
        
        Spark.get("/employee/:id", (req,res) -> {
            logger.debug("Got request to get employee with id : {}", req.params(":id"));
            return "Retrieved Employee No " + req.params(":id");
        });
        
        Spark.post("/employee/:id", (req,res) -> {
            logger.debug("Got request to add employee with id : {}", req.params(":id"));
            return "Added Employee No " + req.params(":id");
        });
    }

}

That was simple. Wasn't it? You want to deploy it in production like an actual web application in form of war you need to follow a bit different steps -

http://sparkjava.com/documentation#embedded-web-server

Saturday, 20 January 2018

Difference between a forward proxy and a reverse proxy server

Background

Most of the companies out there have a proxy in between their corporate traffic and internet. This could be for multiple reasons - network security being one of them. In my previous post I showed how to set up a squid proxy -

How to set up a squid Proxy with basic username and password authentication in Ubuntu

That was basically a forward proxy. There are other type of proxies called reverse proxies. In this post we will see difference between them and how they work.

Proxy in lay man terms mean someone acting on behalf of someone else. This is the main principle behind forward and reverse proxy.

Forward Proxy :

Working :

Forward proxy sits between client machines and an origin server. Client machines make a request to the forward proxy with target as the origin server. Forward proxy then makes a request to the origin server, gets the response and sends it back to the clients. Clients in this case need to be explicitly configured to use this kind of forward proxy.

So to summarize a forward proxy retrieves data from another website (origin server) on behalf of the clients.

Example :

Consider three computers - A, B and C. Now A want to request a website hosted on computer C. In normal case it would directly be

A -> C

where computer A directly asks C for the website. However in case of Forward proxy there is an intermediate computer B. Computer A makes request to this computer B instead of directly making request to C. Computer B now makes a request to C gets the website and returns it back to the A. So the path would be

A -> B -> C.

When :

There can be multiple cases in which a forward proxy might be useful. Some are -

Client machines (Computer A in above case) are behind some firewall and have no access to internet and thereby no access to origin server.
A company wants to block some of the malicious sites. They do this on the forward proxy and make sure all client make request via this proxy.
A forward proxy also has feature to cache requests so that the response time is minimum.

Reverse Proxy :

Working :

Forward proxy was used to shield the client machines where as a reverse proxy is used to shield a origin server. So client machines make call to the reverse proxy as if they are the origin servers. Reverse proxy now makes a call to the actual origin server and returns the response back to the client.

Example :
Let's consider a similar example of 3 computers - A,B and C. Again in a normal scenario A would directly request website from C.

A -> C

In case of reverse proxy there is a computer B which hides C behind it. A makes call to B instead and B fetches the website from C and returns it back to A. So the path is again -

A -> B -> C

When:

Provide internet users access to a server that is behind the firewall.
Load balance backend servers.
Typical CDN deployment. Proxy server would tell the client the nearest CDN server.

Difference between a proxy and a reverse proxy server

If you see the example above in case of both forward and reverse proxy path is always -

A -> B -> C

In case of forward proxy B shields machine A by fetching content by C itself and sending back to A. Where as in case of reverse proxy B shields C by fetching the data from C and sending it back to A.

In case of forward proxy, C would think that B is the machine sending it request where there could be multiple A's behind B. Similarly in case of reverse proxy A would think that it is sending request to C but it would actually be a request to B and B would in turn talk to multiple C and send back the request to A.

NOTE: It's just nomenclature. As you take a basic reverse proxy setup and start bolting on more pieces like authentication, rate limiting, dynamic config updates, and service discovery, people are more likely to call that an API gateway (https://www.nginx.com/blog/building-microservices-using-an-api-gateway/).

Thursday, 18 January 2018

How to set up a squid Proxy with basic username and password authentication in Ubuntu

Background

Most of the big companies have their own proxies through which all the company data is routed through. This ensure malicious sites are blocked and all other traffic is audited via proper authentication.

To give a little background on Squid proxy -

Squid is a caching and forwarding HTTP web proxy. It has a wide variety of uses, including speeding up a web server by caching repeated requests, caching web, DNS and other computer network lookups for a group of people sharing network resources, and aiding security by filtering traffic. Although primarily used for HTTP and FTP, Squid includes limited support for several other protocols including Internet Gopher, SSL,[6] TLS and HTTPS. Squid does not support the SOCKS protocol.

Squid was originally designed to run as a daemon on Unix-like systems. A Windows port was maintained up to version 2.7. New versions available on Windows use the Cygwin environment.[7] Squid is free software released under the GNU General Public License.

Source : Wiki

Installing Squid proxy on Ubuntu

To install squid server simply run following command in your terminal -

sudo apt install squid

Squid run as daemon service in Ubuntu. You can execute following command to see the status of this service -

service squid status

It will show you if squid service is running or not.

Some important file paths are -

/etc/sqid : This is where your squid configuration resides
/var/log/squid : This is where your squid logs reside
/usr/lib/squid3,/usr/lib/squid : This is where your squid modules or libraries reside.

Now that we have Squid proxy installed. Let's configure it.

Squid configuration is located at -

/etc/squid/squid.conf

Before you make changed to this file make a copy of this and store it aside. Use following commands to do that -

sudo cp /etc/squid/squid.conf /etc/squid/squid.conf.original
sudo chmod a-w /etc/squid/squid.conf.original

This essentially created a copy of squid.conf called squid.conf.original and removed all write access to it so that no one can accidentally write it.

Default TCP port that Squid listens to is 3128. Go ahead and change it to 8888. I prefer using 8888 port since this is used by other proxies as well like Charles and Fiddler. To do this find a line called

http_port 3128

and change it to

http_port 8888

Next you need to provide rules to allow and disallow traffic. If you want to just allow trafic from your local machine you can add the following lines to the configuration -

acl localhost src 127.0.0.1/32
http_access allow localhost

acl is nothing but access control list. it's a keyword that states acl is starting. Next localhost is the name that is used to indentify the acl. I have named it localhost but it can be anything. Next we have src which is used to identify local IP addresses. Other options are -

srcdomain : used for declaring local domain,
dst : used for public IP &
dstdomain : used for public domain name

Next we have http_access that will basically take action provide in it's next word on the acl we define. In this we we are saying allow and for acl named localhost that we defined above. So Squid proxy is going to allow all http traffic from local machine (i.e with IP 127.0.0.1)

Last line you can add as -

http_access deny all

which says you deny all other traffic. So the way acl's work is -

For each request that Squid receives it will look through all the http_access statements in order until it finds a line that matches. It then either accepts or denys depending on your setting. The remaining rules are ignored.

This was basic settings for squid proxy. Now let's see how we can add an authentication to this scheme.

Post configuration you can just restart the squid service -

service squid restart

You can also view the service logs for this in file-

less /var/log/squid/cache.log

And you can view the access logs in file -

less /var/log/squid/access.log

How to set up a squid Proxy with basic username and password authentication?

For this you can add following lines to your squid configuration file squid.conf -

auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/passwords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED
http_access allow authenticated

ident_lookup_access deny all
http_access deny all

Above configuration will ensure all traffic is authenticated. The username/password that would be needed to provide access will be stored in a file - /etc/squid/passwords. We will now see how we can create this file.

To generate username/passwrod you need to use a command called htpasswd. You can install this using -

apt-get install apache2-utils

Next to generate username/password type in following command -

sudo htpasswd -c /etc/squid/passwords YOUR_USERNAME

Replace YOUR_USERNAME with the user name you want. Eg admin. You will be prompted for password for this username twice. Once done your user is all setup. You can use this credentials to access your proxy.

NOTE : htpasswd stores the password hashed.

One done you can restart your squid service -

service squid restart

My conf file looks like below -

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports


http_port 8888

coredump_dir /var/spool/squid

refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
refresh_pattern .               0       20%     4320

auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/passwords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED
http_access allow authenticated


ident_lookup_access deny all
http_access deny all

Now you can test this by adding a proxy in firefox and trying to go to a http URL.

Add username/password that you just created before and the URL should be accessible.

Sunday, 10 September 2017

Installing and Using OpenGrok on Mac OS X

Background

In previous couple of posts we saw how we can setup git repositories, install git client and maintain your codebase.

In this post we will see how to set up opengrok. This is a code base search tool that you can use to index and search your huge complex codebase. I am going to show this on my mac laptop. If you want to setup a proper server please refer to official documentation.

Installing and Using OpenGrok on Mac OS X

I am going to use Homebrew to do most of the setup here. If you are not aware of homebrew then you can read -

Tips and Tricks for your MacBook

Couple of things you need to install before are -

A servlet container like GlassFish or Tomcat to run and deploy your grok server. I will use tomcat.
Exuberant Ctags for analysis.

You can run following commands to set these up -

brew update
brew install tomcat
brew install ctags

You can just type catalina to see tomcat is properly installed -

Next set environment variable as follows -

export OPENGROK_TOMCAT_BASE=/usr/local/Cellar/tomcat/8.5.20/libexec

For path you can refer to catalina screenshot above. This environment variable will basically tell where grok needs to be deployed.

Download the latest opengrok binary from-

https://github.com/OpenGrok/OpenGrok/releases

I am using opengrok-1.1-rc13.tar.gz.

Next go yo your opengrok bin directory. In my case it is -

/Users/athakur/Documents/Softwares/opengrok-1.1-rc13/bin

and run -

./OpenGrok deploy

This will deploy grok code on your tomcat container. Now start the tomcat container

You can now access it via -

http://localhost:8080/source/

The error you see is ok since we have not provided our codebase source directory yet.

Noe lets add source directory. My code is in-

~/Documents/git/DataStructures

NOTE : DataStructures is a local copy of my github repo -

https://github.com/aniket91/DataStructures/

I am going to maintain all codebase references in

~/local_repos/src/

So create a directory and add soft links as below -

Now it's time to define your code directory that opengrok can understand. So define another environment variable -

export OPENGROK_INSTANCE_BASE=/Users/athakur/local_repos

That's now lets index this content. To index it go to you opengrok bin directory and run -

./OpenGrok index.

You can see it automatically creates directory it needs. Just make sure it has appropriate permissions -

That's it you can refresh grok page and start searching code.

NOTE : For every update to your actual repository or for any new repository getting added you need to call ./Opengrok index to index it. You can probably write a cron job that does an automatic pull of your repository and runs index on it.

Saturday, 16 May 2020

Background

How to add code Syntax highlighting to your blogger blog?

Optimizations

Configuration

Related Links

Saturday, 27 July 2019

Background

How to install IntelliJ Idea plugin from local disk

Directories used by the IDE to store plugins

Related Links

Background

Idea

Creating an IntelliJ plugin

Distributing the plugin

Related Links

Tuesday, 27 February 2018

Background

Create a gif from a video using FFmpeg

Related Links

Friday, 23 February 2018

Background

Rescaling images with FFmpeg

Related Links

Thursday, 1 February 2018

Background

Setup

How to restore a corrupted or deleted partition with TestDisk and Ubuntu Live

Screenshots

Related Links

Sunday, 28 January 2018

Background

Setup

Getting Started with Java Spark

Related Links

Saturday, 20 January 2018

Background

Forward Proxy :

Reverse Proxy :

Difference between a proxy and a reverse proxy server

Related Links

Thursday, 18 January 2018

Background

Installing Squid proxy on Ubuntu

How to set up a squid Proxy with basic username and password authentication?

Related Links

Sunday, 10 September 2017

Background

Installing and Using OpenGrok on Mac OS X

Related Links