Using Fabric Python command-line tool

What is Fabric?

You can visit the fabric site . It gives the most apt description of what fabric is.

Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

More specifically, Fabric is:

• A tool that lets you execute arbitrary Python functions via the command line
• A library of subroutines (built on top of a lower-level library) to make executing shell commands over SSH easy and Pythonic.

Prerequisites

1. Assuming you have fair knowledge of python basics. If not please go over the basics in the tutorial - The Python Tutorial
2. Also you mus have python installed. If not refer to the guide -  The Hitchhiker’s Guide to Python!
1. Installing Python on Mac OS 
2. Installing Python on Windows
3. Installing Python on Linux 

Note : Fabric is supported only by Python versions (2.5-2.7).

Installing Fabric

 Before you install Fabric you need to install modules needed to install other modules. For that you can either install - 

1. pip or
2. setuptools

I am going to use pip to install fabric as it is the latest one. To install Fabric use the following command -

 pip install fabric

and your fabric should be installed.  You can see various related installation ways here. You can see the installed module in  

• <Python installation directory>\Lib\site-packages .

For me it is located in 

• C:\Python34\Lib\site-packages\fabric

You can see the fab. exe file in

• C:\Python34\Scripts

which we have already added in the PATH environment variable [This is the same place where you have pip and easy_install executable]

Getting Started

Lets start with our usual "Hello World!"  example. Create a file called fabfile.py and add following content in it - 

def helloworld():
        """This is a test hello world fab method"""
        print("Hello World!")

To see list of available fabric commands do  - 

• fab -l

For above file you should see something like - 

[athakur@localhost athakur]$ fab -l
Available commands:

    helloworld  This is a test hello world fab method

Next it's time to execute the command itself. Execute

• fab helloworld

and you should see Hello World! printed on the console.

Note :  Any method that starts with an underscore (_) is a private method and cannot be used directly as fab command. It will even not be listed in fab -l command.

Consider following code -

def helloworld():
        """This is a test hello world fab method"""
        print("Hello World")
        _private_helloworld()

def _private_helloworld():
        """ This is private method and should not be listed with fab -l"""
        print("Private hello World")

Execute fab -l  and you should again see the same output as above i.e you should not see _private_helloworld method listed there.

Executing fab helloworld should print following now -

[athakur@localhost athakur]$ fab helloworld
Hello World
Private hello World
Done.

Note : The fab tool simply imports your fabfile and executes the function or functions you instruct it to. There’s nothing magic about it – anything you can do in a normal Python script can be done in a fabfile!

Fabfile discovery

Fabric is capable of loading Python modules (e.g. fabfile.py) or packages (e.g. a fabfile/ directory containing an __init__.py). By default, it looks for something named (to Python’s import machinery) fabfile - so either fabfile/ or fabfile.py.

The fabfile discovery algorithm searches in the invoking user’s current working directory or any parent directories. Thus, it is oriented around “project” use, where one keeps e.g. a fabfile.py at the root of a source code tree. Such a fabfile will then be discovered no matter where in the tree the user invokes fab.

(More Details) 

Fabric Tasks with arguments

Put the following  content in fabfile.py

def helloworld(name="Aniket"):
        """This is a test hello world fab method. Syntax : fab helloworld:name=<your name>"""
        print("Hello World from %s!" %name)
        _private_helloworld()

def _private_helloworld():
        """ This is private method and should not be listed with fab -l"""
        print("Private hello World")

and run 

• fab helloworld

You should get

[athakur@localhost athakur]$ fab helloworld
Hello World from Aniket!
Private hello World
Done.

Or you can execute by giving name argument as

• fab helloworld:name="John" or simply
• fab helloworld:"John"

and you should get output as

[athakur@localhost athakur]$ fab helloworld:"John"
Hello World from John!
Private hello World
Done.

Running fabric tasks on other machines

Replace your fabfile with following contents :

from fabric.api import run

def host_name():
        run('uname -a')

and then run it. You should br prompted for hostname and then password for the user you are already logged in as.

[athakur@localhost athakur]$ fab host_name
No hosts found. Please specify (single) host string for connection: localhost
[localhost] run: uname -a
[localhost] Login password for 'athakur':
[localhost] out: Linux localhost 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Done.
Disconnecting from localhost... done.

Troubleshooting

In Windows while installing Fabric you may get following error - 

building 'Crypto.Random.OSRNG.winrandom' extension
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath
error: Microsoft Visual C++ 10.0 is required (Unable to find vcvarsall.bat).

If you get this you will have to install Microsoft Visual C++ compiler. See Stack overflow answers in the 1st link of related Sections. This is windows specific issue. If you are using Linux this should not occur.

Or You can install MinGW (Minimalist GNU for Windows) as I did. You need to install msys package under MinGW

 and add following entries in your PATH env variable.

• C:\MinGW\bin
• C:\MinGW\msys\1.0\bin [This is where you will find chmod executable]

Then run your command from normal windows command prompt.

Related Links

• error: Unable to find vcvarsall.bat (SO)
• How do I install PyCrypto on Windows?(SO)
• How To Use Fabric To Automate Administration Tasks And Deployments
• Using Fabric for SSH

Using Async Tasks in Android

Background

The whole android story behind this posts revolves around a single concept - the UI Thread. Lets understand what is this UI thread and what makes it so important. 

• Each application has a main thread which is also called UI thread.
• All the application components that are a part of same process will use this same UI thread. 
•  All the android life cycle methods, the system callbacks, user interactions etc are handled by this UI thread.

Now you see why this main thread (UI Thread) is so important. You should avoid performing expensive time consuming operations on main thread as it will block your application altogether. This is one of the reasons why you may see ANR (App not responding) messages on you android device.

Note : Generally all components in an android application will run in same process unless you explicitly specify components to run in different processed in the manifest file.

So in this post we will see how to run your time consuming processes in Async task which basically runs your time consuming process in a new thread.

Why you should not do time consuming processing on UI Thread?

I am creating a simple that has a download button. To simulate a time consuming operation I am simply going to make the main thread sleep for 5 seconds.

My MainActivity code is as follows - 

package com.opensourceforgeeks.asynctaskdemo;

import android.app.Activity;
import android.os.Bundle;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.Button;
/**
 * @author athakur
 */
public class MainActivity extends Activity {

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        Button downloadButton = (Button) findViewById(R.id.download_button);
        downloadButton.setOnClickListener(new OnClickListener() {
           
            @Override
            public void onClick(View v) {
                try {
                    //sleep for 5 seconds
                    Thread.sleep(5000);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        });
    }
}
 

After you install the app on your android device or emulator  click on download button. You should see the following screen - 

Note :  As usual I am skipping showing the resource code in the posts assuming you should be comfortable creating one now. If not you can always go back and see basic post -

• Android "Hello World" Application development tutorial

So there you go. You got an ANR (App Not responding). This means your code is very bad and poor quality. Now lets see how we can make use of Async task to get rid of this ANR.

Using Aysnc Task for time consuming  processing

I am going to slightly change the code now. Will add a textView at the top which will show the download status - stop, progress and completed. 

Code is as follows - 

package com.opensourceforgeeks.asynctaskdemo;

import android.app.Activity;
import android.os.AsyncTask;
import android.os.Bundle;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.Button;
import android.widget.TextView;
/**
 * @author athakur
 */
public class MainActivity extends Activity {

    Button downloadButton;
    TextView downloadStatus;
    
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        downloadButton = (Button) findViewById(R.id.download_button);
        downloadStatus = (TextView) findViewById(R.id.downlaodStatus);
        downloadButton.setOnClickListener(new OnClickListener() {
            
            @Override
            public void onClick(View v) {
                new DownloadAsynctask().execute("TestInput");
            }
        });
    }
    
    class DownloadAsynctask extends AsyncTask<String, Integer, Boolean> {    
        @Override
        protected String doInBackground(String... params) {
            downloadStatus.setText(R.string.download_progress);
            try {
                //sleep for 5 seconds
                Thread.sleep(5000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            downloadStatus.setText(R.string.download_completed);
            return null;
        }
        
    }
}

Ok Go ahead and install the app and click on the download button.

Opps! Your app would crash with following error.

04-12 16:05:51.241: E/AndroidRuntime(11317): FATAL EXCEPTION: AsyncTask #1
04-12 16:05:51.241: E/AndroidRuntime(11317): Process: com.opensourceforgeeks.asynctaskdemo, PID: 11317
04-12 16:05:51.241: E/AndroidRuntime(11317): java.lang.RuntimeException: An error occured while executing doInBackground()
....
04-12 16:05:51.241: E/AndroidRuntime(11317): Caused by: android.view.ViewRootImpl$CalledFromWrongThreadException: Only the original thread that created a view hierarchy can touch its views.
....

But what happened here?

Note : Only the thread that created and rendered a View can alter it and that thread is main UI thread. As I mentioned before Async Task starts a new thread and doInBackground method essentially runs in that thread. So you cannot update any of the UI elements from this thread. You can only do that from UI thread.

Next natural question would be how will that be possible? Is there a callback? Well, there are. There are other methods in AsyncTask that we can override to leverage it. For example methods like onPreExecute(), onProgressUpdate() and onPostExecute() run in UI thread and can be used to update the UI elements.

So let us make that changes. Few changes that I am going to make in the upcoming code - 

• Instead of making thread sleep for 5 seconds in one go I am going to make thread sleep 5 times each for 1 sec to show how onProgressUpdate() methods works.
• I will override onPreExecute(), onProgressUpdate(), and onPostExecute()  to show how they work. Typically I will update the downloadStatus TextView that we had in above code with corresponding download status.

Code is as follows -

package com.opensourceforgeeks.asynctaskdemo;

import android.app.Activity;
import android.os.AsyncTask;
import android.os.Bundle;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.Button;
import android.widget.TextView;
/**
 * @author athakur
 */
public class MainActivity extends Activity {

    Button downloadButton;
    TextView downloadStatus;
    
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        downloadButton = (Button) findViewById(R.id.download_button);
        downloadStatus = (TextView) findViewById(R.id.downlaodStatus);
        downloadButton.setOnClickListener(new OnClickListener() {
            
            @Override
            public void onClick(View v) {
                new DownloadAsynctask().execute("TestInput");
            }
        });
    }
    
    class DownloadAsynctask extends AsyncTask<String, Integer, Boolean> {    

        @Override
        protected void onPreExecute() {
            super.onPreExecute();
            downloadStatus.setText(R.string.download_progress);
        }

        @Override
        protected void onProgressUpdate(Integer... values) {
            super.onProgressUpdate(values);
            downloadStatus.setText(getString(R.string.download_progress) + " : " + values[0] + "%");
        }

        @Override
        protected Boolean doInBackground(String... params) {
            
            //sleep for 5 seconds
            for (int i=0;i<5;i++) {
                try {
                    Thread.sleep(1000);
                    publishProgress((i+1)*20);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                    return false;
                }
            }

            return true;
        }
        

        @Override
        protected void onPostExecute(Boolean result) {
            super.onPostExecute(result);
            if(result) {
                downloadStatus.setText(R.string.download_completed);
            }
            else {
                downloadStatus.setText(R.string.download_incompleted);
            }
            
        }
        
    }
}

And as expected this time your application should not crash and you should see following set of screens on click of download button -

Note : Skipping few screenshots here. You should see update progress at 20%, 40%, 60% and 80%.

Few important points

• Notices the generics in Aysnctask definition? -  class DownloadAsynctask extends AsyncTask<String, Integer, Boolean>. 
• Here first generic argument String is the type of argument that you will give in execute() method when starting async task. We have used new DownloadAsynctask().execute("TestInput"); This is the argument that doInBackground() methods receives - protected Boolean doInBackground(String... params)
• Next generic argument in Integer. It is basically what you would receive in 
  onProgressUpdate() method. The argument in this method will be array of generic value specified as 2nd generic argument. You can see protected void onProgressUpdate(Integer... values) 
• Last we have Boolean. This is basically the value that 
  doInBackground() methods returns to onPostExecute() methods. You can see - return true; in the doInBackground method and protected void onPostExecute(Boolean result).
•  doInBackground() may call publishProgress() as we have used above which will make a call back to onProgressUpdate() to update the progress of background thread.

Related Links

• Android life cycle methods

Difference between functions and methods.

Background

Let me be honest here. I always thought they are the same and used them interchangeably but as it turns out they are not. Each has it's significance or scope.  I stumbled on this question today when I started with python again after a long time. I simply googled "methods in python" and the results on different pages  were using different terminology. And that's when it struck me that methods and functions are indeed different and we should be careful to use each in proper context. I will tell what to use it context of few programming languages but lets start with the understanding process.

I believe programming can be fun.  Above is my 1st meme so please bear with me :)

Difference between Functions and Methods

 Ever tried to explain anyone what a method or function is ? You may have read some definition back in college. Anyway lets try to explain that first.

So a function has a name (called as function name). There it has the data that is passed to it that the function can operate on (called arguments). There arguments are generally provided in parenthesis "()" adjacent to the function name. And lastly there is method body that has the logic to execute when this code module is called. If you ask about a method you may probably answer the same way. 

But there is a difference. 

• Functions are standalone where as method are associated with an Object. When you call a method it is in scope of that object which has it's own blueprint (called class).  
• Another difference is that all arguments in function are explicit where as method had one implicit argument and that is reference to the object that the method belongs to. For example methods in Java can use this  in the method body or in python you can use self.

Just think of "Methods" as object oriented way of calling "Functions.

Note :  I can understand if you argue back saying what about static methods in Java? They are not associated with any instance and we definitely cannot use this in static methods. So are they still methods? Well answer is Yes. You will still call them methods. As I said "Methods" are object oriented way of calling "Functions". I know this is a lame argument but that is what it is. Do let me know in comments if you have some other view point. Would love to discuss this :)

Example in Python

I will give example in python due to which this all started for me. We can have both functions and methods in python. Hope it helps to understand this better.

Functions in python :

def test(name, place="India") :
    """ This is a test function"""
    print("Hi " + name + "! Welcome to " + place)


test("Aniket", "USA")
test("John")

Methods in python : 

class TestClass:
    def test(self, name, place="India"):
        """ This is a test method """
        print("Hi " + name + "! Welcome to " + place)

testObj = TestClass()
testObj.test("Aniket", "USA")
testObj.test("John")
 

And you should see output as (in both cases) : 

General Terminology

• For Java, there are only methods.
• For C, there are only functions.
• For C++ or Python it would depend on whether or not you're in a class.

Note : I know I have labelled this post as "programming concept" but it is more of a terminology use. There were many instance when I answered on Stack overflow using these words interchangeably and ended up corrected by someone and each time I am like "What.... how does it matter?". Well it does .. specially when you are in a circle of programmers/developers. So better late than never :)  you know now!

Related Links

• Difference between a method and a function(SO)
• Python Functions
• What is a “method” in Python?(SO)
• Functional programming(Wiki)

Difference between JSPs and Servlets

Background

Both JSPs and Servlets are server side technologies based on Java to generate web content dynamically that is then rendered on the requesting client which is typically your web browser. In this post we will see How JSPs work and what is exactly the difference between JSPs and Servlets.

Java Servlet

Servlet is a Java module that listens for HTTP requests like GET, POST etc. Once request is received by the servlet it will return appropriate response for it. A server code would look like - 

import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;
/**
* @author athakur
*/
public class TestServlet extends HttpServlet {

  public void init() throws ServletException
  {
      //Servlet initialization code go here
  }

  public void doGet(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException
  {
      response.setContentType("text/html");
      PrintWriter out = response.getWriter();
      out.println("<b>Hello World!</b>");
  }
  
  public void destroy()
  {
      // Any code to be executed when servlet is destroyed
  }
}

and your web.xml will have mapping of this servlet and the URL the servlet handles - 

    <servlet>
        <servlet-name>TestServletName</servlet-name>
        <servlet-class>TestServlet</servlet-class>
    </servlet>

    <servlet-mapping>
        <servlet-name>TestServletName</servlet-name>
        <url-pattern>/HelloWorld</url-pattern>
    </servlet-mapping> 

So when ever tomcat server receives a request for URL like http://localhost:8080/projectName/HelloWorld you should see response as Hello World! in bold on a webpage that is rendered on your web browser.

As I said before think of a servlet as Java module that intercepts HTTP request and provides the response for it. It is like HTML in Java .

JSP pages

JSP  (Java server pages) - They are like Java in HTML. A simple JSP would look like below - 

<html>
<head><title>Test JSP</title></head>
<body>
  <%
    String greeting = "Hello World!";
  %>
  <b><%= greeting %></b>
</body>
</html>

As you can see it is  HTML content with embedded Java logic. Lets now see how JSPs actually work?

How JSPs work?

1. When JSP container gets request for a JSP (Request URL would be something like http://localhost:8080/projectName/greetings.jsp) it checks whether JSP is compiled or not. If it is compiled it check for last modification time. If JSP page is modified after the JSP was previously compiled then JSP will be compiled again.
2. Before compilation JSP is actually converted to a java servlet. This generated content is in Java just like a normal servlet we saw above. This servlet is then compiled into a class file.
3. Request is then forwarded to this generated servlet which then handles the request and returns the response.

 The relevant generated servlet content in step 2 above for JSP same page that I have described above would be something like -

out.write("<html>\r\n  ");
String greeting = "Hello World!";
out.write("<b>");
out.print( greeting );
out.write("</b> \r\n");
out.write("</html>\r\n");

 You should be able to see the generated java file under 

• work\Catalina\localhost\ProjectName\greetings.java

Related Links

• Java JSP vs servlet?(SO)
• What is the difference between JSF, Servlet and JSP?(SO)