Wednesday, November 20, 2013

Storing Centralized Mobile Data with Parse and Kinvey

Storing centralized data is important to mobile apps and simple websites alike but many developers don't have the resources to set up their own middle and back end tiers. Independent mobile developers especially can benefit from cheap and simple storage options that they don't need to manage themselves. I've recently learned about several back end as a service providers that allow developers to skip maintaining their own middle tier and databases and store data straight in a third party through API calls. While some might balk at giving someone else full control over storage, the services can allow for great cost savings over setting everything up yourself. Additionally, most providers support both mobile apps and websites coded in different languages. The usage, pricing and features of the providers varies certainly vary so its helpful to compare to see what might be a good fit. I myself have been seeking a good storage option for some of my Android mobile efforts so I thought I'd take a look at a couple of the options available today.

Parse

Parse is a backend data service that has SDKs for mobile devices and web sites alike. It has a free tier for up to a million data requests or pushes a month and a $199 dollar a month paid tier in addition to an Enterprise tier. It is pretty easy to get up and running.

The first step is to get set up on Parse's website with an account and download the SDK. You can register a project and get your application id and client key that you will need in the code. Once your all set up you are ready to code. You start by initializing Parse with the app id and key:

Parse.initialize(this, "myappid","mykey");
Once initialized you can start accessing data. We'll start by creating a simple user account and signing in:
ParseUser user = new ParseUser();
user.setUsername("Tim");
user.setPassword("Tester");
user.setEmail("tim@test.com");

user.signUpInBackground(new SignUpCallback() {
    @Override
    public void done(ParseException arg0) {
        if (arg0 == null) {
            //sign up was successful
        }
    }
});

Parse methods have synchronous and asynchronous methods so you can also call the standard signUp method if you don't care for multi-threading. Once you have user accounts created you can let users log in. Once again this has synchronous and asynchronous versions:

ParseUser.logInInBackground("Tim", "Tester", new LogInCallback() {
    public void done(ParseUser user, ParseException e) {
        if (user != null) {
            //do work here
        } else {
            //log in failed          }
        }
    }
});   

If you aren't storing the current user and need to get it at a later point you can call getCurrentUser. You can also log them out the current user without having a pointer to them with the logout function:

ParseUser user = ParseUser.getCurrentUser();
ParseUser.logout();

Storing Objects

Next we will look at how data can be stored and retrieved. This can be done with or without user accounts. Objects are created using ParseObject and giving a name for the specific object you want to create. You can then add as many fields to it as you like. The following demonstrates saving user reviews and attaching them to the logged in user:

ParseUser user = ParseUser.getCurrentUser();
ParseObject review1 = new ParseObject("Review");
review1.put("title", "Sam's Restaurant");
review1.put("content", "It was great.");
review1.put("user", user); //this association is optional
review1.save();

ParseObject review2 = new ParseObject("Review");
review2.put("title", "Mary's Diner");
review2.put("content", "Service was slow.");
review2.put("user", user); //this association is optional
review2.save();

We now have two objects called Review associated with the current user. The association allows us to easily retrieve the specific user's records:

ParseQuery query = ParseQuery.getQuery("Review");
query.whereEqualTo("user", user);
List reviews = query.find();
for (ParseObject review : reviews) {
    String title = review.get("title");
    ...
}

If we wanted to further narrow down results from the query we could add additional filters. Retrieved objects can of course be updated by calling save or saveInBackground:

review.put("content", "Not so good."); //update content
review.saveInBackground();

Finally you can delete an object by using delete or deleteInBackground:

review.deleteInBackground();

User Security

Another feature of Parse is the ability to set the security on objects. For example, if we only want to allow the logged in user read and write access to a specific review, we can do this:

ParseUser user = ParseUser.getCurrentUser();
ParseACL acl = new ParseACL();
acl.setReadAccess(user, true);
acl.setWriteAccess(user, true);
review.setACL(acl); //apply the control list to the review
review.save();

You can add as many users to the read and write permissions on an ACL as you like. You can also call setPublicReadAccess(true) or setPublicWriteAccess(true) if you want to allow all users to have one of those permissions.

Browsing Data

A nice feature of Parse is an online data browser allowing you to track your access and your data. You can access it by logging on to the parse home page and choosing Access Dashboard. Once inside you can view users and objects, ACLs, notifications and information on how often data is accessed.

There is lots more to try of course but this is just to demonstrate the basics. Their useful android guide can be found here.

Kinvey

Kinvey offers a lot of similar features to parse but has a more generous pricing structure centered around the number of users. Like Parse it is a great starting point for simple projects. The first step is to register on the Kinvey site and download the SDK. Like Parse, Kinvey supports many different clients such as BackBone, HTML5 and Node.js. Once you have the SDK and your app key and secret, you can create your client object:

String appKey = "xxxx";
String appSecret = "xxxxx";
final Client kinveyClient = new Client.Builder(appKey, appSecret, this.getApplicationContext()).build();

After creating our client, we need to call logout to make sure no default user is logged in:

kinveyClient.user().logout().execute();

Now we can create user's similar to Parse. This will also log the user in which is why its important to call logout first:

kinveyClient.user().create("Tim", "Tester", new KinveyUserCallback() {
    public void onFailure(Throwable t) {
        //log error
    }
    public void onSuccess(User u) {
        //user code here            
    }
});

To log the user in in the future we can call the login function:

kinveyClient.user().login("Tim", "Tester", new KinveyUserCallback() {
    public void onFailure(Throwable t) {
        //log error
    }
    public void onSuccess(User u) {
        //user code here             
    }
}); 

Storing Data

Data is stored in Kinvey by creating objects that extend from GenericJson. You list your fields as private and then can access them like dictionary objects using get and put methods. You can optionally use the Key annotation to specify the exact name of the field you want it to be stored in:

public class ReviewEntity extends GenericJson {
    @Key("Title")
    private String title;
    private String content;
}

To create and save the object, we can do the following:

ReviewEntity review1 = new ReviewEntity();
review1.put("title", "Corner Bistro");
review1.put("content", "Interior was dirty.");
AsyncAppData reviews = kinveyClient.appData("reviewCollection",ReviewEntity.class);
  
reviews.save(review1, new KinveyClientCallback() {
    @Override
    public void onFailure(Throwable e) {
        //log error here
    }

    @Override
    public void onSuccess(ReviewEntity r) {
    }
});

Once an object is saved, a string id is automatically created and can be fetched with get("_id"). You can also specify your own property to map to it by using the @Key("_id") annotation when defining your entity class. This property is helpful if you need to retrieve an object based on its id.

Querying

You can easily fetch data from a collection by using the entities id string in the getEntity call or by creating a Query and calling get. The following shows a simple query to get back a review:

Query query = kinveyClient.query();
query.equals("title","Corner Bistro");
reviews.get(query, new KinveyListCallback() {
    @Override
    public void onSuccess(ReviewEntity result[]) {
        //use restaurant here
    }
    @Override
    public void onFailure(Throwable error) { 
       //handle error here
    }
});

You can also skip the query parameter and get all the records back. To retrieve a specific object, you can call getEntity and specify the objects string id.

User Security

Similar to Parse you can also set up access control lists to allow read or write access to data. If we want to apply read permissions on our review entity, we can modify our class like this:

public class ReviewEntity extends GenericJson {

    private String title;
    private String content;

    @Key("_acl")
    private KinveyMetaData.AccessControlList acl;
 
    public KinveyMetaData.AccessControlList getACL() {
        return acl;
    }
 
    public ReviewEntity() {
        acl = new KinveyMetaData.AccessControlList();
    }
}

Now when creating a review, we can specify that only our current user is to be able to read it. After logging in, user holds Tim:

ReviewEntity review1 = new ReviewEntity();
review1.put("title", "Secured Review");
review1.put("content", "Only viewable by Tim");
ArrayList users = new ArrayList();
users.add(user.getId());
review1.getACL().setGloballyReadable(false);
review1.getACL().setRead(users);

Now we will be able to query this object like we did above. If we log in with someone else and search for that title, we will get an empty list. Kinvey supports a lot more as you can imagine. For more useful things to do you can refer to this site.

Conclusion

So this was a basic look at Kinvey and Parse. Both are great for providing data backends mobile apps that do not have the infrastructure or ability to manage their own database. There are several more out there such as StackMob or StorageRoom and which is best for you may likely come down to pricing and usage terms.

Friday, November 15, 2013

Handling Activity Re-creation in Android Fragments

I like using ActionBar tabs in Android for simple applications where I want to quickly switch between two or three views. Fragments are easy to create and switch between. One of the challenges I have found however is properly handling orientation changes or activity pauses. Since activities are recreated when display changes occur, I have seen artifacts such as tabs drawing on top of each other and data getting erased for certain kinds of controls. I have found a couple of things to help me provide a consistent user experience.

First, its very important to have a properly coded Tab Listener. The listener is what controls how tabs are added or removed to their parent based on their selection. Its most important of course to attach and detach but I found that its important to first look for the tab in the fragment manager and using it in place of the local variable if its found. Performing this step instead of just checking to see if the local variable is set prevented issues of tabs drawing on top of each other:

public class TabListener implements ActionBar.TabListener {
    private android.app.Fragment mFragment;  
    private final Activity mActivity;  
    private final String mTag;
    private Class mClass;
 
    public TabListener(Activity activity, String tag, Class cl) {  
        mActivity = activity;  
        mTag = tag;  
        mClass = cl;
    }  

    public void onTabSelected(Tab tab, android.app.FragmentTransaction ft) {
        Fragment preInitFragment = mActivity.getFragmentManager().findFragmentByTag(mTag);
        if ((mFragment == null) && (preInitFragment == null)) { 
     mFragment = android.app.Fragment.instantiate(mActivity, mClass.getName());
     ft.add(android.R.id.content, mFragment, mTag);         
        } else if (mFragment != null) {
            ft.attach(mFragment);
        }
        else if (preInitFragment != null) {
            ft.attach(preInitFragment);
            mFragment = preInitFragment;
        } 
    }  

    public void onTabUnselected(Tab tab, android.app.FragmentTransaction ft) {
 if (mFragment != null) {  
     // Detach the fragment, because another one is being attached  
     ft.detach(mFragment);  
        }  
    }  

    public void onTabReselected(Tab tab, android.app.FragmentTransaction ft) {  
    }
}

So nothing fancy above, just an extra check to make use of the existing tab if its found.

A second issue I have with fragments (and activities in general) is that they get recreated when orientation changes happen. Controls that might get filled programmatically such as a listbox you might be filling will not automatically reload with the last value like a text box will. Thankfully, fragments support overriding onSaveInstanceState and onActivityCreated (similar to onRestoreInstanceState) so that custom values can be stored and retrieved. For example I had a list of items you could add to from a separate list that I wanted to keep track of. The following shows how you can store an array of integers representing the indexes of the selected items and then reload them in to the adapter when the activity is recreated:

    @Override
    public void onSaveInstanceState(Bundle savedInstanceState) {
     super.onSaveInstanceState(savedInstanceState);

        ArrayList<Integer> indexList = new ArrayList<Integer>();
        for (int i=0; i < contactListAdapter.getCount(); i++) {
            int id = contactListAdapter.getItem(i).getId();
            //get its index
            for (int counter=0; counter < masterList.size(); counter++) {
         if (masterList.get(counter).getId() == id) {
                    indexList.add(counter);
                    break;
                }
            }
        }
        savedInstanceState.putIntegerArrayList("contactList", indexList);
    }


    @Override
    public void onActivityCreated(Bundle savedInstanceState) {
     super.onActivityCreated(savedInstanceState);
     
     if ((savedInstanceState != null) && (savedInstanceState.containsKey("contactList"))) {
            ArrayList<Integer> indexes = savedInstanceState.getIntegerArrayList("contactList");
            contactListAdapter.clear();
            for (int index : indexes) {
         contactListAdapter.addItem(masterList.get(index));
            }
     }
    }

It is also possible of course to write your own parcelable class to store the actual selected data from the adapter so as not to have to build an index list but I thought this was a little easier. Once re-created you are good to go.

Saturday, November 9, 2013

Setting up the Cubieboard 2 with Linux

The Raspberry Pi has become very popular among hobbyists and has given rise to many similar boards. One such is the Cubieboard. The Cubieboard is a single board computer made in China using an AllWinner system on a chip. The original was released in 2012 with a single Cortex A8 ARM chip clocked at 1GHz, 512 MB of RAM, a 10/100 megabit ethernet port and 4 GB of NAND flash RAM with Android pre-installed. In 2013, the Cubieboard 2 was released containing 1 GB of RAM and a dual core Cortex A8 followed by the Cubietruck which added 2GB of RAM and gigabit ethernet. A nice feature of all Cubieboards is the presence of a SATA connector, allowing for better disk performance compared to the flash or micro sd memory cards. Also, they are all under $100 with the original costing around $49, the Cubieboard 2 going for about $69 and the latest version costing around $89. I purchased a Cubieboard 2 to play with finding it really good bang for the buck.

The Cubieboard is roughly the size of a credit card and won't take up much space on a desk. The board comes with a USB to DC connctor which I hooked up to 2 amp USB wall charger I bought so I would not need to rely on another computer to power it. There are 2 USB ports so its easy to hook it up to a USB mouse and keyboard. It also features an HDMI out for connecting to monitors. You can power up the board as is and run Android 4 after connecting everything or you can replace the OS with something else. There are two ways you can accomplish this. First you can use Berryboot to write an Operating System to a micro sd card and then boot from the card as described here. Additionally you can flash the NAND memory that comes with it and replace the default Operating System. I chose the latter option.

Flashing the Cubieboard 2

The first step I took was to download and install PhoenixSuit from the CubieBoard site to my Windows 7 PC. This included a driver for recognizing the Cubieboard on my PC. I then downloaded a Linux image from the site to flash on to the board. I had trouble finding one that PhoenixSuit would accept and ultimately settled on Lubuntu desktop 1.06 from here. I then hooked up the Cubieboard to my PC using a micro USB to USB cable which holding down the FEL button at the same time (located right under the micro USB port). After it loaded the driver and recognized the board, I then disconnected it and launched PhoenixSuit.

At the top of the screen is a menu with an item labeled Firmware. I clicked on that and then browsed to the image I downloaded (lubuntu-desktop-nand.img.gz). I then reconnected the Cubieboard to the PC while holding down the FEL button and the program automatically started to flash the device. At the popup that initially appears, I chose Yes. After about 10 minutes or so, the job was done and I disconnected the board.

Running Lubuntu

Unfortunately as of right now I cannot get the HDMI output to work with Lubuntu. Instead, using my PC I used Putty to connect to it using Cubieboard2 as the address. The default user name and password are linaro and linaro. The first step I took was to set the IP address to something static to make it easier to work with other devices on the network. I edited the /etc/network/interfaces file to change the eth0 option to the following:

iface eth0 inet static
    address 192.168.1.50
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.1

Then I restarted the networking service:

sudo service networking restart

After this I decided to try out the SATA port. I purchased a 320 GB 2.5 drive and a cheap external rack to mount it in. After hooking up the drive using the provided SATA cable, I checked my devices using fdisk and saw it listed as /dev/sda:

sudo fdisk -l

I then used fdisk to create a single partition on the drive:

sudo fdisk /dev/sda

At the prompt I entered n to tell it to create a new partition. I then chose primary with partition number of 1. I left the default values for the sectors blank so it would use the entire disk. At the main prompt I entered w to write the partition table to the disk. When this was done I exited.

With the partition created I was then able to format and mount the drive. I created an ext3 partition mounted at /storage with the following:

sudo mkfs.ext3 -b 4096 /dev/sda1
sudo mount /dev/sda1 /storage

Finally I added the following entry to /etc/fstab so it would mount after booting:

/dev/sda1   /storage   ext3    defaults 1   1

There is still plenty more to try with this board. Many distributions are available for Berryboot that may allow for a better experience and there is help available on the forums. This is an easy starting point though and will hopefully encourage others to try out these boards.

Tuesday, November 5, 2013

Creating Mobile Webpages with Django

Supporting both mobile and desktop browsers is an important part of any live website. A mobile webpage should be simpler and display the bare minimum of necessary information in a small space. Frameworks like ASP.NET MVC 4 have built in features that allow for detection of mobile devices so you can display different pages depending on how the site is being viewed. I wanted to look at some options for doing this in Django.

Django-mobile 0.3.0

This package is easy to install and doesn't require many steps to be used. Once working it adds a property to request object called flavour that you can query to see if its coming from a mobile device or not. The package can be found here. Setup is easy as described on the page itself. I used pip to install it and then added the necessary changes to settings.py:

TEMPLATE_LOADERS = (
    'django_mobile.loader.Loader',
    ...

TEMPLATE_CONTEXT_PROCESSORS = (
    'django_mobile.context_processors.flavour',
)

MIDDLEWARE_CLASSES = (
    ...
    'django_mobile.middleware.MobileDetectionMiddleware',
    'django_mobile.middleware.SetFlavourMiddleware',

INSTALLED_APPS = (
    ...
    'django_mobile',

Now when defining our function in views.py, we can simply look for the new property and pass it on.

def home(request):
  mobilePage = 0
  if request.flavour == "mobile":
    mobilePage = 1

  ...
  context = { 'mobilePage': mobilePage }
  return render(request, 'home', context)

Now we can use the property in our template. We can render custom tags inside the control statements to change the whole experience of the page based on the device:

<html>
<body>
    {% if mobilePage == 1 %}
    Welcome to the mobile version.
    {% else %}
    Welcome to the regular version.
    {% endif %}
</body>
</html>

Django-mobi

Another easy to use package is called Django-mobi and can be found here. Simply put the package inside your project and you can easily reference it in your code using an attribute. A bonus to this package is that it has the ability to exempt certain devices from being detected as described on their home page. To use the package we just need to add it to settings.py:

MIDDLEWARE_CLASSES = (
    ...
    'mobi.MobileDetectionMiddleware',
Now we can use it in our code. In practice, it works just like the example above:

from mobi.decorators import detect_mobile
...
@detect_mobile
def home(request):
    mobilePage = 0
    if request.mobile:
        mobilePage = 1

    context = { 'mobilePage': mobilePage }
    return render(request, 'home', context)
These are two decent options I found for customizing Django sites. There could be other ways as well but I felt these worked pretty nicely.

Thursday, October 3, 2013

Building Sites Quickly with Node.js and AngularJS

There are increasingly many options for getting sites up and running quickly. This week I thought I'd combine two options that I have worked with previously, Node.js and AngularJS. Using my ec2 server I decided to work out a few demonstrations.

At its simplest, Node.js acts as an HTML module that can serve simple content to a caller and provide consistent connections for messaging or chatting. Through the power of its modules, it can be used to build powerful sites. We'll start by creating a simple one using the express module. Express is a great module that builds on the connect module, giving us features for creating simple or complex sites as well as for creating rest services.

Creating a Simple Page

If it's not installed already, Node.js can be installed using:

sudo yum install npm

Once installed, the first step is to create a directory for your project. Then add a file called package.json to list our dependencies. It can look like this:

 {
  "name": "TestSite",
  "description": "My Test Site",
  "version": "0.0.1",
  "private": true,
  "dependencies": {
   "express": "3.4",
   "jade": "0.35.0"
  }
 }

Once that is created, run:

npm install This will fetch our dependencies. You could also run these separately:
npm install express
npm install jade
Now we are ready to create some simple content. First create a file called index.html. It can be simple and look like this:
<html>
<body>
  <div>
    <h3>Hello from Node.js</h3>
  </div>
</body>
</html>

Now we can create our server side javascript. We create a file called server.js which looks like this:

var express = require("express");
var app = express();
app.use(express.static(__dirname));
app.listen(8888);

Here we tell node to use the express module and we create a new express object. We then tell it to serve static content from our base directory with the __dirname keyword and then listen on port 8888. Now we can start the node server with the following:

node server.js

You will now be able to call your index.html page in the root of your server at port 8888 for example:

http://mytestsite.com:8888/index.html

Adding Asynchronous Calls with AngularJS

Now that we have a simple page hosted, we can add javascript and asynchronous calls to it. To accomplish this we will use AngularJS. AngularJS is an MVVM framework in javascript that makes it easy to bind data and events to your views, fetch data asynchronously, validate your UIs and a lot more. We can get it on our server like this:

wget https://ajax.googleapis.com/ajax/libs/angularjs/1.0.8/angular.min.js

To use it, we will first modify our index.html page. Using extra attributes in our HTML elements we can tell angular what controller we want to hook up to and what events and model items our controls map to:

<html ng-app>
<head>
  <script src="angular.min.js"></script>
  <script src="index.js"></script>
</head>
<body>
  <h3>Hello from Node.js</h3>
  <div ng-controller="TestController">
    <span ng-bind="txtdata"></span><br/>
    <button ng-click="getData()">Get Message</button>
  </div>
</body>
</html>

Here we took our original page and added some mark up. The ng-app label tells it this uses AngularJS while the ng-controller attribute specifies what controller to use. To give dynamic content to our span tag, we use ng-bind. Finally, we use ng-click to link a button to an event in the controller. We will use this event to fill data in to the span tag. Once the modifications are done we can create our javascript that will contain the AngularJS code. Inside index.js we put:

function TestController($scope,$http) {
  $scope.txtdata = "";
  $scope.getData = function() {
    $http.get('/getdata/').success(function(result) {
      $scope.txtdata = result.message;
    });
  }
}

This code defines our controller TestController which matches what we are looking for in our HTML. We define the data we want to bind to, txtdata, and the function getData that our button click binds to. Inside the function we call a url on our root site /getdata which we will add to node js next. The result is then stored in the txtdata variable. To allow for the /getdata call to the server side, we'll modify our server.js file. We'll simulate an object to return but it could just as easily come from a database call:

function MyCustomObject() {
  this.message = "Test messsage";
}

var myObj = new MyCustomObject();
 
app.get('/getdata', function(req, res) {
  res.send(myObj);
});

This code allows us to listen on ourpath/getdata and returns a json object with the message property. Now you can run the node server again and re-test the index.html file. Clicking the button should force a call back to the node server and return the json with the message property.

Using Jade

Jade is a templating engine that express can use to render markup as HTML and is easy to use. To duplicate the index.html file from above, we can create index.jade in our root directory and fill it like this:

doctype 5
html(lang="en" ng-app)
  body
    h3 Hello from Node.js
    #container(ng-controller="TestController")
      span(ng-bind="txtdata")
      br
      button(ng-click="getData()").
        Get Text

    script(src='angular.min.js')
    script(src='index.js')

This markup will create all the HTML tags for us and add the attributes listed in the parentheses. At the end we include our javascript files from earlier though they could also be placed in the header. To hookup the jade rendering so we can display this, we can add the following to our server.js file:

app.set('view engine', 'jade');
app.set('views', __dirname);
app.get('/index', function(req, res) {
  res.render('index.jade');
});

Now if we run this and go to our root path /index it will render a page indentical to index.html with a working angular controller.

So hopefully this shows how quick and powerful Node.js and AngularJS are. For fans of javascript, they are a nice starting point for quickly getting a site up and running. This demonstration can easily be extending by using a modules such as Mongoose or Helenus to connect to databases.

Wednesday, September 25, 2013

Securing Your Django Rest Service with OAuth2

Once you have a rest service that you want others to access, you probably want to secure it. There are many ways this can be done but I thought I'd look in to the OAuth authentication standard. OAuth is a specification specifically for Authorization and provides patterns for distributing access tokens that can be checked to protect server side resources.

For example, in a simple case, you may have one server trying to access a service or resource from another service. First, the calling server would create an account on the server owning the service and obtain its own client id and client secret which would need to be stored securely. Once it has its own secret, it can send its id and secret to the owning service for an authorization check and receive an access token in exchange. That access token can then be used to retrieve data from the server.

In a more complex example, a mobile app may want to pull data from a rest service. A mobile app cannot truly store a client secret and consider it private so extra steps need to be taken. To accomplish this, an account would need to exist on the server owning the service for any user that wants access. The mobile app would first direct the user to a log in page on the remote server owning the service where they either log in or create an account there if one doesn't exist. Once they have an account and enter their credentials, the server will redirect back to a url specified by the app and pass it an authorization code. The app would listen for the redirect and take the access code from the redirect. Now the app can call the server again passing its client id, client secret and access code and get an oauth token back. Now the mobile app can then use this token to make requests to the service. The use of a log in page and redirect url are important as we don't want the mobile app to need to store the username or password for security reasons and we don't want to pass back the authorization code to just anyone. It is usally recommended to somehow register the redirect url with the remote server to prevent third parties from passing random urls that they control.

For more details on these scenarios, refer to the spec here.

For this tutorial we will focus on the simple case of a website or service requesting data from another service. Our consumer service is assumed to be securely storing its client id and secret. As OAuth2 is just a specification, there are multiple implementation libraries out there. For this demonstration we will use the django-oauth2-provider (https://github.com/caffeinehit/django-oauth2-provider). Assuming you have pip, you can install it like this:

sudo pip install django-oauth2-provider

Once we have the provider installed, we want to allow django to use it. Modify settings.py by adding to INSTALLED_APPS:

INSTALLED_APPS = (
   ...
   'provider',
   'provider.oauth2',
   ...
)

Now we want to add the following entry to urls.py:

...
url(r'^oauth2/', include('provider.oauth2.urls', namespace = 'oauth2')),
...

This code sets up a url to route token requests for oauth2. We will use that shortly. First we need to set up a client id and secret that we can use to request a token. For this we will use the admin console and a backend datastore to store our information. In my case, django is configured to use my local MySql database. Make sure your database is running and enter the console like this:

python manage.py shell

Now in the shell you can enter your script to create a new user, associate it with a client and get the clients id and secret:

from provider.oauth2.models import Client
from django.contrib.auth.models import User
user = User.objects.create_user('john', 'john@djangotest.com', 'abcd')
c = Client(user=user, name="mysite client", client_type=1, url="http://djangotest.com")
c.save()

Then you can use c.client_id and c.client_secret to get the information for the new client. In this case, the id is:

'c513118ee3b176805722'
and the secret is:
'd4e5bca2996c8c543349cf0ce140bcd73c86450c'

With this information we can now make calls to the url we set up above to get a token. For example, if your url is http://johnssite.com and django is running on port 80 then you would send a post or get request to:

http://johnssite.com/oauth2/accesstoken

The data to post would look like this:

client_id=c513118ee3b176805722&client_secret=d4e5bca2996c8c543349cf0ce140bcd73c86450c&grant_type=password&username=john&password=abcd&scope=write

This should return a token you can use for your requests.

Now we can fix our service to make use of incoming tokens. The first step is to make sure https is on for your service. This is important to protect the information you are sending to the authorization service. Next we will modify our service to expect authentication. We can add the following code to what we previously had in views.py for our rest service:

import datetime
from provider.oauth2.models import AccessToken
...
if request.method == 'GET':
    restaurants = dbconn['restaurants']
    rests = []

    key = request.META.get('HTTP_AUTHORIZATION')
    if not key:
        return Response({ 'Error': 'No token provided.'})
    else:
        try:
            token = AccessToken.objects.get(token=key)
            if not token:
                return Response({'Error': 'Access denied.'})

            if token.expires < datetime.datetime.now():
                return Response({'Error': 'Token has expired.'})
        except AccessToken.DoesNotExist, e:
            return Response({'Error': 'Token does not exist.'})

     ...

In this code we first take the token that should have been passed in to the authorization header. If you are expecting it as an argument, you can modify the code accordingly. If we have a token, we next check that it exists in the token store that issued it and then check that it didn't expire. If all tests pass, the code flow continues to get the data and return it.

So that's it. Using the django-oauth2-provider library makes it pretty straight forward to create a token authorization service. With only a little extra effort we added basic security to our site.

Thursday, January 10, 2013

Communicating With Hadoop Through RabbitMQ

For those who have a need to start hadoop jobs from other systems, RabbitMQ is an easy to use possibility. RabbitMQ is an open source and freely available messaging system that works with many languages such as Java, Ruby, Python and C#. Its simple to install and connect machines across a network. We'll start by setting up RabbitMQ on the linux machine that will run our Hadoop job. Install guides can be found here: http://www.rabbitmq.com/download.html. For ubuntu Linux, I simply used the following command:

sudo apt-get install rabbitmq-server

You can check that the broker is installed and running using this command:

rabbitmqctl status

The broker's job is to listen for messages coming from this or other machines and route them to the proper queues. If the broker is not running, your machine will not be able to communicate with other queues. Once you have determined the service is running, you can create your listener program. The listener's job is to wait for messages sent to our queue and then start the appropriate job upon receipt. Your listening program can be as fancy as you like, for testing purposes you can just create a basic Java application with a main entry point. Before writing the program you will need to download the libraries from here:

http://www.rabbitmq.com/java-client.html

When creating your program you will need to add references to the jar files: rabbitmq-client.jar, commons-cli-1.1.jar and commons-io-1.2.jar. To listen to a message queue doesn't take much code. First we set up our connection to our local broker:

    ConnectionFactory factory = new ConnectionFactory();
    factory.setHost("localhost");
    Connection connection = factory.newConnection();
    Channel channel = connection.createChannel();
Next we can specify our queue to listen to which will be created if it doesn't already exist.
    //create the queue if it doesn't already exist
    channel.queueDeclare(queueName, false, false, false, null);            
    QueueingConsumer consumer = new QueueingConsumer(channel);
    //listen to the queue
    channel.basicConsume(queueName, true, consumer);
Finally we can retrieve the message like this:
    QueueingConsumer.Delivery delivery = consumer.nextDelivery();
    String message = new String(delivery.getBody());

Once we retrieve the message we can determin a course of action. In this case, we wish to start our job using the tool runner:

ToolRunner.run(new Configuration(), new MyMapReduceJob(), args);

The full client code looks like this:

    try
    {
        String queueName = "TESTQUEUE";
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("localhost");
        Connection connection = factory.newConnection();
        Channel channel = connection.createChannel();

        channel.queueDeclare(queueName, false, false, false, null);
            
        QueueingConsumer consumer = new QueueingConsumer(channel);
        channel.basicConsume(queueName, true, consumer);

        while (true) {
            QueueingConsumer.Delivery delivery = consumer.nextDelivery();
            String message = new String(delivery.getBody());
            if (message.equalsIgnoreCase("runjob")) {
                int res = ToolRunner.run(new Configuration(), new MyMapReduceJob(), args);
                break;
            }
        }
    }
    catch (Exception ex) {
        System.out.println(ex.getMessage());
    }
Publishing Messages to Hadoop

Now with the client set up and listening we can create a sender. For this article we'll assume the message is being sent from .NET on a windows server. The first step is to install the RabbitMQ server. Instructions can be found here:

http://www.rabbitmq.com/install-windows.html

Currently this requires setting up Erlang first. Once the server is installed you can download the client library for .NET which can be found here:

http://www.rabbitmq.com/dotnet.html

Once everything is ready you can create a new C# console application to send messages. The code from .NET is similarly only a few lines. We create a connection using the connection factory which should point to the name of the server receiving the messages.

    ConnectionFactory factory = new ConnectionFactory();
    factory.Protocol = Protocols.FromEnvironment();
    factory.HostName = "UbuntuMachine";
    IConnection conn = factory.CreateConnection();

Next we create a model and bind to the queue.

    using (IModel model = conn.CreateModel())
    {
        model.ExchangeDeclare("exch", ExchangeType.Direct);
        model.QueueBind("TESTQUEUE", "exch", "key");
    }

Finally we send our message as a byte array.

    byte[] messageBody = Encoding.UTF8.GetBytes("runjob");
    model.BasicPublish("exch", "key", null, messageBody);

The final completed code looks like this:

    ConnectionFactory factory = new ConnectionFactory();
    factory.Protocol = Protocols.FromEnvironment();
    factory.HostName = "UbuntuMachine";
    using (IConnection conn = factory.CreateConnection())
    {
        using (IModel model = conn.CreateModel())
        {
            model.ExchangeDeclare("exch", ExchangeType.Direct);
            model.QueueBind("TESTQUEUE", "exch", "key");

            Console.WriteLine("Sending message.");
            byte[] messageBody = Encoding.UTF8.GetBytes("runjob");
            model.BasicPublish("exch", "key", null, messageBody);
        }
    }

And that is it. Now we can send messages to Hadoop from different environments using a simple message queue. This can be especially useful for distributed ETL system using tools such as SSIS or Pentaho.