Monday, July 16, 2018

Containerizing Django and MySQL with Kubernetes Part 2: Deploying with Minikube

In part 1 I created a simple site with a database and placed it in containers. In this part we'll use Minikube with Kubernetes to manage the containers locally. Kubernetes is a free and open source system for managing containers and Minikube is a utility to allow us to run a local Kubernetes setup. The first step is to install kubectl:
       
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl

Next we install Minikube:
       
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Now we need to install virtual box to work with our local Kubernetes:
       
sudo apt-get install virtualbox

Now we can start minikube. For simplicity, we want our docker commands to use the docker running in Minikube. The following will start Minikube and set environment variables to make this happen:
       
minikube start
eval $(minikube docker-env)

To deploy our custom Dockerfiles, we want to use the Docker running in Minikube so we want to rebuild the images so they reside in the Minikube's repository. From the django directory we can build it like before but without sudo:
       
docker build -t simplesite .

And then the same from the mysql directory:
       
docker build -t mysqlmain .

To deploy our containers to Minikube we will create yaml files that will be read by the kubectl utility. The yaml allow us to define Pods - groupings of one or more containers that we want to run together, Services - which allow us to define how to access our pods internally or externally, and Replication Sets - controllers that allow us to define how many instances of a Pod we want to run.

Before we define our Pods and Services, we want to utilize the secret functionality of Kubernetes to hold our database credentials so we don't have to store them in plain text in the yaml. We start by getting base64 encodings of our username and password:

       
echo -n 'testuser' | base64
dGVzdHVzZXI=
echo -n 'password' | base64
cGFzc3dvcmQ=
echo -n 'rootpass' | base64
cm9vdHBhc3M=

Then we create a new file called secret.yaml with the base64 values:
       
apiVersion: v1
kind: Secret
metadata:
  name: mysql-secret
data:
  username: dGVzdHVzZXI=
  password: cGFzc3dvcmQ=
  root_password: cm9vdHBhc3M=

Now we can pass this file to kubectl to store the credentials internally for use with the containers we'll create next.
       
kubectl create -f secret.yaml

Next we need a yaml file to create a MySQL pod and expose it as a service so that our main site can access it. We'll call it mysql-main.yaml make use of the root password secret:
       
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: mysqlmain
    role: db
  name: mysqlmain
spec:
  containers:
    - name: mysqlmain
      image: mysqlmain
      imagePullPolicy: IfNotPresent
      env:
          - name: MYSQL_ROOT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql-secret
                key: root_password
      ports:
        - containerPort: 3036
      volumeMounts:
        - mountPath: /mysqldata
          name: data
  volumes:
    - name: data
      emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: mysqlmain
    role: service
  name: mysqlmain
spec:
  ports:
    - port: 3306
      targetPort: 3306
      name: port1
  selector:
    role: db

Now we can create and start the pod and service:
       
kubectl create -f mysql-main.yaml

You can look up the service information here to ensure its running:
       
kubectl get services

Finally we can create the yaml for our website. We'll call it simplesite.yaml. This makes use of the username and password secrets we previously created:
       
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: simplesite
    role: master
  name: simplesite
spec:
  containers:
    - name: simplesite
      image: simplesite
      imagePullPolicy: IfNotPresent
      env:
        - name: DJANGO_SETTINGS_FILE
          value: "simplesite.settings_kubernetes"
        - name: MYSQL_HOST_DNS
          value: "mysqlmain.default.svc.cluster.local"
        - name: MAIN_DB_NAME
          value: "kubernetes_test"
        - name: MAIN_DB_USER
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: username
        - name: MAIN_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
      ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  labels:
    name: simplesite
    role: service
  name: simplesite
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
      name: port1
  selector:
    role: master
  type: NodePort

Now we can create and start the pod and service:
       
kubectl create -f simplesite.yaml

And to see that it is running:
       
kubectl get services

Now we want to get the url of the running site. You can use the following minikube to retrieve the url it is listening on:
       
minikube service simplesite --url

In my case its http://192.168.99.100:30620

You might get an error about the host not being authorized. In this case we will need to update the site's hosts setting to allow the hosts root. I left this in to demonstrate how to connect to a running container. You can access the running container with the following:

       
kubectl exec -it simplesite -- /bin/bash

And then edit the /simplesite/simplesite/settings.py file:
       
ALLOWED_HOSTS = ['*']

Then restart our service:
       
service uwsgi reload

Now we can exit the container and refresh the page in the browser. The final thing we will do is set up replicas of our website. You'll want to update the ALLOWED_HOSTS in the source settings file and rebuild the docker image for simplesite before continuing. For simplicity, we will use the Kubernetes Deployment object which is the recommended way to control containers. We will create a new yaml called simplesite-deployment.yaml:

       
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: simplesite
  labels:
    name: simplesite
spec:
  replicas: 3
  template:
    metadata:
      labels:
        name: simplesite
    spec:
      containers:
      - name: simplesite
        image: simplesite
        imagePullPolicy: IfNotPresent
        env:
          - name: DJANGO_SETTINGS_FILE
            value: "simplesite.settings_kubernetes"
          - name: MYSQL_HOST_DNS
            value: "mysqlmain.default.svc.cluster.local"
          - name: MAIN_DB_NAME
            value: "kubernetes_test"
          - name: MAIN_DB_USER
            valueFrom:
              secretKeyRef:
                name: mysql-secret
                key: username
          - name: MAIN_DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql-secret
                key: password
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: simplesite
  labels:
    name: simplesite
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    name: simplesite

This file will create 3 containers of our site and a service to load balance access to them. Before running it we should stop the existing service:
       
kubectl delete services simplesite
kubectl delete pods simplesite

And now we can create our new containers:
       
kubectl create -f simplesite-deployment.yaml

We can again retrieve the url of the service with:
       
minikube service simplesite --url

We now have a load balanced site with a MySQL backend. Further steps to improve this would include using a volume to persist the MySQL data in case the container is re-created later.

Tuesday, June 26, 2018

Containerizing Django and MySQL with Kubernetes Part 1

I've been fidding with Docker and Docker Swarm a bit for the past couple of years and wanted to see how easy it was to use Kubernetes. I thought I'd start with a simple Django project I could deploy locally using Minikube. The tutorial uses Django 1.10 and 18.03.1-ce. Assuming a blank slate, we can start by installing Docker.

sudo apt-get update

sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo apt-key fingerprint 0EBFCD88

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

sudo apt-get update

sudo apt-get install docker-ce

Next we can create a Django app to use for testing Docker. The app will allow us to add names to a local database.

sudo mkdir /kubernetestest
sudo chown dan:dan /kubernetestest
cd /kubernetestest

django-admin startproject simplesite

Next I created a models directory in the app directory with a people.py file with the following:

from django.db import models

class People(models.Model):
    class Meta:
        db_table = "people"
        managed = True
    people_id = models.AutoField(primary_key=True)
    name = models.CharField(max_length=50)

The main page code in views.py page looks like this:

from models.people import People

def index(request):
    people = []
    try:
        #if posting, add a new person to the database
        if request.method == 'POST':
            People.objects.create(name=request.POST['name'])

        #fetch all names from the database to display on the page
        people = People.objects.all()
    except Exception as ex:
        pass

    return render(request, "index.html", { "people": people })

And the template looks like this:

<html>
<body>
    Current Values
    <ul>
    {% for person in people %}
        <li>{{person.name}}</li>
    {% endfor %}
    </ul>

    <form method="POST">
        {% csrf_token %}
        Add a value:<br/>
        <input type="text" name="name"/><br/>
        <input type="submit" value="Add"/>
    </form>

</body>
</html>

For the initial run, I created a sqlite database so the site can be run locally:


python manage.py makemigrations simplesite
python manage.py sqlmigrate simplesite 0001
python manage.py migrate

Now we can run our server and test it:

python manage.py runserver

Now that the basic app is runable as a site, we can think about how to containerize. I wanted the containerized solution to use NGINX and uWSGI instead of just the built in Django webserver so Docker would need to set them up complete with config files. In a production solution you might use separate containers for these services but I combined them with Django for simplicity. I created a directory parallel to the Django website called setupscripts with the basic uWSGI and NGINX config files. The uWSGI config sits in simplesite.ini:


[uwsgi]
touch-reload = /tmp/simplesite
socket = 127.0.0.1:3031
enable-threads = true
single-interpreter = true
chmod-socket = 770
chown-socket www-data:www-data
workers = 15
uid = www-data
gid = www-data
chdir = /simplesite
wsgi-file=/simplesite/simplesite/simplesite.wsgi
for-readline = /simplesite/uwsgi_vars.ini
  env = %(_)
endfor =
module = django.core.handlers.wsgi:WSGIHander()
buffer-size = 16384

The environment variables that uWSGI will pass to Django will come from a config file that will be created when the container starts. The NGINX settings are in a file called nginx-default:

server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;

        location / {
          include uwsgi_params;
          uwsgi_pass 127.0.0.1:3031;
        }

}

Finally we need a wsgi file in our Django app for uWSGI to use. The simplesite.wsgi file goes in the main app directory:

import os, sys
import json
import socket

apache_configuration = os.path.dirname(__file__)
project = os.path.dirname(apache_configuration)
workspace = os.path.dirname(project)
toplevel = os.path.dirname(workspace)
sys.path.append(workspace)
sys.path.append(project)
sys.path.append(toplevel)

#if not set by uwsgi, default to base settings
if not "DJANGO_SETTINGS_MODULE" in os.environ:
    os.environ["DJANGO_SETTINGS_MODULE"] = "simplesite.settings"

if "MYSQL_HOST_DNS" in os.environ:
    os.environ["MAIN_DB_HOST"] = socket.gethostbyname(os.environ["MYSQL_HOST_DNS"])
else:
    os.environ["MAIN_DB_HOST"] = "localhost"

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

The file makes sure the app defaults to the base settings file if no alternate settings file is supplied in the environment variables. The actual Dockerfile to house the Djano app sits in the root project directory along with manage.py. It looks like this:

FROM ubuntu:latest

RUN apt-get update
RUN apt-get install -y python python-dev python-pip build-essential
RUN apt-get install -y nginx uwsgi uwsgi-plugin-python curl nano net-tools
RUN apt-get install -y libmysqlclient-dev python-mysqldb

RUN /usr/bin/pip install Django==1.10.8

RUN mkdir /simplesite && chown www-data:www-data /simplesite

ADD setupscripts/init.sh /tmp/init.sh
ADD setupscripts/nginx-default /etc/nginx/sites-available/default
ADD setupscripts/simplesite.ini /etc/uwsgi/apps-available/simplesite.ini
ADD manage.py /simplesite
COPY simplesite /simplesite/simplesite

RUN ln -s /etc/uwsgi/apps-available/simplesite.ini /etc/uwsgi/apps-enabled/simplesite.ini
RUN touch /tmp/simplesite

RUN ["chmod", "+x", "/tmp/init.sh"]
ENTRYPOINT ["/tmp/init.sh"]

CMD ["nginx", "-g", "daemon off;"]

Now we need to create the init.sh file that the container will call when it starts. The file goes in the setupscripts directory and creates the file with the needed uWSGI variables from environment variables and then runs NGINX:

#!/bin/bash

chown -R www-data:www-data /simplesite

touch /simplesite/uwsgi_vars.ini

service uwsgi restart

exec "$@"

Finally we can build our docker container:

sudo docker build -t simplesite .

To test it:


sudo docker run -d -p 8095:80 simplesite

To run it, go to http://localhost:8095/. You should be able to add names to the database using the text entry box on the form.

The next step is to set up our MySQL database so we can use it in place of sqlite. I created a new Dockerfile in a parallel directory to the Django project root:

FROM mysql:5.7.15

ADD schema.sql /docker-entrypoint-initdb.d

EXPOSE 3306

Next I created a schema.sql file in the same directory to handle the base setup for our new db:

create database kubernetes_test;
create user 'testuser'@'%' identified by 'password';
grant all privileges on kubernetes_test.* to 'testuser'@'%' with grant option;
use kubernetes_test;
create table people (people_id integer not null auto_increment primary key, name varchar(50));

Now we can create the docker image from the directory where the Dockerfile is:


sudo docker build -t mysqlmain .

Now we need to update our Django container to be able to optionally use a MySQL database. I created a new settings file in the Django app and called it settings_docker.py:

from settings import *
import os

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.mysql",
        "NAME": os.getenv("MAIN_DB_NAME"),
        "USER": os.getenv("MAIN_DB_USER"),
        "PASSWORD": os.getenv("MAIN_DB_PASSWORD"),
        "HOST": os.getenv("MAIN_DB_HOST"),
    }
}

Environment variables allow us to not hard code any assumptions about the database. The init.sh file can now be updated to set the variables our Django container will need.

#!/bin/bash

chown -R www-data:www-data /simplesite

touch /simplesite/uwsgi_vars.ini

echo "MYSQL_HOST_DNS=$MYSQL_HOST_DNS" >>/simplesite/uwsgi_vars.ini
echo "MAIN_DB_NAME=$MAIN_DB_NAME" >>/simplesite/uwsgi_vars.ini
echo "MAIN_DB_USER=$MAIN_DB_USER" >>/simplesite/uwsgi_vars.ini
echo "MAIN_DB_PASSWORD=$MAIN_DB_PASSWORD" >>/simplesite/uwsgi_vars.ini

if [ -z "$DJANGO_SETTINGS_FILE" ]; then
    echo "DJANGO_SETTINGS_MODULE=simplesite.settings" >>/simplesite/uwsgi_vars.ini
    python /simplesite/manage.py migrate
    chown www-data:www-data /simplesite/db.sqlite3
else
    echo "DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_FILE" >>/simplesite/uwsgi_vars.ini
fi

service uwsgi restart

exec "$@"

We can now re-create our docker image:


sudo docker build -t simplesite .

Next we will use docker compose to test them out together. You can install docker compose with the following if you don't already have it:


sudo curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Now we can create a docker-compose.yml file in a separate directory:

version: '3'
services:
  db:
    image: "mysqlmain"
    environment:
     - MYSQL_ROOT_PASSWORD=password
  web:
    image: "simplesite"
    ports:
     - "8095:80"
    links:
     - db:mysqlmain
    environment:
     - DJANGO_SETTINGS_FILE=settings_docker
     - MYSQL_HOST_DNS=mysqlmain
     - MAIN_DB_NAME=kubernetes_test
     - MAIN_DB_USER=testuser
     - MAIN_DB_PASSWORD=password

This file will set some environment variables for us such as the settings file Django should use and the credentials to connect to our database. To create our containers, run:

sudo docker-compose up

And now the site should again be reachable at http://localhost:8095/. In the part 2 I'll show how to use kubernetes locally to manage the containers.

Wednesday, November 20, 2013

Storing Centralized Mobile Data with Parse and Kinvey

Storing centralized data is important to mobile apps and simple websites alike but many developers don't have the resources to set up their own middle and back end tiers. Independent mobile developers especially can benefit from cheap and simple storage options that they don't need to manage themselves. I've recently learned about several back end as a service providers that allow developers to skip maintaining their own middle tier and databases and store data straight in a third party through API calls. While some might balk at giving someone else full control over storage, the services can allow for great cost savings over setting everything up yourself. Additionally, most providers support both mobile apps and websites coded in different languages. The usage, pricing and features of the providers varies certainly vary so its helpful to compare to see what might be a good fit. I myself have been seeking a good storage option for some of my Android mobile efforts so I thought I'd take a look at a couple of the options available today.

Parse

Parse is a backend data service that has SDKs for mobile devices and web sites alike. It has a free tier for up to a million data requests or pushes a month and a $199 dollar a month paid tier in addition to an Enterprise tier. It is pretty easy to get up and running.

The first step is to get set up on Parse's website with an account and download the SDK. You can register a project and get your application id and client key that you will need in the code. Once your all set up you are ready to code. You start by initializing Parse with the app id and key:

Parse.initialize(this, "myappid","mykey");
Once initialized you can start accessing data. We'll start by creating a simple user account and signing in:
ParseUser user = new ParseUser();
user.setUsername("Tim");
user.setPassword("Tester");
user.setEmail("tim@test.com");

user.signUpInBackground(new SignUpCallback() {
    @Override
    public void done(ParseException arg0) {
        if (arg0 == null) {
            //sign up was successful
        }
    }
});

Parse methods have synchronous and asynchronous methods so you can also call the standard signUp method if you don't care for multi-threading. Once you have user accounts created you can let users log in. Once again this has synchronous and asynchronous versions:

ParseUser.logInInBackground("Tim", "Tester", new LogInCallback() {
    public void done(ParseUser user, ParseException e) {
        if (user != null) {
            //do work here
        } else {
            //log in failed          }
        }
    }
});   

If you aren't storing the current user and need to get it at a later point you can call getCurrentUser. You can also log them out the current user without having a pointer to them with the logout function:

ParseUser user = ParseUser.getCurrentUser();
ParseUser.logout();

Storing Objects

Next we will look at how data can be stored and retrieved. This can be done with or without user accounts. Objects are created using ParseObject and giving a name for the specific object you want to create. You can then add as many fields to it as you like. The following demonstrates saving user reviews and attaching them to the logged in user:

ParseUser user = ParseUser.getCurrentUser();
ParseObject review1 = new ParseObject("Review");
review1.put("title", "Sam's Restaurant");
review1.put("content", "It was great.");
review1.put("user", user); //this association is optional
review1.save();

ParseObject review2 = new ParseObject("Review");
review2.put("title", "Mary's Diner");
review2.put("content", "Service was slow.");
review2.put("user", user); //this association is optional
review2.save();

We now have two objects called Review associated with the current user. The association allows us to easily retrieve the specific user's records:

ParseQuery query = ParseQuery.getQuery("Review");
query.whereEqualTo("user", user);
List reviews = query.find();
for (ParseObject review : reviews) {
    String title = review.get("title");
    ...
}

If we wanted to further narrow down results from the query we could add additional filters. Retrieved objects can of course be updated by calling save or saveInBackground:

review.put("content", "Not so good."); //update content
review.saveInBackground();

Finally you can delete an object by using delete or deleteInBackground:

review.deleteInBackground();

User Security

Another feature of Parse is the ability to set the security on objects. For example, if we only want to allow the logged in user read and write access to a specific review, we can do this:

ParseUser user = ParseUser.getCurrentUser();
ParseACL acl = new ParseACL();
acl.setReadAccess(user, true);
acl.setWriteAccess(user, true);
review.setACL(acl); //apply the control list to the review
review.save();

You can add as many users to the read and write permissions on an ACL as you like. You can also call setPublicReadAccess(true) or setPublicWriteAccess(true) if you want to allow all users to have one of those permissions.

Browsing Data

A nice feature of Parse is an online data browser allowing you to track your access and your data. You can access it by logging on to the parse home page and choosing Access Dashboard. Once inside you can view users and objects, ACLs, notifications and information on how often data is accessed.

There is lots more to try of course but this is just to demonstrate the basics. Their useful android guide can be found here.

Kinvey

Kinvey offers a lot of similar features to parse but has a more generous pricing structure centered around the number of users. Like Parse it is a great starting point for simple projects. The first step is to register on the Kinvey site and download the SDK. Like Parse, Kinvey supports many different clients such as BackBone, HTML5 and Node.js. Once you have the SDK and your app key and secret, you can create your client object:

String appKey = "xxxx";
String appSecret = "xxxxx";
final Client kinveyClient = new Client.Builder(appKey, appSecret, this.getApplicationContext()).build();

After creating our client, we need to call logout to make sure no default user is logged in:

kinveyClient.user().logout().execute();

Now we can create user's similar to Parse. This will also log the user in which is why its important to call logout first:

kinveyClient.user().create("Tim", "Tester", new KinveyUserCallback() {
    public void onFailure(Throwable t) {
        //log error
    }
    public void onSuccess(User u) {
        //user code here            
    }
});

To log the user in in the future we can call the login function:

kinveyClient.user().login("Tim", "Tester", new KinveyUserCallback() {
    public void onFailure(Throwable t) {
        //log error
    }
    public void onSuccess(User u) {
        //user code here             
    }
}); 

Storing Data

Data is stored in Kinvey by creating objects that extend from GenericJson. You list your fields as private and then can access them like dictionary objects using get and put methods. You can optionally use the Key annotation to specify the exact name of the field you want it to be stored in:

public class ReviewEntity extends GenericJson {
    @Key("Title")
    private String title;
    private String content;
}

To create and save the object, we can do the following:

ReviewEntity review1 = new ReviewEntity();
review1.put("title", "Corner Bistro");
review1.put("content", "Interior was dirty.");
AsyncAppData reviews = kinveyClient.appData("reviewCollection",ReviewEntity.class);
  
reviews.save(review1, new KinveyClientCallback() {
    @Override
    public void onFailure(Throwable e) {
        //log error here
    }

    @Override
    public void onSuccess(ReviewEntity r) {
    }
});

Once an object is saved, a string id is automatically created and can be fetched with get("_id"). You can also specify your own property to map to it by using the @Key("_id") annotation when defining your entity class. This property is helpful if you need to retrieve an object based on its id.

Querying

You can easily fetch data from a collection by using the entities id string in the getEntity call or by creating a Query and calling get. The following shows a simple query to get back a review:

Query query = kinveyClient.query();
query.equals("title","Corner Bistro");
reviews.get(query, new KinveyListCallback() {
    @Override
    public void onSuccess(ReviewEntity result[]) {
        //use restaurant here
    }
    @Override
    public void onFailure(Throwable error) { 
       //handle error here
    }
});

You can also skip the query parameter and get all the records back. To retrieve a specific object, you can call getEntity and specify the objects string id.

User Security

Similar to Parse you can also set up access control lists to allow read or write access to data. If we want to apply read permissions on our review entity, we can modify our class like this:

public class ReviewEntity extends GenericJson {

    private String title;
    private String content;

    @Key("_acl")
    private KinveyMetaData.AccessControlList acl;
 
    public KinveyMetaData.AccessControlList getACL() {
        return acl;
    }
 
    public ReviewEntity() {
        acl = new KinveyMetaData.AccessControlList();
    }
}

Now when creating a review, we can specify that only our current user is to be able to read it. After logging in, user holds Tim:

ReviewEntity review1 = new ReviewEntity();
review1.put("title", "Secured Review");
review1.put("content", "Only viewable by Tim");
ArrayList users = new ArrayList();
users.add(user.getId());
review1.getACL().setGloballyReadable(false);
review1.getACL().setRead(users);

Now we will be able to query this object like we did above. If we log in with someone else and search for that title, we will get an empty list. Kinvey supports a lot more as you can imagine. For more useful things to do you can refer to this site.

Conclusion

So this was a basic look at Kinvey and Parse. Both are great for providing data backends mobile apps that do not have the infrastructure or ability to manage their own database. There are several more out there such as StackMob or StorageRoom and which is best for you may likely come down to pricing and usage terms.

Friday, November 15, 2013

Handling Activity Re-creation in Android Fragments

I like using ActionBar tabs in Android for simple applications where I want to quickly switch between two or three views. Fragments are easy to create and switch between. One of the challenges I have found however is properly handling orientation changes or activity pauses. Since activities are recreated when display changes occur, I have seen artifacts such as tabs drawing on top of each other and data getting erased for certain kinds of controls. I have found a couple of things to help me provide a consistent user experience.

First, its very important to have a properly coded Tab Listener. The listener is what controls how tabs are added or removed to their parent based on their selection. Its most important of course to attach and detach but I found that its important to first look for the tab in the fragment manager and using it in place of the local variable if its found. Performing this step instead of just checking to see if the local variable is set prevented issues of tabs drawing on top of each other:

public class TabListener implements ActionBar.TabListener {
    private android.app.Fragment mFragment;  
    private final Activity mActivity;  
    private final String mTag;
    private Class mClass;
 
    public TabListener(Activity activity, String tag, Class cl) {  
        mActivity = activity;  
        mTag = tag;  
        mClass = cl;
    }  

    public void onTabSelected(Tab tab, android.app.FragmentTransaction ft) {
        Fragment preInitFragment = mActivity.getFragmentManager().findFragmentByTag(mTag);
        if ((mFragment == null) && (preInitFragment == null)) { 
     mFragment = android.app.Fragment.instantiate(mActivity, mClass.getName());
     ft.add(android.R.id.content, mFragment, mTag);         
        } else if (mFragment != null) {
            ft.attach(mFragment);
        }
        else if (preInitFragment != null) {
            ft.attach(preInitFragment);
            mFragment = preInitFragment;
        } 
    }  

    public void onTabUnselected(Tab tab, android.app.FragmentTransaction ft) {
 if (mFragment != null) {  
     // Detach the fragment, because another one is being attached  
     ft.detach(mFragment);  
        }  
    }  

    public void onTabReselected(Tab tab, android.app.FragmentTransaction ft) {  
    }
}

So nothing fancy above, just an extra check to make use of the existing tab if its found.

A second issue I have with fragments (and activities in general) is that they get recreated when orientation changes happen. Controls that might get filled programmatically such as a listbox you might be filling will not automatically reload with the last value like a text box will. Thankfully, fragments support overriding onSaveInstanceState and onActivityCreated (similar to onRestoreInstanceState) so that custom values can be stored and retrieved. For example I had a list of items you could add to from a separate list that I wanted to keep track of. The following shows how you can store an array of integers representing the indexes of the selected items and then reload them in to the adapter when the activity is recreated:

    @Override
    public void onSaveInstanceState(Bundle savedInstanceState) {
     super.onSaveInstanceState(savedInstanceState);

        ArrayList<Integer> indexList = new ArrayList<Integer>();
        for (int i=0; i < contactListAdapter.getCount(); i++) {
            int id = contactListAdapter.getItem(i).getId();
            //get its index
            for (int counter=0; counter < masterList.size(); counter++) {
         if (masterList.get(counter).getId() == id) {
                    indexList.add(counter);
                    break;
                }
            }
        }
        savedInstanceState.putIntegerArrayList("contactList", indexList);
    }


    @Override
    public void onActivityCreated(Bundle savedInstanceState) {
     super.onActivityCreated(savedInstanceState);
     
     if ((savedInstanceState != null) && (savedInstanceState.containsKey("contactList"))) {
            ArrayList<Integer> indexes = savedInstanceState.getIntegerArrayList("contactList");
            contactListAdapter.clear();
            for (int index : indexes) {
         contactListAdapter.addItem(masterList.get(index));
            }
     }
    }

It is also possible of course to write your own parcelable class to store the actual selected data from the adapter so as not to have to build an index list but I thought this was a little easier. Once re-created you are good to go.

Saturday, November 9, 2013

Setting up the Cubieboard 2 with Linux

The Raspberry Pi has become very popular among hobbyists and has given rise to many similar boards. One such is the Cubieboard. The Cubieboard is a single board computer made in China using an AllWinner system on a chip. The original was released in 2012 with a single Cortex A8 ARM chip clocked at 1GHz, 512 MB of RAM, a 10/100 megabit ethernet port and 4 GB of NAND flash RAM with Android pre-installed. In 2013, the Cubieboard 2 was released containing 1 GB of RAM and a dual core Cortex A8 followed by the Cubietruck which added 2GB of RAM and gigabit ethernet. A nice feature of all Cubieboards is the presence of a SATA connector, allowing for better disk performance compared to the flash or micro sd memory cards. Also, they are all under $100 with the original costing around $49, the Cubieboard 2 going for about $69 and the latest version costing around $89. I purchased a Cubieboard 2 to play with finding it really good bang for the buck.

The Cubieboard is roughly the size of a credit card and won't take up much space on a desk. The board comes with a USB to DC connctor which I hooked up to 2 amp USB wall charger I bought so I would not need to rely on another computer to power it. There are 2 USB ports so its easy to hook it up to a USB mouse and keyboard. It also features an HDMI out for connecting to monitors. You can power up the board as is and run Android 4 after connecting everything or you can replace the OS with something else. There are two ways you can accomplish this. First you can use Berryboot to write an Operating System to a micro sd card and then boot from the card as described here. Additionally you can flash the NAND memory that comes with it and replace the default Operating System. I chose the latter option.

Flashing the Cubieboard 2

The first step I took was to download and install PhoenixSuit from the CubieBoard site to my Windows 7 PC. This included a driver for recognizing the Cubieboard on my PC. I then downloaded a Linux image from the site to flash on to the board. I had trouble finding one that PhoenixSuit would accept and ultimately settled on Lubuntu desktop 1.06 from here. I then hooked up the Cubieboard to my PC using a micro USB to USB cable which holding down the FEL button at the same time (located right under the micro USB port). After it loaded the driver and recognized the board, I then disconnected it and launched PhoenixSuit.

At the top of the screen is a menu with an item labeled Firmware. I clicked on that and then browsed to the image I downloaded (lubuntu-desktop-nand.img.gz). I then reconnected the Cubieboard to the PC while holding down the FEL button and the program automatically started to flash the device. At the popup that initially appears, I chose Yes. After about 10 minutes or so, the job was done and I disconnected the board.

Running Lubuntu

Unfortunately as of right now I cannot get the HDMI output to work with Lubuntu. Instead, using my PC I used Putty to connect to it using Cubieboard2 as the address. The default user name and password are linaro and linaro. The first step I took was to set the IP address to something static to make it easier to work with other devices on the network. I edited the /etc/network/interfaces file to change the eth0 option to the following:

iface eth0 inet static
    address 192.168.1.50
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    gateway 192.168.1.1

Then I restarted the networking service:

sudo service networking restart

After this I decided to try out the SATA port. I purchased a 320 GB 2.5 drive and a cheap external rack to mount it in. After hooking up the drive using the provided SATA cable, I checked my devices using fdisk and saw it listed as /dev/sda:

sudo fdisk -l

I then used fdisk to create a single partition on the drive:

sudo fdisk /dev/sda

At the prompt I entered n to tell it to create a new partition. I then chose primary with partition number of 1. I left the default values for the sectors blank so it would use the entire disk. At the main prompt I entered w to write the partition table to the disk. When this was done I exited.

With the partition created I was then able to format and mount the drive. I created an ext3 partition mounted at /storage with the following:

sudo mkfs.ext3 -b 4096 /dev/sda1
sudo mount /dev/sda1 /storage

Finally I added the following entry to /etc/fstab so it would mount after booting:

/dev/sda1   /storage   ext3    defaults 1   1

There is still plenty more to try with this board. Many distributions are available for Berryboot that may allow for a better experience and there is help available on the forums. This is an easy starting point though and will hopefully encourage others to try out these boards.

Tuesday, November 5, 2013

Creating Mobile Webpages with Django

Supporting both mobile and desktop browsers is an important part of any live website. A mobile webpage should be simpler and display the bare minimum of necessary information in a small space. Frameworks like ASP.NET MVC 4 have built in features that allow for detection of mobile devices so you can display different pages depending on how the site is being viewed. I wanted to look at some options for doing this in Django.

Django-mobile 0.3.0

This package is easy to install and doesn't require many steps to be used. Once working it adds a property to request object called flavour that you can query to see if its coming from a mobile device or not. The package can be found here. Setup is easy as described on the page itself. I used pip to install it and then added the necessary changes to settings.py:

TEMPLATE_LOADERS = (
    'django_mobile.loader.Loader',
    ...

TEMPLATE_CONTEXT_PROCESSORS = (
    'django_mobile.context_processors.flavour',
)

MIDDLEWARE_CLASSES = (
    ...
    'django_mobile.middleware.MobileDetectionMiddleware',
    'django_mobile.middleware.SetFlavourMiddleware',

INSTALLED_APPS = (
    ...
    'django_mobile',

Now when defining our function in views.py, we can simply look for the new property and pass it on.

def home(request):
  mobilePage = 0
  if request.flavour == "mobile":
    mobilePage = 1

  ...
  context = { 'mobilePage': mobilePage }
  return render(request, 'home', context)

Now we can use the property in our template. We can render custom tags inside the control statements to change the whole experience of the page based on the device:

<html>
<body>
    {% if mobilePage == 1 %}
    Welcome to the mobile version.
    {% else %}
    Welcome to the regular version.
    {% endif %}
</body>
</html>

Django-mobi

Another easy to use package is called Django-mobi and can be found here. Simply put the package inside your project and you can easily reference it in your code using an attribute. A bonus to this package is that it has the ability to exempt certain devices from being detected as described on their home page. To use the package we just need to add it to settings.py:

MIDDLEWARE_CLASSES = (
    ...
    'mobi.MobileDetectionMiddleware',
Now we can use it in our code. In practice, it works just like the example above:

from mobi.decorators import detect_mobile
...
@detect_mobile
def home(request):
    mobilePage = 0
    if request.mobile:
        mobilePage = 1

    context = { 'mobilePage': mobilePage }
    return render(request, 'home', context)
These are two decent options I found for customizing Django sites. There could be other ways as well but I felt these worked pretty nicely.

Thursday, October 3, 2013

Building Sites Quickly with Node.js and AngularJS

There are increasingly many options for getting sites up and running quickly. This week I thought I'd combine two options that I have worked with previously, Node.js and AngularJS. Using my ec2 server I decided to work out a few demonstrations.

At its simplest, Node.js acts as an HTML module that can serve simple content to a caller and provide consistent connections for messaging or chatting. Through the power of its modules, it can be used to build powerful sites. We'll start by creating a simple one using the express module. Express is a great module that builds on the connect module, giving us features for creating simple or complex sites as well as for creating rest services.

Creating a Simple Page

If it's not installed already, Node.js can be installed using:

sudo yum install npm

Once installed, the first step is to create a directory for your project. Then add a file called package.json to list our dependencies. It can look like this:

 {
  "name": "TestSite",
  "description": "My Test Site",
  "version": "0.0.1",
  "private": true,
  "dependencies": {
   "express": "3.4",
   "jade": "0.35.0"
  }
 }

Once that is created, run:

npm install This will fetch our dependencies. You could also run these separately:
npm install express
npm install jade
Now we are ready to create some simple content. First create a file called index.html. It can be simple and look like this:
<html>
<body>
  <div>
    <h3>Hello from Node.js</h3>
  </div>
</body>
</html>

Now we can create our server side javascript. We create a file called server.js which looks like this:

var express = require("express");
var app = express();
app.use(express.static(__dirname));
app.listen(8888);

Here we tell node to use the express module and we create a new express object. We then tell it to serve static content from our base directory with the __dirname keyword and then listen on port 8888. Now we can start the node server with the following:

node server.js

You will now be able to call your index.html page in the root of your server at port 8888 for example:

http://mytestsite.com:8888/index.html

Adding Asynchronous Calls with AngularJS

Now that we have a simple page hosted, we can add javascript and asynchronous calls to it. To accomplish this we will use AngularJS. AngularJS is an MVVM framework in javascript that makes it easy to bind data and events to your views, fetch data asynchronously, validate your UIs and a lot more. We can get it on our server like this:

wget https://ajax.googleapis.com/ajax/libs/angularjs/1.0.8/angular.min.js

To use it, we will first modify our index.html page. Using extra attributes in our HTML elements we can tell angular what controller we want to hook up to and what events and model items our controls map to:

<html ng-app>
<head>
  <script src="angular.min.js"></script>
  <script src="index.js"></script>
</head>
<body>
  <h3>Hello from Node.js</h3>
  <div ng-controller="TestController">
    <span ng-bind="txtdata"></span><br/>
    <button ng-click="getData()">Get Message</button>
  </div>
</body>
</html>

Here we took our original page and added some mark up. The ng-app label tells it this uses AngularJS while the ng-controller attribute specifies what controller to use. To give dynamic content to our span tag, we use ng-bind. Finally, we use ng-click to link a button to an event in the controller. We will use this event to fill data in to the span tag. Once the modifications are done we can create our javascript that will contain the AngularJS code. Inside index.js we put:

function TestController($scope,$http) {
  $scope.txtdata = "";
  $scope.getData = function() {
    $http.get('/getdata/').success(function(result) {
      $scope.txtdata = result.message;
    });
  }
}

This code defines our controller TestController which matches what we are looking for in our HTML. We define the data we want to bind to, txtdata, and the function getData that our button click binds to. Inside the function we call a url on our root site /getdata which we will add to node js next. The result is then stored in the txtdata variable. To allow for the /getdata call to the server side, we'll modify our server.js file. We'll simulate an object to return but it could just as easily come from a database call:

function MyCustomObject() {
  this.message = "Test messsage";
}

var myObj = new MyCustomObject();
 
app.get('/getdata', function(req, res) {
  res.send(myObj);
});

This code allows us to listen on ourpath/getdata and returns a json object with the message property. Now you can run the node server again and re-test the index.html file. Clicking the button should force a call back to the node server and return the json with the message property.

Using Jade

Jade is a templating engine that express can use to render markup as HTML and is easy to use. To duplicate the index.html file from above, we can create index.jade in our root directory and fill it like this:

doctype 5
html(lang="en" ng-app)
  body
    h3 Hello from Node.js
    #container(ng-controller="TestController")
      span(ng-bind="txtdata")
      br
      button(ng-click="getData()").
        Get Text

    script(src='angular.min.js')
    script(src='index.js')

This markup will create all the HTML tags for us and add the attributes listed in the parentheses. At the end we include our javascript files from earlier though they could also be placed in the header. To hookup the jade rendering so we can display this, we can add the following to our server.js file:

app.set('view engine', 'jade');
app.set('views', __dirname);
app.get('/index', function(req, res) {
  res.render('index.jade');
});

Now if we run this and go to our root path /index it will render a page indentical to index.html with a working angular controller.

So hopefully this shows how quick and powerful Node.js and AngularJS are. For fans of javascript, they are a nice starting point for quickly getting a site up and running. This demonstration can easily be extending by using a modules such as Mongoose or Helenus to connect to databases.