Building a side project is fun. Launching it is fun. Improving and measuring it is fun. But marketing it is aweful!

Marketing your side project means you're not coding, instead you're walking around the interwebs with your pants down trying your hardest to get people to not only try your little project but to also get beyond that by tweeting about it, Facebook status update about it, blog about it or use whatever devices inside it to help the viral spread. Now that! ...is freckin hard.

I'm struggling to even get my best friends and my wife to even try my side projects. I can't blame them, unlike a lemonade stand at a farmers market it's very impersonal. When I tried to get my buddies to try Around The World several did but only very briefly and granted some few did give me feedback but it's really not much to go by.

So, today I'm launching the start of my new web marketing strategy: Begging

Or rather, politely asking people to help me. Instead of using the usual "we" or "our" language I'm referring to it in first person instead. The platform for this strategy experiment is on HUGEpic and it looks like this: hugepic.io/yourhelp/

I'm recently built a feature into HUGEpic that once a month emails everyone who uploaded a picture a little summary of their upload and the number of hits and comments and boldly in the footer of this email there's a link to the /yourhelp/ page (see screenshot above).

Let's see how this works out. Mostly likely it'll be just another noise in the highways of peoples' internet lifes but perhaps it can become successful too.

Mind you, the motives of all of this is for my "insert-sideproject-name-here" to become successful. And by successful I mean popular and lots of traffic. None of my side projects make me any money which makes it easier to beg. However, none of them make any money for the people I'm asking for help. Perhaps that's what could be the version 2.0 of my web marketing strategy.

New feature just landed! Now you can embed pictures from HUGEpic so it can be on your own site. See example below.

So to do this I opted for the simplest solution possible. It's basically just an iframe to the regular URL but with ?embedded=1 set. What this does is that it removes all buttons except the zoom navigation buttons. There are some other configurable things like like hide_download_counter=1|0 and hide_annotations=1|0. At the moment there's no UI to change these options but at least the functionality is there in case somebody wants it.

Here's what it can look like

One particular little feature I think is neat is that whilst your previewing your embedded code and you zoom in and pan around on your image, the position and zoom level is automatically inserted into the HTML code. The way this is done is by this pattern:

setInterval(pluck_position, 2 * 1000);

function pluck_position() {
    var url = iframe[0].contentWindow.location.href;
    var numbers = url.match(/\/([0-9\.]+)\/([-0-9\.]+)\/([-0-9\.]+)/g)[0];
    var zoom = parseFloat(numbers.split('/')[1]);
    var lat = parseFloat(numbers.split('/')[2]);
    var lng = parseFloat(numbers.split('/')[3]);
    ...
}

Code is here

To try it, click on any picture then click the little "Permalink" icon (the icon that looks like a chain link) in the upper right hand corner and follow the link that appears.

For example, here's an upload of a huge Minecraft world:

So here's my latest little fun side-project: HUGEpic.io http://hugepic.io

It's a web app for uploading massive pictures and looking at them like maps.

The advantages with showing pictures like this are:

• you only download what you need
• you can send a permanent link to a picture at a particular location with a particular zoom level
• you can draw annotations on a layer on top of the image

All the code is here on Github and as you can see it's a Tornado that uses two databases: MongoDB and Redis and when it connects to MongoDB it uses the new Tornado specific driver called Motor which is great.

Before I get to the juicy client side stuff, let me talk about something awesome in between Tornado and Javascript, namely: RQ
It's an awesomely simple python message queue that only works with Python, Redis and on UNIXy systems. All checks. My only real experience with message queues has honestly been with Celery which is also great but a right pain compared to RQ. With RQ, all I do is reduce the heavy tasks down to pure python functions. For example, make_thumbnail() then I simply do this:

from utils import make_thumbnail
from rq import Queue
queue = Queue(connection=self.redis)
job = q.enqueue(
    make_thumbnail,
    image,
    width,
    extension,
    self.application.settings['static_path']
)

and that's it. Starting rqworker on the command line (from somewhere where __import__('utils.make_thumbnail') makes sense) and we're off!

You might think that using a message queue is all fancy pants and just something I need to bother myself with because of Tornado's eventloop nature. But no, it's so much more than that. When a massive 5 Mb JPG is uploaded, a little algorithm is figuring out roughly how many zoom levels that can be used and what ALL 256x256 tiles are going to be for each resized version of the original. Then it needs to generate a thumbnail to represent that JPG and all the tiles and the thumbnail need to go through an optimizer (I'm using jpegoptim and optipng).
Lastly, to be able to serve all tiles from a fast CDN I have to upload every single tile to a Reduced Redundancy Amazon S3 storage that the Amazon CloudFront CDN is hooked up to. This might sometimes fail due to network hickups and must be resilient to continue where it left off.

All of that stuff takes a very long time but it's made it much easier and much more comfortable thanks to RQ.

Now, on the front end. The genesis inspiration to this was a library called Polymaps which isn't bad but when I later switched to Leaftlet I was blown away. It was lighter, smoother running and has an absolutely stunning API that even I could understand.

And that led me to find another amazingly neat library, for Leaflet, called Leaflet.draw which makes it really easy to add tools for drawing on the pictures. You can draw lines, rectangles, circles, polygons and drop markers. And for all of them it was relatively easy to bind cute popup bubbles so you can type in comments like this or this.

And lastly, there's Filepicker. It's a brilliant web service that simply takes care of your uploads. Uploading a 8 Mb JPG through a little file upload form not only takes an incredibly long time, it's fragile and has no good default UX. Filepicker takes care of all of that and makes it possible to upload files the way you want it. For example, if you use Google Drive to back up your massie pictures, Filepicker can handle that. And Dropbox. And Box. And of course, regular drag-and-drop uploads but with a lovely progress bar indicator and thumbnail preview.

There's also upload by simply entering a URL. So, try find a picture on Google Images click on one, then in the right-hand bar right-click the URL and "Copy Link Location" and paste that into HUGEpic to test.

So for a weekend project that has taken only a couple of weeks I'm quite proud. My hopes for big success is nil but it has been a great learning experience mixing interesting client-side programming, web programming and intereting CPU bound and networking challenges.

Oh, and did I mention it works great on mobile too? Even the file uploading part. Thanks to Filepicker.

Here are two perfectly good ways to turn 123456789 into "123,456,789":

import locale

def f1(n):
    locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
    return locale.format('%d', n, True)

def f2(n):
    r = []
    for i, c in enumerate(reversed(str(n))):
        if i and (not (i % 3)):
            r.insert(0, ',')
        r.insert(0, c)
    return ''.join(r)

assert f1(123456789) == '123,456,789'
assert f2(123456789) == '123,456,789'    

Which one do you think is the fastest?

Easy, write a benchmark:

from time import time

for f in (f1, f2):
    t0 = time()
    for i in range(1000000):
        f(i)
    t1 = time()
    print f.func_name, t1 - t0, 'seconds'

And, drumroll, the results are:

peterbe@mpb:~$ python benchmark.py
f1 19.4571149349 seconds
f2 6.30253100395 seconds

The f2 one looks very plain and a good candidate for PyPy:

peterbe@mpb:~$ pypy dummy.py
f1 14.367814064 seconds
f2 0.77246594429 seconds

...which is 800% speed boost which is cute. It's also kinda ridiculous that each iteration of f2 takes 0.0000008 seconds. What's that!?

An obvious albeit somewhat risky optimization on f1 is this:

import locale
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
def f1(n):
    return locale.format('%d', n, True)

...and now we get:

peterbe@mpb:~$ python dummy.py
f1 16.3811080456 seconds
f2 6.14097189903 seconds

Before you say it, yes I'm aware the locale can do much more but I was just curious and I scratched it.

UPDATE

Dave points out the built in function format (which was added in Python 2.6). So let's add it and kick ass!

def f3(i):
    return format(i, ',')

And we run the tests again:

peterbe@mpb:~$ python dummy.py
f1 16.4227910042
f2 6.13625884056
f3 0.892002105713
peterbe@mpb:~$ pypy dummy.py
f1 4.61941003799
f2 0.720993041992
f3 0.26224398613

There's your winner!

I'm quite fond of hastebin.com. It's fast. It's reliable. And it's got nice keyboard shortcuts that work for my taste.

So, I created a little program to quickly throw things into hastebin. You can have one too:

First create ~/bin/hastebinit and paste in:

#!/usr/bin/python

import urllib2
import os
import json

URL = 'http://hastebin.com/documents'

def run(*args):
    if args:
        content = [open(x).read() for x in args]
        extensions = [os.path.splitext(x)[1] for x in args]
    else:
        content = [sys.stdin.read()]
        extensions = [None]

    for i, each in enumerate(content):
        req = urllib2.Request(URL, each)
        response = urllib2.urlopen(req)
        the_page = response.read()
        key = json.loads(the_page)['key']
        url = "http://hastebin.com/%s" % key
        if extensions[i]:
            url += extensions[i]
        print url

if __name__ == '__main__':
    import sys
    sys.exit(run(*sys.argv[1:]))

Then run: chmod +x ~/bin/hastebinit

Now you can do things like:

$ cat ~/myfile | hastebinit
$ hastebinit < ~/myfile
$ hastebinit ~/myfile myotherfile

Hopefully it'll one day help at least one more soul out there!

So, the cool thing about Tornado the Python web framework is that it's based on a single thread IO loop. Aka Eventloop. This means that you can handle high concurrency with optimal performance. However, it means that can't do things that take a long time because then you're blocking all other users.

The solution to the blocking problem is to then switch to asynchronous callbacks which means a task can churn away in the background whilst your web server can crack on with other requests. That's great but it's actually not that easy. Writing callback code in Tornado is much more pleasant than say, Node, where you actually have to "fork" off in different functions with different scope. For example, here's what it might look like:

class MyHandler(tornado.web.RequestHandler):
    @asynchronous
    @gen.engine
    def get(self):
        http_client = AsyncHTTPClient()
        response = yield gen.Task(http_client.fetch, "http://example.com")
        stuff = do_something_with_response(response)
        self.render("template.html", **stuff)

It's pretty neat but it's still work. And sometimes you just don't know if something is going to be slow or not. If it's not going to be slow (e.g. fetching a simple value from a fast local database) you don't want to do it async anyway.

So on Around The World I have a whole Admin dashboard where I edit questions, upload photos and run various statistical reports which can be all very slow. Since the only user who is using the Admin is me, I don't care if it's not very fast. So, I don't have to worry about wrapping things like thumbnail pre-generation in asynchronous callback code. But I don't either want to block the rest of the app where every single request has to be fast. Here's how I solve that.

First, I start 4 different processors across 4 different ports:

127.0.0.1:10001
127.0.0.1:10002
127.0.0.1:10003
127.0.0.1:10004

Then, I decide that 127.0.0.1:10004 will be dedicated to slow blocking ops only used by the Admin dashboard.

In Nginx it's easy. Here's the config I used (simplified for clarity)

upstream aroundtheworld_backends {
    server 127.0.0.1:10001;
    server 127.0.0.1:10002;
    server 127.0.0.1:10003;
}

upstream aroundtheworld_admin_backends {
    server 127.0.0.1:10004;
}

server {
    server_name aroundtheworldgame.com;
    root /var/lib/tornado/aroundtheworld;
    ...

    try_files /maintenance.html @proxy;

    location /admin {
        proxy_pass http://aroundtheworld_admin_backends;
    }

    location @proxy {
        proxy_pass http://aroundtheworld_backends;
    } 

    ...
}

With this in play it means that I can write slow blocking code without worry about blocking other users than myself.

This might sound lazy and that I should use an asynchronous library for all my DB, net and file system access but mind you that's not without its own risks and trouble. Most of the DB access is built to be very very simple queries that are always really light and almost always done over and database index that is fully in RAM. Doing it like this I can code away without much complexity and yet never have to worry about making the site slow.

UPDATE

I changed the Nginx config to use the try_files directive instead. Looks nicer.

After about a month of weekend development the League of Friends is finally finished.

Usually on games like this, if it has a highscore list you might find yourself at number 3,405,912 and the people at the top of the highscore list are people you've never heard of so what's the point of comparing yourself with them?

On Around The World, you select your own friends for your league. Everyone you invite get an email asking if they want to accept it mutually. If you want to invite someone who isn't already on Around The World, you can type in their email address and complete an email that gets sent to that friend on your behalf from Around The World.

Also with this, you can click on any of your travelling friends and get lots more details about their progress. It doesn't reveal anything about how smart or not smart that friend is so you never have to worry about looking stupid because it never reveals with easy questions you accidentally got wrong.

In yesterdays DjangoCon BDFL Keynote Adrian Holovaty called out that Django needs a Real-Time story. Well, here's a response to that: django-sockjs-tornado

Immediately after the keynote I went and found a comfortable chair and wrote this app. It's basically a django app that allows you to run a socketserver with manage.py like this:

python manage.py socketserver

Now, you can use all of SockJS to write some really flashy socket apps. In Django! Using Django models and stuff. The example included shows how to write a really simple chat application using Django models. check out the whole demo here

If you're curious about SockJS read the README and here's one of many good threads about the difference between SockJS and socket.io.

The reason I could write this app so quickly was because I have already written a production app using sockjs-tornado so the concepts were familiar. However, this app has (at the time of writing) not been used in any production. So mind you it might still need some more love before you show your mom your django app with WebSockets.

I've finally had time to sort out the django-mongokit code so it's now fully compatible with Django 1.4.

Because I don't personally use the project I've sort of got myself lost in various patches from some awesome contributors who keep it in check.

Also, thanks to Marc Abramowitz who added a tox.ini which I've now updated to also test python 2.7 and Django 1.4

Go forth and build awesome Django apps with MongoKit which is still think is the best wrapper available on PyMongo out there.

Last month I built a very basic mobile app using jQuery Mobile. It's still available here.

One flaw I found with jQuery Mobile is that it's a bit slow. Scrolling up and down feels sluggish. It looks pretty though.
This time around I opted instead to use Twitter Bootstrap instead. (inspired by some other blog post with a similar experience)

Also, with this version I've set up a domain name: uslicensespotter.com
The source code is available here and there are some interesting pieces of this that I'd like to delve into deeper.

Optimization

This new app feels faster and scrolling is smoother. There is still some parts where it doesn't feel snappy. The primary part is when you click one of the buttons. It works something like this:

// binding (simplified)
  $('form a.btn').click(function(event) {
    var $el = $(this);
    var id = $el.attr('id');
    if ($el.hasClass('btn-success')) {
      switch_off($el);
      State.remove(id);
    } else {
      switch_on($el, new Date());
      State.add(id);
    }
    return false;
  });

// the switch_on function
  function switch_on($el, timestamp) {
    $el.addClass('btn-success');
    $('i', $el).addClass('icon-white').removeClass('icon-remove').addClass('icon-check');
    $('span', $el).text(timeSince(timestamp));
  }

There are two potentials for making this faster.

1. Instead of using element.click(...) I can use touch events which supposedly fire better thus starting the actual animation earlier.
   
2. Instead of modifying the element (adding a class, adding a class, removing a class, adding a class, changing the text) what I could do is to remove the element from the DOM, manipulate it and then re-insert it back into the DOM. That way the (mobile) browser doesn't have to re-render after each manipulation.
   

My next task is to apply both of these ideas and try to somehow keep and eye on which makes the biggest impact.

Deployment

One neat new feature with this new code is the deployment script, build.py. The actual code might be a bit of a mess but the way it works for me is great.

What I do is that I work on the app by editing dev.html and when I deploy I run python build.py and it opens dev.html and concatenates and minifies all CSS and Javascript ready for ideal caching. This reduces the static assets down to just three files weighing only 78Kb altogether.

(Note: even more can be done such as switching to ZeptoJS and removing bootstrap stuff I don't need with a custom download)

What the script does is that it parses the dev.html template with lxml and applies whatever transformations it needs to do such as generating the Apple touch images eg. this one

I think a lot of mobile web developers can benefit from this script. Maybe you don't want to copy it verbatim but you might want to copy it and do the transformations you want to do.