Dawned on me that the Django auth user model that ships with Django is like the string built-in of a high level programming language. With the string built-in it's oh so tempting to add custom functionality to it like a fancy captialization method or some other function that automatically strips whitespace or what not. Yes, I'm looking at you Prototype for example.

By NOT doing that, and leaving it as it is, you automatically manage to Keep It Simple Stupid and your application code makes sense to the next developer who joins your project.

I'm not a smart programmer but I'm a smart developer in that I'm good at keeping things pure and simple. It means I can't show off any fancy generators, monads or metaclasses but it does mean that fellow coders who follow my steps can more quickly hit the ground running.

My colleagues and I now have more than ten Django projects that rely on, without overriding, the django.contrib.auth.models.User class and there has been many times where I've been tempted to use it as a base class or something instead but in retrospect I'm wholeheartedly happy I didn't. The benefit isn't technical; it's a matter of teamwork and holistic productivity.

django-mongokit is the project you want to use if you want to connect your Django project to your MongoDB database via the pymongo Python wrapper. An alternative (dare I say competing alternative) is MongoEngine which is bridge between Django and straight to pymongo. The immediate difference you notice is the syntax. django-mongokit looks like MongoKit syntax and MongoEngine looks like Django ORM. They both accomplish pretty much the same thing. So, which one is fastest?

First of all, remember this? where I showed how django-mongokit sped past the SQL ORM like a lightning bullet. Well appears MongoEngine is even faster.

That's an average of 23% faster for all three operations!

About 5 years ago Martijn Faasen wrote the wonderful blog article What is Pythonic. One thing that I feel is extremely Pythonic is to not compare certain thing to other things when Python has built-in understanding of what false or true means.

Having reviewed/read a lot of beginner code or senior code but of people coming from lower-level languages I often see this:

To be Pythonic is to understand that Python evaluates all of these to false. All built in types have a perfectly sensible boolean operator which is automatically used in an if statement or an embedded if statement in a list comprehension. Keep it clean a pure just like this to check for true:

And if you have your custom class such as the example just above with the pseudo "ORM" it's easy to extend it by writing your own custom __bool__ like this:

By playing along with Python just the way Guido indented it you can abstract yourself from being overly dependent of types. By doing the shorthand notation a variable that is otherwise a list can be None if it's not set and your code will continue to work.

All the above might not be true for more explicit lower-level languages like C++ but it sure is Pythonic in Python and that's a good thing.

Ludvig Ericson ("Sweden-based backend-centric super-programmer.") yesterday released the most important Python package you'll ever see this year. Sort of. It animates a little fish on your terminal that goes back and forth across the screen.

Maybe I'm exaggerating a bit. This is the kind of superficial hype that made Rails successful at least. What the package is really useful for is a great start for those who want to do those fancy writes to the terminal without linebreaks. Spoiler alert:

 sys.stderr.write("\x1b[2K\r" + fish + "\r")

I've learned a couple of things this week on deploying my first site to use a user friendly OpenID.

My first revelation was when I realized that Google and Yahoo! have solved the usability stumbling block that you can use them as providers without having to know a personally unique URL. For example, for Yahoo! it's just http://yahoo.com which means that you don't need to offer a cryptic URL form and you can just show it as a logo image.

The second thing is that Google's hybrid OpenID + OAuth isn't as complicated as it sounds. It's basically a light extension to the OpenID "protocol" whereby you say, "while you're at it, also give me a OAuth token please so that I can connect back into Google's services later". What's important to understand though is that if you use this you need to know the "scope". scope is a URL to a service. Google Docs is a service for example and you need to search the web to figure out what the scope URL is for that service.

The third revelation was when I understood the difference between Simple Registration Extension (SREG) and Attribute Exchange (AX). Basically, AX is a newer more modern alternative and SREG was the first one. AX is better but some OpenID providers don't yet support it. Google for example, only supports AX. Key to be able to support not just Google's OpenID but any OpenID is that you can request both AX and SREG and whichever one works will be returned.

The fourth thing that helped a lot to understand was the Google's OpenID has a bug in its implementation of Attribute Exchange. Actually, perhaps it's a deliberate design choice they've made but in my opinion a bad one. Unless you say you require email, firstname, lastname, country etc. it won't return it. If you use the if_available directive you won't get it. Another bug/bad design choice is that Google seems to not forward the country attribute. It can happily do first- and last name but not country even if the documentation claims so.

The fifth thing is that python-openid is a lot easier to work with than you think. You don't need to do any crazy network checks or callbacks. For initiating the challenge all you're effectively doing is creating a long URL. If you don't like the API methods python openid offers, just add your own with:

 redirect_url += '&openid.ax.mode=fetch_request' # etc.

After so many years since OpenID arrived, I'm only now excited about it. It's tonnes easier to implement than OAuth and now it's actually really pleasant to use as an end user.

uwsgi is the latest and greatest WSGI server and promising to be the fastest possible way to run Nginx + Django. Proof here But! Is it that simple? Especially if you're involving Django herself.

So I set out to benchmark good old threaded fcgi and gunicorn and then with a source compiled nginx with the uwsgi module baked in I also benchmarked uwsgi. The first mistake I did was testing a Django view that was using sessions and other crap. I profiled the view to make sure it wouldn't be the bottleneck as it appeared to take only 0.02 seconds each. However, with fcgi, gunicorn and uwsgi I kept being stuck on about 50 requests per second. Why? 1/0.02 = 50.0!!! Clearly the slowness of the Django view was thee bottleneck (for the curious, what took all of 0.02 was the need to create new session keys and putting them into the database).

So I wrote a really dumb Django view with no sessions middleware enabled. Now we're getting some interesting numbers:

 fcgi (threaded)              640 r/s
 fcgi (prefork 4 processors)  240 r/s (*)
 gunicorn (2 workers)         1100 r/s
 gunicorn (5 workers)         1300 r/s
 gunicorn (10 workers)        1200 r/s (?!?)
 uwsgi (2 workers)            1800 r/s
 uwsgi (5 workers)            2100 r/s
 uwsgi (10 workers)           2300 r/s

 (* this made my computer exceptionally sluggish as CPU when through the roof)

Following on from yesterday's blog about How and why to use django-mongokit I extended the exampleproject which is inside the django-mongokit project with another app called exampleapp_sql which does the same thing as the exampleapp but does it with SQL instead. Then I added a very simple benchmarker app in the same project and wrote three functions:

1. One to create 10/100/500/1000 instances of my class
2. One to edit one field of all 10/100/500/1000 instances
3. One to delete each of the 10/100/500/1000 instances

Here I'm going to explain how to combine Django and MongoDB using MongoKit and django-mongokit.

MongoDB is a document store built for high speed and high concurrency with a very good redundancy story. It's an alternative to relational databases (e.g. MySQL) that is what Django is tightly coupled with in it's ORM (Object Relation Mapping) and what it's called now is ODM (Object Document Mapping) in lack of a better acronym. That's where MongoKit comes in. It's written in Python and it connects to the MongoDB database using a library called pymongo and it turns data from the MongoDB and turns it into instances of classes you have defined. MongoKit has nothing to do with Django. That's where django-mongokit comes in. Written by yours truly.

IssueTrackerMassContainer is a simple Zope product that is used to put a bunch of IssueTrackerProduct instances into. It doesn't add much apart from a nice looking dashboard that lists all recent issues and then with an AJAX poll it keeps updating automatically.

But what it was doing was it recursively put together all issues across all issue trackers, sorting them and then returning only the first 20. Fine, but once the numbers start to add up it can become a vast sort operation to deal with.

In my local development copy of 814 issues, by the use of pympler and time() I was able to go from 7 Mb taking 2 seconds down to using only 8 Kb and taking 0.05 seconds.

This cost me some hair-pulling today as I was trying to write a custom test runner for a Django project I'm working on that creates a test Xapian database just for running the tests. Basically, you can't do this:

Because if you do you end up getting these strange DatabaseOpeningError exceptions. So, here's how you do it:

Hopefully by blogging about this some other poor coder will save some time.

YUI Compressor apparently supports compressing CSS. Cool! I had to try it and what's even more cool is that it's the only other CSS minifier/compressor that doesn't choke on CSS hacks (the ones I tried). The only other CSS minifier/compressor is my very own slimmer. So, let's see what the difference is.

Running the YUI Compressor 10 times on a relatively small file takes 0.3 seconds on average. Running the same with python 2.6 and slimmer.css_slimmer takes 0.1 seconds on average. I think most of this time is spent loading the jar file than the actual time of running the compression.

I have some code that checks that a file is treated differently if some time has passed. In other words, it reacts if the modification time is more than it was before. The code uses os.stat(filename)[stat.ST_MTIME] to do this. The challenge was to mock os.stat so that I can pretend some time has passed without having to wait. Here was my first solution which made running many tests sloooow:

So, here's how I mock os.stat to avoid having to do the time.sleep(1) in the test:

I hope this helps someone else who's trying to do the same. It took me some time to figure out that os.stat is used by lots of various sub routines in the os module so it's important to only mock the relevant argument otherwise you might get unexpected problems.

One really annoying thing about PIL is that it's importable as ImageandPIL. It leads me and other newbies to think if it's different. I don't want choices:

 Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
 [GCC 4.3.3] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import PIL
 >>> import Image
 >>> ?

When PIL/Image is put into standard lib, can we call the module: imaging?

I've just finished my first Twitter app. It's basically a just a about using OAuth to allow people to sign up to KungFuPeople.com without having to pick yet another password.

I simply took the oauth.py module by Leah Culver and wrapped it with some useful functions taken from a similar Twitter app we've done at work.

Unlike other Twitter apps for this one I'm using Twitter solely for handling authorization and authentication. That means that it has to work with the existing user + profile functionality but just side-step the sign up and login.

Next goal: Google OAuth

JSMIN - The Javascript Minifier is written in C and it does an excellent job of minifying Javascript code. After all, Douglas Crockford wrote it. I noticed there's a Python implementation of it so I wanted to see how it stacks up against my slimmer which is also written in Python.

For sake of argument I compiled the C version and ran that in my little benchmark and did so by using the subprocess module. Also, for the sake of comparison I threw in a run with YUI Compressor. Here are some quick results:

 On js/signup-core.js
 --------------------
 js_slimmer
 from 9708 to 6905 in 0.0245039463043 seconds
 jsmin
 from 9708 to 6720 in 0.0850019454956 seconds
 jsmin.c
 from 9708 to 6721 in 0.0026159286499 seconds
 yuicompressor
 from 9708 to 6102 in 0.914173126221 seconds

 On js/zoom.js 
 -------------
 js_slimmer
 from 5920 to 3712 in 0.0106379985809 seconds
 jsmin
 from 5920 to 3582 in 0.0582370758057 seconds
 jsmin.c
 from 5920 to 3583 in 0.00282216072083 seconds
 yuicompressor
 from 5920 to 2771 in 0.839382171631 seconds

 On js/diypack.js
 ----------------
 js_slimmer
 from 21559 to 14059 in 0.0409741401672 seconds
 jsmin
 from 21559 to 13655 in 0.177556037903 seconds
 jsmin.c
 from 21559 to 13656 in 0.00346994400024 seconds
 yuicompressor
 from 21559 to 11638 in 0.891603946686 seconds

So, roughly, slimmer is 4 times faster than jsmin.py but fails to minify a couple of bytes. jsmin.c is about 6 times faster than slimmer.py but is awkward since it's in C. I guess jsmin.c is the way forward when you want speed and the best result. slimmer has the advantage of being all in python and PyPi and contains functions for CSS, HTML and XHTML as well.

It's clear the YUI Compressor does a wicked job at minifying but by running a .jar file every time in a subprocess is crazily slow if that matters for you.

If you're on the python-uk mailing list you will have already seen this but if you're not, here we go.

Fry-IT is hosting a Code Dojo in our offices. It's free and open to anyone. My colleague Nicholas has written up a little bit about what a Code Dojo is which should get you excited.

Details are available on this page which is also the place to go to secure your place. Currently there are 12 people who say they're coming and we've decided to cap the geek influx to 30 people.

Cheers,
Peter-san

I've been experimenting with the new Google Reverse Geocoding which allows you to get the location name and country and stuff from a latitude/longitude coordinate.

What I've been doing is comparing this with GeoNames. GeoNames is available from geopy in the reverse-geocode branch.

I wrote down a list of about 15 lat/long points and the result I expect from them (taken from an existing app I'm contemplating switching to Google Reverse Geocoding for) and ran a batch of timed tests on. These results might satisfy the impatient:

 FAILURES:
 geonames_json        0
 google               0
 geonames             12

 TOTAL TIMES:
 geonames_json        2.43582677841        0.143283928142 seconds/request
 google               2.24999976158        0.132352927152 seconds/request
 geonames             1.78063511848        0.104743242264 seconds/request

By popular demand I've made my little pyinotify wrapper available for download. It's nothing fancy really but damn useful and productive.

It relies on inotify (so you're stuffed on OSX and Windows) which makes it very fast and efficient (as opposed to periodic polling and file modification time comparisons).

At the moment it's actually quite generic for any command and any file but I'm hoping to take this to the next level with some magic dust that automatically only runs unit tests that fail or something. We'll see what happens.

Gun to your head; what would it take to make setuptools as a package author easy to use?

I've spent far too long time today trying to create a package for a little piece of code I've written. Because I can never remember all the bizarre options and commands to setup.py I tried to do it by following Tarek Ziade's wonderful Expert Python Programming but I still got stuck.

Granted, I did not read the f**n manual. Why should I have to? I've got more important things to do such as eating cookies and watching tv.

By blogging about it I can pretty much guarantee that someone will comment and say "Hey, why didn't you use the pypi/alreadyexists package which does the same thing but better". I couldn't find one after a quick search and I felt the hacker mood creeping up on my begging me to (re)invent it.

premailer.py takes a HTML page, finds all CSS blocks and transforms these into style attributes. For example, from this:

You get this:

Why is this useful? When you're writing HTML emails. Like this newsletter app that I'm working on.

I just wrote it late yesterday and it needs lots of work to impress but for the moment it works for me. If I take the time to tidy it up properly I'll turn it into a package. Assuming there isn't one already :)

UPDATE

No available on github.com and as a PyPi package