I'm brand spanking new to the node.js web application development. The framework I'm currently using is express which seems OK. So I've got an app that consists of 1 static HTML file, a lot of Javscript/CSS/image resources and some express GET and POST views that return small snippets of HTML. All data will be loaded with AJAX to avoid having to use any HTML templating on first load. What's cool about this is that it's soo fast! Everything except the JSON data can be loaded from an Nginx server.

At the moment I've got a light static HTML page that loads about 240Kb of Javascript and CSS (jQuery UI is big) and a couple of bytes of JSON data pulled from Node. As a little anal perfectionism I put an Nginx server in front so that Node doesn't have to serve any of the static files. To get that you have to have a Nginx site enabled that looks like this:

 server {
    root /home/peterbe/task-calendar/static;
    location / {
      if (-f $request_filename) {
          add_header X-Static hit;
          access_log   off;
      }
      if (!-f $request_filename) {
          proxy_pass http://127.0.0.1:8000; # where Node is running
          add_header X-Static miss;
      }
    }
 }

I think much of the fun of working with this app is that it's a delight to see it load in the browser without any sluggishness or delay. Lovely!

Dawned on me that the Django auth user model that ships with Django is like the string built-in of a high level programming language. With the string built-in it's oh so tempting to add custom functionality to it like a fancy captialization method or some other function that automatically strips whitespace or what not. Yes, I'm looking at you Prototype for example.

By NOT doing that, and leaving it as it is, you automatically manage to Keep It Simple Stupid and your application code makes sense to the next developer who joins your project.

I'm not a smart programmer but I'm a smart developer in that I'm good at keeping things pure and simple. It means I can't show off any fancy generators, monads or metaclasses but it does mean that fellow coders who follow my steps can more quickly hit the ground running.

My colleagues and I now have more than ten Django projects that rely on, without overriding, the django.contrib.auth.models.User class and there has been many times where I've been tempted to use it as a base class or something instead but in retrospect I'm wholeheartedly happy I didn't. The benefit isn't technical; it's a matter of teamwork and holistic productivity.

I currently live and work from home in Raleigh, North Carolina. Bing maps has a cool feature called "Bird's eye view".

About a month ago I add a new feature to django-static that makes it possible to define a function that all files of django-static goes through.

First of all a quick recap. django-static is a Django plugin that you use from your templates to reference static media. django-static takes care of giving the file the optimum name for static serving and if applicable compresses the file by trimming all whitespace and what not. For more info, see The awesomest way possible to serve your static stuff in Django with Nginx

The new, popular, kid on the block for CDN (Content Delivery Network) is Amazon Cloudfront. It's a service sitting on top of the already proven Amazon S3 service which is a cloud file storage solution. What a CDN does is that it registers a domain for your resources such that with some DNS tricks, users of this resource URL download it from the geographically nearest server. So if you live in Sweden you might download myholiday.jpg from a server in Frankfurk and if you live in North Carolina, USA you might download the very same picture from Virgina, USA. That assures the that the distance to the resource is minimized. If you're not convinced or sure about how CDNs work check out THE best practice guide for faster webpages by Steve Sounders (it's number two)

A disadvantage with Amazon Cloudfront is that it's unable to negotiate with the client to compress downlodable resources with GZIP. GZIPping a resource is considered a bigger optimization win than using CDN. So, I continue to serve my static CSS and Javascript files from my Nginx but put all the images on Amazon Cloudfront. How to do this with django-static? Easy: add this to your settings:

Then you need to write that function that get's a chance to do something with every static resource that django-static prepares. Here's a naive first version:

Isn't it amazingly positive news that Warren Buffet and Melinda and Bill Gates have put up the $600 billion challenge which is "asking the nation's billionaires to pledge to give at least half their net worth to charity". And if you haven't already read about it, Warren Buffet pledges 99% of this company stock to charity. All good news but what's really interesting is reading peoples comments on the CNN page. A handful pick:

"Interesting article. It is saddening, however, to ponder just how much of this crowd's wealth was made through unfair business practices, worker exploitation, price fixing, etc. I suppose philanthropy on the back end is a nice afterthought, though, and certainly earns more praise from the public than would lessening their profit margins at the get-go."

"Pay their taxes first, then contribute with after tax money."

"If I may be cynical. Perhaps these super rich people should have done more for the people that worked for them so that they made more money and the leaders made a little less. Buffet owns companies that make goods in second and third world countries at some of the lowest possible wages."

"$1000 in the hands of ONE could be investment money. $1000 distributed $1 to ONE THOUSAND could get each a Coke (no fries)."

But also, there are some more "positive" comments:

"There is sooo much negativity in this country! I don't care what anyone of you says...Bill and Melinda came from Blue Collar....and now they are giving back and I think it's awesome!"

"I think what they are doing is very admirable. The Gates Foundation is the reason I was able to pay for college. People need to not criticize what they do with their money, at least they are trying to make a difference."

In conclusion from skimming the comments it's pretty obvious that people in the USA are angry and bitter. What is there to complain about? Really? Poor Obama, he's doing a great job but with all this resentment sizzling around it's going to be very hard if even "extreme philanthropy" gets butchered like this.

Yesterday I found out that Transport for Londonlifted all restrictions for commercial use of its data that it has made available for developers.

In lack of better imagination I decided to attack the Live Traffic Cameras data and whipped up this little app: tflcameras.peterbe.com

It basically shows a map of London and then shows all the spots where traffic cameras are installed so that you can click on them. The data is updated every 3 hours I think but I haven't checked that claim yet. Use this if you're a London commuter and want to check the traffic before you hit the road.

Oh, and this app uses the geo location stuff so that I know where to zoom in first. But if you're not based in London it zooms in over Trafalgar square by default.

At Euro DjangoCon I met lots of people and talked a lot about MongoDB as the backend. I even did a presentation on the subject which led to a lot of people asking me more questions about MongoDB.

I did mention to some people that one of the drawbacks of using MongoDB which doesn't have transactions is that you have to create and destroy the collections (like SQL tables) each time for every single test runs. I thought this was slow. It's not

Today I've been doing some more profiling and testing and debugging and I can conclude that it's not a problem. Creating the database has a slight delay but it's something you only have to do once and actually it's very fast. Here's how I tear down the collections in between each test:

For example, running test of one of my apps looks like this:

So, don't fear writing lots of individual unit tests. MongoDB will not slow you down.

Having lived under a rock for a while I've managed to miss this great new feature in Gmail: Muting or ignoring conversions

From their help text:

"you've no doubt been subjected to the 'thread that just won't die!' If you're part of a long message conversation that isn't relevant, you can mute the conversation to keep all future additions out of your inbox."

That is such a smart feature. Interestingly I didn't even think there was a solution to that problem. I sure I have many times needed something like this. Now, let's hope I can remember to actually use this feature.

Straight from DjangoCon 2010 here in Berlin. Slides from my talk on "Using MongoDB in your Django app - implications and benefits" are available as a HTML5 web page so you'll need one of those fancy browsers like Chrome to be able to view it. Sorry.

django-mongokit is the project you want to use if you want to connect your Django project to your MongoDB database via the pymongo Python wrapper. An alternative (dare I say competing alternative) is MongoEngine which is bridge between Django and straight to pymongo. The immediate difference you notice is the syntax. django-mongokit looks like MongoKit syntax and MongoEngine looks like Django ORM. They both accomplish pretty much the same thing. So, which one is fastest?

First of all, remember this? where I showed how django-mongokit sped past the SQL ORM like a lightning bullet. Well appears MongoEngine is even faster.

That's an average of 23% faster for all three operations!

About 5 years ago Martijn Faasen wrote the wonderful blog article What is Pythonic. One thing that I feel is extremely Pythonic is to not compare certain thing to other things when Python has built-in understanding of what false or true means.

Having reviewed/read a lot of beginner code or senior code but of people coming from lower-level languages I often see this:

To be Pythonic is to understand that Python evaluates all of these to false. All built in types have a perfectly sensible boolean operator which is automatically used in an if statement or an embedded if statement in a list comprehension. Keep it clean a pure just like this to check for true:

And if you have your custom class such as the example just above with the pseudo "ORM" it's easy to extend it by writing your own custom __bool__ like this:

By playing along with Python just the way Guido indented it you can abstract yourself from being overly dependent of types. By doing the shorthand notation a variable that is otherwise a list can be None if it's not set and your code will continue to work.

All the above might not be true for more explicit lower-level languages like C++ but it sure is Pythonic in Python and that's a good thing.

The lovely people of Packt Publishing asked me to review Karen Tracey's latest book Django 1.1 Testing and Debugging.

I didn't actually read the book but rather skimmed it, apart from some selected parts and from what I read it's obvious that Karen has an ability to write to people who are not experts on the subject. Years of being a top contributor on the Django users mailing list must have something to do with it.

But here's the cracker. I didn't learn anything from this book (actually, I wasn't aware of the pp command in the pdb debugger). Is that a complaint about the book? No! It just means that the book was aimed at beginners and apparently I'm not a beginner any more. Great!

One thing I would have liked to see is more about testing strategy since this is something beginners often have problems with. I don't know if there even is such a word as "testing strategy" but I'm referring to the thinking behind what to test and more importantly sometimes what not to test. Beginners have a tendency to write tests for the most specific things and thus spending all their time assuring the most unrealistic scenarios are covered. Also, a lot of beginner tests I see check basic things like types which the semi-compiler will just automatically cover for you. Perhaps for a beginner, just getting some tests up and running this is a big step forward.

I'm a little bit disappointed that my lovely gorun wasn't mentioned in the book :) Perhaps the next version Karen?

This is fairly obvious stuff I guess but it has troubled me for a long time. Some programs on Linux don't spit out their results to stdout. Instead they start a little program similar to less. So what is a console nerd to do?

Pipe it cat! I don't know why I've never thought of this before:

 $ psql -l | cat

Today I finally got around to launching a new version of the IssueTrackerProduct. After a years worth of bug fixes and new features added the most exciting one to me is the Monthly summary feature.

Gosh! I really regret I've never properly collected email addresses for people who actually use the IssueTrackerProduct. I was using a SourceForge mailing list for a while but that got to spam ridden I gave up on it a long time ago.

Check out how long the list of change is on version 0.11

Feedback mucho appreciatado.

So I upgraded to the latest Ubuntu Lucid Lynx 10.04 the other day and to my horror it removed Python 2.4 and Python 2.5. I rely more on those programs than I do on some silly Facebook connecting social widget crap. On my laptop I have lots of Zopes requiring Python 2.4 and I have about 10 active Django projects that rely on Python2.5. This fuckup by Ubuntu caused me to write this complaint.

So my estimeed colleague and Linux wiz Jan Kokoska helped me set things straight by showing me how to downgrade these packages to Karmic version and how to pin them in the apt preferences. First of all, make your /etc/apt/source.list look like this:

 deb http://gb.archive.ubuntu.com/ubuntu/ karmic main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ karmic-updates main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic-updates main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ karmic-backports main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic-backports main restricted universe multiverse

 deb http://security.ubuntu.com/ubuntu karmic-security main restricted universe multiverse
 deb-src http://security.ubuntu.com/ubuntu karmic-security main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse

 deb http://gb.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
 deb-src http://gb.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse

 deb http://security.ubuntu.com/ubuntu lucid-security main restricted universe multiverse
 deb-src http://security.ubuntu.com/ubuntu lucid-security main restricted universe multiverse

If you know what you're doing you might have other additional sources in there then keep those as is. Next thing to do is to update and upgrade:

 # apt-get update
 # apt-get dist-upgrade

You should now see that it's intending to upgrade a bunch of juicy packages like python2.4-dev for example. To check that python2.4 is now getting in from Karmic run this:

 $ apt-cache madison python2.4

Now for the trick that really makes the difference:

 # apt-get install python2.4=2.4.6-1ubuntu3.2.9.10.1 python2.4-dbg=2.4.6-1ubuntu3.2.9.10.1 \
 python2.4-dev=2.4.6-1ubuntu3.2.9.10.1 python2.4-doc=2.4.6-1ubuntu3.2.9.10.1 \
 python2.4-minimal=2.4.6-1ubuntu3.2.9.10.1

The command is quite self-explanatory. You use the equal sign to basically say what version you want to install. If you now for example want to install something like python-profiler for your Python 2.4 since this isn't available as a PyPi package. First, find out what version you have to install:

 $ apt-cache madison python-profiler | grep karmic

From that list you'll get a bunch of versions. Chose the one from karmic-updates or karmic-security. Then install it:

 # apt-get install python-profiler=2.6.4-0ubuntu1

Now, to avoid this causing a conflict and thus be removed the next time you do an upgrade you need to pin it. Create a file called /etc/apt/preferences and put the following into it:

 Package: python-profiler
 Pin: version 2.6.4-0ubuntu1
 Pin-Priority: 999

And that concludes it. A word of warning from Jan:

"he slight problem is that with this setup, suppose a big security flaw was found in python-imaging and got patched in karmic that is still supported... you wouldn't get the package update. That is because it's pinned and while asterisks can be used in the version number, we don't know in advance what the version will match and what the Lucid version that we don't want will match"

"so you basically lose security upgrades for affected packages"

"minor annoyance when you have one or two packages on a laptop, but a big deal if you have a dozen packages on 100 VMs on server"

Having written about this helps me remember it myself for the next time I need it. Also, hopefully it will help other people who get bitten by this. Hopefully this will shame the Canonical guys into action so that the next time they don't haste their deprecation process and actually think about who's using their products. I bet a majority of Ubuntu's users care more about programming or something like that than they do about the ability to buy music on Ubuntu One or whatever it's called.

Ludvig Ericson ("Sweden-based backend-centric super-programmer.") yesterday released the most important Python package you'll ever see this year. Sort of. It animates a little fish on your terminal that goes back and forth across the screen.

Maybe I'm exaggerating a bit. This is the kind of superficial hype that made Rails successful at least. What the package is really useful for is a great start for those who want to do those fancy writes to the terminal without linebreaks. Spoiler alert:

 sys.stderr.write("\x1b[2K\r" + fish + "\r")

It works. Sort of.

The screenshot attached here doesn't really show it but the nav at the bottom is supposed to stay at the bottom as you scroll but that doesn't seem to work on the iPad. Shows me up for not using solid basic (X)HTML.

Ever since we started building SnapExpense we ask people we meet how they do their business/travel expenses at their company. Almost always they say their software is annoyingly bad and it's a right pain in the ass to process their expenses. Today I heard another such horror story from a friend about: GlobalExpense (www.globalexpense.com). Apparently it's dreadfully slow to use and it never remembers or adapts to what you have entered previously.

So my friend went to the website on my computer to show me how crap it was but that didn't work because GlobalExpense doesn't work in Firefox! What?! On SnapExpense, about 30% of our visitors use Firefox (with 31% using Internet Explorer).

I'm glad to see that the competition is lagging behind. Gives me heart. Especially as I have recently added Google OpenID log in to SnapExpense so you can log in or register with your Google (or Yahoo! or any OpenID provider) account straight away.

This is going to be useful to keep handy when I move to America.

I've learned a couple of things this week on deploying my first site to use a user friendly OpenID.

My first revelation was when I realized that Google and Yahoo! have solved the usability stumbling block that you can use them as providers without having to know a personally unique URL. For example, for Yahoo! it's just http://yahoo.com which means that you don't need to offer a cryptic URL form and you can just show it as a logo image.

The second thing is that Google's hybrid OpenID + OAuth isn't as complicated as it sounds. It's basically a light extension to the OpenID "protocol" whereby you say, "while you're at it, also give me a OAuth token please so that I can connect back into Google's services later". What's important to understand though is that if you use this you need to know the "scope". scope is a URL to a service. Google Docs is a service for example and you need to search the web to figure out what the scope URL is for that service.

The third revelation was when I understood the difference between Simple Registration Extension (SREG) and Attribute Exchange (AX). Basically, AX is a newer more modern alternative and SREG was the first one. AX is better but some OpenID providers don't yet support it. Google for example, only supports AX. Key to be able to support not just Google's OpenID but any OpenID is that you can request both AX and SREG and whichever one works will be returned.

The fourth thing that helped a lot to understand was the Google's OpenID has a bug in its implementation of Attribute Exchange. Actually, perhaps it's a deliberate design choice they've made but in my opinion a bad one. Unless you say you require email, firstname, lastname, country etc. it won't return it. If you use the if_available directive you won't get it. Another bug/bad design choice is that Google seems to not forward the country attribute. It can happily do first- and last name but not country even if the documentation claims so.

The fifth thing is that python-openid is a lot easier to work with than you think. You don't need to do any crazy network checks or callbacks. For initiating the challenge all you're effectively doing is creating a long URL. If you don't like the API methods python openid offers, just add your own with:

 redirect_url += '&openid.ax.mode=fetch_request' # etc.

After so many years since OpenID arrived, I'm only now excited about it. It's tonnes easier to implement than OAuth and now it's actually really pleasant to use as an end user.