Peterbe.com

So I upgraded to the latest Ubuntu Lucid Lynx 10.04 the other day and to my horror it removed Python 2.4 and Python 2.5. I rely more on those programs than I do on some silly Facebook connecting social widget crap. On my laptop I have lots of Zopes requiring Python 2.4 and I have about 10 active Django projects that rely on Python2.5. This fuckup by Ubuntu caused me to write this complaint.

So my estimeed colleague and Linux wiz Jan Kokoska helped me set things straight by showing me how to downgrade these packages to Karmic version and how to pin them in the apt preferences. First of all, make your /etc/apt/source.list look like this:

deb http://gb.archive.ubuntu.com/ubuntu/ karmic main restricted universe multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic main restricted universe multiverse

deb http://gb.archive.ubuntu.com/ubuntu/ karmic-updates main restricted universe multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic-updates main restricted universe multiverse

deb http://gb.archive.ubuntu.com/ubuntu/ karmic-backports main restricted universe multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ karmic-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu karmic-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu karmic-security main restricted universe multiverse

deb http://gb.archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse

deb http://gb.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse

deb http://gb.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
deb-src http://gb.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse

deb http://security.ubuntu.com/ubuntu lucid-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu lucid-security main restricted universe multiverse

If you know what you're doing you might have other additional sources in there then keep those as is. Next thing to do is to update and upgrade:

# apt-get update
# apt-get dist-upgrade

You should now see that it's intending to upgrade a bunch of juicy packages like python2.4-dev for example. To check that python2.4 is now getting in from Karmic run this:

$ apt-cache madison python2.4

Now for the trick that really makes the difference:

# apt-get install python2.4=2.4.6-1ubuntu3.2.9.10.1 python2.4-dbg=2.4.6-1ubuntu3.2.9.10.1 \
python2.4-dev=2.4.6-1ubuntu3.2.9.10.1 python2.4-doc=2.4.6-1ubuntu3.2.9.10.1 \
python2.4-minimal=2.4.6-1ubuntu3.2.9.10.1

The command is quite self-explanatory. You use the equal sign to basically say what version you want to install. If you now for example want to install something like python-profiler for your Python 2.4 since this isn't available as a PyPi package. First, find out what version you have to install:

$ apt-cache madison python-profiler | grep karmic

From that list you'll get a bunch of versions. Chose the one from karmic-updates or karmic-security. Then install it:

# apt-get install python-profiler=2.6.4-0ubuntu1

Now, to avoid this causing a conflict and thus be removed the next time you do an upgrade you need to pin it. Create a file called /etc/apt/preferences and put the following into it:

Package: python-profiler
Pin: version 2.6.4-0ubuntu1
Pin-Priority: 999

And that concludes it. A word of warning from Jan:

"he slight problem is that with this setup, suppose a big security flaw was found in python-imaging and got patched in karmic that is still supported... you wouldn't get the package update. That is because it's pinned and while asterisks can be used in the version number, we don't know in advance what the version will match and what the Lucid version that we don't want will match"

"so you basically lose security upgrades for affected packages"

"minor annoyance when you have one or two packages on a laptop, but a big deal if you have a dozen packages on 100 VMs on server"

Having written about this helps me remember it myself for the next time I need it. Also, hopefully it will help other people who get bitten by this. Hopefully this will shame the Canonical guys into action so that the next time they don't haste their deprecation process and actually think about who's using their products. I bet a majority of Ubuntu's users care more about programming or something like that than they do about the ability to buy music on Ubuntu One or whatever it's called.

Ludvig Ericson ("Sweden-based backend-centric super-programmer.") yesterday released the most important Python package you'll ever see this year. Sort of. It animates a little fish on your terminal that goes back and forth across the screen.

Maybe I'm exaggerating a bit. This is the kind of superficial hype that made Rails successful at least. What the package is really useful for is a great start for those who want to do those fancy writes to the terminal without linebreaks. Spoiler alert:

sys.stderr.write("\x1b[2K\r" + fish + "\r")

It works. Sort of.

The screenshot attached here doesn't really show it but the nav at the bottom is supposed to stay at the bottom as you scroll but that doesn't seem to work on the iPad. Shows me up for not using solid basic (X)HTML.

Ever since we started building SnapExpense we ask people we meet how they do their business/travel expenses at their company. Almost always they say their software is annoyingly bad and it's a right pain in the ass to process their expenses. Today I heard another such horror story from a friend about: GlobalExpense (www.globalexpense.com). Apparently it's dreadfully slow to use and it never remembers or adapts to what you have entered previously.

So my friend went to the website on my computer to show me how crap it was but that didn't work because GlobalExpense doesn't work in Firefox! What?! On SnapExpense, about 30% of our visitors use Firefox (with 31% using Internet Explorer).

I'm glad to see that the competition is lagging behind. Gives me heart. Especially as I have recently added Google OpenID log in to SnapExpense so you can log in or register with your Google (or Yahoo! or any OpenID provider) account straight away.

This is going to be useful to keep handy when I move to America.

I've learned a couple of things this week on deploying my first site to use a user friendly OpenID.

My first revelation was when I realized that Google and Yahoo! have solved the usability stumbling block that you can use them as providers without having to know a personally unique URL. For example, for Yahoo! it's just http://yahoo.com which means that you don't need to offer a cryptic URL form and you can just show it as a logo image.

The second thing is that Google's hybrid OpenID + OAuth isn't as complicated as it sounds. It's basically a light extension to the OpenID "protocol" whereby you say, "while you're at it, also give me a OAuth token please so that I can connect back into Google's services later". What's important to understand though is that if you use this you need to know the "scope". scope is a URL to a service. Google Docs is a service for example and you need to search the web to figure out what the scope URL is for that service.

The third revelation was when I understood the difference between Simple Registration Extension (SREG) and Attribute Exchange (AX). Basically, AX is a newer more modern alternative and SREG was the first one. AX is better but some OpenID providers don't yet support it. Google for example, only supports AX. Key to be able to support not just Google's OpenID but any OpenID is that you can request both AX and SREG and whichever one works will be returned.

The fourth thing that helped a lot to understand was the Google's OpenID has a bug in its implementation of Attribute Exchange. Actually, perhaps it's a deliberate design choice they've made but in my opinion a bad one. Unless you say you require email, firstname, lastname, country etc. it won't return it. If you use the if_available directive you won't get it. Another bug/bad design choice is that Google seems to not forward the country attribute. It can happily do first- and last name but not country even if the documentation claims so.

The fifth thing is that python-openid is a lot easier to work with than you think. You don't need to do any crazy network checks or callbacks. For initiating the challenge all you're effectively doing is creating a long URL. If you don't like the API methods python openid offers, just add your own with:

redirect_url += '&openid.ax.mode=fetch_request' # etc.

After so many years since OpenID arrived, I'm only now excited about it. It's tonnes easier to implement than OAuth and now it's actually really pleasant to use as an end user.

According to my analytics the most popular Google search for getting to Crosstips.org is exactly crosstips.org which clearly proves that a lot of people type in the domain name in the search field these days. I do it too.

The second most popular search is "word whomp" which Crosstips has a dedicated page for. What's cute about that is that it was just a little side project I threw in yet it has grown to become one of the most popular features. You can never predict these kinds of things. I think the next thing I'm going to add is a Hangman solver which shouldn't be too hard.

So fellow Word Whomp cheaters, go for it!

I have a Zope web app that uses hand coded SQL (PostgreSQL) statements. Similar to the old PHP. My only excuse for not using an ORM was that this project started in 2005 and at the time SQLAlchemy seemed like a nerdy mess with more undocumented quirks than it was worth losing hair over.

Anyway, several statements use ILIKE to get around the problem of making things case insensitive. Something like the Django ORM uses UPPER to get around it. So I wonder how much the ILIKE statement slows down compared to UPPER and the indexed equal operator. Obviously, neither ILIKE or UPPER can use an index.

Long story short, here are the numbers for selecting on about 10,000 index records:

# ONE
EXPLAIN ANALYZE SELECT ... FROM ... WHERE name = 'Abbaye';
Average: 0.14 milliseconds

# TWO
EXPLAIN ANALYZE SELECT ... FROM ... WHERE  UPPER(name::text) = UPPER('Abbaye');
Average: 18.89 milliseconds

# THREE
EXPLAIN ANALYZE SELECT ... FROM ... WHERE  name ILIKE 'Abbaye';
Average: 24.63 milliseconds

First of all, the conclusion is to use UPPER instead of ILIKE if you don't need to do regular expressions. Secondly, if at all possible try to use the equal operator first and foremost and only reside on the case insensitive one if you really need to.

Lastly, in PostgreSQL is there a quick and easy to use drop in alternative to make an equality operatorish thing for varchars that are case insensitive?

UPDATE 2014

There is an answer to the question just above: citext

In lack of a fancier solution here's how I solved a problem of knowing who was logged in when an error occurred. I'm building a Intranet like system for a close group of people and if an error occurs I get an email that reminds me to add more tests. So I fix the bugs and upgrade the server. But I often want to know what poor sucker was logged in at the time the exception happened so that I can email them and say something like "Hi! I noticed your stumbled across a bug. My bad. Just wanted to let you know I've fixed that now"

So to do this I installed a silly little piece of middleware:

from django.conf import settings
class ExceptionExtraMiddleware(object):
   def process_exception(self, request, exception):
       if settings.DEBUG:
           return
       try:
           logged_in_info = ''
           if request.user and request.user.is_authenticated():
               logged_in_info = "%s" % request.user
               if request.user.email:
                   logged_in_info += ' %s' % request.user.email
               if request.user.first_name or request.user.last_name:
                   logged_in_info += ' (%s %s)' % \
                     (request.user.first_name, request.user.last_name)
           if logged_in_info:
               request.META['ERROR-X-LOGGED-IN'] = logged_in_info
       except:
           # don't make matters worse in these sensitive times
           logging.debug("Unable to debug who was logged in", exc_info=True)

This means that when I get an email with the traceback and snapshot of the request object I get this included:

...
'ERROR-X-LOGGED-IN': u'anita (Anita Test)',
...

UPDATE

The code above had a bug in it. Doing an if on request.user will return true even if there is no logged in user. The safest thing is to change it to:

if request.user and request.user.is_authenticated():

uwsgi is the latest and greatest WSGI server and promising to be the fastest possible way to run Nginx + Django. Proof here But! Is it that simple? Especially if you're involving Django herself.

So I set out to benchmark good old threaded fcgi and gunicorn and then with a source compiled nginx with the uwsgi module baked in I also benchmarked uwsgi. The first mistake I did was testing a Django view that was using sessions and other crap. I profiled the view to make sure it wouldn't be the bottleneck as it appeared to take only 0.02 seconds each. However, with fcgi, gunicorn and uwsgi I kept being stuck on about 50 requests per second. Why? 1/0.02 = 50.0!!! Clearly the slowness of the Django view was thee bottleneck (for the curious, what took all of 0.02 was the need to create new session keys and putting them into the database).

So I wrote a really dumb Django view with no sessions middleware enabled. Now we're getting some interesting numbers:

fcgi (threaded)              640 r/s
fcgi (prefork 4 processors)  240 r/s (*)
gunicorn (2 workers)         1100 r/s
gunicorn (5 workers)         1300 r/s
gunicorn (10 workers)        1200 r/s (?!?)
uwsgi (2 workers)            1800 r/s
uwsgi (5 workers)            2100 r/s
uwsgi (10 workers)           2300 r/s

(* this made my computer exceptionally sluggish as CPU when through the roof)