Crosstips.org

My fun Crossword solver project. Crosstips.org & Krysstips.se

Kung Fu

Fujian White Crane Kung Fu

Fry-IT

Fry-IT is the company I work for

Photos

Photoalbum, both old and new.

Zope

What I have and am doing with Zope

Receptsamlingen

In Swedish only. About my "Collection of Recipes" website.

Contact me

My contact details and how to contact me.

 

KungFuPeople.com
Do you train Kung Fu?
Or know someone who does?
Then check out KungFuPeople.com


Mobile version of this page Mobile version of this page

RSS

Hot topics

by Peter Bengtsson: Lack of transactions is definitely a key pain point. It changes your code a...

Speed test between django_mongokit and postgresql_

by Peter Bengtsson: I'm sure there are parameters to make it faster but there are parameters to...

Speed test between django_mongokit and postgresql_

by Peter Bengtsson: Both when it retrieves for editing and for deleting it does selects by key....

Speed test between django_mongokit and postgresql_

by Eas: Michael, there are other postgres tuning parameters that help with throughp...

Speed test between django_mongokit and postgresql_

by Lior Gradstein: You made an error in one of the assignements, you wrote cpu_hrz, so in the ...

How and why to use django-mongokit (aka. Django to

by Alex: You're comparing 1000 transactions vs. 1000 inserts to a database that does...

Speed test between django_mongokit and postgresql_

by Michael Pasternak: I'm following your posts carefully :-) It seems, that "good old SQL" is of...

Speed test between django_mongokit and postgresql_

by Massimiliano Torromeo: What about SELECTs?...

Speed test between django_mongokit and postgresql_

by Gudbergur Erlendsson: I think it's lacking DRY. This is more beautiful IMO: #models.py from mong...

How and why to use django-mongokit (aka. Django to

by Peter Bengtsson: I could. Perhaps when time allows. Feel free to fork it on github if you fe...

Speed test between django_mongokit and postgresql_

Old entries


September, 2009
London Frock Exchange launched
My first Twitter app - KungFuPeople.com
Comparing jsmin and slimmer
Python Code Dojo London - 17 Sep 2009
"Hello John. It's Gordon Brown."
7 of the World's Most Irresponsible Companies

August, 2009
Cgunit - Online Gallery
To sub-select or not sub-select in PostgreSQL
Custom CacheMiddleware that tells Javascript a page is cached in Django
What a super user-friendly menu!
Table Of Countries Showing Drive Direction
The Secret to SEO Search Engine Optimization
Calling all kung fu people - kungfupeople.com
Google Reverse Geocoding vs. GeoNames
gg - wrapping git-grep
Public calendars on Google Calendar
More optimization of Peterbe.com - CSS sprites

2009
2008
2007
2006
2005
2004
2003

 

You're viewing blogs from Django only.

View all different categories

9th of March

Speed test between django_mongokit and postgresql_psycopg2

Following on from yesterday's blog about How and why to use django-mongokit I extended the exampleproject which is inside the django-mongokit project with another app called exampleapp_sql which does the same thing as the exampleapp but does it with SQL instead. Then I added a very simple benchmarker app in the same project and wrote three functions:

  1. One to create 10/100/500/1000 instances of my class
  2. One to edit one field of all 10/100/500/1000 instances
  3. One to delete each of the 10/100/500/1000 instances


>Read the whole text (325 more words)

8th of March

How and why to use django-mongokit (aka. Django to MongoDB)

How and why to use django-mongokit Here I'm going to explain how to combine Django and MongoDB using MongoKit and django-mongokit.

MongoDB is a document store built for high speed and high concurrency with a very good redundancy story. It's an alternative to relational databases (e.g. MySQL) that is what Django is tightly coupled with in it's ORM (Object Relation Mapping) and what it's called now is ODM (Object Document Mapping) in lack of a better acronym. That's where MongoKit comes in. It's written in Python and it connects to the MongoDB database using a library called pymongo and it turns data from the MongoDB and turns it into instances of classes you have defined. MongoKit has nothing to do with Django. That's where django-mongokit comes in. Written by yours truly.


>Read the whole text (1551 more words)

16th of January

Bookmarklet to replace the current domain with localhost:8000

If you, like me, have various projects that do things like OAuth on Twitter or Google or you have a development site that goes to PayPal. So you're doing some Django development on http://localhost:8000/foo and click, for example, to do an OAuth on Twitter with an app you have there. Then Twitter will redirect you back to the live site with which you've set it up. But you're doing local development so you want to go back to http://localhost:8080/... instead.

Add this bookmarklet: to localhost:8000 to your browser Bookmarks toolbar and it does exactly that.

Here's its code in more verbose form:

 (function() { 
    a = function(){
      location.href = window.location.href.replace(/http:\/\/[^\/]+\//,
             'http://localhost:8000/')
    };
    if (/Firefox/.test(navigator.userAgent)) { 
      setTimeout(a,0)
    } else {
       a()
    }
  })()

16th of December

Migrating with South on a field that uses auto_now_add=True

I have a Django model that looks something like this:

 class MyModel(models.Model):
    modify_date = models.DateTimeField(auto_now=True)
    ...

Retroactively now I wanted to add a field called add_date which uses the auto_now_add=True trick. The migration used in this project is South which is great but doesn't work very well with the auto_now_add=True because the field doesn't have a straight forward default. So, first I changed the field to this:

 class MyModel(models.Model):
    modify_date = models.DateTimeField(auto_now=True)
    add_date = models.DateTimeField(auto_now_add=True, null=True)
    ...

Notice the null=True which is important. Then I used startmigration to generate the code for the forward and backward to which I added a this stuff:

 class Migration:

    def forwards(self, orm):

        db.add_column('myapp_mymodel', 'add_date', orm['myapp.mymodel:add_date'])
        for each in MyModel.objects.all():
            # since MyModel is referenced elsewhere I can work out the oldest date
            oldest_date = get_oldest_related_date(each, 
                               default=each.modify_date)
            each.add_date = oldest_date
            each.save()

That way all old records will have the date (not entirely accurate but good enough) and all new records will automatically get a date. Is there a better way? I bet, but I don't know how to do it.

15th of November

Those Crazy Chinese

Those Crazy Chinese My friend Chris West has built a great new site called Those Crazy Chinese which describes itself like this:

"Chinese (Mandarin) is a beautiful and highly literal language - directly translated, many words have entertaining and occasionally logical meanings."

It's built in Django and it integrates to Twitter so if you're on Twitter just follow it there to get the latest additions of new interesting and amusing literal translations. This website is mainly geared towards people who are, like Chris, learning Mandarin.

16th of October

Messed up columns in Django Admin

In case this bites someone else like it bit more and chewed off many many minutes of debugging time.

If you ever get weird columns in your Django Administration interface, I now know why that happens. See this screenshot example:

Messed up columns in Django Admin

This happens when you've defined a TEMPLATE_STRING_IF_INVALID in your settings.py. I always put in my local_settings.py this line:

 TEMPLATE_STRING_IF_INVALID = '{{ %s }}'

So that I can quickly see which variable references in template code is potential typos. I'm not a big fan of the implicit magic of equating absence to False/None so I try to avoid the confusion altogether.

12th of October

Automatically strip whitespace in Django forms

The current project I'm working has at the time of writing 20 different forms (90% model forms) instantiated in different scenarios. Django doesn't automatically strip whitespace in text based fields. So instead of doing this:

 class ContactMarketingForm(forms.ModelForm):
    class Meta:
        model = ContactMarketing
        exclude = ('contact',)

    def clean_notes(self):
        return self.cleaned_data['notes'].strip()

    def clean_name(self):
        return self.cleaned_data['name'].strip()

Instead I wrote a common class for all of my form classes to use:

 class _BaseForm(object):
    def clean(self):
        for field in self.cleaned_data:
            if isinstance(self.cleaned_data[field], basestring):
                self.cleaned_data[field] = self.cleaned_data[field].strip()
        return self.cleaned_data

 class BaseModelForm(_BaseForm, forms.ModelForm):
    pass

 class ContactMarketingForm(BaseModelForm):
    class Meta:
        model = ContactMarketing
        exclude = ('contact',)

Now all text inputs and textareas are automatically whitespace stripped. Perhaps useful for other Djangonauts.

26th of September

London Frock Exchange launched

London Frock Exchange launched Today we launched The London Frock Exchange which is a joint project between Fry-IT, Charlotte Davies and Sarah Caverhill

Elevator sales pitch: Unlike other clothes swapping sites, with Charlotte, Sarah and Rani as an expert hub in the middle you don't swap straight across; no you swap one frock in and can choose a frock (of equal value) from the pool of frocks.

Fry-IT is co-founding this venture and hope it'll make us billionaires by the end of the year (They take a small admin fee of £25 for sending you a frock back but sending it in is free with freepost). It's been great fun to work on it over the last couple of months as it means we (Fry-IT is a all-male highly technical company) have had to learn about sizes, body shapes and trying to learn how a female web audience thinks. The ladies have done a great job of seeding it with lots and lots of frocks all of which you can wear in a matter of days if you just swap one of equal value in first. Enjoy!

24th of August

Custom CacheMiddleware that tells Javascript a page is cached in Django

Here I'm going to explain a solution I had to make for a site I recently launched. Basically, I wanted to cache the whole page in memcache and set the appropriate Expires and Cache-Control headers so that my view was only rendered once an hour and parts of the page needs to be unique (i.e. "Hi, logged in as xxxx")

The advantages is great: The page loads fast, content is stored in memcache every hour, page still appears to be dynamic.

The disadvantages are not so great: the AJAX loads fast but causes a flicker

Basically, I wrote a custom decorator called custom_cache_page(<delay in seconds>) that works like the normal cache_page(<delay in seconds>) decorator available in stock Django. However, my decorator inserts a piece of HTML into the rendered HTML (before it's stored in memcache) that I later use to update certain elements of the page with AJAX instead of server side.


>Read the whole text (519 more words)

18th of August

Calling all kung fu people - kungfupeople.com

Calling all kung fu people - kungfupeople.com Tonight we're launching our new Kung Fu website: kungfupeople.com

My friend Chris and I have been busy building a website where people who do kung fu can put themselves on a map to say where they train kung fu, what style they do and what kung fu club they belong to. The site is very much centred on having a world map and each little pin on the map is one kung fu martial artist.

This site is build in Django and is based on work that was done to build Django People originally developed by Simon Willison. We took his original code and revamped it almost completely.

Our goal is to slowly build up a world map of people from all sorts of clubs and styles and hopefully one day become the best place on the Internet for understanding what clubs are available where and what styles different people do. The site has been in an "alpha testing" phase now for a couple of weeks and even though we still have lots of ideas and cool features to add we believe it's ready to go live.

So if you train kung fu or know someone who trains kung fu go to our website and add yourself to the map

1st of June

My dislike for booleans and that impact on the Django Admin

I've got this model in Django:

 class MyModel(models.Model):
    completed_date = models.DateTimeField(null=True)

My dislike for booleans and that impact on the Django Admin By using a DateTimeField instead of a BooleanField I'm able to record if an instance is completed or not and when it was completed. A very common pattern in relational applications. Booleans are brief but often insufficient. (Check out Ned Batchelder's Booleans suck)

To make it a bit more convenient (and readable) to work with I added this method:

 class MyModel(models.Model):
    completed_date = models.DateTimeField(null=True)

    @property
    def completed(self):
        return self.completed_date is not None

That's great! Now I can do this (use your imagination now):

 >>> from myapp.models import MyModel
 >>> instance = MyModel.objects.all()[0]
 >>> instance.completed
 False
 >>> instance.completed_date = datetime.datetime.now()
 >>> instance.save()
 >>> instance.completed
 True

I guess I could add a setter too.

But Django's QuerySet machinery doesn't really tie in with the ORM Python classes until the last step so you can't use these property methods in your filtering/excluding. What I want to do is to be able to do this:

 >>> from myapp.models import MyModel
 >>> completed_instances = MyModel.objects.filter(completed=True)
 >>> incomplete_instances = MyModel.objects.filter(completed=False)

To be able to do that I had to add special manager which is sensitive to the parameters it gets and changes them on the fly. So, the manager plus model now looks like this:

 class SpecialManager(models.Manager):
    """turn certain booleanesque parameters into date parameters"""

    def filter(self, *args, **kwargs):
        self.__transform_kwargs(kwargs)
        return super(SpecialManager, self).filter(*args, **kwargs)

    def exclude(self, *args, **kwargs):
        self.__transform_kwargs(kwargs)
        return super(SpecialManager, self).exclude(*args, **kwargs)

    def __transform_kwargs(self, kwargs):
        bool_name, date_name = 'completed', 'completed_date'
        for key, value in kwargs.items():
            if bool_name == key or key.startswith('%s__' % bool_name):
                if kwargs.pop(key):
                    kwargs['%s__lte' % date_name] = datetime.now()
                else:
                    kwargs[date_name] = None

 class MyModel(models.Model):
    completed_date = models.DateTimeField(null=True)

    @property
    def completed(self):
        return self.completed_date is not None

Now, that's fine but there's one problem. For the application in hand, we're relying on the admin interface a lot. Because of the handy @property decorator I set on the method completed() I now can't include completed into the admin's list_display so I have to do this special trick:

 class MyModelAdmin(admin.ModelAdmin):
    list_display = ('is_completed',)

    def is_completed(self, object_):
        return object_.completed
    is_completed.short_description = u'Completed?'
    is_completed.boolean = True

Now, I get the same nice effect in the admin view where this appears as a boolean. The information is still there about when it was completed if I need to extract that for other bits and pieces such as an advanced view or auditing. Pleased!

Now one last challenge with the Django admin interface was how to filter on these non-database-fields? It's been deliberately done so that you can't filter on methods but it's slowly changing and with some hope it'll be in Django 1.2. But I'm not interested in making my application depend on a patch to django.contrib but I really want to filter in the admin. We've already added some custom links and widgets to the admin interface.

After a lot of poking around and hacking together with my colleague Bruno Renié we came up with the following solution:

 class MyModelAdmin(admin.ModelAdmin):
    list_display = ('is_completed',)

    def is_completed(self, object_):
        return object_.completed
    is_arrived.short_description = u'Completed?'
    is_arrived.boolean = True

    def changelist_view(self, request, extra_context=None, **kwargs):
        from django.contrib.admin.views.main import ChangeList
        cl = ChangeList(request, self.model, list(self.list_display),
                        self.list_display_links, self.list_filter,
                        self.date_hierarchy, self.search_fields, 
                        self.list_select_related,
                        self.list_per_page,
                        self.list_editable, self)
        cl.formset = None

        if extra_context is None:
            extra_context = {}

        if kwargs.get('only_completed'):
            cl.result_list = cl.result_list.exclude(completed_date=None)
            extra_context['extra_filter'] = "Only completed ones"

        extra_context['cl'] = cl
        return super(SendinRequestAdmin, self).\
          changelist_view(request, extra_context=extra_context)

    def get_urls(self):
        from django.conf.urls.defaults import patterns, url
        urls = super(SendinRequestAdmin, self).get_urls()
        my_urls = patterns('',
                url(r'^only-completed/$', 
                    self.admin_site.admin_view(self.changelist_view),
                     {'only_completed':True}, name="changelist_view"),
        )
        return my_urls + urls

Granted, we're not getting the nice filter widget on the right hand side in the admin interface this time but it's good enough for me to be able to make a special link to /admin/myapp/mymodel/only-completed/ and it works just like a normal filter.

Ticket 5833 is quite busy and has been going on for a while. It feels a daunting task to dig in and contribute when so many people are already ahead of me. By writing this blog entry hopefully it will help other people who're hacking on their Django admin interfaces who, like me, hate booleans.

27th of May

Introducing django-spellcorrector

I've now made a vastly improved spellcorrector specifically tied into Django and it's models. It's the old class as before but hooked up to models so Django can take care of persisting the trained words. Again, I have to give tribute to Peter Norvig for his inspirational blog How to Write a Spelling Corrector which a large majority of my code is based in. At least in the tricky parts.

What's nice about this little app is that it's very easy to plug in and use. You just download it, put it on your PATH and include it in your INSTALLED_APPS. Then from another app you do something like this:

 from spellcorrector.views import Spellcorrector
 sc = Spellcorrector()
 sc.load() # nothing will happen the first time

 sc.train(u"peter")
 print sc.correct(u"petter") # will print peter
 sc.save()

 sc2 = Spellcorrector()
 sc2.load()
 print sc2.correct(u"petter") # will print peter


>Read the whole text (79 more words)

22nd of May

Crossing the world - new feature on Crosstips

Crossing the world - new feature on Crosstips I've added a very fun new feature on Crosstips called Crossing the world which shows real-time searches happening all over the world. Admittedly the traffic on Crosstips isn't particularly high, (At the time of writing, 1 search every 2 minutes) so you might have to sit there for a while until something happens. It's strangely addictive to watch it.

To do this I had to use all sorts of buzz words. AJAX, function cache decorators, GeoIP and Google Maps. I'm currently using the free version of GeoIP City Lite which seems to work on a large majority of all captured IP addresses. And since the map is sufficiently zoomed out you can't really tell how inaccurate it is.

One little detail I'm quite proud of is how the AJAX code understands how to change interval between lookups. Each time the server responds with something, the interval is reduced down but if there aren't any new searches the interval slowly increases again. This is done to minimize the number of useless server requests but at the same time try to make it react often if there are plenty of things to show. The next feature to add is Comet (like AJAX but push instead of pull).

Now if we could only get some more action on the site!! Tell all your grand-people to use this site when they get stuck on solving crossword puzzles!

UPDATE

I've just learnt that GeoIP is already shipped in GeoDjango so I've basically reinvented half a wheel :(

11th of February

Krysstips.se

Krysstips.seKrysstips.se is my latest hobby project that I'm launching. It's a Django app that has a huge database of Swedish names and words in various forms. With this tool, when you get stuck doing a crossword you can get hints. E.g. "a 5 letter word that starts with p and second to last letter 'e'"

I'm now looking for an English database that I'm allowed to load and I'll write an English equivalent of the site. Any help appreciated! In fact, if you have any other language you'd like to add I'm more than happy to join forces.

This site started as a funny side project just to see if it was possible. I crunched the database file of words into a PostgreSQL database in a couple of hours and I wrote the actual search algorithm in another couple of hours. After that, spread over many late evenings and procrastinating tea breaks, I've spent, in total, about 5 man days doing all the other fluff which was: "stealing" a CSS, making it internationalized, translating the code and templates to Swedish, AJAXifying the search, adding the statistics and deploying it.

I'm not expecting high load or to make money on it but it's been done and I'm quite proud of it so I might as well announce it to the world.

17th of January

Nasty surprise of Django and gettext

I've done a site that is initially only going to be in Swedish. So I set the LANGUAGE_CODE like this:

 # Language code for this installation. All choices can be found here:
 http://www.i18nguy.com/unicode/language-identifiers.html
 LANGUAGE_CODE = 'sv-SE'

I then go ahead and mark my templates and code and run django-admin.py makemessages -l sv-SE. Then I fire up Poedit to translate the .po file and then compile to make the .mo file, so now I had a file called <my project>/locale/sv-SE/LC_MESSAGES/django.mo but it just wouldn't work!! I tried debugging django.utils.translation with little success. Eventually through trial-and-error I found that if I change it to "sv" instead of "sv-SE" it works.

Is this a problem with gettext or a problem somewhere in Django? It might cause equally much headache for other developers so I wouldn't mind that we get to the bottom of this.

16th of December

My first YSlow Grade A (100)!

My first YSlow Grade A (100)! Just put in the remaining necessary cache headers on my kungfu club mobile site and I ran a YSlow! test on it and it got a 100! I've never had that before.

Admittedly, this is like cheating since the site is for mobile use it's kept very very simple and has no external dependencies so no need for a CDN. Getting a Grade A on YSlow! is very hard if the site needs to really blink. This one is built dedicated for small mobile phone screens on slow connections. I think now the bottleneck isn't in the rendering but in the connection latency for the remote database.

It also got a 4.9 (max 5.0) on ready.mobi which is the highest I've ever achieved there.


>Read the whole text (95 more words)

9th of December

Nasty surprise of Django cache

Caching in Django absolutely rocks. Much so because of its simplicity which lowers the threshold to the height of a snowflake. However the simplicity can cause some unexpected surprises.

I ran my site and clicked around until I found a bug I wanted to fix. Then I wrote the test and ran the testrunner but I was getting some really weird behavior. The view looked like this:

 @never_cache
 def club_page(request, clubname):
    club = _find_club(clubname)
    if club is None:
        raise Http404('Could not find the club')

    classes = cache.get('club_page_classes')
    if classes is None:
        classes = ClubClass.objects.filter(club=club).order_by('start_time')
        cache.set('club_page_classes', classes, CACHE_TIMEOUT)
    ...

What happened (and what took me a while to figure out) was that the memcache was still active and being used when running the tests since it's only within the hour that I started running the tests with completely different data. Here's how I solved the problem:

 class ViewsTestCase(unittest.TestCase):
     ...
     def test_club_page(self):
        """ test rendering the club page """
        cache.delete('club_page_classes')
        ...

There must be an easier way to patch the testrunner to reset all cache before running the tests. Anybody?

14th of November

How to unit test the innards of a Django view function

Seconds ago I got this running and haven't yet fully understood what I've just done but the results are exactly what I need and this is going to be great.

Basically, in Django you have views like this:

 def _render(template, data, request):
    return render_to_response(template, data,
              context_instance=RequestContext(request))

 @login_required
 def club_page(request, year):
    variable1 = year / 4
    variable2 = variable1 * 2
    return _render("foo.html", locals(), request)

Now in my unit tests I don't want to have to call the view function and then have to dissect the resulting HTML just to figure out if the view function prepared the correct variables. So, here's my solution to this problem:


>Read the whole text (410 more words)

25th of October

Django vs. Java

From the django-users mailinglist which I'm becoming more and more helpful in:

 > Could you share approximately how big your project is? I know it's
 > hard to find a real measure for this, but how about number of database
 > tables?

 A project I worked on over the summer used a Database that was 130
 tables, and getting 1gb updates every 2 minutes. I was witting a new
 web app to do calculations on the data and the company wanted to use
 Java since thats what they knew best and had spend huge amounts of
 money (1 mil +) to support with Sun Servers, and such. But I knew
 python and django would be a better fit for this particular app, but
 the boss wouldnt listen. So we had 10 Developers working on the Java
 version (Including me) and over 3 months we got it about 85% done,
 though it had no unit tests. During the same three months, I worked on
 my own time after work and basically had no life for the whole time, I
 was able to get the web app 100% complete with unit tests. That
 convinced my boss that Django was a good fit.

 The site is an internal app that I cannot give access to (And I
 actually had to get permission to give what info I have), but I can
 say that Django is a suitable framework for what you are looking for. 

Christ! 10 developers and no unit test!? Someone should remind them that you don't write unit tests for your bosses pleasure but for your own sanity and productivity.

I know that this quote is totally unscientific since Dj, as he says, can't back it up but it's a story interesting enough.

15th of June

Another brownie point for Django

Another brownie point for DjangoI've been working with Django a lot lately and while I can't contribute to the code base until my project is done, I can contribute money.

Been browsing the Django mailinglist and found this guy (link above) and some other people saying they're willing to donate money towards the OS effort that is Django. That's nuts and is a failed practice but it does mean a lot. Zope had this about 4-5 years ago too but that was then. Clearly the heat is all on Django (and Rails admittedly) at the moment. Well done to all involved!

I've been doing quite a bit of Django this weekend and this instant quick rush I got from getting started has gone off and now starts to become just normal trudging. At the moment it's the templating language that annoys the hell out of me.

In conclusion: Today Django won another point in the race for my attention.