Peterbe.com

A blog and website by Peter Bengtsson

Filtered home page! Currently only showing blog entries under the category: Django. Clear filter

If you, like me, have various projects that do things like OAuth on Twitter or Google or you have a development site that goes to PayPal. So you're doing some Django development on http://localhost:8000/foo and click, for example, to do an OAuth on Twitter with an app you have there. Then Twitter will redirect you back to the live site with which you've set it up. But you're doing local development so you want to go back to http://localhost:8080/... instead.

Add this bookmarklet: to localhost:8000 to your browser Bookmarks toolbar and it does exactly that.

Here's its code in more verbose form:

(function() { 
   a = function(){
     location.href = window.location.href.replace(/http:\/\/[^\/]+\//,
            'http://localhost:8000/')
   };
   if (/Firefox/.test(navigator.userAgent)) { 
     setTimeout(a,0)
   } else {
      a()
   }
 })()

I have a Django model that looks something like this:

class MyModel(models.Model):
   modify_date = models.DateTimeField(auto_now=True)
   ...

Retroactively now I wanted to add a field called add_date which uses the auto_now_add=True trick. The migration used in this project is South which is great but doesn't work very well with the auto_now_add=True because the field doesn't have a straight forward default. So, first I changed the field to this:

class MyModel(models.Model):
   modify_date = models.DateTimeField(auto_now=True)
   add_date = models.DateTimeField(auto_now_add=True, null=True)
   ...

Notice the null=True which is important. Then I used startmigration to generate the code for the forward and backward to which I added a this stuff:

class Migration:

   def forwards(self, orm):

       db.add_column('myapp_mymodel', 'add_date', orm['myapp.mymodel:add_date'])
       for each in MyModel.objects.all():
           # since MyModel is referenced elsewhere I can work out the oldest date
           oldest_date = get_oldest_related_date(each, 
                              default=each.modify_date)
           each.add_date = oldest_date
           each.save()

That way all old records will have the date (not entirely accurate but good enough) and all new records will automatically get a date. Is there a better way? I bet, but I don't know how to do it.

Those Crazy Chinese My friend Chris West has built a great new site called Those Crazy Chinese which describes itself like this:

"Chinese (Mandarin) is a beautiful and highly literal language - directly translated, many words have entertaining and occasionally logical meanings."

It's built in Django and it integrates to Twitter so if you're on Twitter just follow it there to get the latest additions of new interesting and amusing literal translations. This website is mainly geared towards people who are, like Chris, learning Mandarin.

In case this bites someone else like it bit more and chewed off many many minutes of debugging time.

If you ever get weird columns in your Django Administration interface, I now know why that happens. See this screenshot example:

Messed up columns in Django Admin

This happens when you've defined a TEMPLATE_STRING_IF_INVALID in your settings.py. I always put in my local_settings.py this line:

TEMPLATE_STRING_IF_INVALID = '{{ %s }}'

So that I can quickly see which variable references in template code is potential typos. I'm not a big fan of the implicit magic of equating absence to False/None so I try to avoid the confusion altogether.

The current project I'm working has at the time of writing 20 different forms (90% model forms) instantiated in different scenarios. Django doesn't automatically strip whitespace in text based fields. So instead of doing this:

class ContactMarketingForm(forms.ModelForm):
   class Meta:
       model = ContactMarketing
       exclude = ('contact',)

   def clean_notes(self):
       return self.cleaned_data['notes'].strip()

   def clean_name(self):
       return self.cleaned_data['name'].strip()

Instead I wrote a common class for all of my form classes to use:

class _BaseForm(object):
   def clean(self):
       for field in self.cleaned_data:
           if isinstance(self.cleaned_data[field], basestring):
               self.cleaned_data[field] = self.cleaned_data[field].strip()
       return self.cleaned_data

class BaseModelForm(_BaseForm, forms.ModelForm):
   pass

class ContactMarketingForm(BaseModelForm):
   class Meta:
       model = ContactMarketing
       exclude = ('contact',)

Now all text inputs and textareas are automatically whitespace stripped. Perhaps useful for other Djangonauts.

London Frock Exchange launched Today we launched The London Frock Exchange which is a joint project between Fry-IT, Charlotte Davies and Sarah Caverhill

Elevator sales pitch: Unlike other clothes swapping sites, with Charlotte, Sarah and Rani as an expert hub in the middle you don't swap straight across; no you swap one frock in and can choose a frock (of equal value) from the pool of frocks.

Fry-IT is co-founding this venture and hope it'll make us billionaires by the end of the year (They take a small admin fee of £25 for sending you a frock back but sending it in is free with freepost). It's been great fun to work on it over the last couple of months as it means we (Fry-IT is a all-male highly technical company) have had to learn about sizes, body shapes and trying to learn how a female web audience thinks. The ladies have done a great job of seeding it with lots and lots of frocks all of which you can wear in a matter of days if you just swap one of equal value in first. Enjoy!

Here I'm going to explain a solution I had to make for a site I recently launched. Basically, I wanted to cache the whole page in memcache and set the appropriate Expires and Cache-Control headers so that my view was only rendered once an hour and parts of the page needs to be unique (i.e. "Hi, logged in as xxxx")

The advantages is great: The page loads fast, content is stored in memcache every hour, page still appears to be dynamic.

The disadvantages are not so great: the AJAX loads fast but causes a flicker

Basically, I wrote a custom decorator called custom_cache_page(<delay in seconds>) that works like the normal cache_page(<delay in seconds>) decorator available in stock Django. However, my decorator inserts a piece of HTML into the rendered HTML (before it's stored in memcache) that I later use to update certain elements of the page with AJAX instead of server side. Enough talking, let's look at the code!:

from django.utils.decorators import decorator_from_middleware
from django.middleware.cache import CacheMiddleware
class CustomCacheMiddleware(CacheMiddleware):
   def __init__(self, cache_delay=0, *args, **kwargs):
       super(CustomCacheMiddleware, self).__init__(*args, **kwargs)
       self.cache_delay = cache_delay

   def process_response(self, request, response):
       if self.cache_delay:
           extra_js = '<script type="text/javascript">var '\
                      'CACHE_CONTROL=%s;</script>' %\
                       self.cache_delay
           response.content = response.content.replace(u'</body>',
                                             u'%s\n</body>' % extra_js)

       return super(CustomCacheMiddleware, self
                   ).process_response(request, response)

custom_cache_page = decorator_from_middleware(CustomCacheMiddleware)

if settings.DEBUG:
   def custom_cache_page(delay):
       def rendered(view):
           def inner(request, *args, **kwargs):
               return view(request, *args, **kwargs)
           return inner
       return rendered

@custom_cache_page(60 * 60 * 1) # 1 hours
def my_expensive_but_cacheable_view(request):
   # only run max. once every hour
   complex_and_cpu_intensive_calculation()
   ...

What you now get is that in every page that is server-side cached in memcache (and set Expires and Cache-Control headers) get a little piece of Javascript code inserted into the rendered HTML. Now, all I need to do is to write some jQuery code that loads the navigation menu dynamically but instigate it from a AJAX request. Your mileage here might vary (put this in your base.html or whatever you call it):

$(function() {
  if (typeof CACHE_CONTROL != "undefined" &amp;&amp; CACHE_CONTROL) {
     // the page is cached, need to use AJAX to load what should be dynamic
     $('#nav').load('/_nav.html');
  }
});

I usually prefix all my views with an underscore when they only return a limited chunk of HTML rather than a whole HTML document.

Discussion

98% of my pages are deliberately not cached because they either aren't expensive to render or really can't be cached because of logged in users or whatnot.

This solution has the added benefit of being totally contained in Django, so it doesn't require any work on the nginx/apache/lighttpd front end but if you really want some speed (e.g. getting nginx talking directly to memcache) you can still use the above solution to get what you want.

My little decorator can be improved admittedly. You might perhaps add a dynamic check if the page should not be cache for example. Or perhaps some other available trick for invalidating the cache if you're really need to. Or perhaps some other tricks can be made to automate the Javascript AJAX loading.

Calling all kung fu people - kungfupeople.com Tonight we're launching our new Kung Fu website: kungfupeople.com

My friend Chris and I have been busy building a website where people who do kung fu can put themselves on a map to say where they train kung fu, what style they do and what kung fu club they belong to. The site is very much centred on having a world map and each little pin on the map is one kung fu martial artist.

This site is build in Django and is based on work that was done to build Django People originally developed by Simon Willison. We took his original code and revamped it almost completely.

Our goal is to slowly build up a world map of people from all sorts of clubs and styles and hopefully one day become the best place on the Internet for understanding what clubs are available where and what styles different people do. The site has been in an "alpha testing" phase now for a couple of weeks and even though we still have lots of ideas and cool features to add we believe it's ready to go live.

So if you train kung fu or know someone who trains kung fu go to our website and add yourself to the map

I've got this model in Django:

class MyModel(models.Model):
   completed_date = models.DateTimeField(null=True)

My dislike for booleans and that impact on the Django Admin By using a DateTimeField instead of a BooleanField I'm able to record if an instance is completed or not and when it was completed. A very common pattern in relational applications. Booleans are brief but often insufficient. (Check out Ned Batchelder's Booleans suck)

To make it a bit more convenient (and readable) to work with I added this method:

class MyModel(models.Model):
   completed_date = models.DateTimeField(null=True)

   @property
   def completed(self):
       return self.completed_date is not None

That's great! Now I can do this (use your imagination now):

>>> from myapp.models import MyModel
>>> instance = MyModel.objects.all()[0]
>>> instance.completed
False
>>> instance.completed_date = datetime.datetime.now()
>>> instance.save()
>>> instance.completed
True

I guess I could add a setter too.

But Django's QuerySet machinery doesn't really tie in with the ORM Python classes until the last step so you can't use these property methods in your filtering/excluding. What I want to do is to be able to do this:

>>> from myapp.models import MyModel
>>> completed_instances = MyModel.objects.filter(completed=True)
>>> incomplete_instances = MyModel.objects.filter(completed=False)

To be able to do that I had to add special manager which is sensitive to the parameters it gets and changes them on the fly. So, the manager plus model now looks like this:

class SpecialManager(models.Manager):
   """turn certain booleanesque parameters into date parameters"""

   def filter(self, *args, **kwargs):
       self.__transform_kwargs(kwargs)
       return super(SpecialManager, self).filter(*args, **kwargs)

   def exclude(self, *args, **kwargs):
       self.__transform_kwargs(kwargs)
       return super(SpecialManager, self).exclude(*args, **kwargs)

   def __transform_kwargs(self, kwargs):
       bool_name, date_name = 'completed', 'completed_date'
       for key, value in kwargs.items():
           if bool_name == key or key.startswith('%s__' % bool_name):
               if kwargs.pop(key):
                   kwargs['%s__lte' % date_name] = datetime.now()
               else:
                   kwargs[date_name] = None

class MyModel(models.Model):
   completed_date = models.DateTimeField(null=True)

   @property
   def completed(self):
       return self.completed_date is not None

Now, that's fine but there's one problem. For the application in hand, we're relying on the admin interface a lot. Because of the handy @property decorator I set on the method completed() I now can't include completed into the admin's list_display so I have to do this special trick:

class MyModelAdmin(admin.ModelAdmin):
   list_display = ('is_completed',)

   def is_completed(self, object_):
       return object_.completed
   is_completed.short_description = u'Completed?'
   is_completed.boolean = True

Now, I get the same nice effect in the admin view where this appears as a boolean. The information is still there about when it was completed if I need to extract that for other bits and pieces such as an advanced view or auditing. Pleased!

Now one last challenge with the Django admin interface was how to filter on these non-database-fields? It's been deliberately done so that you can't filter on methods but it's slowly changing and with some hope it'll be in Django 1.2. But I'm not interested in making my application depend on a patch to django.contrib but I really want to filter in the admin. We've already added some custom links and widgets to the admin interface.

After a lot of poking around and hacking together with my colleague Bruno ReniƩ we came up with the following solution:

class MyModelAdmin(admin.ModelAdmin):
   list_display = ('is_completed',)

   def is_completed(self, object_):
       return object_.completed
   is_arrived.short_description = u'Completed?'
   is_arrived.boolean = True

   def changelist_view(self, request, extra_context=None, **kwargs):
       from django.contrib.admin.views.main import ChangeList
       cl = ChangeList(request, self.model, list(self.list_display),
                       self.list_display_links, self.list_filter,
                       self.date_hierarchy, self.search_fields, 
                       self.list_select_related,
                       self.list_per_page,
                       self.list_editable, self)
       cl.formset = None

       if extra_context is None:
           extra_context = {}

       if kwargs.get('only_completed'):
           cl.result_list = cl.result_list.exclude(completed_date=None)
           extra_context['extra_filter'] = "Only completed ones"

       extra_context['cl'] = cl
       return super(SendinRequestAdmin, self).\
         changelist_view(request, extra_context=extra_context)

   def get_urls(self):
       from django.conf.urls.defaults import patterns, url
       urls = super(SendinRequestAdmin, self).get_urls()
       my_urls = patterns('',
               url(r'^only-completed/$', 
                   self.admin_site.admin_view(self.changelist_view),
                    {'only_completed':True}, name="changelist_view"),
       )
       return my_urls + urls

Granted, we're not getting the nice filter widget on the right hand side in the admin interface this time but it's good enough for me to be able to make a special link to /admin/myapp/mymodel/only-completed/ and it works just like a normal filter.

Ticket 5833 is quite busy and has been going on for a while. It feels a daunting task to dig in and contribute when so many people are already ahead of me. By writing this blog entry hopefully it will help other people who're hacking on their Django admin interfaces who, like me, hate booleans.

I've now made a vastly improved spellcorrector specifically tied into Django and it's models. It's the old class as before but hooked up to models so Django can take care of persisting the trained words. Again, I have to give tribute to Peter Norvig for his inspirational blog How to Write a Spelling Corrector which a large majority of my code is based in. At least in the tricky parts.

What's nice about this little app is that it's very easy to plug in and use. You just download it, put it on your PATH and include it in your INSTALLED_APPS. Then from another app you do something like this:

from spellcorrector.views import Spellcorrector
sc = Spellcorrector()
sc.load() # nothing will happen the first time

sc.train(u"peter")
print sc.correct(u"petter") # will print peter
sc.save()

sc2 = Spellcorrector()
sc2.load()
print sc2.correct(u"petter") # will print peter

This is my first django-*** app I've wrapped up and released in github so I'm not even near certain that I've done it right.

I'll try to keep the project updated and more approachable but right now it works in one of my production sites so I'm confident the code is ok.

Screenshots

Introducing django-spellcorrector

Introducing django-spellcorrector

Introducing django-spellcorrector