Optimization story involving something silly I call "dict+"

13 June 2011   0 comments   Python, MongoDB


Powered by Fusion×

Here's a little interesting story about using MongoKit to quickly draw items from a MongoDB

So I had a piece of code that was doing a big batch update. It was slow. It took about 0.5 seconds per user and I sometimes had a lot of users to run it for.

The code looked something like this:

 for play in db.PlayedQuestion.find({'user.$id': user._id}):
    if play.winner == user:
    elif play.draw:

Because the model PlayedQuestion contains DBRefs MongoKit will automatically look them up for every iteration in the main loop. Individually very fast (thank you indexes) but because of the number of operations very slow in total. Here's how to make it much faster:

   for play in db.PlayedQuestion.collection.find({'user.$id': user._id}):

The problem with this is that you get dict instances for each which is more awkward to work with. I.e. instead of `play.winner` you have use `play['winner'].id`. Here's my solution that makes this a lot easier:

class dict_plus(dict):

  def __init__(self, *args, **kwargs):
       if 'collection' in kwargs:  # excess we don't need
       dict.__init__(self, *args, **kwargs)

   def _wrap_internal_dicts(self):
       for key, value in self.items():
           if isinstance(value, dict):
               self[key] = dict_plus(value)

   def __getattr__(self, key):
       if key.startswith('__'):
           raise AttributeError(key)
       return self[key]


 for play in db.PlayedQuestion.collection.find({'user.$id': user._id}):
    play = dict_plus(play)
    if play.winner.id == user._id:
    elif play.draw:

Now, the whole thing takes 0.01 seconds instead of 0.5. 50 times faster!!


Thank you for posting a comment

Your email will never ever be published

Related posts

Launching Kwissle.com 04 June 2011
Chinese tea sampler pack now on sale 16 June 2011
Don't forget your sets in Python! 10 March 2017
Optimization of QuerySet.get() with or without select_related 03 November 2016
How to no-mincss links with django-pipeline 03 February 2016
mozjpeg installation and sample 10 October 2015
Examples of mozjpeg savings 01 September 2015
Introducing optisorl 18 August 2015
Python slow-down of exception handling or condition checking 14 May 2015
AJAX or not 22 December 2014
Gzip rules the world of optimization, often 09 August 2014
Optimizing MozTrap 04 June 2014