Playing with filestream_iterator

30 May 2007   0 comments   Zope

Mind That Age!

This blog post is 10 years old! Most likely, its content is outdated. Especially if it's technical.

Powered by Fusion×

There are several ways of serving static files from Zope. The simplest way is to just do something like this:

size = os.stat(file_path)[stat.ST_SIZE]
return open(file_path, 'rb').read()

The disadvantage with this is if the file is say >1Mb, the whole 1Mb needs to be loaded into RAM before it can be sent. This is surely garbage collected afterwards but if you serve many of these at the same time, there's obviously the risk that the RAM gets too full.

The alternative is to use filestream_iterator from the ZPublisher which is an iterator that sends out chunks of the file at a time. Note that to use the filestream_iterator you must set the Content-Length header first. Here's how to use it:

from ZPublisher.Iterators import filestream_iterator
size = os.stat(file_path)[stat.ST_SIZE]
return filestream_iterator(file_path, 'rb')

I did a little unofficial benchmark to test if there's any speed difference between the two approaches. I'm currently only interested in serving photographs (jpg) in this app and I'm sure other results would be found if you did bigger filesystem files. These jpgs are in average about 200 Kb. During the course of writing this I tought I'd also try using sendfile as well to see if that makes it any faster:

Serving 526 jpgs with...
function             average total time (5 tests)
  open()               5.07750701904
  filestream_iterator  4.32704296112
  sendfile             4.431672620774

Interesting. Using filestream_iterator shaves off a few milliseconds but it's over many many servings. Conclusion, the benefit of filestream_iterator is not speed, it's RAM (un)usage.

Note. From reading the LocalFS source code I noticed that they only bother with the filestream_iterator if the filesize is greater than 128Kb. I copied that too into my app now but don't understand why. Is there a disadvantage of serving tiny files with filestream_iterator() Anybody?

Worth noting that sendfile wasn't any faster but I suppose you only see a speed boost with sendfile on much larger files.


Thank you for posting a comment

Your email will never ever be published

Related posts

American foreign policy 25 May 2007
<button> not supported on Pocket IE 02 June 2007
Local NodeJS development environment with Nginx 01 September 2010
More optimization of - CSS sprites 05 August 2009
Gzip and Slimmer optimization anecdote 30 January 2007
Entering multilanguage data in Zope 26 November 2006