Major performance fix on file searches

19 November 2005   0 comments   Zope, IssueTrackerProduct

Powered by Fusion×

A week ago I ran some ad hoc benchmarks on various suspect functions in the IssueTrackerProduct and came to a clear and simple conclusion: searching is the bottleneck and within the search it's the searching for file attachments that take all of the time.

If you're interested and open minded, here's the results of that benchmark This sparked some thoughts. First I wrote a filename splitter which isn't rocket science but I'm proud to say that it's use is brilliant. Before, the find-by-file function in the IssueTrackerProduct used a plain old find() test like this:

filename.find('foo') > -1

This is very fast but not very intelligent. For example it'll with match on foobar.txt and plainfooter.gif. So, what I did instead was to create a KeywordIndex and index all the splitted filenames in that index.

What you have to remember is that KeywordIndexes are case sensitive so when I populate the KeywordIndex I have to lowercase everything. But, now let's get to the performance fix.

The problem was that before, I used a ZopeFind() to find files by filename and ZopeFind() is slooooowww compared to a catalog search. Before I switched on the new code that searches the filename splitted keyword index I did a coule of searches on an issuetracker with about 200 issues and a couple of searches on an issuetracker with more than 10,000 issues. The results can be seen here: itp-benchmark-medium-before.log and itp-benchmark-huge-before.log Quite terrible isn't it. Then I switched on the new code that is much more intelligent and hopefully faster. The results can be seen here: itp-benchmark-medium-after.log and itp-benchmark-huge-after.log



As you can see, that's an enourmous increase in speed. For the medium sized issuetracker, that's a 600% increase and for the huge issuetracker that's a 2000% speed increase. Oops! This shouldn't be called optimization maybe. Perhaps bug fixing is a better word :)


Thank you for posting a comment

Your email will never ever be published

Related posts

Automatically refreshing issue 17 November 2005
createElement('a') with a javascript href 21 November 2005