Peterbe.com

IssueTrackerProduct now officially abandoned

30 March 2012 6 comments   Zope, IssueTrackerProduct

In 2001 I started my first and perhaps most successful Open Source project I've ever made: IssueTrackerProduct. After nearly a decade of maintaining it I have now officially abandoned it.

It all started when I needed a way to track feedback on my personal website. That's why it was originally called "SiteTrackerProduct". I needed something where I could collect bug reports and any other pieces of feedback and then process it in some structured fashion. It was therefore very important that it would be possible to run the application open for anonymous access. People should be able to submit bugs and issues without having to create an account. You see, kids, back in that day it was actually very common that sites would force users to register and create accounts even just because the content owner wanted it. These days, it's common knowledge that to get people to open up and share anything for others to benefit you make it absolutely trivial to jump straight in without having to see a registration page that looks like a tax return form.

Now, since I long ago abandoned the Zope2 application server technology stack and I no longer use IssueTrackerProduct for anything real it's no longer feasible to maintain this project. In the last five years or so we were actually using it actively to track all projects at Fry-IT where I used to work. I have to say, even though we did grow out of it, it was actually successful. It handled the load (after some much needed patches towards optimization) and it was easy for people to actually use since unlike many other bug trackers, it focused on the non-technical end user first and foremost. As much as possible was done to make it trivial to type in your bug or issue and it automatically took care of all notifications and access rights.

Being a personal Open Source project, over the years, it became a melting pot for experimenting and perfecting various new ideas. Many of them we take for granted today but back then it was quite novel if I may say so. This includes:

• ability to auto-save unfinished form inputs (added before Gmail had it)
• automatic updates of the content without reload made it possible to see other people participating as you're typing
• ability to reply directly to email notifications without having to open a web browser
• an advanced via-the-web programmable interface for adding and modifying custom fields (e.g. "Customer reference code")
• full-text search combined with ability to search on specific fields by key
• file attachments that are images automatically appear as little thumbnails
• file attachments that have text become searchable (e.g Word documents)
• advanced filtering where you can easily decide to search inclusive or exclusive on certain fields
• persistent filtering automatically saved and share'able between different users
• programmable search filters that are coded in Python which made it possible to create very specific reports
• ability to export and import bugs from and to Excel for offline processing

Writing all of this, I can not resist to get a bit nostalgic. I did sink A LOT of time into this project. Today when I look back at the code and almost feel sick seeing all the mistakes that I made. Much of the ugliness of the code can be attributed partially to the fact that I often used and abused the code to add new features. Also, because we often needed some features (since it was used to manage all of our projects) "yesterday" and then it was hard to justify doing things "properly". For example, the main .py file is over 14,000 lines of code!

I did called it "perhaps most successful Open Source project I've ever made" in the first sentence. The reason for that is that over the years many many people have downloaded it and installed and let it be used by thousands of users. That's something to be proud of.

Anyway! It's time to move on. So long and thank you for all the fish!

The code is still available at github.com/peterbe/IssueTrackerProduct

To all Zope developers: Does this sound familiar?

08 March 2011 2 comments   Zope

I was reading this article about linkfluence moving from CouchDB to Riak

Does that sound familiar, fellow Zope developer? I know a lot about ZODB but little about CouchDB. One thing that a lot of people don't know about ZODB is that it's very fast and I think this is true about CouchDB too. Speed isn't the same as a raw speed of inserts/queries because with the concurrency variable added the story gets a lot more complex.

It's the exact same perspectives I've always had on ZODB:

1) It's really convenient and powerful

2) It being a single HUGE file makes it hard to scale

3) Versioning can be nifty but it's often not needed and causes headache with the packing

4) It works great but when it cracks it cracks hard and cryptically

Massive improvement on sorting a fat list

28 February 2010 6 comments   Zope, Python, IssueTrackerProduct

http://www.issuetrackerproduct.com/Download#IssueTrackerMassContainer

IssueTrackerMassContainer is a simple Zope product that is used to put a bunch of IssueTrackerProduct instances into. It doesn't add much apart from a nice looking dashboard that lists all recent issues and then with an AJAX poll it keeps updating automatically.

But what it was doing was it recursively put together all issues across all issue trackers, sorting them and then returning only the first 20. Fine, but once the numbers start to add up it can become a vast sort operation to deal with.

In my local development copy of 814 issues, by the use of pympler and time() I was able to go from 7 Mb taking 2 seconds down to using only 8 Kb and taking 0.05 seconds.

Here's the initial naive version of the code:

    def getRecentIssues(self, since=None, recursive=True, batch_size=20, batch_start=0):
       """ return a list of all the most recent issues """
       issues = self._getAllIssues(self.getRoot())
       if since is not None:
           ... # checking variable since
           issues = [x for x in issues
                     if float(x.getModifyDate()) > since]

       issues.sort(lambda x,y: cmp(y.getModifyDate(), x.getModifyDate()))
       return issues[int(batch_start):int(batch_size)]

So, instead of making a fat list of issue objects, just turn it into a list of the things we really need. The second version:

    def getRecentIssues(self, since=None, recursive=True, batch_size=20, batch_start=0):
       """ return a list of all the most recent issues """
       root = self.getRoot()
       issues = self._getAllIssues(root)
       issues = [(x.getModifyDate(), '/'.join(x.getPhysicalPath())) for x in issues]
       if since is not None:
           ... # checking variable since
           issues = [(t,i) for (t,i) in issues
                     if float(t) > since]

       issues.sort()
       issues.reverse()
       issue_paths = [x[1] for x in issues[int(batch_start):int(batch_size)]]
       return [root.unrestrictedTraverse(p) for p in issue_paths]

The issue method getModifyDate() returns a Zope DateTime instance which is ridiculously nifty datetime implementation but it sucks in memory use and performance. See this blog about how it sucks compared to mxDateTime and standard lib datetime. So, this time, turn it into a float and then sort. Final version:

    def getRecentIssues(self, since=None, recursive=True, batch_size=20, batch_start=0):
       """ return a list of all the most recent issues """
       root = self.getRoot()
       issues = self._getAllIssues(root)
       issues = [(float(x.getModifyDate()), '/'.join(x.getPhysicalPath())) 
                 for x in issues]
       if since is not None:
           ... # checking variable since
           issues = [(t,i) for (t,i) in issues
                     if t > since]

       issues.sort()
       issues.reverse()
       issue_paths = [x[1] for x in issues[int(batch_start):int(batch_size)]]
       return [root.unrestrictedTraverse(p) for p in issue_paths]

And the results for my local copy of 818 issues?:

Version 1:
   7842736 bytes (7.5 Mb)
   2.1547999382 seconds

Version 2:
   834880 bytes (0.79 Mb)
   0.210245847702 seconds

Version 3:
   87448 bytes (85 Kb)
   0.0538010597229 seconds

Granted, Zope will release this memory by the garbage collector but why even let it get that big if you have concurrent hits or if anything gets stuck for longer than necessary. Python 2.4 can free memory used but not return it to the operating system to reuse unless the process dies (this was fixed in Python 2.5).

That's a memory usage improvement of about 90 fold and a speed improvement of about 40 fold.

Fry-IT has huge issuetracker instances and at the time of writing keeps 6668 "active" issues across all projects.

Custom Fields in IssueTrackerProduct documentation written

05 June 2009 1 comment   Zope, IssueTrackerProduct

http://www.issuetrackerproduct.com/News/custom-fields-documentation

The Custom Fields feature started as a consultancy job in which we agreed the work can be open sourced as part of IssueTrackerProduct so I never got around to write an sensible high level documentation for it. Now I have! From the news piece about it:

"Custom Fields was a feature that was released almost a year ago but didn't have much documentation. Especially easy documentation that describes what it is and how it can be used. That has changed now.

In Custom Fields it is now described what they are and how they can become useful to you. It's such a powerful tool that very few "competing" issue/bug tracking systems can offer."

The written documentation is here: Custom Fields

Feedback appreciated.

Lesson learned: Unicodifying request variables in Zope

16 April 2009 3 comments   Zope

This cost me a good hour of debugging so I thought I'd share it in case anybody else stumbles across the same problem. In the end, to solve my problem I had to add debug statements to StringIO.py to be able to find out where in my Zope Page Template a non-unicode string with non-ascii characters appeared and messed things up.

The error I was getting was this, which I suspect several Zope developers have encountered before:

UnicodeDecodeError: \
'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)

The traceback only mentions files in the innards of ZPT of which none you can really do anything about. We all know that the key to avoid Unicode error is to be consistent. You can do this:

>>> '\xc3' + 'string'
'\xc3string'
>>> u'\xc3' + u'string'
u'\xc3string'

But you can't do this:

>>> '\xc3' + u'string'
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: 
ordinal not in range(128)

So, how did I get these non-unicode strings into my application in first place. Simple, I have a search parameter q and end up with a URL like this:

/bla/bla?q=régime

And the template had this little innocent piece of code:

<input tal:attributes="value request/q"/>

That's what f'ed everything up. So, I ended up having to add this:

<input tal:attributes="value python:context.unicodify(request['q'])"/> 

With this little helper function in the base class:

def unicodify(self, s):
   if isinstance(s, str):
       return unicode(s, 'utf8')
   return s

So, hopefully by writing this it will help someone else making the same trivial mistake and not wasting their evening with sporadic print statements all over their frameworks code.

When '_properties' gets stuck as a persistent attribute

01 October 2008 1 comment   Zope

Doing some on-site consulting on an old Zope CMS that has been developed by many different developers over many years. It's pretty good and has lots of powerful features but over the years certain things have been allowed to slip. One problem was that you couldn't click the "Properties" tab. The reason was that it was trying to fetch properties that didn't exist anymore. What had happened was that the class attribute _properties (which is used by the "Properties" tab in the ZMI) had been stored as a persistent attribute. Here's how to solve that:

def manage_fixPropertiesProblem(self):
    """ fix so _properties becomes a class attribute instead """
    if '_properties' in self.__dict__.keys():
        del self._properties

    return "Awesome!"