Peterbe.com

In 2001 I started my first and perhaps most successful Open Source project I've ever made: IssueTrackerProduct. After nearly a decade of maintaining it I have now officially abandoned it.

It all started when I needed a way to track feedback on my personal website. That's why it was originally called "SiteTrackerProduct". I needed something where I could collect bug reports and any other pieces of feedback and then process it in some structured fashion. It was therefore very important that it would be possible to run the application open for anonymous access. People should be able to submit bugs and issues without having to create an account. You see, kids, back in that day it was actually very common that sites would force users to register and create accounts even just because the content owner wanted it. These days, it's common knowledge that to get people to open up and share anything for others to benefit you make it absolutely trivial to jump straight in without having to see a registration page that looks like a tax return form.

Now, since I long ago abandoned the Zope2 application server technology stack and I no longer use IssueTrackerProduct for anything real it's no longer feasible to maintain this project. In the last five years or so we were actually using it actively to track all projects at Fry-IT where I used to work. I have to say, even though we did grow out of it, it was actually successful. It handled the load (after some much needed patches towards optimization) and it was easy for people to actually use since unlike many other bug trackers, it focused on the non-technical end user first and foremost. As much as possible was done to make it trivial to type in your bug or issue and it automatically took care of all notifications and access rights.

Being a personal Open Source project, over the years, it became a melting pot for experimenting and perfecting various new ideas. Many of them we take for granted today but back then it was quite novel if I may say so. This includes:

• ability to auto-save unfinished form inputs (added before Gmail had it)
• automatic updates of the content without reload made it possible to see other people participating as you're typing
• ability to reply directly to email notifications without having to open a web browser
• an advanced via-the-web programmable interface for adding and modifying custom fields (e.g. "Customer reference code")
• full-text search combined with ability to search on specific fields by key
• file attachments that are images automatically appear as little thumbnails
• file attachments that have text become searchable (e.g Word documents)
• advanced filtering where you can easily decide to search inclusive or exclusive on certain fields
• persistent filtering automatically saved and share'able between different users
• programmable search filters that are coded in Python which made it possible to create very specific reports
• ability to export and import bugs from and to Excel for offline processing

Writing all of this, I can not resist to get a bit nostalgic. I did sink A LOT of time into this project. Today when I look back at the code and almost feel sick seeing all the mistakes that I made. Much of the ugliness of the code can be attributed partially to the fact that I often used and abused the code to add new features. Also, because we often needed some features (since it was used to manage all of our projects) "yesterday" and then it was hard to justify doing things "properly". For example, the main .py file is over 14,000 lines of code!

I did called it "perhaps most successful Open Source project I've ever made" in the first sentence. The reason for that is that over the years many many people have downloaded it and installed and let it be used by thousands of users. That's something to be proud of.

Anyway! It's time to move on. So long and thank you for all the fish!

The code is still available at github.com/peterbe/IssueTrackerProduct

I was reading this article about linkfluence moving from CouchDB to Riak

Does that sound familiar, fellow Zope developer? I know a lot about ZODB but little about CouchDB. One thing that a lot of people don't know about ZODB is that it's very fast and I think this is true about CouchDB too. Speed isn't the same as a raw speed of inserts/queries because with the concurrency variable added the story gets a lot more complex.

It's the exact same perspectives I've always had on ZODB:

1) It's really convenient and powerful

2) It being a single HUGE file makes it hard to scale

3) Versioning can be nifty but it's often not needed and causes headache with the packing

4) It works great but when it cracks it cracks hard and cryptically

This cost me a good hour of debugging so I thought I'd share it in case anybody else stumbles across the same problem. In the end, to solve my problem I had to add debug statements to StringIO.py to be able to find out where in my Zope Page Template a non-unicode string with non-ascii characters appeared and messed things up.

The error I was getting was this, which I suspect several Zope developers have encountered before:

UnicodeDecodeError: \
'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)

The traceback only mentions files in the innards of ZPT of which none you can really do anything about. We all know that the key to avoid Unicode error is to be consistent. You can do this:

>>> '\xc3' + 'string'
'\xc3string'
>>> u'\xc3' + u'string'
u'\xc3string'

But you can't do this:

>>> '\xc3' + u'string'
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: 
ordinal not in range(128)

So, how did I get these non-unicode strings into my application in first place. Simple, I have a search parameter q and end up with a URL like this:

/bla/bla?q=régime

And the template had this little innocent piece of code:

<input tal:attributes="value request/q"/>

That's what f'ed everything up. So, I ended up having to add this:

<input tal:attributes="value python:context.unicodify(request['q'])"/> 

With this little helper function in the base class:

def unicodify(self, s):
   if isinstance(s, str):
       return unicode(s, 'utf8')
   return s

So, hopefully by writing this it will help someone else making the same trivial mistake and not wasting their evening with sporadic print statements all over their frameworks code.

Doing some on-site consulting on an old Zope CMS that has been developed by many different developers over many years. It's pretty good and has lots of powerful features but over the years certain things have been allowed to slip. One problem was that you couldn't click the "Properties" tab. The reason was that it was trying to fetch properties that didn't exist anymore. What had happened was that the class attribute _properties (which is used by the "Properties" tab in the ZMI) had been stored as a persistent attribute. Here's how to solve that:

def manage_fixPropertiesProblem(self):
    """ fix so _properties becomes a class attribute instead """
    if '_properties' in self.__dict__.keys():
        del self._properties

    return "Awesome!"

I've just released a new little project in Python for tracking memory usage in Zope2 applications with the added benefit that you can hopefully see what URL causes which memory usage "jumps". Hopefully this can help Zope2 developers find out what causes RAM bloat but can also help in helping you optimize your application by early in the development process find out what uses too much RAM. I wouldn't be surprised that there is already a program that does something like this. I've just never seen one. Also by putting this out as an Open Source project and blogging about it hopefully more clever people than me will come forward and point out the right way to do things.

I've also used Google Code this time to manage the project. I've used it before but only for hosting a public SVN for the IssueTrackerProduct SVN. I have to say that I was quite impressed with Google Code this second time. I think it's still fundamentally wrong to confuse people with by offering both download and SVN checkout. I did both this time but I think I might give up on the downloads because who out there, who understands that he/she needs to debug RAM usage, doesn't know how to use SVN?

Finally a little disclaimer: By writing about this here, preparing it on Google Code and writing a README.txt file I've now spent more time "managing" the project than I have on coding it. It's an early test release which hopefully will stir up some ideas for genuine important improvements. I had fun coding it as well since this is my first attempt with Flot which has been great to work with. You get very quick and powerful results. Lastly, I haven't tested this in anything but 32-bit Ubuntu Linux and Firefox.

Here is a sample report: 2008-05-30_16.47.32__3.8_minutes

Martijn Faasen is my hero. Not only is an absolutely brilliant coder he's also able talk so that mortals understand.

This is why I like Grok

What he's replying about is mainly the question "What does Grok give me that, say, django does not?"

And, this is why I dislike Grok

Yes, you clever people. It's the same link. For some reason all the great documentation goes into replies on the mailing list rather than into a concise web page with cookbook, book and styled and funny tutorials. Why is that? They've actually made it quite easy now to enter documentation on grok.zope.org with the new Plone site.

An equally important question is: Why don't I do something about it rather than to complain? Well, I've written one how-to at least. My other "excuse" is that I'm not yet an expert enough and hence writing good documentation takes a very long time.

I think there's an important philosophical and political issue at hand. The Grok community is filled with really clever people who are very senior in the web development industry who like using mailing lists and perhaps more importantly, don't need documentation since they can study source code and unit tests to answer their questions. I know this is a sensitive statement but I'll take my chances since it implies that these guys are smarter (or perhaps just more time on their hands).

My internal battle of which new web framework to put my energy into continues. Today (thanks to Martijn's post) Grok earned one more point.