A blog and website by Peter Bengtsson
30 March 2012 6 comments Zope, IssueTrackerProduct
In 2001 I started my first and perhaps most successful Open Source project I've ever made: IssueTrackerProduct. After nearly a decade of maintaining it I have now officially abandoned it.
It all started when I needed a way to track feedback on my personal website. That's why it was originally called "SiteTrackerProduct". I needed something where I could collect bug reports and any other pieces of feedback and then process it in some structured fashion. It was therefore very important that it would be possible to run the application open for anonymous access. People should be able to submit bugs and issues without having to create an account. You see, kids, back in that day it was actually very common that sites would force users to register and create accounts even just because the content owner wanted it. These days, it's common knowledge that to get people to open up and share anything for others to benefit you make it absolutely trivial to jump straight in without having to see a registration page that looks like a tax return form.
Now, since I long ago abandoned the Zope2 application server technology stack and I no longer use IssueTrackerProduct for anything real it's no longer feasible to maintain this project. In the last five years or so we were actually using it actively to track all projects at Fry-IT where I used to work. I have to say, even though we did grow out of it, it was actually successful. It handled the load (after some much needed patches towards optimization) and it was easy for people to actually use since unlike many other bug trackers, it focused on the non-technical end user first and foremost. As much as possible was done to make it trivial to type in your bug or issue and it automatically took care of all notifications and access rights.
Being a personal Open Source project, over the years, it became a melting pot for experimenting and perfecting various new ideas. Many of them we take for granted today but back then it was quite novel if I may say so. This includes:
- ability to auto-save unfinished form inputs (added before Gmail had it)
- automatic updates of the content without reload made it possible to see other people participating as you're typing
- ability to reply directly to email notifications without having to open a web browser
- an advanced via-the-web programmable interface for adding and modifying custom fields (e.g. "Customer reference code")
- full-text search combined with ability to search on specific fields by key
- file attachments that are images automatically appear as little thumbnails
- file attachments that have text become searchable (e.g Word documents)
- advanced filtering where you can easily decide to search inclusive or exclusive on certain fields
- persistent filtering automatically saved and share'able between different users
-
programmable search filters that are coded in Python which made it possible to create very specific reports
- ability to export and import bugs from and to Excel for offline processing
Writing all of this, I can not resist to get a bit nostalgic. I did sink A LOT of time into this project. Today when I look back at the code and almost feel sick seeing all the mistakes that I made. Much of the ugliness of the code can be attributed partially to the fact that I often used and abused the code to add new features. Also, because we often needed some features (since it was used to manage all of our projects) "yesterday" and then it was hard to justify doing things "properly". For example, the main .py file is over 14,000 lines of code!
I did called it "perhaps most successful Open Source project I've ever made" in the first sentence. The reason for that is that over the years many many people have downloaded it and installed and let it be used by thousands of users. That's something to be proud of.
Anyway! It's time to move on. So long and thank you for all the fish!
The code is still available at github.com/peterbe/IssueTrackerProduct
08 March 2011 2 comments Zope
I was reading this article about linkfluence moving from CouchDB to Riak
"
Why we move away from CouchDBWe were already aware of Riak before we started using CouchDB, but we werent sure about trusting a new product at this point, so we decided, after some benchmark, to go for CouchDB.
After the first couple of months, it was obvious that this was a bad choice.
Our main problems with CouchDB is scalability, versionning and stability.
Once we store a document in CouchDB, we modify it at least twice after the original write. Each modification generates a new version of the document. This feature is nice for some use-cases, but we dont need it, and theres no way to disable it, so the size of our databases started to become really important. Youll probably tell me hey, you know you can compact your database ?, and Ill answer sure. The trouble is that we never managed to get it to compact an entire database without crashing (well, to be honest, with the last version of CouchDB we finally managed to compact one database).
The second issue is that one database == one file. When you have multiple small databases, this is fine. When you a have only a few databases, and some grow to more than 1TB, the problems keep growing too (and its a real pain to backup).
We also had a lot of random crashes with CouchDB, even if the last version was quite stable."
Does that sound familiar, fellow Zope developer? I know a lot about ZODB but little about CouchDB. One thing that a lot of people don't know about ZODB is that it's very fast and I think this is true about CouchDB too. Speed isn't the same as a raw speed of inserts/queries because with the concurrency variable added the story gets a lot more complex.
It's the exact same perspectives I've always had on ZODB:
1) It's really convenient and powerful
2) It being a single HUGE file makes it hard to scale
3) Versioning can be nifty but it's often not needed and causes headache with the packing
4) It works great but when it cracks it cracks hard and cryptically
28 February 2010 6 comments Python
, Zope, IssueTrackerProduct
IssueTrackerMassContainer is a simple Zope product that is used to put a bunch of IssueTrackerProduct instances into. It doesn't add much apart from a nice looking dashboard that lists all recent issues and then with an AJAX poll it keeps updating automatically.
But what it was doing was it recursively put together all
issues across all issue trackers, sorting them and then returning only the first 20. Fine, but once the numbers start to add up it can become a vast sort operation to deal with.
In my local development copy of 814 issues, by the use of pympler and time() I was able to go from 7 Mb taking 2 seconds down to using only 8 Kb and taking 0.05 seconds.
Here's the initial naive version of the code:
def getRecentIssues(self, since=None, recursive=True, batch_size=20, batch_start=0):
""" return a list of all the most recent issues """
issues = self._getAllIssues(self.getRoot())
if since is not None:
... # checking variable since
issues = [x for x in issues
if float(x.getModifyDate()) > since]
issues.sort(lambda x,y: cmp(y.getModifyDate(), x.getModifyDate()))
return issues[int(batch_start):int(batch_size)]
So, instead of making a fat list of issue objects, just turn it into a list of the things we really need. The second version:
def getRecentIssues(self, since=None, recursive=True, batch_size=20, batch_start=0):
""" return a list of all the most recent issues """
root = self.getRoot()
issues = self._getAllIssues(root)
issues = [(x.getModifyDate(), '/'.join(x.getPhysicalPath())) for x in issues]
if since is not None:
... # checking variable since
issues = [(t,i) for (t,i) in issues
if float(t) > since]
issues.sort()
issues.reverse()
issue_paths = [x[1] for x in issues[int(batch_start):int(batch_size)]]
return [root.unrestrictedTraverse(p) for p in issue_paths]
The issue method getModifyDate() returns a Zope DateTime instance which is ridiculously nifty datetime implementation but it sucks in memory use and performance. See this blog about how it sucks compared to mxDateTime and standard lib datetime. So, this time, turn it into a float and then sort. Final version:
def getRecentIssues(self, since=None, recursive=True, batch_size=20, batch_start=0
):
""" return a list of all the most recent issues """
root = self.getRoot()
issues = self._getAllIssues(root)
issues = [(float(x.getModifyDate()), '/'.join(x.getPhysicalPath()))
for x in issues]
if since is not None:
... # checking variable since
issues = [(t,i) for (t,i) in issues
if t > since]
issues.sort()
issues.reverse()
issue_paths = [x[1] for x in issues[int(batch_start):int(batch_size)]]
return [root.unrestrictedTraverse(p) for p in issue_paths]
And the results for my local copy of 818 issues?:
Version 1:
7842736 bytes (7.5 Mb)
2.1547999382 seconds
Version 2:
834880 bytes (0.79 Mb)
0.210245847702 seconds
Version 3:
87448 bytes (85 Kb)
0.0538010597229 seconds
Granted, Zope will release this memory by the garbage collector but why even let it get that big if you have concurrent hits or if anything gets stuck for longer than necessary. Python 2.4 can free memory used but not return it to the operating system to reuse unless the process dies (this was fixed in Python 2.5).
That's a memory usage improvement of about 90 fold and a speed improvement of about 40 fold.
Fry-IT has huge issuetracker instances and at the time of writing keeps 6668 "active" issues across all projects.
05 June 2009 1 comment Zope,
IssueTrackerProduct
The Custom Fields feature started as a consultancy job in which we agreed the work can be open sourced as part of IssueTrackerProduct so I never got around to write an sensible high level documentation for it. Now I have! From the news piece about it:
"Custom Fields was a feature that was released almost a year ago but didn't have much documentation. Especially easy documentation that describes what it is and how it can be used. That has changed now.
In Custom Fields it is now described what they are and how they can become useful to you. It's such a powerful tool that very few "competing" issue/bug tracking systems can offer."
The written documentation is here: Custom Fields
Feedback appreciated.
16 April 2009 3 comments Zope
This cost me a good hour of debugging so I thought I'd share it in case anybody else stumbles across the same problem. In the end, to solve my problem I had to add debug statements to StringIO.py
to be able to find out where in my Zope Page Template a non-unicode string with non-ascii characters appeared and messed things up.
The error I was getting was this, which I suspect several Zope developers have encountered before:
UnicodeDecodeError: \
'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)
The traceback only mentions files in the innards of ZPT of which none you can really do anything about. We all know that the key to avoid Unicode error is to be consistent. You can do this:
>>> '\xc3' + 'string'
'\xc3string'
>>> u'\xc3' + u'string'
u'\xc3string'
But you can't do this:
>>> '\xc3' + u'string'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0:
ordinal not in range(128)
So, how did I get these non-unicode strings into my application in first place. Simple, I have a search parameter q and end up with a URL like this:
And the template had this little innocent piece of code:
<input tal:attributes="value request/q"/>
That's what f'ed everything up. So, I ended up having to add this:
<input tal:attributes="value python:context.unicodify(request['q'])"/>
With this little helper function in the base class:
def unicodify(self, s):
if isinstance(s, str):
return unicode(s, 'utf8')
return s
So, hopefully by writing this it will help someone else making the same trivial mistake and not wasting their evening with sporadic print statements all over their frameworks code.
01 October 2008 1 comment Zope
Doing some on-site consulting on an old Zope CMS that has been developed by many different developers over many years. It's pretty good and has lots of powerful features but over the years certain things have been allowed to slip. One problem was that you couldn't click the "Properties" tab. The reason was that it was trying to fetch properties that didn't exist anymore. What had happened was that the class attribute _properties
(which is used by the "Properties" tab in the ZMI) had been stored as a persistent attribute. Here's how to solve that:
def manage_fixPropertiesProblem(self):
""" fix so _properties becomes a class attribute instead """
if '_properties' in self.__dict__.keys():
del self._properties
return "Awesome!"
30 May 2008 0 comments Zope
I've just released a new little project in Python for tracking memory usage in Zope2 applications with the added benefit that you can hopefully see what URL causes which memory usage "jumps". Hopefully this can help Zope2 developers find out what causes RAM bloat but can also help in helping you optimize your application by early in the development process find out what uses too much RAM. I wouldn't be surprised that there is already a program that does something like this. I've just never seen one. Also by putting this out as an Open Source project and blogging about it hopefully more clever people than me will come forward and point out the right way to do things.
I've also used Google Code this time to manage the project. I've used it before but only for hosting a public SVN for the IssueTrackerProduct SVN
. I have to say that I was quite impressed with Google Code this second time. I think it's still fundamentally wrong to confuse people with by offering both download and SVN checkout. I did both this time but I think I might give up on the downloads because who out there, who understands that he/she needs to debug RAM usage, doesn't know how to use SVN?
Finally a little disclaimer: By writing about this here, preparing it on Google Code and writing a README.txt file I've now spent more time "managing" the project than I have on coding it. It's an early test release which hopefully will stir up some ideas for genuine important improvements. I had fun coding it as well since this is my first attempt with Flot which has been great to work with. You get very quick and powerful results. Lastly, I haven't tested this in anything but 32-bit Ubuntu Linux and Firefox.
Here is a sample report: 2008-05-30_16.47.32__3.8_minutes
29 April 2008 3 comments Python, Zope
Tonight I released an experimental version of the IssueTrackerProduct that is packed with new cool stuff. I call this an experimental release (but I run it on my production systems) because it's got so many new features.
During the course of preparing for this release and writing the news item I deployed the latest version to real.issuetrackerproduct.com and immediately noticed two bugs I to do with user names. So I immediately fixed those and prepared a new release minutes after. I expect to release another more stable version within a few weeks.
11 April 2008 2 comments Zope
Martijn Faasen is my hero. Not only is an absolutely brilliant coder he's also able talk so that mortals understand.
This is why I like Grok
What he's replying about is mainly the question "What does Grok give me that, say, django does not?"
And, this is why I dislike Grok
Yes, you clever people. It's the same link. For some reason all the great documentation goes into replies on the mailing list rather than into a concise web page with cookbook, book and styled and funny tutorials. Why is that? They've actually made it quite easy now to enter documentation on grok.zope.org with the new Plone site.
An equally important question is: Why don't I do something about it rather than to complain? Well, I've written one how-to at least. My other "excuse" is that I'm not yet an expert enough and hence writing good documentation takes a very long time.
I think there's an important philosophical and political issue at hand. The Grok community is filled with really clever people who are very senior in the web development industry who like using mailing lists and perhaps more importantly, don't need documentation since they can study source code and unit tests to answer their questions. I know this is a sensitive statement but I'll take my chances since it implies that these guys are smarter (or perhaps just more time on their hands).
My internal battle of which new web framework to put my energy into continues. Today (thanks to Martijn's post) Grok earned one more point.
09 April 2008 2 comments Zope
Don't ask why I'm developing products for Zope 2.7 but I had to and I should have been more careful with these oldtimers.
I kept getting this error:
TypeError: expected 2 arguments, got 1
(notice the strange double space after the : colon) This is different from the standard python TypeError when you get the parameters wrong which looks like this TypeError: __init__() takes exactly 2 arguments (1 given).
The line it complained this happened looked like this:
class MyTool(SimpleItem, UniqueObject, OtherClass):
id = 'some_tool'
meta_type = 'some meta type'
def __init__(self, id='some_tool'):
self.id = id # <--- THIS WAS THE CULPRIT LINE APPARENTLY!!
I couldn't understand what the hell was wrong on that line! Clearly it wasn't a normal Python error. Here's the explaination: That OtherClass was a new-style class inheriting from object. It looked like this:
class OtherClass(object):
...
When I changed that to:
The whole thing started to work. Long lesson learnt, don't use new-style classes mixed in into Zope 2.7.
