Filtered by IssueTrackerProduct
In 2001 I started my first and perhaps most successful Open Source project I've ever made: IssueTrackerProduct. After nearly a decade of maintaining it I have now officially abandoned it.
It all started when I needed a way to track feedback on my personal website. That's why it was originally called "SiteTrackerProduct". I needed something where I could collect bug reports and any other pieces of feedback and then process it in some structured fashion. It was therefore very important that it would be possible to run the application open for anonymous access. People should be able to submit bugs and issues without having to create an account. You see, kids, back in that day it was actually very common that sites would force users to register and create accounts even just because the content owner wanted it. These days, it's common knowledge that to get people to open up and share anything for others to benefit you make it absolutely trivial to jump straight in without having to see a registration page that looks like a tax return form.
Now, since I long ago abandoned the Zope2 application server technology stack and I no longer use IssueTrackerProduct for anything real it's no longer feasible to maintain this project. In the last five years or so we were actually using it actively to track all projects at Fry-IT where I used to work. I have to say, even though we did grow out of it, it was actually successful. It handled the load (after some much needed patches towards optimization) and it was easy for people to actually use since unlike many other bug trackers, it focused on the non-technical end user first and foremost. As much as possible was done to make it trivial to type in your bug or issue and it automatically took care of all notifications and access rights.
Being a personal Open Source project, over the years, it became a melting pot for experimenting and perfecting various new ideas. Many of them we take for granted today but back then it was quite novel if I may say so. This includes:
- ability to auto-save unfinished form inputs (added before Gmail had it)
- automatic updates of the content without reload made it possible to see other people participating as you're typing
- ability to reply directly to email notifications without having to open a web browser
- an advanced via-the-web programmable interface for adding and modifying custom fields (e.g. "Customer reference code")
- full-text search combined with ability to search on specific fields by key
- file attachments that are images automatically appear as little thumbnails
- file attachments that have text become searchable (e.g Word documents)
- advanced filtering where you can easily decide to search inclusive or exclusive on certain fields
- persistent filtering automatically saved and share'able between different users
- programmable search filters that are coded in Python which made it possible to create very specific reports
- ability to export and import bugs from and to Excel for offline processing
Writing all of this, I can not resist to get a bit nostalgic. I did sink A LOT of time into this project. Today when I look back at the code and almost feel sick seeing all the mistakes that I made. Much of the ugliness of the code can be attributed partially to the fact that I often used and abused the code to add new features. Also, because we often needed some features (since it was used to manage all of our projects) "yesterday" and then it was hard to justify doing things "properly". For example, the main .py file is over 14,000 lines of code!
I did called it "perhaps most successful Open Source project I've ever made" in the first sentence. The reason for that is that over the years many many people have downloaded it and installed and let it be used by thousands of users. That's something to be proud of.
Anyway! It's time to move on. So long and thank you for all the fish!
The code is still available at github.com/peterbe/IssueTrackerProduct
I just got back from the Helpdesk & IT Support Show in Olympia (Kensington, London). My main impression is: there are many, big players in this industry.
My pet project, the IssueTrackerProduct is very basic in comparison to some of these companies products. Although it's often used in help desk situations the kind of help desk solutions I've seen today are way different. For many of them, it's all about integrating various systems such as asset management, call logging, configuration management, knowledge management, etc. It seems that the actual help desk apps seems to have to be low priority compared to getting all pieces to fit together.
I walked one lap around the perimeter of the big hall so first I saw the little stands. There they called it "Help desk". As I moved in towards the centre of the hall where the really big players had really big stands, big plasma screens and lightly dressed young ladies a different word was used: "Service management". I talked to a really friendly chap from ICCM Solutions and asked him honestly: what is Service Management? He gave me two answers. One short, one longer. The short version was: the smaller less advanced providers haven't evolved and still call it help desk; the bigger players who have evolved more call it service management. The longer version was that the expression "help desk" is more associated with IT and problems that need immediate solutions. Service management is more about monitoring and controlling: Configs, Incidents, Problems and Change. In that order. It's all business/marketing lingo that don't mean much but at the time he explained it it actually started to make sense. In simplicity, they just want to get away from the word help desk because it's got ingrained annotations to it and they want to do something else.
Very few of the screenshots and demos I saw impressed me in the detail work. I saw endless horrible help desk issue/incident entering screens where some poor user has to select the right fields to fill in amongst a screen packed of different input fields of various sorts. Many systems had issue/incident reports and lists again packed with far too much screen-noise. Some of them had quite a slick and neat design but a lot of them had windows GUI apps that look like a video-rent-shop-epos system from 1995.
Although I never saw it in action, HelpSTAR had (supposedly) a clever solution. You can apparently drag and drop emails from your MS Outlook into their help desk system. That email is then logged, archived and can now be responded to and become a email correspondence ticket. They also had (supposedly) an interesting templating system. That makes it easy to reply to frequently asked questions that people send in to a help desk.
One promising company that I talked to was Logicalware and their product MailManager. Their brochure-work was, unlike all others, really down to earth and easy to read and understand. MailManager is, like the IssueTrackerProduct, written in Zope and is Open Source too. The IssueTrackerProduct has email ticketing capabilities but that's not its core functionality. With MailManager, I think email ticketing is the only thing it does and does it well. Keep up the good work guys!
In conclusion, my IssueTrackerProduct does do much but the few simple things it does it does very well. If you don't have £10,000 to spend on licenses, consultants and headaches and just need a simple system for logging issues, pick my system
A week ago I ran some ad hoc benchmarks on various suspect functions in the IssueTrackerProduct and came to a clear and simple conclusion: searching is the bottleneck and within the search it's the searching for file attachments that take all of the time.
If you're interested and open minded, here's the results of that benchmark
This sparked some thoughts. First I wrote a filename splitter which isn't rocket science but I'm proud to say that it's use is brilliant. Before, the find-by-file function in the IssueTrackerProduct used a plain old
find() test like this:
filename.find('foo') > -1
This is very fast but not very intelligent. For example it'll with match on
plainfooter.gif. So, what I did instead was to create a KeywordIndex and index all the splitted filenames in that index.
What you have to remember is that KeywordIndexes are case sensitive so when I populate the KeywordIndex I have to lowercase everything. But, now let's get to the performance fix.
The problem was that before, I used a
ZopeFind() to find files by filename and
ZopeFind() is slooooowww compared to a catalog search. Before I switched on the new code that searches the filename splitted keyword index I did a coule of searches on an issuetracker with about 200 issues and a couple of searches on an issuetracker with more than 10,000 issues. The results can be seen here:
itp-benchmark-medium-before.log and itp-benchmark-huge-before.log
Quite terrible isn't it. Then I switched on the new code that is much more intelligent and hopefully faster. The results can be seen here: itp-benchmark-medium-after.log and itp-benchmark-huge-after.log
itp-search-Real-before.log 0.692274004221 itp-search-new-tracker-before.log 7.29415249824 itp-search-Real-after.log 0.0011158519321 itp-search-new-tracker-after.log 0.00395481482796
As you can see, that's an enourmous increase in speed. For the medium sized issuetracker, that's a 600% increase and for the huge issuetracker that's a 2000% speed increase. Oops! This shouldn't be called optimization maybe. Perhaps bug fixing is a better word :)
I think there's a lot of potential in screencasting. As far as I've understood, screencasting is when you make a movie recoding in some manner of what happens on the computer screen. To test this I downloaded Camtasia Studio 3 and as a demo I created a new
Issue Tracker instance on www.issuetrackerproduct.com. The next time I do this I'll make sure I plan what I want to do instead of just making it up after I've started. On this windows computer that I tried this it lagged so incredibly much that it was to move the mouse because it didn't move smoothly. Perhaps there are some further options to free up some resources to make it run better.
If you want to see the result (as a Flash movie) follow this link (1.2Mb)
When you produce the screencast there are loads of different export options which is pretty cool. Although I don't care much for Windows Media or Quicktime because it doesn't work on Linux.
This is actually quite exciting because I can create screencasts of all the How Tos on www.issuetrackerproduct.com if possible. This Camtasia cost some $300 but I test it some more and it continues to impress I'm more than happy to pay that.
I've made a quick graph showing how the releases of the IssueTrackerProduct increases steadily in size with every new version. Since the first release, 10 months ago, the release package has more than doubled in size. Much of that is due to the new templates that have been added and some icons.
Is this a positive trend? Yes of course! A lot of the new code isn't just additional fancy features. Sometimes it takes a couple of extra bytes just dealing with stuff under the hood that has no impact whatsoever on the interface. The actual numbers aren't really a measure that can be used, but what is really important to notice is the solid and steady growth. This year has been very busy for me with work but I've always tried to squeeze in a bit of open source work too in the mornings, evenings and lunch breaks. I wish I could focus more on the IssueTrackerProduct but so far we haven't come up with a direct way of getting rich on it so it still remains just a "hobby" for me an my company.