IssueTrackerMassContainer is a simple Zope product that is used to put a bunch of IssueTrackerProduct instances into. It doesn't add much apart from a nice looking dashboard that lists all recent issues and then with an AJAX poll it keeps updating automatically.

But what it was doing was it recursively put together all issues across all issue trackers, sorting them and then returning only the first 20. Fine, but once the numbers start to add up it can become a vast sort operation to deal with.

In my local development copy of 814 issues, by the use of pympler and time() I was able to go from 7 Mb taking 2 seconds down to using only 8 Kb and taking 0.05 seconds.

The Custom Fields feature started as a consultancy job in which we agreed the work can be open sourced as part of IssueTrackerProduct so I never got around to write an sensible high level documentation for it. Now I have! From the news piece about it:

"Custom Fields was a feature that was released almost a year ago but didn't have much documentation. Especially easy documentation that describes what it is and how it can be used. That has changed now.

In Custom Fields it is now described what they are and how they can become useful to you. It's such a powerful tool that very few "competing" issue/bug tracking systems can offer."

The written documentation is here: Custom Fields

Feedback appreciated.

This cost me a good hour of debugging so I thought I'd share it in case anybody else stumbles across the same problem. In the end, to solve my problem I had to add debug statements to StringIO.py to be able to find out where in my Zope Page Template a non-unicode string with non-ascii characters appeared and messed things up.

The error I was getting was this, which I suspect several Zope developers have encountered before:

The traceback only mentions files in the innards of ZPT of which none you can really do anything about. We all know that the key to avoid Unicode error is to be consistent. You can do this:

But you can't do this:

So, how did I get these non-unicode strings into my application in first place. Simple, I have a search parameter q and end up with a URL like this:

And the template had this little innocent piece of code:

That's what f'ed everything up. So, I ended up having to add this:

With this little helper function in the base class:

So, hopefully by writing this it will help someone else making the same trivial mistake and not wasting their evening with sporadic print statements all over their frameworks code.

Doing some on-site consulting on an old Zope CMS that has been developed by many different developers over many years. It's pretty good and has lots of powerful features but over the years certain things have been allowed to slip. One problem was that you couldn't click the "Properties" tab. The reason was that it was trying to fetch properties that didn't exist anymore. What had happened was that the class attribute _properties (which is used by the "Properties" tab in the ZMI) had been stored as a persistent attribute. Here's how to solve that:

I've just released a new little project in Python for tracking memory usage in Zope2 applications with the added benefit that you can hopefully see what URL causes which memory usage "jumps". Hopefully this can help Zope2 developers find out what causes RAM bloat but can also help in helping you optimize your application by early in the development process find out what uses too much RAM. I wouldn't be surprised that there is already a program that does something like this. I've just never seen one. Also by putting this out as an Open Source project and blogging about it hopefully more clever people than me will come forward and point out the right way to do things.

I've also used Google Code this time to manage the project. I've used it before but only for hosting a public SVN for the IssueTrackerProduct SVN. I have to say that I was quite impressed with Google Code this second time. I think it's still fundamentally wrong to confuse people with by offering both download and SVN checkout. I did both this time but I think I might give up on the downloads because who out there, who understands that he/she needs to debug RAM usage, doesn't know how to use SVN?

Finally a little disclaimer: By writing about this here, preparing it on Google Code and writing a README.txt file I've now spent more time "managing" the project than I have on coding it. It's an early test release which hopefully will stir up some ideas for genuine important improvements. I had fun coding it as well since this is my first attempt with Flot which has been great to work with. You get very quick and powerful results. Lastly, I haven't tested this in anything but 32-bit Ubuntu Linux and Firefox.

Here is a sample report: 2008-05-30_16.47.32__3.8_minutes

Tonight I released an experimental version of the IssueTrackerProduct that is packed with new cool stuff. I call this an experimental release (but I run it on my production systems) because it's got so many new features.

During the course of preparing for this release and writing the news item I deployed the latest version to real.issuetrackerproduct.com and immediately noticed two bugs I to do with user names. So I immediately fixed those and prepared a new release minutes after. I expect to release another more stable version within a few weeks.

Martijn Faasen is my hero. Not only is an absolutely brilliant coder he's also able talk so that mortals understand.

This is why I like Grok

What he's replying about is mainly the question "What does Grok give me that, say, django does not?"

And, this is why I dislike Grok

Yes, you clever people. It's the same link. For some reason all the great documentation goes into replies on the mailing list rather than into a concise web page with cookbook, book and styled and funny tutorials. Why is that? They've actually made it quite easy now to enter documentation on grok.zope.org with the new Plone site.

An equally important question is: Why don't I do something about it rather than to complain? Well, I've written one how-to at least. My other "excuse" is that I'm not yet an expert enough and hence writing good documentation takes a very long time.

I think there's an important philosophical and political issue at hand. The Grok community is filled with really clever people who are very senior in the web development industry who like using mailing lists and perhaps more importantly, don't need documentation since they can study source code and unit tests to answer their questions. I know this is a sensitive statement but I'll take my chances since it implies that these guys are smarter (or perhaps just more time on their hands).

My internal battle of which new web framework to put my energy into continues. Today (thanks to Martijn's post) Grok earned one more point.

Don't ask why I'm developing products for Zope 2.7 but I had to and I should have been more careful with these oldtimers.

I kept getting this error:

(notice the strange double space after the : colon) This is different from the standard python TypeError when you get the parameters wrong which looks like this TypeError: __init__() takes exactly 2 arguments (1 given).

The line it complained this happened looked like this:

I couldn't understand what the hell was wrong on that line! Clearly it wasn't a normal Python error. Here's the explaination: That OtherClass was a new-style class inheriting from object. It looked like this:

When I changed that to:

The whole thing started to work. Long lesson learnt, don't use new-style classes mixed in into Zope 2.7.

Since I've seen so many poor solutions to this problem I thought I'd share mine. Here's how I make printer friendly pages.

1. Add a method in your base class that looks like this:

2. Change all your Page Templates to refer to a method rather than a macro directly so that pages like index_html.zpt start like this:

I've seen the hard coded way too many times where people do something like this <html metal:use-macro="here/main_template/macros/master"> which gives you no flexibility.

3. Now make a copy of main_template.zpt called print_main_template.zpt and the most important change is to make print.css load by default. Here's what it should look like this somewhere inside the <head> tag:

Note how the print.css link tag is now not conditional. Before in main_template.zpt it should have looked like this:

And note how the order is stacked just to be extra safe to weird browsers that don't understand the media condition.

As a last optional feature you should add is to add these lines at the bottom of the template 'print_main_template.zpt':

Another tip is to add something like this to the footer because it becomes useful when you look at a printed copy:

4. Now rewrite the method getMainTemplate() to become usefully intelligent:

5. Prepare the interface now for the printer friendly page. This can be done in two different ways. One way is to put a link in the footer or byline like this:

Or if you want to force a particular page to always be printer friendly, for example print_invoice.zpt then write it like this:

As a final point; how you solve your web design with screen.css and print.css varies. One way is to define multiple css files each suitable for individual things like this example shows:

An alternative solution is to don't expect print.css to stand on its own two legs but only be a supplement of the general css file. When doing this you're probably just going to want to override some things and hide some other things like this example from a 'print.css':

To conclude

This gives you a robust framework for enabling printer friendly pages that are quite different from the main template and doing it like this means that you don't have add conditional hacks to your main template that displays certain things if in printer friendly mode or not.

Most importantly, this gives you the framework for adding other versions of main template. For example these:

• mobile_main_template.zpt (guess what for)
• minimal_main_template.zpt (for things like Help page popups)

A healthy and fair use of METAL macros is also key to asserting that you don't have to repeat yourself too much in the copies of main_template.pt.

Good luck!

Here's one of many things I've learnt today at PyCon. Inspired by code that Grig Gheorghiu showed in his slides on automated testing, you can monkey patch a standard library that your application is using in your unit tests to, in my case, mock a remote service without having to run a server. I've done lots of monkey-patching in Zope but then I've only been monkey patching individual methods or attributes of imported classes. This is very similar to that. Here's what my application does:

Adjacent to this I have a unit/integration test that looks like this:

I've installed a lot of Zope instances on my laptop since version 2.7.3 and out of curiosity and desperate need for more hard drive space I thought I'd log rotate them all with the standard Linux logrotate program.

Before doing the log rotate, the total size of all my event.log files came to about 290Mb! After running logrotate (twice of course to go from event.log.1 to event.log.2.gz) the total size become 20Mb. Not a huge significance in the world of gigabyte hard drives but at least something.

This took me a while to grok and perhaps by mentioning it here, it'll prevent other people from making the same mistake as I did and perhaps preventing myself from doing the same mistake again.

In the ZCatalog, when you set up indexes you can give them a name and an index attribute. If you omit the index attribute, it'll try to hook into the objects by the name of the index. For example, if you set the index to be title with no indexed attribute it'll fetch the title attribute of the objects it catalogs. But if you set the indexed attribute to be something like idx_getTitle you can do something like this in your class:

The same can not be done with indexes of type DateIndex.

Yesterday I released IssueTrackerProduct 0.8.0 which I'm quite happy about. It's been 10 months since the last release and although there are still some outstanding feature requests and non-urgent bugs I'm happy to get this out of the door.

To all those doing the upgrade, remember to press the Update Everything button!

Together with my colleague Jan we today tried to fix a problem with Zope ZEO server that wouldn't start. Very strange we thought since another ZEO server on the same machine was working fine. We compared zope.conf files and found nothing obvious. Here was the error you get with 'bin/runzeo':

 root@da-ovz-vm99182:/var/lib/zope-2.8.9/aragdb-zeo1# ./bin/runzeo 
 Error: error opening file //var/lib/zope-2.8.9/aragdb-zeo1/etc/zeo.conf: 
 [Errno ftp error] no host given

What?? ftp error??

Long story short, the error was this line in 'runzeo':

 CONFIG_FILE="/${INSTANCE_HOME}/etc/zeo.conf"

which should have been this:

 CONFIG_FILE="${INSTANCE_HOME}/etc/zeo.conf"

Not so easy to spot because the CONFIG_FILE line is something you rarely look closely at.

I saw this today on the Zope mailing list:

 TypeError: open() takes at least 1946389116 arguments (4 given)

For anybody who's ever used Python you'll find this funny.

To set the Expires header in my web application I used to use the Zope DateTime function rfc822() which doesn't return the date in GMT format. Here's how I used it:

Then I found out (from using YSlow) that it's better to use the GMT format (RFC 1123), and here's how to do that in Zope:

(notice that even though my locale is here in London, because of the summer time an hour is added)

There are several ways of serving static files from Zope. The simplest way is to just do something like this:

The disadvantage with this is if the file is say >1Mb, the whole 1Mb needs to be loaded into RAM before it can be sent. This is surely garbage collected afterwards but if you serve many of these at the same time, there's obviously the risk that the RAM gets too full.

The alternative is to use filestream_iterator from the ZPublisher which is an iterator that sends out chunks of the file at a time. Note that to use the filestream_iterator you must set the Content-Length header first. Here's how to use it:

This week we improved MOBi with a neat little feature that makes it possible to decide what the SMS response should be to inbound SMS on your own server.

To test this you need to be UK resident and willing to part with the 25p it costs to receive the response. Send peterbe is NN (where NN is a number, how old you think I am) to 83211 and await a result. I've set up the sub word "is *" to forward all inbound SMS to an address on www.peterbe.com and here on this server I run the following Python Script:

Today I bumped into a nasty challenge when trying to copy a Zope image from ZODB to the filesystem. The reason why it wasn't easy was that Zope splits the file up into chunks if it's upload size is more than 65Kb (2^16 bytes) which I guess it does for being able to "stream" the file back into the browser when viewing the image instead of sending one large blob.

So, this didn't work:

...if the data is big because sometimes imgobj.data is a string (the binary data) and sometimes is a ImplicitAcquirerWrapper that you have to iterate over.

Lukasz Lakomy is the latest fulltime addition to Fry-IT, my company. Lukasz is a kickass Plone and Zope developer and has just launched his own little website which is done in Plone.

The content is still a bit on the light side but he's already managed to start hosting his wonderful little ZPTDebugger which has now reached a 1.0 release. This product wasn't created because it was possible but because he actually needed it. Applications that come from that kind of motivation always end up being much more useful.

Keep up the good work colleague!

I've wanted to Gzip the static content (CSS & Javascript) on my sites for a long time but never found a good enough solution. mod_gzip is out of the question because as far as I've understood it it does a compression on the fly, every time you request the file.

Other solutions have disappointed me because enabling gzip compression has been for all content. I don't want my HTML files gzipped because they're rendered on the fly based on business logic plus by compressing the HTML files. Long story short my yet-to-be-released app now serves the following files from Zope but are only compressed and whitespace slimmed once per server restart:

 FILE               ORIG SIZE  NEW SIZE   REDUCTION
 screen.css             15224      2738   556%
 print.css               2633       885   298%
 jquery-latest.js*      57712     18130   318%
 jquery-latest.pack.js  20461     10513   195%
 common.js               3803      1131   336%
 complete-expense.js    18184      2847   639%
 Total                 118017     36244   326%

 * only used in debug mode

I often use the Zope management interface but not to do much ZMI like stuff. Just simple things like creating new base objects, changing properties or testing a change to a template. For some reason I seem to lack the ability to spell the word manage_main. I can't remember how many times I've accidently typed manage-main , mange_main, manag_main, manage)_main etc.

There's got to be an end to that! Hence the MUnderscorePatch monkey patch. Now, all I have to do is to type in m_ at the end of the URL and it becomes the same as manage_main. Ie. I go to http://localhost:8080/ then I append m_ so that it becomes http://localhost:8080/m_ and it redirects automatically to http://localhost:8080/manage_main

Silly but actually quite useful for me. Want to download it too?

Here are a few lessons learnt when sending HTML emails in Zope with unicode data. I'm still a beginner with this stuff and still probably have a lot to learn. What I've achived now works but might break with different encodings or on different systems so don't assume that these patterns will work in your setup.

To the send the email I'm using the default MailHost that comes with Zope 2.8.5. I call it's send() passing the subject line a second time because the subject line is already in the body string.

The most valuable piece of magic to learn and remember from this is that when you construct the multi-part message you have to attach the plain text before the html text. Thank you Lukasz Lakomy for that tip!

Just wanted to let you know that I've written a blog on my company's website about the Zope 3 training course I've attended in Denmark.

I was reading Of snakes and rubies; Or why I chose Python over Ruby which is an article comparing Python and Ruby. The blog article has loads of comments. This one I liked in particular

"For people writing good production code in PHP, I'm not trying to pile on - I know you get disrespected enough. It's possible to write well-structured sites on PHP, and lots of you do. It's just that in PHP, anyone can learn how to create spaghetti code in 10 minutes, but you have to learn a decent-sized body of knowledge to gain the ability to separate presentation from logic. By contrast, when novices get started in Rails, they're using good structure by default. They have to make an effort to screw it up."

This is my first release of ZTinyMCE. ZTinyMCE makes it easy to use TinyMCE, the best Open Source WYSIWYG editor in my opinion.

We at Fry-IT use this a lot for our websites as shown in the screenshots. The ZTinyMCE product works only with Zope2. Once you have it installed and you have created a ZTinyMCE instance in your root plus a ZTinyMCE Configuration object called "tinymce.conf" all you have to do to convert a plain page of HTML with textareas to a plain page of HTML with WYSIWYG textareas is this:

 <script tal:replace="structure here/tinymce.conf">
 </script>

If you're into Zope python product stuff, read on, otherwise don't bother.

Thanks to Brian Lloyd (Zope corp) and Florent Guillaume (Nuxeo) I now know how to set security declarations on a class outside the class. It doesn't work like a normal python class (new or old style) which was a bit of a surprise. This is how you do it in a "normal" python class:

Email replies and AJAX is the highlight of the latest IssueTrackerProduct release.

If you set up your POP3 for email replies you can simply hit reply on the notifications that goes out and continue an issue discussing without leaving the comfort of Outlook/Thunderbird/Gmail/Hotmail.

Another underestimated important feature is the automatically refreshing issue where the content is periodically refreshed (if the content changes) whilst you're on the page so that you know you're always looking at the very latest copy before you take any action.

For the people who've dared to use the CVS, they've probably already seen the new keyboard shortcuts that is inspired by Gmail. It's very addictive to not have to scroll up and down to reach the navigation or to be able to quickly jump to another issue without having to click on List Issue.

And for the Gettings Things Done people you might like how you now can get a list of "You next action issues" on the Home page. This is a clever list that attempts to figure out what you need to do next in that order.

A week ago I ran some ad hoc benchmarks on various suspect functions in the IssueTrackerProduct and came to a clear and simple conclusion: searching is the bottleneck and within the search it's the searching for file attachments that take all of the time.

If you're interested and open minded, here's the results of that benchmark This sparked some thoughts. First I wrote a filename splitter which isn't rocket science but I'm proud to say that it's use is brilliant. Before, the find-by-file function in the IssueTrackerProduct used a plain old find() test like this:

 filename.find('foo') > -1

This is very fast but not very intelligent. For example it'll with match on foobar.txt and plainfooter.gif. So, what I did instead was to create a KeywordIndex and index all the splitted filenames in that index.

Just wanted to make sure you all know about my latest invention :) It's an AJAX implementation on the IssueTrackerProduct that updates an issue page you might be looking at if it's changed on the server. This will become very useful if there are several people working on the tracker and you don't want to make sure you're reading the very latest when you post your followup.

This accomplished by having a periodic Javascript (about 30 seconds at the time of writing) that constantly does a timestamp check against the server. If the timestamp changes, the issue you're viewing will most likely have changed and that's where AJAX kicks in. The AJAX script reloads certain parts of the page and also changes the title of the page (observe that in the screencast).

One thing I just noticed about the screencast is that the debug-mode was switched on which means that there's lot's of geeky stuff at the bottom of the Internet Explorer window. Oops!