IssueTrackerMassContainer is a simple Zope product that is used to put a bunch of IssueTrackerProduct instances into. It doesn't add much apart from a nice looking dashboard that lists all recent issues and then with an AJAX poll it keeps updating automatically.

But what it was doing was it recursively put together all issues across all issue trackers, sorting them and then returning only the first 20. Fine, but once the numbers start to add up it can become a vast sort operation to deal with.

In my local development copy of 814 issues, by the use of pympler and time() I was able to go from 7 Mb taking 2 seconds down to using only 8 Kb and taking 0.05 seconds.

The Custom Fields feature started as a consultancy job in which we agreed the work can be open sourced as part of IssueTrackerProduct so I never got around to write an sensible high level documentation for it. Now I have! From the news piece about it:

"Custom Fields was a feature that was released almost a year ago but didn't have much documentation. Especially easy documentation that describes what it is and how it can be used. That has changed now.

In Custom Fields it is now described what they are and how they can become useful to you. It's such a powerful tool that very few "competing" issue/bug tracking systems can offer."

The written documentation is here: Custom Fields

Feedback appreciated.

This cost me a good hour of debugging so I thought I'd share it in case anybody else stumbles across the same problem. In the end, to solve my problem I had to add debug statements to StringIO.py to be able to find out where in my Zope Page Template a non-unicode string with non-ascii characters appeared and messed things up.

The error I was getting was this, which I suspect several Zope developers have encountered before:

The traceback only mentions files in the innards of ZPT of which none you can really do anything about. We all know that the key to avoid Unicode error is to be consistent. You can do this:

But you can't do this:

So, how did I get these non-unicode strings into my application in first place. Simple, I have a search parameter q and end up with a URL like this:

And the template had this little innocent piece of code:

That's what f'ed everything up. So, I ended up having to add this:

With this little helper function in the base class:

So, hopefully by writing this it will help someone else making the same trivial mistake and not wasting their evening with sporadic print statements all over their frameworks code.

Doing some on-site consulting on an old Zope CMS that has been developed by many different developers over many years. It's pretty good and has lots of powerful features but over the years certain things have been allowed to slip. One problem was that you couldn't click the "Properties" tab. The reason was that it was trying to fetch properties that didn't exist anymore. What had happened was that the class attribute _properties (which is used by the "Properties" tab in the ZMI) had been stored as a persistent attribute. Here's how to solve that:

I've just released a new little project in Python for tracking memory usage in Zope2 applications with the added benefit that you can hopefully see what URL causes which memory usage "jumps". Hopefully this can help Zope2 developers find out what causes RAM bloat but can also help in helping you optimize your application by early in the development process find out what uses too much RAM. I wouldn't be surprised that there is already a program that does something like this. I've just never seen one. Also by putting this out as an Open Source project and blogging about it hopefully more clever people than me will come forward and point out the right way to do things.

I've also used Google Code this time to manage the project. I've used it before but only for hosting a public SVN for the IssueTrackerProduct SVN. I have to say that I was quite impressed with Google Code this second time. I think it's still fundamentally wrong to confuse people with by offering both download and SVN checkout. I did both this time but I think I might give up on the downloads because who out there, who understands that he/she needs to debug RAM usage, doesn't know how to use SVN?

Finally a little disclaimer: By writing about this here, preparing it on Google Code and writing a README.txt file I've now spent more time "managing" the project than I have on coding it. It's an early test release which hopefully will stir up some ideas for genuine important improvements. I had fun coding it as well since this is my first attempt with Flot which has been great to work with. You get very quick and powerful results. Lastly, I haven't tested this in anything but 32-bit Ubuntu Linux and Firefox.

Here is a sample report: 2008-05-30_16.47.32__3.8_minutes

Tonight I released an experimental version of the IssueTrackerProduct that is packed with new cool stuff. I call this an experimental release (but I run it on my production systems) because it's got so many new features.

During the course of preparing for this release and writing the news item I deployed the latest version to real.issuetrackerproduct.com and immediately noticed two bugs I to do with user names. So I immediately fixed those and prepared a new release minutes after. I expect to release another more stable version within a few weeks.

Martijn Faasen is my hero. Not only is an absolutely brilliant coder he's also able talk so that mortals understand.

This is why I like Grok

What he's replying about is mainly the question "What does Grok give me that, say, django does not?"

And, this is why I dislike Grok

Yes, you clever people. It's the same link. For some reason all the great documentation goes into replies on the mailing list rather than into a concise web page with cookbook, book and styled and funny tutorials. Why is that? They've actually made it quite easy now to enter documentation on grok.zope.org with the new Plone site.

An equally important question is: Why don't I do something about it rather than to complain? Well, I've written one how-to at least. My other "excuse" is that I'm not yet an expert enough and hence writing good documentation takes a very long time.

I think there's an important philosophical and political issue at hand. The Grok community is filled with really clever people who are very senior in the web development industry who like using mailing lists and perhaps more importantly, don't need documentation since they can study source code and unit tests to answer their questions. I know this is a sensitive statement but I'll take my chances since it implies that these guys are smarter (or perhaps just more time on their hands).

My internal battle of which new web framework to put my energy into continues. Today (thanks to Martijn's post) Grok earned one more point.

Don't ask why I'm developing products for Zope 2.7 but I had to and I should have been more careful with these oldtimers.

I kept getting this error:

(notice the strange double space after the : colon) This is different from the standard python TypeError when you get the parameters wrong which looks like this TypeError: __init__() takes exactly 2 arguments (1 given).

The line it complained this happened looked like this:

I couldn't understand what the hell was wrong on that line! Clearly it wasn't a normal Python error. Here's the explaination: That OtherClass was a new-style class inheriting from object. It looked like this:

When I changed that to:

The whole thing started to work. Long lesson learnt, don't use new-style classes mixed in into Zope 2.7.

Since I've seen so many poor solutions to this problem I thought I'd share mine. Here's how I make printer friendly pages.

1. Add a method in your base class that looks like this:

2. Change all your Page Templates to refer to a method rather than a macro directly so that pages like index_html.zpt start like this:

I've seen the hard coded way too many times where people do something like this <html metal:use-macro="here/main_template/macros/master"> which gives you no flexibility.

3. Now make a copy of main_template.zpt called print_main_template.zpt and the most important change is to make print.css load by default. Here's what it should look like this somewhere inside the <head> tag:

Note how the print.css link tag is now not conditional. Before in main_template.zpt it should have looked like this:

And note how the order is stacked just to be extra safe to weird browsers that don't understand the media condition.

As a last optional feature you should add is to add these lines at the bottom of the template 'print_main_template.zpt':

Another tip is to add something like this to the footer because it becomes useful when you look at a printed copy:

4. Now rewrite the method getMainTemplate() to become usefully intelligent:

5. Prepare the interface now for the printer friendly page. This can be done in two different ways. One way is to put a link in the footer or byline like this:

Or if you want to force a particular page to always be printer friendly, for example print_invoice.zpt then write it like this:

As a final point; how you solve your web design with screen.css and print.css varies. One way is to define multiple css files each suitable for individual things like this example shows:

An alternative solution is to don't expect print.css to stand on its own two legs but only be a supplement of the general css file. When doing this you're probably just going to want to override some things and hide some other things like this example from a 'print.css':

To conclude

This gives you a robust framework for enabling printer friendly pages that are quite different from the main template and doing it like this means that you don't have add conditional hacks to your main template that displays certain things if in printer friendly mode or not.

Most importantly, this gives you the framework for adding other versions of main template. For example these:

• mobile_main_template.zpt (guess what for)
• minimal_main_template.zpt (for things like Help page popups)

A healthy and fair use of METAL macros is also key to asserting that you don't have to repeat yourself too much in the copies of main_template.pt.

Good luck!

Here's one of many things I've learnt today at PyCon. Inspired by code that Grig Gheorghiu showed in his slides on automated testing, you can monkey patch a standard library that your application is using in your unit tests to, in my case, mock a remote service without having to run a server. I've done lots of monkey-patching in Zope but then I've only been monkey patching individual methods or attributes of imported classes. This is very similar to that. Here's what my application does:

Adjacent to this I have a unit/integration test that looks like this:

I've installed a lot of Zope instances on my laptop since version 2.7.3 and out of curiosity and desperate need for more hard drive space I thought I'd log rotate them all with the standard Linux logrotate program.

Before doing the log rotate, the total size of all my event.log files came to about 290Mb! After running logrotate (twice of course to go from event.log.1 to event.log.2.gz) the total size become 20Mb. Not a huge significance in the world of gigabyte hard drives but at least something.

This took me a while to grok and perhaps by mentioning it here, it'll prevent other people from making the same mistake as I did and perhaps preventing myself from doing the same mistake again.

In the ZCatalog, when you set up indexes you can give them a name and an index attribute. If you omit the index attribute, it'll try to hook into the objects by the name of the index. For example, if you set the index to be title with no indexed attribute it'll fetch the title attribute of the objects it catalogs. But if you set the indexed attribute to be something like idx_getTitle you can do something like this in your class:

The same can not be done with indexes of type DateIndex.

Yesterday I released IssueTrackerProduct 0.8.0 which I'm quite happy about. It's been 10 months since the last release and although there are still some outstanding feature requests and non-urgent bugs I'm happy to get this out of the door.

To all those doing the upgrade, remember to press the Update Everything button!

Together with my colleague Jan we today tried to fix a problem with Zope ZEO server that wouldn't start. Very strange we thought since another ZEO server on the same machine was working fine. We compared zope.conf files and found nothing obvious. Here was the error you get with 'bin/runzeo':

 root@da-ovz-vm99182:/var/lib/zope-2.8.9/aragdb-zeo1# ./bin/runzeo 
 Error: error opening file //var/lib/zope-2.8.9/aragdb-zeo1/etc/zeo.conf: 
 [Errno ftp error] no host given

What?? ftp error??

Long story short, the error was this line in 'runzeo':

 CONFIG_FILE="/${INSTANCE_HOME}/etc/zeo.conf"

which should have been this:

 CONFIG_FILE="${INSTANCE_HOME}/etc/zeo.conf"

Not so easy to spot because the CONFIG_FILE line is something you rarely look closely at.

I saw this today on the Zope mailing list:

 TypeError: open() takes at least 1946389116 arguments (4 given)

For anybody who's ever used Python you'll find this funny.

To set the Expires header in my web application I used to use the Zope DateTime function rfc822() which doesn't return the date in GMT format. Here's how I used it:

Then I found out (from using YSlow) that it's better to use the GMT format (RFC 1123), and here's how to do that in Zope:

(notice that even though my locale is here in London, because of the summer time an hour is added)

There are several ways of serving static files from Zope. The simplest way is to just do something like this:

The disadvantage with this is if the file is say >1Mb, the whole 1Mb needs to be loaded into RAM before it can be sent. This is surely garbage collected afterwards but if you serve many of these at the same time, there's obviously the risk that the RAM gets too full.

The alternative is to use filestream_iterator from the ZPublisher which is an iterator that sends out chunks of the file at a time. Note that to use the filestream_iterator you must set the Content-Length header first. Here's how to use it:

This week we improved MOBi with a neat little feature that makes it possible to decide what the SMS response should be to inbound SMS on your own server.

To test this you need to be UK resident and willing to part with the 25p it costs to receive the response. Send peterbe is NN (where NN is a number, how old you think I am) to 83211 and await a result. I've set up the sub word "is *" to forward all inbound SMS to an address on www.peterbe.com and here on this server I run the following Python Script:

Today I bumped into a nasty challenge when trying to copy a Zope image from ZODB to the filesystem. The reason why it wasn't easy was that Zope splits the file up into chunks if it's upload size is more than 65Kb (2^16 bytes) which I guess it does for being able to "stream" the file back into the browser when viewing the image instead of sending one large blob.

So, this didn't work:

...if the data is big because sometimes imgobj.data is a string (the binary data) and sometimes is a ImplicitAcquirerWrapper that you have to iterate over.

Lukasz Lakomy is the latest fulltime addition to Fry-IT, my company. Lukasz is a kickass Plone and Zope developer and has just launched his own little website which is done in Plone.

The content is still a bit on the light side but he's already managed to start hosting his wonderful little ZPTDebugger which has now reached a 1.0 release. This product wasn't created because it was possible but because he actually needed it. Applications that come from that kind of motivation always end up being much more useful.

Keep up the good work colleague!