This cost me a good hour of debugging so I thought I'd share it in case anybody else stumbles across the same problem. In the end, to solve my problem I had to add debug statements to StringIO.py to be able to find out where in my Zope Page Template a non-unicode string with non-ascii characters appeared and messed things up.

The error I was getting was this, which I suspect several Zope developers have encountered before:

UnicodeDecodeError: \
'ascii' codec can't decode byte 0xc3 in position 46: ordinal not in range(128)

The traceback only mentions files in the innards of ZPT of which none you can really do anything about. We all know that the key to avoid Unicode error is to be consistent. You can do this:

>>> '\xc3' + 'string'
'\xc3string'
>>> u'\xc3' + u'string'
u'\xc3string'

But you can't do this:

>>> '\xc3' + u'string'
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: 
ordinal not in range(128)

So, how did I get these non-unicode strings into my application in first place. Simple, I have a search parameter q and end up with a URL like this:

/bla/bla?q=régime

And the template had this little innocent piece of code:

<input tal:attributes="value request/q"/>

That's what f'ed everything up. So, I ended up having to add this:

<input tal:attributes="value python:context.unicodify(request['q'])"/> 

With this little helper function in the base class:

def unicodify(self, s):
   if isinstance(s, str):
       return unicode(s, 'utf8')
   return s

So, hopefully by writing this it will help someone else making the same trivial mistake and not wasting their evening with sporadic print statements all over their frameworks code.

Anonymous - 16 April 2009 [«« Reply to this]
Wouldn't bla/bla?q:ustring:utf8=régime be a simpler solution ?
Peter Bengtsson - 17 April 2009 [«« Reply to this]
Sure, ...but uglier. I often use, but rarely rely on Zope's magic type casting. In the same sense I often use but don't rely on Javascript
Damian - 19 April 2009 [«« Reply to this]
Hello, finally found your blog :) and it is very interesting. I got question, what is better Python or PHP in web development?


Your email will never ever be published