Home
Mobile version of this pageby bobpooch: This works great. The key is security.apply(), that was the piece I was mi...
Setting security declarations to Zope classesby zoof: I noticed that these functions don't work with lists of lists (i.e., lists ...
Fastest way to uniqify a list in Pythonby Peter Bengtsson: yeah, you're right. I was lazy....
Most unusual letters in English languageby Peter Bengtsson: Says the Jpg is broken every time I try to open it....
Most unusual letters in English languageby K: Perhaps a better data source would be texts from project Gutenberg or Wikip...
Most unusual letters in English languageby Dougal Matthews: http://utilitymill.com/utility/Keyboard_HeatMap Not great but kinda fun....
Most unusual letters in English languageby Eric: By "3 times more common" do you mean "4 times as common"? The numbers make...
Most unusual letters in English languageby Peter Bengtsson: You're actually right. I just took a huge list of words without caring for ...
Most unusual letters in English languageby Michal Bartoszkiewicz: You should include the frequency of each word in the calculation – you prob...
Most unusual letters in English languageby Yish: Where did you get your list of words? Also is it useful to account for the...
Most unusual letters in English languageYou're viewing blogs from Python only.
I needed to find out what are the least used letters in the English language. I pulled down a list of about 100,000+ English words, split them all and made a list of about 1,000,000 letters. Sorted them by usage and came up with this as the result:
esiarntoldcugpmhbyfkwvzxjq
It would be interesting to make a heatmap of this over an image of a QWERTY keyboard.
Read the whole text (103 more words)
I was reading David Cramer's tip to use JSONField in Django to be able to store arbitrary fields in a SQL database. Nice. But is it fast enough? Well, I can't answer that but I did look into the difference in read/write performance between simplejson, cPickle and marshal.
Only reading:
Reading and writing:
Clearly marshal is faster but to quote the documentation:
"Warning: The marshal module is not intended to be secure against erroneous or maliciously constructed data. Never unmarshal data received from an untrusted or unauthenticated source."
Clearly simplejson is a very fast reader and the JSON format has the delicious advantage that it's "human readable" (compared to the others).
NOTE! I spent about 5 minutes putting together the script and about 10 minutes writing this so feel free to doubt it's scientific accuracy.
Read the whole text (139 more words)
I've now written my first Git hook. For the people who don't know what Git is you have either lived under a rock for the past few years or your not into computer programming at all.
The hook is a post-commit hook and what it does is that it sends the last commit message up to a twitter account I called "friedcode". I guess it's not entirely useful but for you who want to be loud about your work and the progress you make I guess it can make sense. Or if you're a team and you want to get a brief overview of what your team mates are up to. For me, it was mostly an experiment to try Git hooks and pytwitter. Here's how I did it:
Read the whole text (252 more words)When you write unit tests in Python you can use these widgets:
That's fine but is it "pythonic" enough? The alternative is to do with with "pure python". Eg:
I'm sure there are several benefits with using the unittest methods that I don't understand but I understand the benefits of brevity and readability. The more tests you write the more tedious it becomes to write self.assertEquals(..., ...) every time. In my own code I prefer to use simple assert statements rather than the verbose unittest alternative. Partially because I'm lazy and partially because they read better and the word assert is highlit in red in my editor so it just looks nicer from a distance.
Perhaps some much more clever people than me can explain what a cardinal sin it is to not use the unittest methods over the lazy more pythonic ones.
Incidentally, during the course of jotting down this blog I reviewed some old inherited code and changed this:
into this:
Isn't that just nicer to use/read/write?
I lost about half an hour just moments ago debugging this and pulling out a fair amount of hair. I had some code that looked like this:
It looked so simple but further up the tree I never got any entries with type="bool" even though I knew there were boolean values in the dictionary.
The pitfall I fell into was this:
Not entirely obvious if you ask me. The solution in my case was just to change the order of the if and the elif so that bool is tested first.
I'm just playing with the impressive lxml.html package. It makes it possible to easily work with HTML trees and manipulate them.
I had this crazy idea of a "DOM stripper" that removes all but specified elements from an HTML file. For example you want to keep the contents of the <head> tag intact but you just want to keep the <div id="content">...</div> tag thus omitting <div id="banner">...</div> and <div id="nav">...</div>. domstripper now does that. This can be used for example as a naive proxy that tranforms a bloated HTML page into a more stripped down smaller version suitable for say mobile web browsers. It's more a proof of concept that anything else.
To test you just need a virtual python environment and the right system libs to needed to install lxml. This worked for me:
Now you can use it like this:
Best to just play with it and see if makes sense. I'm not saying this is an amazing package but it goes to show what can be done with lxml.html and the extremely user friendly CSS selectors.
I have for a long time wondered why I'm supposed to use this in the top of my executable python files:
#!/usr/bin/env python
Today I figured out why.
The alternative, which you see a lot around is something like this:
#!/usr/bin/python
Here's why it's better to use env rather than the direct path to the executable: virtualenv. Perhaps there are plenty of other reasons the Linux experts can teach me but this is now my first obvious benefit of doing it the way I'm supposed to do it.
If you create a virtualenv, enter it and activate it so that writing:
$ python
starts the python executable of the virtual environment, then this will be respected if you use the env shebang header. Good to know.
Both in Zope2 and in Django you need to work with attributes called id. This is a shame since it's such a huge pitfall. Despite having done Python programming for so many years I today fell into this pitfall twice!! The pitfall is that id is a builtin function, not a suitable variable name. The reason is that I was changing a complex app to use something called the UUID as the indentifier instead of the ID which happened to be a name of a primary key in a table.
This meant lots of changes and I tested and tested and kept getting really strange errors. I took the whole thing apart and put it back together when I discovered my error of checking if variable id was set or not. id, if undefined, defaults to the builtin function id() which will always return true on bool(id).
It's been a long day. I'm going home. Two newbie mistakes in one programming session. I'm sure I'm not the only one who's been trapped by this.
I've never really understood the impact of new-style Python classes and what it means to your syntax until now. With new-style classes you can use the super() builtin, otherwise you can't. This works for new-style classes:
If you want to do the same for old-style classes you simply can't use super() so you'll have to do this:
Strange that I've never realised this before. The reason I did now was that I had to back-port some code into Zope 2.7 which doesn't support setting security on new-style classes.
Now I need to do some reading on new-style classes because clearly I haven't understood it all.
Inspired by Google's way of working I today put together a little script in Python for splitting a search. The idea is that you can search by entering certain keywords followed by a colon like this:
And this will be converted into two parts:
You can configure which keywords should be recognized and to make things simple, you can basically set this to be the columns you have to do advanced search on in your application. For example (from_date,to_date)
Feel free to download and use it as much as you like. You might not agree completely with it's purpose and design so you're allowed to change it as you please.
Here's how to use it:
Tonight I released an experimental version of the IssueTrackerProduct that is packed with new cool stuff. I call this an experimental release (but I run it on my production systems) because it's got so many new features.
During the course of preparing for this release and writing the news item I deployed the latest version to real.issuetrackerproduct.com and immediately noticed two bugs I to do with user names. So I immediately fixed those and prepared a new release minutes after. I expect to release another more stable version within a few weeks.
I'm going to Chicago on Wednesday for the PyCon 2008 conference. I'm going to stay at the Crowne Plaza (or whatever it was called) like many of the other people at the conference.
This is what I look like:
If you see this mug, go up to it and say Hi. It speaks British, Swedish and some American and loves food, beer and tea which might be helpful to know if you would feel like to talk more to it. Its interests for this conference are: Grok, Zope, Django, Plone, buildout, automated testing, agile development and Javascript. Its main claim-to-fame is an Open Source bug/issue tracker program called IssueTrackerProduct which it is more than delighted to talk about.
I've never been to Chicago before and I'm really excited about Tuesday night as I've bought tickets to a Chicago Bulls NBA game (basketball). All other nights I'm hoping to socialise, get drunk, get full and get down and dirty nerdy all week. See you there!
I just read the feature article "Command line programs are classes, too!" by Doug Hellmann in the January 2008 issue of Python Magazine about his program CommandLineApp and I've tried it out on one of my old Python programs where I do the opt parsing manually with getopt. The results are beautiful and quick. It's sprinkled with Doug specific magic but I quickly got over that when I saw out easy it was to work with. There are still a few questions of things I didn't manage to work out but that will unfortunately have to wait.
If anything, the worst thing about this library is that it's not part of the standard library so either you have to tell people to sudo easy_install CommandLineApp in the instructions or include it yourself in your packages if you prefer to ship things with a kitchen sink included.
If you want to check it out in action, either subscribe to the magazine (and support the effort) or just download csvcat
I was working on a unittest which when it failed would say "this string != that string" and because some of these strings were very long (output of a HTML lib I wrote which spits out snippets of HTML code) it became hard to spot how they were different. So I decided to override the usual self.assertEqual(str1, str2) in Python's unittest class instance with this little baby:
It's far from perfect and doesn't really work when you've got Unicode characters that the terminal you use can't print properly. It might not look great on strings that are really really long but I'm sure that's something that can be solved too. After all, this is just a quick hack that helped me spot that the difference between one snippet and another was that one produced <br/> and the other produced <br />. Below are some examples of this utility function in action.
Read the whole text (145 more words)I need a mini calculator in my web app so that people can enter basic mathematical expressions instead of having to work it out themselfs and then enter the result in the input box. I want them to be able to enter "3*2" or "110/3" without having to do the math first. I want this to work like a pocket calculator such that 110/3 returns a 36.6666666667 and not 36 like pure Python arithmetic would. Here's the solution which works but works like Python:
Read the whole text (361 more words)I recently wrote a Grok application that implements a REST API for Atom Publishing so that I can connect a website I have via my new Nokia phone has LifeBlog which uses the Atom API to talk to the server.
Anyway, the authentication on Atom is WSSE (good introduction article) which basically works like this:
This is one of the pieces in a request header called Authorization which can look something like this:
What I did was I wrote a simple Python script to mimic what the Nokia does but from a script. The script creates a password digest using these python modules: sha, binascii and base64 and then fires off a POST request. Here's thing, if you generate this header with base64.encodestring(ascii_string) you get something like this:
Notice the extra newline character at the end of the base64 encoded string. This is perfectly valid and is decoded easily with base64.decodestring(base64_string) by the Grok app. Everything was working fine when I tried posting to http://localhost:8080/++rest++atompub/snapatom and my application successfully authenticated the dummy user. I was happy.
Then I set this up properly on atom.someotherdomain.com which was managed by Apache who internally rewrote the URL to a Grok on localhost:8080. The problem now was that the Authentication header value was broken into two lines because of the newline character and then the whole request was rejected by Apache because some header values came without a : semi-colon.
The solution was to not use base64.encodestring() and base64.decodestring() but to instead use base64.urlsafe_b64encode() and base64.urlsafe_b64decode(). Let me show you:
If you're still reading, then hopefully you won't make the same mistake as I did and wasting time on trying to debug Apache. The lesson learned from this is to use the URL safe base64 header values and not the usual ones.
Geopy is a great little Python library for working with geocoding and distances using various online services such as Google's geocoder API.
Today I spent nearly half an hour trying to debug what was going on with my web application since I was getting this strange error:
Read the whole text (156 more words)Unlike previous incarnations of Spellcorrector not it does not by default load the two huge language files for English and Swedish. Alternatively/additionally you can load your own language file. The difference between loading a language file and training on your own words is that trained words are always assumed to be correct.
Another major change with this release is that a pickle file is created once the language file or own training file has been parsed once. This works like a cache, if the original text file changes, the pickle file is recreated. The outcome of this is that the first time you create a Spellcorrector instance it takes a few seconds if the language files is large but on the second time it takes virtually no time at all.
Read the whole text (189 more words)From the doc string:
A very spartan attempt of a script that converts HTML to
plaintext.
The original use for this little script was when I send HTML emails out I also
wanted to send a plaintext version of the HTML email as multipart. Instead of
having two methods for generating the text I decided to focus on the HTML part
first and foremost (considering that a large majority of people don't have a
problem with HTML emails) and make the fallback (plaintext) created on the fly.
This little script takes a chunk of HTML and strips out everything except the
<body> (or an elemeny ID) and inside that chunk it makes certain conversions
such as replacing all hyperlinks with footnotes where the URL is shown at the
bottom of the text instead. <strong>words</strong> are converted to *words*
and it does a fair attempt of getting the linebreaks right.
As a last resort, it strips away all other tags left that couldn't be gracefully
replaced with a plaintext equivalent.
Thanks for Fredrik Lundh's unescape() function things like:
'Terms & Conditions' is converted to
'Termss & Conditions'
It's far from perfect but a good start. It works for me for now.
Version at the time of writing this: 0.1.
I wouldn't be surprised if I've reinvented the wheel here but I did plenty of searches and couldn't really find anything like this.
Let's run this for a while until I stumble across some bugs or other inconsistencies which I haven't quite done yet. The one thing I'm really unhappy about is the way I extract the body from the BeautifulSoup parse object. I really couldn't find another better way in the few minutes I had to spare on this.
Feel free to comment on things you think are pressing bugs.
You can download the script here html2plaintext.py version 0.1
UPDATE
I should take a second look at Aaron Swartz's html2text.py script the next time I work on this. His script seems a lot more mature and Aaron is brilliant Python developer.
Like many other Python fellow geeks on Planet Python I too took the Which Programming Language Are You? quiz. Apparently I'm Prolog.
I've never used Prolog and I barely know how it works or what it's syntax looks like. Well, I guess I'll just erase all my current projects and recode them in Prolog from now on. Unpractical but necessary.
I think a lot of Python people have seen Peter Novig's beautiful article about How to Write a Spelling Corrector. So have I and couldn't wait to write my own little version of it to fit my needs.
The changes I added were:
dict instead of a collectionsuggestions()Read the whole text (220 more words)Don't do the silly misstake that I did today. I improved my code to better support unicode by replacing all plain strings with unicode strings. In there I had code that looked like this:
This was changed to:
And it no longer matched since type_ was a normal ascii string. The correct wat to do these things is like this:
Remember:
Suppose you have a list in python that looks like this:
and you want to remove all duplicates so you get this result:
How do you do that? ...the fastest way? I wrote a couple of alternative implementations and did a quick benchmark loop on the various implementations to find out which way was the fastest. (I haven't looked at memory usage). The slowest function was 78 times slower than the fastest function.
Read the whole text (567 more words)This has been a problem for a long time for me. Whenever someone enters a title in my CMS the id of the document is derived from the title. Spaces are replaced with '- and &' is replaced with and etc. The final thing I wanted to do was to make sure the Id is ASCII encoded when it's saved. My original attempt looked like this:
>>> title = u"Klüft skräms inför på fédéral électoral große"
>>> print title.encode('ascii','ignore')
Klft skrms infr p fdral lectoral groe
But as you can see, a lot of the characters are gone. I'd much rather that a word like "Klüft" is converted to "Kluft" which will be more human readable and still correct. My second attempt was to write a big table of unicode to ascii replacements.
It looked something like this:
u'\xe4': u'a', u'\xc4': u'A', etc...
Read the whole text (71 more words)If you have some code that you need to optimise, like some Javascript code that is well commented but costs too many bytes of download for your users then you might want to use my slimmer web service. I'll let a Python example speak for itself:
Read the whole text (181 more words)
I've just added some nice stats to RememberYourFriends.com. It's basically two line charts. One of how many reminders have been sent that week and one of how many reminders have been sent, ever up to a particular week from the very start.
To do this I use the wonderful CharDirector package for Python which is really fast and very easy to implement. These graphs are created on the fly and apart from generating the image it obviously needs to generate the data. Pretty fast I'd say. Will be interesting to see how it will fair when the load starts to get interesting.
Once I get a bit more users I'll start thinking of other funky charts to draw. It's fun.
I've just quickly put together a little script that computes the difference between two texts in a human readable format. The result when you run diff is a bit difficult to understand for a human being and I wanted something more "humane" that quickly summarises what's different on one simple line. Eg. "Added 2 lines, change 1 line".
This little script is going to be part an undo function in our new CMS that I'm working on. Instead of just pinpointing which revision date you want to go back to you'll also be able to see what the differences were between each revision in the undo history for the CMS.
Read the whole text (272 more words)A little while ago I wrote about how I got Jed + TAGS to work thanks the ntags library. I've been using it now for a while and I love it! I doubt there are any IDEs that beats a swift combination of Ctrl+2 followed by Alt+. and you get the definition of a function or variable without losing any focus.
If you're not into programming stop reading now because it's going to get even more technical.
Read the whole text (145 more words)Last week I wrote about a hope of slimming "private functions in Javascript" but realized that it was futile since it's not really possible. It led me to another idea where I rewrite all long-named functions within the code and relabel them at the end. The idea is that code that looks like this:
becomes this:
None of the functionality is ruined and in this case we save about 10bytes.
Read the whole text (147 more words)
In Python you indicate whether a function/method is private by naming it so that it starts with an _ underscore like this:
It means that if you have these two functions in a file called dummy.py and you do something like this:
Seems fair doesn't it. Now is there such similar naming convention and functionality in Javascript?
Read the whole text (240 more words)
