A blog and website by Peter Bengtsson

My friend and colleague Jannis (aka jezdez) Leidel saved my bacon today where I had gotten completely stuck.

So, I have this python2.6 virtualenv and whenever I ran python sdist upload it would upload a really nasty tarball to PyPI. What would happen is that when people do pip install premailer it would file horribly and look something like this:

IOError: [Errno 2] No such file or directory: '/path/to/virtual-env/build/premailer/'

What?!?! If you download the tarball and unpack it you'll see that there definitely is a file in there.

Anyway. What happens, which I didn't realize was that within the .tar.gz file there were these strange copies of files. For example for every there was a etc.

Here's what the file looked like after a tarball had been created:

(premailer26)peterbe@mpb:~/dev/PYTHON/premailer (master)$ tar -zvtf dist/premailer-2.0.2.tar.gz
-rwxr-xr-x  0 peterbe staff     311 Apr 11 15:51 ./._premailer-2.0.2
drwxr-xr-x  0 peterbe staff       0 Apr 11 15:51 premailer-2.0.2/
-rw-r--r--  0 peterbe staff     280 Mar 28 10:13 premailer-2.0.2/._LICENSE
-rw-r--r--  0 peterbe staff    1517 Mar 28 10:13 premailer-2.0.2/LICENSE
-rw-r--r--  0 peterbe staff     280 Apr  9 21:10 premailer-2.0.2/
-rw-r--r--  0 peterbe staff      34 Apr  9 21:10 premailer-2.0.2/
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/._PKG-INFO
-rw-r--r--  0 peterbe staff    7226 Apr 11 15:51 premailer-2.0.2/PKG-INFO
-rwxr-xr-x  0 peterbe staff     311 Apr 11 15:51 premailer-2.0.2/._premailer
drwxr-xr-x  0 peterbe staff       0 Apr 11 15:51 premailer-2.0.2/premailer/
-rwxr-xr-x  0 peterbe staff     311 Apr 11 15:51 premailer-2.0.2/._premailer.egg-info
drwxr-xr-x  0 peterbe staff       0 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/
-rw-r--r--  0 peterbe staff     280 Mar 28 10:13 premailer-2.0.2/
-rw-r--r--  0 peterbe staff    5185 Mar 28 10:13 premailer-2.0.2/
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/._setup.cfg
-rw-r--r--  0 peterbe staff      59 Apr 11 15:51 premailer-2.0.2/setup.cfg
-rw-r--r--  0 peterbe staff     280 Apr  9 21:09 premailer-2.0.2/
-rw-r--r--  0 peterbe staff    2079 Apr  9 21:09 premailer-2.0.2/
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._dependency_links.txt
-rw-r--r--  0 peterbe staff       1 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/dependency_links.txt
-rw-r--r--  0 peterbe staff     280 Apr  9 21:04 premailer-2.0.2/premailer.egg-info/._not-zip-safe
-rw-r--r--  0 peterbe staff       1 Apr  9 21:04 premailer-2.0.2/premailer.egg-info/not-zip-safe
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._PKG-INFO
-rw-r--r--  0 peterbe staff    7226 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/PKG-INFO
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._requires.txt
-rw-r--r--  0 peterbe staff      23 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/requires.txt
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._SOURCES.txt
-rw-r--r--  0 peterbe staff     329 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/SOURCES.txt
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._top_level.txt
-rw-r--r--  0 peterbe staff      10 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/top_level.txt
-rw-r--r--  0 peterbe staff     280 Apr  9 21:21 premailer-2.0.2/premailer/
-rw-r--r--  0 peterbe staff      66 Apr  9 21:21 premailer-2.0.2/premailer/
-rw-r--r--  0 peterbe staff     280 Apr  9 09:23 premailer-2.0.2/premailer/
-rw-r--r--  0 peterbe staff    3315 Apr  9 09:23 premailer-2.0.2/premailer/
-rw-r--r--  0 peterbe staff     280 Apr  8 16:22 premailer-2.0.2/premailer/
-rw-r--r--  0 peterbe staff   15368 Apr  8 16:22 premailer-2.0.2/premailer/
-rw-r--r--  0 peterbe staff     280 Apr  8 16:22 premailer-2.0.2/premailer/
-rw-r--r--  0 peterbe staff   37184 Apr  8 16:22 premailer-2.0.2/premailer/

Strangly, this only happened in a Python 2.6 environment. The problem went away when I created a brand new Python 2.7 enviroment with the latest setuptools.

So basically, the fault lies with OSX and a strange interaction between OSX and tar.
This answer does a much better job explaining this "flaw".

So, the solution to the problem is to create the distribution like this instead:

$ COPYFILE_DISABLE=true python sdist

If you do that, you get a healthy lookin tarball that actually works to pip install. Thanks jezdez for pointing that out!

Buggy is a singe-page webapp that relies entirely on the Bugzilla Native REST API. And it works offline. Sort of. I say "sort of" because obviously without a network connection you're bound to have outdated information from the bugzilla database but at least you'll have what you had when you went offline.

When you post a comment from Buggy, the posted comment is added to an internal sync queue and if you're online it immediately processes that queue. There is, of course, always a risk that you might close a bug when you're in a tunnel or on a plane without WiFi and when you later get back online the sync fails because of some conflict.

The reason I built this was partly to scratch an itch I had ("What's the ideal way possible for me to use Bugzilla?") and also to experiment with some new techniques, namely AngularJS and localforage.


So, the way it works is:

  1. You pick your favorite product and components.

  2. All bugs under these products and components are downloaded and stored locally in your browser (thank you localforage).

  3. When you click any bug it then proceeds to download its change history and its comments.

  4. Periodically it checks each of your chosen product and components to see if new bugs or new comments have been added.

  5. If you refresh your browser, all bugs are loaded from a local copy stored in your browser and in the background it downloads any new bugs or comments or changes.

  6. If you enter your username and password, an auth token is stored in your browser and you can thus access secure bugs.

I can has charts

Pros and cons

The main advantage of Buggy compared to Bugzilla is that it's fast to navigate. You can instantly filter bugs by status(es), components and/or by searching in the bug summary.

The disadvantage of Buggy is that you can't see all fields, file new bugs or change all fields.

The code

The code is of course open source. It's available on and released under a MPL 2 license.

The code requires no server. It's just an HTML page with some CSS and Javascript.

Everything is done using AngularJS. It's only my second AngularJS project but this is also part of why I built this. To learn AngularJS better.

Much of the inspiration came from the CSS framework Pure and one of their sample layouts which I started with and hacked into shape.

The deployment

Because Buggy doesn't require a server, this is the very first time I've been able to deploy something entirely on CDN. Not just the images, CSS and Javascript but the main HTML page as well. Before I explain how I did that, let me explain about the script.

I really wanted to use Grunt but it just didn't work for me. There are many positive things about Grunt such as the ease with which you can easily add plugins and I like how you just have one "standard" file that defines how a bunch of meta tasks should be done. However, I just couldn't get the concatenation and minification and stuff to work together. Individually each tool works fine, such as the grunt-contrib-uglify plugin but together none of them appeared to want to work. Perhaps I just required too much.

In the end I wrote a script in python that does exactly what I want for deployment. Its features are:

  • Hashes in the minified and concatenated CSS and Javascript files (e.g. vendor-8254f6b.min.js)
  • Custom names for the minified and concatenated CSS and Javascript files so I can easily set far-future cache headers (e.g. /_cache/vendor-8254f6b.min.js)
  • Ability to fold all CSS minified into the HTML (since there's only one page, theres little reason to make the CSS external)
  • A Git revision SHA into the HTML of the generated ./dist/index.html file
  • All files in ./client/static/ copied intelligently into ./dist/static/
  • Images in CSS to be given hashes so they too can have far-future cache headers

So, the way I have it set up is that, on my server, I have a it run python and that generates a complete site in a ./dist/ directory. I then point Nginx to that directory and run it under Then I set up a Amazon Cloudfront distribution to that domain and then lastly I set up a CNAME for to point to the Cloudfront distribution.

The future

I try my best to maintain a TODO file inside the repo. That's where I write down things to come. (it's also works as a changelog) since I also use this file to write down what's been done.

One of the main features I want to add is the ability to add bugs that are outside your chosen products and components. It'll be a "fake" component called "Misc". This is for bugs outside the products and components you usually monitor and work in but perhaps bugs you've filed or been assigned to. Or just other bugs you're interested in in general.

Another major feature to work on is the ability to choose to see more fields and ability to edit these too. This will require some configuration on the individual users' behalf. For example, some people use the "Target Milestone" a lot. Some use the "Importance" a lot. So, some generic solution is needed to accomodate all these non-basic fields.

And last but not least, the Bugzilla team here at Mozilla is working on a very exciting project that allows you to register a certain list of bugs with a WebSocket and have it push to you as soon as these bugs change. That means that I won't have to periodically query bugzilla every 30 seconds if certain bugs have changed but instead get instant notifications when they do. That's going to be major! I confidently speculate that that will be implemented some time summer this year.

Give it a go. What are you waiting for? :) Go to, pick your favorite products and components and try to use it for a week.

I do not deny it. I'm a YouTube fiend. I very rarely watch YouTube on my computer but a lot on my Apple TV and only tablet. It's

Here are some of my favorite YouTube channels that I subscribe to and encourage you to do the same if you aren't already and if there's something it appears you'll like too.


1. MinutePhysics

They started as clips that were around 1 minute but are now of variable length. I just adore Henry's voice and the topics he chooses. The animations are cute and even though seasoned with silly cat and dog references they really help to explain some of the most advanced subjects in physics.

Incidentally, this was the first channel I subscribed to once I figured that's the best way to get recurring content from channels I really liked.


2. Numberphile

This is a Brady Haran production that speaks directly to my mathematical aspirations. These aspirations aren't to solve any complex calculus problems but to keep that almost mystic infatuation alive I have with mathematics. There's something wonderfully down to earth and kind about the content which challenge you without patronizing you. By the way, my favorite interviewee, James Grime has his own channel now called singingbanana and also, by the way, and amazingly unattractive website.


3. Veritasium

Derek Muller is a brilliant video maker. Most of his videos are about science and it's mainly Derek holding his camera at arms length filming his pleasant face and talking about the perception or understanding of science. More so than the science itself. Actually some videos are not about how people (miss)understand science but speak directly to you and those are just brilliant. Usually sufficiently advanced to really get reallying thinking hard.

CGP Grey

4. CGP Grey

The only, of my top favorite channels, that is not about natural science. These videos are on social science subjects you might never have thought to think about and not only that, but each and every one digs deep and misses very few facts. Similarly to SciShow, these videos require your full attention. Because what you learn from them is often so very valuable, I've revisited many videos. Some more than twice.


5. MinuteEarth

This is Henry Reich's (see above about MinutePhysics) second channel and the name of the channels fully describes what the videos are about. The animations are really magnificantly simple and rich at the same time. The subject matters in this videos are generally less advanced that those in MinutePhysics but often full of really interesting factoids to keep up your sleeve for dinner parties.


6. SciShow

Hang Green is a gem! His geeky and passionate mannerisms is worth it just on its own. But you have to pay full attention because Hank speaks very fast. There is though an important undertone that isn't immediately obvious. There is this feeling of deeply researched facts. Even though you only understand a small part of it all (not to mention how little you remember!) it's inspiring that someone takes the time to do all the research.
A lot of subject matters are science oriented but more popular sciencey.

Sixty Symbols

7. Sixty Symbols

Another Brady Haran production, but this time more about physics and than Numberphile which is more about mathematics. Almost all videos are Brady interviewing doctors and professors in physics at the University of Nottingham. All very humble and approachable interviewees that, perhaps thanks to Brady's brilliant questions, the subjects are understandable but also very exciting because they're usually on matters that are very advanced and something more to look forward to than to enjoy in the moment.

PHD Comics

8. Piled Higher and Deeper (PHD Comics)

This is a newcomer and I include it because they're of such high quality and adorable animations. To be honest I don't think I really understand what the various videos have in common. For example, one recent video is on quantum entanglement and another on the Dead Sea scrolls. Either way, every video is professional and highly enjoyable.

There are more channels I subscribe to and enjoy very much but the above list are my favorite ones. For example, I watch Jamie Oliver's Food Tube videos just as often but that's somehow more "obvious".

Actually I have many more channels on science and a bunch of computers and programming but I'm just simply not as passionate about them as I are with the channels mentioned above.

I really hope that by writing this it will inspire one or two fellow science nerdy readers to also discover some of the channels mentioned here.

Last week I built a little tools called github-pr-triage. It's a single page app that sits on top of the wonderful GitHub API v3.

Its goal is to try to get an overview of what needs to happen next to open pull requests. Or rather, what needs to happen next to get it closed. Or rather, who needs to act next to get it closed.

It's very common, at least in my team, that someone puts up a pull request, asks someone to review it and then walks away from it. She then doesn't notice that perhaps the integrated test runner fails on it and the reviewer is thinking to herself "I'll review the code once the tests don't fail" and all of a sudden the ball is not in anybody's court. Or someone makes a comment on a pull request that the author of the pull requests misses in her firehose of email notifictions. Now she doesn't know that the comment means that the ball is back in her court.

Ultimately, the responsibility lies with the author of the pull request to pester and nag till it gets landed or closed but oftentimes the ball is in someone elses court and hopefully this tool makes that clearer.

Here's an example instance:

Currently you can use for any public Github repo but if too many projects eat up all the API rate limits we have I might need to narrow it down to use mozilla repos. Or, you can simply host your own. It's just a simple Flask server

About the technology

I'm getting more and more productive with Angular but I still consider myself a beginner. Saying that also buys me insurance when you laugh at my code.

So it's a single page app that uses HTML5 pushState and an angular $routeProvider to make different URLs.

The server simply acts as a proxy for making queries to and and the reason for that is for caching.

Every API request you make through this proxy gets cached for 10 minutes. But here's the clever part. Every time it fetches actual remote data it stores it in two caches. One for 10 minutes and one for 24 hours. And when it stores it for 24 hours it also stores its last ETag so that I can make conditional requests. The advantage of that is you quickly know if the data hasn't changed and more importantly it doesn't count against you in the rate limiter.

What a book!
I defend that it took months to finish it with; it's not easy reading, I only read on the short train commute 3 days a week and I'm a really really slow reader.

Moby Dick book cover
Even though it was hard going at times I generally enjoyed every page. Some passages were like reading Latin but with English words. Some pages where thrilling and some pages where beautiful as poetry. Some short passages were so amazing that you have to stop and just take a quick smile-break.

Unlike many people I actually didn't know how the book plays out or how it ends and I'm NOT going to spoil that here, in case you too want to read it too, not knowing how it ends. The only thing I regret is reading the editors introduction which revealed something crucial to the plot line without any warning.

It was not until afterwards when I read about the book on Wikipedia (link contains spoilers) that I appreciated the many sub-plots and sub-contexts. For example, the many metaphysical and theological undertones. For one thing (this is NOT a spoiler), if you're going to read it pay extra attention to peoples' names.

One thing I can reveal is that the book is basically three books but you don't really notice that when it goes from one to the other. I can not imagine a modern day publisher allowing that to happen to a contemporary book with contemporary readers who have less attention span than a gold fish. However, I am glad I've read it because not only is it an entertaining book it's also a good exercise in modern life that not everything has to be so perfect and lean.

And a final tip to you who now feel inspired to read the book for the first time; it's an old-English book with lots of words you won't know and that's fine, but do take the time to look up some of the nautical words related to the ship because they re-appear again and again when you're reading action filled passages. Like bulwark, masthead and starboard.

tl;dr 36

For some time now, I've been running an experiment where I analyze how many different domains any website depends on. For example, you might have Google Analytics on your site (that's and you might have a Facebook Like button (that's and/or and you might serve your images from a CDN (that's That there is 3-4 distinct domains.

Independent of how many requests come from each domain, I wanted to measure how many distinct domains a website depends on so I wrote a script and started collecting random URLs across the web. Most of the time, to get a sample of different URLs I would take the RSS feed on and the RSS feed on Hacker News on a periodic basis.

Network tab on the Dev Tools console for a page on
The results are amazing! Some websites depend on over 100 different domain names!

Take this page on The Toast for example, it depends on 143 different domains. Loading it causes your browser to make 391 requests, download 4.8Mb and takes 29 seconds (in total, not necessarily till you can start reading it). What were they thinking!?!

I think what this means is that website makers will probably continue to make websites like this. What we, as web software engineers, can not tell people it's a bad idea but instead to try to do something about it. It's quite far from my expertise but clearly if you want to make the Internet faster, DNS would be an area to focus on.

Test it out for yourself here: Number of Domains

For people familar with AngularJS, it's almost frighteningly easy to make a live-search on a repeating iterator.

Here's such an example:

Out of the box it just works. If nothing is typed into the search field it returns everything.

A big problem with this is that the pattern matching isn't very good. For example, if you search for ter you get Teresa and Peter.
More realistically you want it to only match with a leading word delimiter. In other words, if you type ter you want it only to match Teresa but not Peter because Peter doesn't start with ter.
So, to remedy that we construct a regular expression on the fly with a leading word delimiter. I.e. \bter.

Here's an example of that:

Now, there's a problem. For every item in the list the regular expression needs to be created and compiled which, when the list is very long, can become incredibly slow.
To remedy that we use $scope.$watch to create a local regular expression which only happens once per update to $

Here's an example of that:

That, I think, is a really good pattern. Unfortunately we've left the simplicity but we now have something snappier.

Unfortunately the example is a little bit contrived because the list of names it filters on is so small but the list could be huge. It could also be that we want to make a more advanced regular expression. For example, you might want to allow multiple words to match so as ter ma should match Teresa Mayers, John Mayor and Maria Connor. Then you could make a regular expression with something like \b(ter|ma).

For seasoned Angularnauts this is trivial stuff but it really helped me make an app much faster and smoother. I hope it helps someones else doing something similar.

Because this bit me harder than I was ready for, I thought I'd make a note of it for the next victim.

In Python 2, suppose you have this:

Python 2.7.5
>>> items = [(1, 'A number'), ('a', 'A letter'), (2, 'Another number')]

Sorting them, without specifying how, will automatically notice that it contains tuples:

Python 2.7.5
>>> sorted(items)
[(1, 'A number'), (2, 'Another number'), ('a', 'A letter')]

This doesn't work in Python 3 because comparing integers and strings is not allowed. E.g.:

Python 3.3.3
>>> 1 < '1'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: int() < str()

You have to convert them to stings first.

Python 3.3.3
>>> sorted(items, key=lambda x: str(x[0]))
[(1, 'A number'), (2, 'Another number'), ('a', 'A letter')]

If you really need to sort by 1 < '1' this won't work. Then you need a more complex key function. E.g.:

Python 3.3.3
>>> def keyfunction(x):
...   v = x[0]
...   if isinstance(v, int): v = '0%d' % v
...   return v
>>> sorted(items, key=keyfunction)
[(1, 'A number'), (2, 'Another number'), ('1', 'Actually a string')]

That's really messy but the best I can come up with at past 4PM on Friday.

I have and have had many sites that I run. They're all some form of side-project.

What they almost all have in common is two things

  1. They have very little traffic (thus not particularly mission critical)
  2. I run everything on one server (no need for "spinning up" new VMs here and there)

Many many years ago, when current interns I work with were mere babies, I started a very simple "procedure".

  1. On the server, in the user directory where the site is deployed, I write a script called something like which is executable and does what the name of the script is: it upgrades the site.

  2. In the server's root home directory I write a script called which also does exactly what the name of the script is: it restarts the service.

  3. On my laptop, in my ~/bin directory I create a script called (*) which runs on the server and runs also on the server.

And here is, if I may say so, the cleverness of this; I use ssh to execute these scripts remotely by simply piping the commands to ssh. For example:

echo "./" | ssh -A
echo "./" | ssh

That's an example I use for Wish List Granted.

This works so darn well, and has done for years, that this is why I've never really learned to use more advanced tools like Fabric, Salt, Puppet, Chef or <insert latest deployment tool name>.

This means that all I need to do run a deployment is just type[ENTER] and the simple little bash scripts takes care of everything else.

The reason I keep these on the server and not on my laptop is simply because that's where they naturally belong and if I'm ssh'ed in and mess around I don't have to exit out to re-run them.

Here's an example of the I use for Wish List Granted:

cd generousfriends
source venv/bin/activate
git pull origin master
find . | grep '\.pyc$' | xargs rm -f
pip install -r requirements/prod.txt
./ syncdb --noinput
./ migrate webapp.main
./ collectstatic --noinput
./ compress --force
echo "Restart must be done by root"

I hope that, by blogging about this, that someone else sees that it doesn't really have to be that complicated. It's not rocket science and most complex tools are only really needed when you have a significant bigger scale in terms of people- and skill-complexity.

In conclusion

Keep it simple.

(*) The reason for the capitalization of my scripts is also an old habit. I use that habit to differentiate my scripts for stuff I install from any third parties.

On Wednesday this week, I managed to get a link to Wish List Granted onto Hacker News. It had enough upvotes to be featured on the front page for a couple of hours. I'm very grateful for the added traffic but not quite so impressed with the ultimate conversions.

  • 4,428 unique visitors
  • 43 Wish Lists created
  • 2 Usersnap pieces of constructive feedback
  • 0 payments made

Google Analytics
So that's 1% conversion of people setting up a wish list. But kinda disappointing that no body ever made a payment. Actually, one friend did make a payment. But he's a colleague and a friend so not a stranger who stumbled onto it from Hacker News.

Also, it's now been 3 days since those 43 wish lists were created and still no payments. That's kinda disappointing too.

I'm starting to fear that Wish List Granted is one of those ideas that people think it's a great idea but have no interest in using.