Peterbe.com

Today Felinx Lee and I launched TornadoGists.org which is a site for discussing gists related to Tornado (python web framework open sourced by Facebook).

Everyone in the Tornado community seems to solve similar problems in different ways. Oftentimes, these solutions are just a couple of lines or so and not something you can really turn into a full package with setup.py and everything.

Sharing a snippet of code is a great way to a) help other people and b) to get feedback on your solutions.

The goal is to make it a very open and active project with lots of contributors. I'll be accepting and reviewing all forks but hopefully control will be opened up to all Tornado developers. Also, since the code is quite generic to any open source project Felinx and I might one day port this to rubygists.org or lispgists.org or something like that. After all, Github does all the heavy lifting and we just wrap it up nicely.

Yay! I've figured out how to properly unit tests my jQuery Mobile app that I'm working on. This app uses localStorage, localSession, lots of AJAX and works both offline and online. Through various hacks and tricks I've managed to mock things so that I can easily test the otherwise asynchronous AJAX calls and I can also test the little quirks of jQuery Mobile such as re-rendering <ul> tags after having changed the DOM tree.

I'm using the QUnit testing framework and I like it. The app isn't launched yet and the code is currently protected but once I get it nailed a bit more I'll blog about it more fully so other people can jump in unit testing their Javascript heavy jQuery Mobile sites too. For now I'm up to 75 tests and it's growing steadily.

Here's a little taster for how I mock the $.mobile, AJAX and the localStorage stuff:

module("Storage and AJAX", {
  setup: function() {
     localStorage.clear();
  },
  teardown: function() {
     localStorage.clear();
  }
});

var MockMobile = function() {
  this.current_page;
};
MockMobile.prototype.changePage = function(location) {
  this.current_page = location;
};

test("test Auth", function() {
  $.mobile = new MockMobile();
  var _last_alert;
  alert = function(msg) {
     _last_alert = msg;
  };
  $.ajax = function(options) {
     switch (options.url) {
      case '/smartphone/checkguid/':
        options.success({ok:true});
        break;
      case '/smartphone/auth/login/':
        if (options.data.password == 'secret')
          options.success({guid:'10001'});
        else
          options.success({error:"Wrong credentials"});
        break;
      default:
        console.log(options.url);
        throw new Error("Mock not prepared (" + options.url + ")");
     }
  };
  var result = Auth.is_logged_in(false);
  equals(result, false);

  Auth.ajax_login('peterbe@example.com', 'other junk');
  ok(_last_alert);
  ...

(Note: this is copied slightly out of context but hopefully it reveals some of the hacks and tricks I use)

Erann Gat reveals why he lost his mojo with Lisp

What caught my attention (for busy people who don't want to read the whole email):

I'm currently studying Lisp myself and it's hard. Really hard. I blame it on being spoiled with a programming language that I can work in without having to read the manual. With python's brilliant introspection I can use the interpreter to find out how a library works just by using help() and dir() without even having to read the source code. (not always true of course)

As we're entering the 21st century, the new contender "Usability" is becoming more and more important. Considering that I've now done Python for more than a decade I remind myself one of the reasons I liked it so much; yes, exactly that: Usability.

What a lovely title for a book! I wanted to read a book about the proper way to write Javascript but I couldn't wait any longer for John Resig's Secrets of the JavaScript Ninja which isn't out in print yet (also a great title by the way).

Eloquent Javascript begins very lightly with the basics of Javascript programming. Variables, scope, data structures and control flow. To be perfectly honest I didn't read it very carefully but I believe I did pick up a thing or two at least. The chapter on error handling was useful but the really interesting chapters were "Functional Programming" and "Object-Oriented Programming". What I love about Marijn's style of writing is that he starts very very simple and builds up the code to be better and better. Not based on what you can do but instead why you should do it. As you go along you can then immediately snap up what the benefits are for yourself. Sometimes it's brevity and sometimes it's for faster performance. My only criticism if I'm allowed is that the jargon is quite a lot keep up with. Especially around "constructors" and "prototypes" which is sometimes easy to forget (especially if you're from another language where these things mean different things).

I'm not great at it but I already knew how to write modules and "classes" so ultimately there wasn't a whole lot to take away from it to be honest. Some tricks such as the inheritance function which Marijn introduced was neat and that might be something I'll copy. Nevertheless, this book showed and educated me in why we do things as modules and stuff which I genuinely appreciated.

Thanks for a great book Marijn! Keep up the good work!

My colleague Lukas and I banged our heads against this for much too long today. So, our SQLAlchemy is was configured like this:

ENV_DB_CONNECTION_DSN = postgresql://localhost:5432/mydatabase

And the database doesn't have a password (local) so I can log in to it like this on the command line:

$ psql mydatabase

Which assumes the username peterbe which is what I'm logged in. So, this is a shortcut for doing this:

$ psql mydatabase -U peterbe

Which, assumes a blank/empty password.

Like so many others you probably have an Nginx server sitting in front of your application server (Django, Zope, Rails). The Nginx server serves static files right off the filesystem and when it doesn't do that it proxy passes the request on to the backend. You might be using proxy_pass, uwsgi or fastcgi_pass or at least something very similar. Most likely you have an Nginx site configure something like this:

server {
   access_log /var/log/nginx/mysite.access.log;
   location ^~ /static/ {
       root /var/lib/webapp;
       access_log off;
   }
   location / {
       proxy_pass http://localhost:8000;
   }
}

What I do is that I add an access log directive that times every request. This makes it possible to know how long every non-trivial request takes for the backend to complete:

server {
   log_format timed_combined '$remote_addr - $remote_user [$time_local]  ' 
                             '"$request" $status $body_bytes_sent '
                             '"$http_referer" "$http_user_agent" $request_time';
   access_log /var/log/nginx/timed.mysite.access.log timed_combined;

   location ^~ /css/ {
       root /var/lib/webapp/static;
       access_log off;
   }
   location / {
       proxy_pass http://localhost:8000;
   }
}

I needed a very basic spell correction string matcher in my current NodeJS project so I wrote a simple class called EditDistanceMatcher that compares a string against another string and matches if it's 1 edit distance away. With it you can do things like Google search's "Did you mean: poop?" when you search for pop.

Note, this code doesn't check popularity of correct words (e.g. "pop" might appear much more often than "poop" so it'll suggest "pop" if you enter "poup"). Anyway this simple snippet from the unit tests will reveal how it works:

     /* The match() method */
     var edm = new EditDistanceMatcher(["peter"]);
     // edm.match returns an array and remember,
     // in javascript ['peter'] == ['peter'] => false
     test.equal(edm.match("petter").length, 1);
     test.equal(edm.match("petter")[0], 'peter');
     test.equal(edm.match("junk").length, 0);

     /* the is_matched() method */
     var edm = new EditDistanceMatcher(["peter"]);
     test.equal(typeof edm.is_matched('petter'), 'boolean');
     test.equal(typeof edm.is_matched('junk'), 'boolean');
     test.ok(edm.is_matched("petter"));
     test.ok(!edm.is_matched("junk"));

The most basic use case is if you have a quiz and you want to accept some spelling mistakes. "What's the capital of Sweden?; STOKHOLM; Correct!"

For the unlazy this NodeJS code can very easily be used in a browser by simply removing the exports stuff.

edit_distance.js

tests/test_edit_distance.js

Note! I wrote this in an airport lounge so I'm sure it can be improved lots more.

I've spent a lot of time trying to figure out which Javascript script loading framework to use. RequireJS or HeadJS. I still don't have an answer. Neither website refers to each other.

In general

• To me, it's important to be able to load and execute some Javascript before downloading Javascript modules that aren't needed to render the initial screen. Makes for a more responsive behaviour and gets pixels drawn quicker for Javascript-heavy sites.
• An understated, massive, benefit to combining multiple .js files into one is sporadic network bottlenecks. Fewer files to download and fewer things can go wrong. These bottlenecks can make a few Kb of a Javascript file take 10 seconds to download.
• Public CDNs (e.g. jQuery from Google's CDN) is an extremely powerful optimization technique. Not only are they extremely fast, it's very likely they're preloaded because some other site uses the exact same URL.
• Where does it say that Javascript has to be loaded in the head? Even html5-boilerplate loads Javascript just before the </body> tag.
• Realistically, in the real world, it's not uncommon that you can't combine all .js files into one. This is not true for web apps that consists of just one HTML file. One page might require A.js, B.js and C.js but another page requires A.js, B.js and D.js. Requires manual thinking whether you should combine A,B,C,D.js or A,B.js + C|D.js. No framework can predict this.
• All loading and browser incompatibility hacks will eventually become obsolete as browsers catch up. Again, requires manual thinking because supporting and ultra-boosting performance might have a different cost today compared to a year from now. The most guilty of this appears to be ControlJS
• I'm confident that optimization in terms of file concatenation and white space optimization does not belong to the framework.
• Apparently iPhone 3.x series can't cache individual files larger than 15Kb (25Kb for iPhone 4.x). That's a very small about if you combine several large modules.
• Accepting the fact of life that sporadic network bottlenecks can kill your page, think hard about asynchronous loading and preserved order. Perhaps ideal is a mix of both. What framework allows that? (both RequireJS and HeadJS it seems)
• Loading frameworks are not for everything and everyone. If you're building something "simple" or landing page like Google's search page frameworks might just get in your way.

RequireJS

• Author well known but his Dojoesque style shines through in RequireJS's syntax and patterns.
• Is only about Javascript. No CSS hacks or other html5ish boilerplates.
• Gets into the realm of module definitions. Neat but do you want the loading framework to get involved in how you prefer to write your code or do you just want it to load your files?
• All the module definition stuff feels excessive for every single project I can imagine but we're entering an era of "web apps" (as opposed to "web sites") so this might need to change.
• What you learn in using RequireJS you can reuse when building NodeJS (a server-side framework). It's also possible to use RequireJS in Rhino (server-side Javascript engine) but personally I haven't reached that level yet.

HeadJS

• Author relatively unknown. quite well known too. Author also of Flowplayer and jQuery Tools.
• Contains a kitchen sink (CSS tricks, modernizer.js) but perhaps they're really quite useful. After all, you don't write your web site in Assembly.
• There's a fork of HeadJS that does just the Javascript stuff. But will it be maintained? And does that defeat the whole point of using HeadJS?
• With its CSS hacks (aka. kitchen sink) HeadJS seems great if you really care about combining HTML5 techniques with Internet Explorer.
• This awesome experiment shows that HeadJS really works and that asynchronous loading can be really powerful. But ask yourself, are you ready to build in an asynchronous way?
• With HeadJS I can label a combined and optimized bundle and load my code once that bundle is loaded. Can I do that with RequireJS? It seems to depend on the filename (minus the .js suffix).
• Makes the assumption that just because a file is loaded the order of execution is a non-issue. This means you might have trouble controlling dependencies during execution. This is a grey area that might or might not matter depending on the complexity of your app.
• A feeling I get is that HeadJS without the CSS kitchen sink stuff reduces to become LabJS or EnhanceJS.

Other alternatives

The ones I can think of are: ControlJS (feels too "hacky" for my taste), CommonJS (not sufficiently "in-browser specific" for my taste) and EnhanceJS (like HeadJS and LabJS but with less power/features)

The one I haven't studied as much is LabJS. It seems more similar to HeadJS in style. Perhaps it deserves more attention but the reason HeadJS got my attention is because it's got a better looking website.

In conclusion

You mileage will vary. The deeper I look into this I feel personal taste comes into play. It's hard enough for a single framework other to write realistic benchmarks; even harder for "evalutators" like myself to benchmark them all. It gets incrementally harder when you take into account the effects of http latency, sporadic network bottlenecks, browser garbage collection and user experience.

Personally I think HeadJS is a smoother transition for general web sites. RequireJS might be more appropriate when write web apps with virtually no HTML and a single URL.

With the risk of starting a war... If you're a Rails/Django/Plone head, consider HeadJS. If you're a mobile web app/NodeJS head consider RequireJS.

UPDATE

Sorry, I now realise that Tero Piirainen actually has built a fair amount of powerful Javascript libraries.

This feels like a bit of a face-plant moment but I've never understood why anyone would use the code module when you can use the pdb when the pdb is like the code module but with less.

What you use it for is to create you own custom shell. Django does this nicely with it's shell management command. I often find myself doing this:

$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from this import that
>>> from module import something
>>> from db import connection
>>> con = connection('0.0.0.0', bla=True)
>>> con.insert(something(that))

And there's certain things I almost always import (depending on the project). So use code to write your own little custom shell loader that imports all the stuff you need. Here's one I wrote real quick. Ultra useful time-saver:

#!/usr/bin/env python
import code, re
if __name__ == '__main__':
   from apps.main.models import *
   from mongokit import Connection
   from pymongo.objectid import InvalidId, ObjectId
   con = Connection()
   db = con.worklog
   print "AVAILABLE:"
   print '\n'.join(['\t%s'%x for x in locals().keys()
                    if re.findall('[A-Z]\w+|db|con', x)])
   print "Database available as 'db'"
   code.interact(local=locals())

This is working really well for me and saving me lots of time. Hopefully someone else finds it useful.

After about two months of evening hacking I'm finally ready to release my latest project: DoneCal.com

It's a simple calendar that doesn't get in your way. You just click on a day and type what you did that day. DoneCal can be an ideal replacement to boring spreadsheet-like timesheets. And unlike regular timesheets/timetrackers with tags you immediately get statistics about how you've spent your time.

I'm personally excited about the Bookmarklet because I practically live in my webbrowser and now I can quickly type what I've just done (could be a piece of support work for a client) with one single click.

If you're a project manager trying to track what your developers are working on, ask them to start tracking time on DoneCal and then ask them to share their calendar with you. They can set up their share so that it only shares on relevant tags.

I'm going to improving it more and more as feedback comes in. Hopefully later this week I'm going to be writing about the technical side of this since this is my first web app built with the uber-fast Tornado framework