24 September 2011 0 comments Tornado
First of all, I should say: I didn't write much of this code. It's copied from Django and modified to work in any Tornado app. It hinges on the same idea as Django that you have to specify what backend you want to use. A backend is an importable string pointing to a class that has the
To begin, here's a sample use case inside a request handler:
class ContactUs(tornado.web.RequestHandler): def post(self): msg = self.get_argument('msg') # NB: you might want to set this once and for all in something # like self.application.settings['email_backend'] backend = 'tornado_utils.send_mail.backends.smtp.EmailBackend' send_email(backend, "New contact form entry", msg + '\n\n--\nFrom our contact form\n', 'email@example.com', ['firstname.lastname@example.org'], ) self.write("Thanks!")
The problem is that SMTP is slow. Even though, in human terms, it's fast, it's still too slow for a non-blocking server that Tornado is. Taking 1-2 seconds to send a message over SMTP means it's blocking every other request to Tornado for 1-2 seconds. The solution is instead save the message on disk in pickled form and use a cron job to pick up the messages and send them by SMTP instead, outside the Tornado process. First do this re-write:
... def post(self): msg = self.get_argument('msg') - backend = 'tornado_utils.send_mail.backends.smtp.EmailBackend' + backend = 'tornado_utils.send_mail.backends.pickle.EmailBackend' ...
Now, write a cron job script that looks something like this:
# send_pickled_messages.py DRY_RUN = False def main(): from tornado_utils.send_mail import config filenames = glob(os.path.join(config.PICKLE_LOCATION, '*.pickle')) filenames.sort() if not filenames: return from tornado_utils.send_mail import backends import cPickle if DRY_RUN: EmailBackend = backends.console.EmailBackend else: EmailBackend = backends.smtp.EmailBackend max_count = 10 filenames = filenames[:max_count] messages = [cPickle.load(open(x, 'rb')) for x in filenames] backend = EmailBackend() backend.send_messages(messages) if not DRY_RUN: for filename in filenames: os.remove(filename)
That code just above is butchered from a more comprehensive script I have but you get the idea. Writing to a pickle file is so fast it's in the lower milliseconds region. However, it depends on disk IO so if you need more speed, write a simple backend that writes instead of saving pickles on disk, make it write to a fast in-memory database like Redis or Memcache.
The code isn't new and it's been battle tested but it's only really been battle tested in the way that my apps use it. So you might stumble across bugs if you use it in a way I haven't tested. However, the code is Open Source and happily available for you to help out and improve.Follow @peterbe on Twitter