Grymt - because I didn't invent Grunt here
18 April 2014
grymt is a python tool that takes a directory full of .html, .css and .js and prepares the html for optimial production use.
For a teaser:
So why did I write my own tool and not use Grunt?!
Glad you asked! The reason is simple: I couldn't get Grunt to work.
Grunt is a framework. It's a place where you say which "recipes" to execute and how. It's effectively a common config framework. Like make.
However, I tried to set up a bunch of recipes in my Gruntfile.js and most of them worked well individually but it was a hellish nightmare to get it all to work together just the way I want it.
For example, the grunt-contrib-uglify is fine for doing the minification but it doesn't work with concatenation and it doesn't deal with taking one input file and outputting to a different file.
Basically, I spent two evenings getting things to work but I could never get exactly what I wanted. So I wrote my own and because I'm quite familiar with this kind of stuff, I did it in Python. Not because it's better than Node but just because I had it near by and was able to quicker build something.
So what sweet features do you get out of grymt?
You can easily make an output file have a hash in the filename. E.g.
vendor-64f7425.min.jsand thus the filename is always unique but doesn't change in between deployments unless you change the files.
It automatically notices which files already have been minified. E.g. no need to minify
somelib.min.jsbut do minify
You can put
$git_revisionanywhere in your HTML and this gets expanded automatically. For example, view the source of buggy.peterbe.com and look at the first 20 lines.
Images inside CSS get rewritten to have unique names (based on files' modified time) so they can be far-future cached aggresively too.
You never have to write down any lists of file names in soome Gruntfile.js equivalent file
$('<img>').attr('src', 'picture.jpg')for example.
The generated (aka. "dist" directory) contains everything you need. It does not refer back to the source directory in any way. This means you can set up your apache/nginx to point directly at the root of your "dist" directory.
So what's the catch?
It's not Grunt. It's not a framework. It does only what it does and if you want it to do more you have to work on grymt itself.
The files you want to analyze, process and output all have to be in a sub directory.
Look at how I've laid out the files here in this project for example. ALL files that you need is all in one sub-directory called
app. So, to run
grymtI simply run:
The HTML files you throw into it have to be plain HTML files. No templates for server-side code.
How do you use it?
pip install grymt
Then you need a directory it can process, e.g
./client/ (assumed to contain a .html file(s)).
For more options, check out
What's in the future of grymt?
If people like it and want to add features, I'm more than happy to accept pull requests. Some future potential feature work:
I haven't needed it immediately, yet, myself, but it would be nice to add things like coffeescript, less, sass etc into pre-processing hooks.
It would be easy to automatically generate and insert a reference to a appcache manifest. Since every file used and mentioned is noticed, we could very accurately generate an appcache file that is less prone to human error.
Spitting out some stats about number bytes saved and number of files reduced.
COPYFILE_DISABLE and python distutils in python 2.6
12 April 2014
My friend and colleague Jannis (aka jezdez) Leidel saved my bacon today where I had gotten completely stuck.
So, I have this python2.6 virtualenv and whenever I ran
python setup.py sdist upload it would upload a really nasty tarball to PyPI. What would happen is that when people do
pip install premailer it would file horribly and look something like this:
... IOError: [Errno 2] No such file or directory: '/path/to/virtual-env/build/premailer/setup.py'
What?!?! If you download the tarball and unpack it you'll see that there definitely is a
setup.py file in there.
Anyway. What happens, which I didn't realize was that within the
.tar.gz file there were these strange copies of files. For example for every
file.py there was a
Here's what the file looked like after a tarball had been created:
(premailer26)peterbe@mpb:~/dev/PYTHON/premailer (master)$ tar -zvtf dist/premailer-2.0.2.tar.gz -rwxr-xr-x 0 peterbe staff 311 Apr 11 15:51 ./._premailer-2.0.2 drwxr-xr-x 0 peterbe staff 0 Apr 11 15:51 premailer-2.0.2/ -rw-r--r-- 0 peterbe staff 280 Mar 28 10:13 premailer-2.0.2/._LICENSE -rw-r--r-- 0 peterbe staff 1517 Mar 28 10:13 premailer-2.0.2/LICENSE -rw-r--r-- 0 peterbe staff 280 Apr 9 21:10 premailer-2.0.2/._MANIFEST.in -rw-r--r-- 0 peterbe staff 34 Apr 9 21:10 premailer-2.0.2/MANIFEST.in -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/._PKG-INFO -rw-r--r-- 0 peterbe staff 7226 Apr 11 15:51 premailer-2.0.2/PKG-INFO -rwxr-xr-x 0 peterbe staff 311 Apr 11 15:51 premailer-2.0.2/._premailer drwxr-xr-x 0 peterbe staff 0 Apr 11 15:51 premailer-2.0.2/premailer/ -rwxr-xr-x 0 peterbe staff 311 Apr 11 15:51 premailer-2.0.2/._premailer.egg-info drwxr-xr-x 0 peterbe staff 0 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/ -rw-r--r-- 0 peterbe staff 280 Mar 28 10:13 premailer-2.0.2/._README.md -rw-r--r-- 0 peterbe staff 5185 Mar 28 10:13 premailer-2.0.2/README.md -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/._setup.cfg -rw-r--r-- 0 peterbe staff 59 Apr 11 15:51 premailer-2.0.2/setup.cfg -rw-r--r-- 0 peterbe staff 280 Apr 9 21:09 premailer-2.0.2/._setup.py -rw-r--r-- 0 peterbe staff 2079 Apr 9 21:09 premailer-2.0.2/setup.py -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._dependency_links.txt -rw-r--r-- 0 peterbe staff 1 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/dependency_links.txt -rw-r--r-- 0 peterbe staff 280 Apr 9 21:04 premailer-2.0.2/premailer.egg-info/._not-zip-safe -rw-r--r-- 0 peterbe staff 1 Apr 9 21:04 premailer-2.0.2/premailer.egg-info/not-zip-safe -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._PKG-INFO -rw-r--r-- 0 peterbe staff 7226 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/PKG-INFO -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._requires.txt -rw-r--r-- 0 peterbe staff 23 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/requires.txt -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._SOURCES.txt -rw-r--r-- 0 peterbe staff 329 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/SOURCES.txt -rw-r--r-- 0 peterbe staff 280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._top_level.txt -rw-r--r-- 0 peterbe staff 10 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/top_level.txt -rw-r--r-- 0 peterbe staff 280 Apr 9 21:21 premailer-2.0.2/premailer/.___init__.py -rw-r--r-- 0 peterbe staff 66 Apr 9 21:21 premailer-2.0.2/premailer/__init__.py -rw-r--r-- 0 peterbe staff 280 Apr 9 09:23 premailer-2.0.2/premailer/.___main__.py -rw-r--r-- 0 peterbe staff 3315 Apr 9 09:23 premailer-2.0.2/premailer/__main__.py -rw-r--r-- 0 peterbe staff 280 Apr 8 16:22 premailer-2.0.2/premailer/._premailer.py -rw-r--r-- 0 peterbe staff 15368 Apr 8 16:22 premailer-2.0.2/premailer/premailer.py -rw-r--r-- 0 peterbe staff 280 Apr 8 16:22 premailer-2.0.2/premailer/._test_premailer.py -rw-r--r-- 0 peterbe staff 37184 Apr 8 16:22 premailer-2.0.2/premailer/test_premailer.py
Strangly, this only happened in a Python 2.6 environment. The problem went away when I created a brand new Python 2.7 enviroment with the latest
So basically, the fault lies with OSX and a strange interaction between OSX and tar.
This superuser.com answer does a much better job explaining this "flaw".
So, the solution to the problem is to create the distribution like this instead:
$ COPYFILE_DISABLE=true python setup.py sdist
If you do that, you get a healthy lookin tarball that actually works to pip install. Thanks jezdez for pointing that out!