03 May 2016
1 comment
Python,
Web development,
Django,
Mozilla
tl;dr; Use raven's ThreadedRequestsHTTPTransport transport class to send Google Analytics pageview trackings asynchronously to Google Analytics to collect pageviews that aren't actually browser pages.
We have an API on our Django site that was not designed from the ground up. We had a bunch of internal endpoints that were used by the website. So we simply exposed those as API endpoints that anybody can query. All we did was wrap certain parts carefully as to not expose private stuff and we wrote a simple web page where you can see a list of all the endpoints and what parameters are needed. Later we added auth-by-token.
Now the problem we have is that we don't know which endpoints people use and, as equally important, which ones people don't use. If we had more stats we'd be able to confidently deprecate some (for easier maintanenace) and optimize some (to avoid resource overuse).
Our first attempt was to use statsd to collect metrics and display those with graphite. But it just didn't work out. There are just too many different "keys". Basically, each endpoint (aka URL, aka URI) is a key. And if you include the query string parameters, the number of keys just gets nuts. Statsd and graphite is better when you have about as many keys as you have fingers on one hand. For example, HTTP error codes, 200, 302, 400, 404 and 500.
Also, we already use Google Analytics to track pageviews on our website, which is basically a measure of how many people render web pages that have HTML and JavaScript. Google Analytic's UI is great and powerful. I'm sure other competing tools like Mixpanel, Piwik, Gauges, etc are great too, but Google Analytics is reliable, likely to stick around and something many people are familiar with.
So how do you simulate pageviews when you don't have JavaScript rendering? The answer; using plain HTTP POST. (HTTPS of course). And how do you prevent blocking on sending analytics without making your users have to wait? By doing it asynchronously. Either by threading or a background working message queue.
Threading or a message queue
If you have a message queue configured and confident in its running, you should probably use that. But it adds a certain element of complexity. It makes your stack more complex because now you need to maintain a consumer(s) and the central message queue thing itself. What if you don't have a message queue all set up? Use Python threading.
To do the threading, which is hard, it's always a good idea to try to stand on the shoulder of giants. Or, if you can't find a giant, find something that is mature and proven to work well over time. We found that in Raven.
Raven is the Python library, or "agent", used for Sentry, the open source error tracking software. As you can tell by the name, Raven tries to be quite agnostic of Sentry the server component. Inside it, it has a couple of good libraries for making threaded jobs whose task is to make web requests. In particuarly, the awesome ThreadedRequestsHTTPTransport. Using it basically looks like this:
import urlparse
from raven.transport.threaded_requests import ThreadedRequestsHTTPTransport
transporter = ThreadedRequestsHTTPTransport(
urlparse.urlparse('https://ssl.google-analytics.com/collect'),
timeout=5
)
params = {
...more about this later...
}
def success_cb():
print "Yay!"
def failure_cb(exception):
print "Boo :("
transporter.async_send(
params,
headers,
success_cb,
failure_cb
)
The call isn't very different from regular plain old requests.post.
About the parameters
This is probably the most exciting part and the place where you need some thought. It's non-trivial because you might need to put some careful thought into what you want to track.
Your friends is: This documentation page
There's also the Hit Builder tool where you can check that the values you are going to send make sense.
Some of the basic ones are easy:
"Protocol Version"
Just set to v=1
"Tracking ID"
That code thing you see in the regular chunk of JavaScript you put in the head, e.g tid=UA-1234-Z
"Data Source"
Optional word you call this type of traffic. We went with ds=api because we use it to measure the web API.
The user ones are a bit more tricky. Basically because you don't want to accidentally leak potentially sensitive information. We decided to keep this highly anonymized.
"Client ID"
A random UUID (version 4) number that identifies the user or the app. Not to be confused with "User ID" which is basically a string that identifies the user's session storage ID or something. Since in our case we don't have a user (unless they use an API token) we leave this to a new random UUID each time. E.g. cid=uuid.uuid4().hex This field is not optional.
"User ID"
Some string that identifies the user but doesn't reveal anything about the user. For example, we use the PostgreSQL primary key ID of the user as a string. It just means we can know if the same user make several API requests but we can never know who that user is. Google Analytics uses it to "lump" requests together. This field is optional.
Next we need to pass information about the hit and the "content". This is important. Especially the "Hit type" because this is where you make your manually server-side tracking act as if the user had clicked around on the website with a browser.
"Hit type"
Set this to t=pageview and it'll show up Google Analytics as if the user had just navigated to the URL in her browser. It's kinda weird to do this because clearly the user hasn't. Most likely she's used curl or something from the command line. So it's not really a pageview but, on our end, we have "views" in the webserver that produce information to the user. Some of it is HTML and some of it is JSON, in terms of output format, but either way they're sending us a URL and we respond with data.
"Document location URL"
The full absolute URL of that was used. E.g. https://www.example.com/page?foo=bar. So in our Django app we set this to dl=request.build_absolute_uri(). If you have a site where you might have multiple domains in use but want to collect them all under just 1 specific domain you need to set dh=example.com.
"Document Host Name" and "Document Path"
I actually don't know what the point of this is if you've already set the "Document location URL".
"Document Title"
In Google Analytics you can view your Content Drilldown by title instead of by URL path. In our case we set this to a string we know from the internal Python class that is used to make the API endpoint. dt='API (%s)'%api_model.__class__.__name__.
There are many more things you can set, such as the clients IP, the user agent, timings, exceptions. We chose to NOT include the user's IP. If people using the JavaScript version of Google Analytics can set their browser to NOT include the IP, we should respect that. Also, it's rarely interesting to see where the requests for a web API because it's often servers' curl or requests that makes the query, not the human.
Sample implementation
Going back to the code example mentioned above, let's demonstrate a fuller example:
import urlparse
from raven.transport.threaded_requests import ThreadedRequestsHTTPTransport
transporter = ThreadedRequestsHTTPTransport(
urlparse.urlparse('https://ssl.google-analytics.com/collect'),
timeout=5
)
# Remember, this is a Django, but you get the idea
domain = settings.GOOGLE_ANALYTICS_DOMAIN
if not domain or domain == 'auto':
domain = RequestSite(request).domain
params = {
'v': 1,
'tid': settings.GOOGLE_ANALYTICS_ID,
'dh': domain,
't': 'pageview,
'ds': 'api',
'cid': uuid.uuid4().hext,
'dp': request.path,
'dl': request.build_request_uri(),
'dt': 'API ({})'.format(model_class.__class__.__name__),
'ua': request.META.get('HTTP_USER_AGENT'),
}
def success_cb():
logger.info('Successfully informed Google Analytics (%s)', params)
def failure_cb(exception):
logger.exception(exception)
transporter.async_send(
params,
headers,
success_cb,
failure_cb
)
How to unit test this
The class we're using, ThreadedRequestsHTTPTransport has, as you might have seen, a method called async_send. There's also one, with the exact same signature, called sync_send which does the same thing but in a blocking fashion. So you could make your code look someting silly like this:
def send_tracking(page_title, request, async=True):
# ...same as example above but wrapped in a function...
function = async and transporter.async_send or transporter.sync_send
function(
params,
headers,
success_cb,
failure_cb
)
And then in your tests you pass in async=False instead.
But don't do that. The code shouldn't be sub-serviant to the tests (unless it's for the sake of splitting up monster-long functions).
Instead, I recommend you mock the inner workings of that ThreadedRequestsHTTPTransport class so you can make the whole operation synchronous. For example...
import mock
from django.test import TestCase
from django.test.client import RequestFactory
from where.you.have import pageview_tracking
class TestTracking(TestCase):
@mock.patch('raven.transport.threaded_requests.AsyncWorker')
@mock.patch('requests.post')
def test_pageview_tracking(self, rpost, aw):
def mocked_queue(function, data, headers, success_cb, failure_cb):
function(data, headers, success_cb, failure_cb)
aw().queue.side_effect = mocked_queue
request = RequestFactory().get('/some/page')
with self.settings(GOOGLE_ANALYTICS_ID='XYZ-123'):
pageview_tracking('Test page', request)
# Now we can assert that 'requests.post' was called.
# Left as an exercise to the reader :)
print rpost.mock_calls
This is synchronous now and works great. It's not finished. You might want to write a side effect for the requests.post so you can have better control of that post. That'll also give you a chance to potentially NOT return a 200 OK and make sure that your failure_cb callback function gets called.
How to manually test this
One thing I was very curious about when I started was to see how it worked if you really ran this for reals but without polluting your real Google Analytics account. For that I built a second little web server on the side, whose address I used instead of https://ssl.google-analytics.com/collect. So, change your code so that https://ssl.google-analytics.com/collect is not hardcoded but a variable you can change locally. Change it to http://localhost:5000/ and start this little Flask server:
import time
import random
from flask import Flask, abort, request
app = Flask(__name__)
app.debug = True
@app.route("/", methods=['GET', 'POST'])
def hello():
print "- " * 40
print request.method, request.path
print "ARGS:", request.args
print "FORM:", request.form
print "DATA:", repr(request.data)
if request.args.get('sleep'):
sec = int(request.args['sleep'])
print "** Sleeping for", sec, "seconds"
time.sleep(sec)
print "** Done sleeping."
if random.randint(1, 5) == 1:
abort(500)
elif random.randint(1, 5) == 1:
# really get it stuck now
time.sleep(20)
return "OK"
if __name__ == "__main__":
app.run()
Now you get an insight into what gets posted and you can pretend that it's slow to respond. Also, you can get an insight into how your app behaves when this collection destination throws a 5xx error.
How to really test it
Google Analytics is tricky to test in that they collect all the stuff they collect then they take their time to process it and it then shows up the next day as stats. But, there's a hack! You can go into your Google Analytics account and click "Real-Time" -> "Overview" and you should see hits coming in as you're testing this. Obviously you don't want to do this on your real production account, but perhaps you have a stage/dev instance you can use. Or, just be patient :)
26 February 2016
0 comments
Python,
Web development,
Mozilla
Last year I developed a web app called "Whatsdeployed". It's one of those rare one-afternoon-hacks that turns out to be really really useful. I use it every [work]day. And I've heard many people say they use it too.
At the time I built it, it only supported comparing multiple instance. E.g. a production and a dev site. Or a test, stage and production. But oftentimes, especially for smaller projects, you might only just have your one deployed site.
So I've now made it possible so you can compare just 1 site against your github.com master branch.
For example: whatsdeployed.io/s-Sir
Or whatsdeployed.io/s-J14
What these do, is simply comparing what git sha revision is deployed on those side-projects, compared to the latest git sha on the master branch on github.com.
27 January 2016
0 comments
Python,
Web development,
Javascript,
Mozilla
Here's the final demo.
What I did was, I used the Bugzilla REST APIs to download all bugs for a specific product. Then I bulk-uploaded then to Autocompeter.com and lastly built a simply web front-end.
When you "download all" bugs with the Bugzilla REST API, it might be capped but I don't know what the limit is. The trick is to not download ALL bugs for the product in one big fat query, but to find out what all components are for that product and then download for each. The Python code is here.
Everyone's Invited to Play
So first you need to sign in on https://autocompeter.com using your GitHub account. Then you can generate a Auth-Key by picking a domain. The domain can be anything really. I picked bugzilla.mozilla.org but you can use whatever you like.
Then, when you have an Auth-Key you need to know the name of the product (or products) and run the script like this:
python download.py 7U4eFYH5cqR15m3ekuxkzaUR Socorro
Once you've done that, fork my codepen and replace the domain and any other references to the product.
Caveats
To make this really useful, you'd have to run it more often. Perhaps you can hook it up to a cron job or something and make it so that you only download, from the REST API, things that have changed since the last time you did a big download. Then you can let the cron job run frequently.
If you want really hot results, you could hook up a server-side service that consumes the Bugzfeed websocket.
Last but not least; this will never list private/secure bugs. Only publically available stuff.
The Future
If people enjoy it perhaps we can change the front-end demo so it's not hardcoded to one specific product ("Socorro" in my case). And it can be made pretty.
And the data would need to be downloaded and re-submitted more frequently. A quick Heroku app mayhaps?
11 November 2015
4 comments
Python,
Web development,
Mozilla
Whatsdeployed was a tool I developed for my work at Mozilla. I think many other organizations can benefit from using it too.
So, on many sites, what we do when deploying a site, is that we note which git sha was used and write that to a file which is then exposed via the web server. Like this for example. If you know that sha and what's at the tip of the master branch on the project's GitHub page, you can build up an interesting dashboard that allows you to see what's available and what's been deployed.
The other really useful case is when you have more than just one environment. For example, you might have a dev, stage and prod environment and, always lastly, the master branch on GitHub. Now you can see what code has been shipped on prod versus your staging environment for example.
This is one of those far too few projects that you build quickly one Friday afternoon and it turns out to be surprisingly useful to a lot of people. I for one, check various projects like this several times per day.
The code is on GitHub and it's basically a tiny bit of Flask with some jQuery doing a couple of AJAX requests. If you enjoy it and use it, please share.
10 October 2015
3 comments
Linux,
Web development,
Mozilla
I've written about mozjpeg before where I showed what it can do to a sample directory full of different kinds of JPEGs. But let's get more real. Let's actually install it and look at one thumbnail and one big photo.
To install, I used the pre-compiled binaries from this wonderful site. Like this:
# wget http://mozjpeg.codelove.de/bin/mozjpeg_3.1_amd64.deb
# dpkg -i mozjpeg_3.1_amd64.deb
# ls -l /opt/mozjpeg/bin/cjpeg
-rwxr-xr-x 1 root root 50784 Sep 3 19:03 /opt/mozjpeg/bin/cjpeg
I don't know why the binary executable becomes called cjpeg but that's fine. Let's put it in $PATH so other users can execute it:
# cd /usr/local/bin
# ln -s /opt/mozjpeg/bin/cjpeg
Now, let's actually use it for something. First we need a realistic lossy thumbnail that we can optimize.
$ wget http://cdn-2916.kxcdn.com/static/cache/eb/f0/ebf08e64e80170dc009e97f6f9681ceb.jpg
This was one of the thumbnails from a previous post called Panasonic Lumix from 2008 or a iPhone 5S from 2014.
Let's optimize!
$ jpeg -outfile ebf08e64e80170dc009e97f6f9681ceb.moz.jpg -optimise ebf08e64e80170dc009e97f6f9681ceb.jpg
$ ls -l ebf08e64e80170dc009e97f6f9681ceb.*
-rw-rw-r-- 1 django django 11391 Sep 26 17:04 ebf08e64e80170dc009e97f6f9681ceb.jpg
-rw-r--r-- 1 django django 9414 Oct 10 01:40 ebf08e64e80170dc009e97f6f9681ceb.moz.jpg
Yay! It's 17.4% smaller. Saving 1.93Kb.
So what do they look like? See for yourself:
I have to zoom in (⌘-+) 3 times until I can see any difference. But remember, the saving isn't massive but the usecase here is a thumbnail.
So, let's do the same with a non-thumbnail. Some huge JPEG.
$ time cjpeg -outfile Lumix-2.moz.jpg -optimise Lumix-2.jpg
real 0m3.285s
user 0m3.122s
sys 0m0.080s
$ ls -l Lumix*
-rw-rw-r-- 1 django django 4880446 Sep 26 17:20 Lumix-2.jpg
-rw-rw-r-- 1 django django 1546978 Oct 10 02:02 Lumix-2.moz.jpg
$ ls -lh Lumix*
-rw-rw-r-- 1 django django 4.7M Sep 26 17:20 Lumix-2.jpg
-rw-rw-r-- 1 django django 1.5M Oct 10 02:02 Lumix-2.moz.jpg
In other words, from 4.7Mb to 1.5Mb. It's 68.3% the size of the original. And the visual difference?
Again, I have to zoom in 3 times to be able to tell any difference and even when I've done that it's hard to tell which is which.
In conclusion, let's go ahead and use mozjpeg to optimize thumbnails.
01 September 2015
5 comments
Web development,
Mozilla
I'm currently working on a Django library that uses mozjpeg to optimize thumbnails that are generated from stored images. I first wanted to get a feel for how good mozjpeg really is.
In my ~/Downloads directory I have all sorts of "junk" from all sorts of saves and experiments. It'll work as a good testbed of relatively random JPEG images of all sorts of sizes and qualities. Without further ado, here's the results:
FILENAME OPTIMIZE ORIGINAL SAVING PERCENT
-----------------------------------------------------------------------------------------
180697_1836563311933_3364808_n.jpg 45.2Kb 50.4Kb 5.1Kb 10.2%
2014-03-20 17.35.39.jpg 2040.1Kb 2207.8Kb 167.7Kb 7.6%
2015-03-04 21.18.16.jpg 1521.5Kb 1629.2Kb 107.7Kb 6.6%
2015-03-04 21.19.16.jpg 1602.4Kb 1720.0Kb 117.6Kb 6.8%
2015-03-04 21.23.16.jpg 1181.7Kb 1272.1Kb 90.4Kb 7.1%
2015-03-05 06.03.00.jpg 1426.7Kb 1557.7Kb 131.0Kb 8.4%
20150626_200629_001.jpg 1566.4Kb 1717.3Kb 151.0Kb 8.8%
20150626_200631.jpg 2157.6Kb 2319.6Kb 162.0Kb 7.0%
Boba_Fett_by_RobD4E.jpg 96.2Kb 104.3Kb 8.1Kb 7.8%
Horse_Play.jpg 170.4Kb 185.2Kb 14.9Kb 8.0%
Image (107).jpg 344.9Kb 390.6Kb 45.7Kb 11.7%
Misc Candle Holder NECA FOTR Balrog Dec2002.jpg 37.1Kb 37.7Kb 0.6Kb 1.5%
Mozilla_Lightbeam.jpg 55.1Kb 79.7Kb 24.6Kb 30.8%
Photo on 12-17-14 at 5.55 PM.jpg 168.5Kb 187.7Kb 19.2Kb 10.2%
dev.jpg 17.5Kb 30.8Kb 13.3Kb 43.2%
dev2.jpg 41.1Kb 54.3Kb 13.3Kb 24.4%
dev3.jpg 35.3Kb 49.0Kb 13.7Kb 28.0%
dev4.jpg 42.0Kb 56.0Kb 14.0Kb 25.0%
dev5.jpg 24.6Kb 37.9Kb 13.2Kb 35.0%
dev6.jpg 28.9Kb 42.8Kb 13.9Kb 32.4%
hr_0570_220_135__0570220135006.jpg 3124.3Kb 3467.8Kb 343.5Kb 9.9%
hr_0570_220_158__0570220158006.jpg 3010.0Kb 3319.1Kb 309.1Kb 9.3%
hr_0570_220_175__0570220175006.jpg 2245.5Kb 2442.6Kb 197.0Kb 8.1%
hr_0570_227_599__0570227599006.jpg 2561.7Kb 2809.8Kb 248.1Kb 8.8%
hr_0596_622_701__0596622701006.jpg 3238.8Kb 3453.6Kb 214.7Kb 6.2%
hr_0596_623_849__0596623849006.jpg 2902.9Kb 3102.1Kb 199.3Kb 6.4%
hr_0622_219_873__0622219873006.jpg 985.3Kb 1066.9Kb 81.7Kb 7.7%
logo.jpg 43.5Kb 51.2Kb 7.7Kb 15.1%
mvm-header.jpg 8.5Kb 12.4Kb 3.9Kb 31.6%
mvm-postcard-picture.jpg 72.2Kb 73.4Kb 1.3Kb 1.7%
overhang_pixels.jpg 3014.3Kb 3370.8Kb 356.4Kb 10.6%
peterbe copy.jpg 4.2Kb 10.4Kb 6.2Kb 59.7%
peterbe.jpg 36.7Kb 44.3Kb 7.5Kb 17.0%
pjt-mcguinty-2.jpg 96.8Kb 101.6Kb 4.8Kb 4.8%
sl1.jpg 28.7Kb 35.4Kb 6.7Kb 18.9%
That's an median of 9.3% (average of 15.3%) savings.
It's not very fast though. Some of the large files take more than a second. In total it took 23.7 seconds to create all of those optimized files. Do what you want with that fact, bear in mind that these are hopefully "once in a lifetime" operations (depending on the ephemerality of your thumbnail storage). Mind you, the really large JPEGs skew that since the median is 72.1 milliseconds and average is 527.0 milliseconds. Also, when I look through the numbers I find that the large JPGs take the longest but had the least benefit in terms of byte savings.
UPDATE
Chris Adams, in the comment below, inspired me to compare my trials with jpegoptim and jpegrescan. So, I took my script that generated a directory of 45 JPEGs and changed it to use jpegoptim and jpegrescan.
The mozjpeg total size of that output directory is 34.1Mb and it took a total of 23.3 seconds (median 76.4 milliseconds).
The jpegoptim & jpegrescan total size of that output directory is 35.6Mb and it took a total of 4.6 seconds (median 32.1 milliseconds).
In other words, roughly speaking mozjpeg is 4.2% more space effective and 58% slower than jpegoptim & jpegrescan.
25 August 2015
0 comments
Web development,
Mozilla
tl;dr Crash-stats is Mozilla's crash reporter dashboard. Simply fixing the static assets made the site 25% faster.
Before http://www.webpagetest.org/result/150820_X5_V5T/
After http://www.webpagetest.org/result/150824_7F_1C3Q/
(The "First Byte Time" is still terrible but that's for another discussion. We're working on a re-write of the underlying data model for that particular report.)
-
Note how the SpeedIndex dropped from 2823 to 2098 which basically means, you can see stuff sooner.
-
The Load Time used to be 5.7 seconds on average. Now it takes 3.5 seconds.
-
It used to weigh 717 KB to load the whole thing. Now it weighs 326 KB.
The only thing we changed was a long overdue correction of static asset headers and Gzip compression. Now, files with unique URLs (e.g. /static/CACHE/css/23a811f100bc.css) have maximum aggressive cache headers. And now all .js, .css and text/html is Gzipped.
Was it easy to do? Hell no!
Does it matter? Hell yeah! We don't have a lot of users or traffic on these reports but the people who use them do this for a living and making the site feel snappier for them would make their lives more productive.
23 April 2015
0 comments
Javascript,
Mozilla
Starting today, (almost) all the thumbnails below the fold on Air Mozilla are not loaded.
The way it works, is that I use a library called Lazyr.js which notices when you scroll down and when certain pictures are going to be in view, it changes the <img> tag's src.
So it basically looks like this:
<article>
<h3>Event 1</h3>
<img src="event1.png">
</article>
<article>
<h3>Event 2</h3>
<img src="event2.png">
</article>
<article>
<h3>Event 3</h3>
<img src="event3.png">
</article>
<article>
<h3>Event 4</h3>
<img src="placeholder.png" data-lazyr="event4.png">
</article>
<article>
<h3>Event 5</h3>
<img src="placeholder.png" data-lazyr="event5.png">
</article>
<article>
<h3>Event 6</h3>
<img src="placeholder.png" data-lazyr="event6.png">
</article>
That means that to load this page it needs to download, only:
event1.png
event2.png
event3.png
placeholder.png
Only 4 images instead of the otherwise 6 (in this example).
When you scroll down to see the rest of the list, it then also downloads:
event4.png
event5.png
event6.png
The actual numbers on Air Mozilla is that there are 10 events page page and I lazy load 6 of them.
You can see the results when comparing this WebPageTest with this one.
There is more work to do though. At the moment, the thumbnails in the sidebar (Trending and Upcoming events) are above the fold when you're browsing but below the fold when you're viewing an individual event. That's something I have yet to implement.
05 March 2015
2 comments
Mozilla
We're proud to announce that we've now published our first Roku channel; Air Mozilla
We actually started this work in the third quarter of 2014 but the review process for adding a channel is really slow. The people we've talked to have been super friendly and provide really helpful feedback as to changes that need to be made. After the first submission, it took about a month for them to get back to us and after some procrastination we submitted it a second time about a month ago and yesterday we found out it's been fully published. I.e. gone live.
Obviously it would be nice if they could get back to us quicker but another thing they could improve is to appreciate that we're a team. All communication with Roku has been to just me and I always have to forward emails or add my teammates as CC when I communicate with them.
Anyway, now we can start on a version 2. We deliberately kept this first version ultra-simple just to prove that it's possible and not being held back due to feature creep.
What we're looking to add in version 2 are, in no particular order:
- Ability to navigate by search
- Ability to sign in and see restricted content
- Adding Trending events
- Ability to see what the upcoming events are
It's going to be much easier to find the energy to work on those features now that we know it's live.
Also, we currently have a problem watching live and archived streams on HTTPS. It's not a huge problem right now because we're not making any restricted content available and we're lucky in that the CDNs we use allow for HTTP traffic equally.
Remember, Air Mozilla is Open Source and we encourage people to jump in and contribute if you want to share your Python, design, Javascript or BrightScript skills.
By the way, the Air Mozilla Roku code is here and there's a README that'll get your started if you want to help out.
19 December 2014
0 comments
Linux,
Web development,
Mozilla
tl;dr Don't run ffmpeg over HTTP(S) and use ffmpegthumbnailer
UPDATE tl;dr Download the file then run ffmpeg with -ss HH:MM:SS first. Don't bother with ffmpegthumbnailer
At work I work on something called Air Mozilla. It's a site for hosting live video broadcasts and then archiving those so they can be retrieved later.
Unlike sites like YouTube we can't take a screencap from the video because many videos are future (aka. "upcoming") videos so instead we use a little placeholder thumbnail (for example, the Rust logo).
However, once it has been recorded we want to switch from the logo to an actual screen capture from the video itself. We set up a cronjob that uses ffmpeg to extract these as JPGs and then the users can go in and select whichever picture they like the best.
This is all work in progress by the way (as of December 2014).
One problem is that we have is that the command for extracting JPGs is really slow. So slow that we can't wrap the subprocess in a Django database connection because it's so slow that the database connection is often killed.
The command to extract them looks something like this:
ffmpeg -i https://cdnexample.com/url/to/file.mp4 -r 0.0143 /tmp/screencaps-%02d.jpg
Where the number r is based on the duration and how many pictures we want out. E.g. 0.0143 = 15 * 1049 where 15 is how many JPGs we want and 1049 is a duration of 17 minutes and 29 seconds.
The script I used first was: ffmpeg1.sh
My first experiment was to try to extract one picture at a time, hoping that way, internally, ffmpeg might be able to optimize something.
The second script I used was: ffmpeg2.sh
The third alternative was to try ffmpegthumbnailer which is an intricate wrapper on ffmpeg and it has the benefit that you can produce slightly higher picture quality too.
The third script I used was: ffmpeg3.sh
And running these three depend very much on the state of my DSL at the time.
For a video clip that is 17 minutes long and a 138Mb mp4 file.
ffmpeg1.sh 2m0.847s
ffmpeg2.sh 11m46.734s
ffmpeg3.sh 0m29.780s
Clearly it's not efficient to do one screenshot at a time.
Because with ffmpegthumbnailer you can tell it not to reduce the picture quality the total weight of the produced JPGs from ffmpeg1.sh was 784Kb and the total weight from ffmpeg3.sh was 1.5Mb.
Just to try again, I ran a similar experiment with a 35 minutes long and 890Mb mp4 file. And this time I didn't bother with ffmpeg2.sh. The results were:
ffmpeg1.sh 18m21.330s
ffmpeg3.sh 2m48.656s
So that means that using ffmpegthumbnailer is about 5 times faster than ffmpeg. Huge difference!
And now, a curveball!
The reason for doing ffmpeg -i https://... was so that we don't have to first download the whole beast and run the command on a local file. However, in light of how so much longer this takes and my disdain to have to install and depend on a new tool (ffmpegthumbnailer) across all servers. Why not download the whole file and run the ffmpeg command locally.
So I download the file and it's slow because of my, currently, terrible home DSL. Then I run and time them again but just a local file instead:
ffmpeg1.sh 0m20.426s
ffmpeg3.sh 0m0.635s
Did you see that!? That's an insane difference. Clearly doing this command over HTTP(S) is a bad idea. It'll be worth downloading it first.
UPDATE
On Stackoverflow, LordNeckBeard gave a great tip of using the -ss option before in the input file and now it's much faster. At this point. I'm no longer interested in having to bother with ffmpegthumbnailer.
Let's fork ffmpeg2.sh into two versions.
ffmpeg2.1.sh same as ffmpeg2.sh but a downloaded file instead of a remote HTTPS URL.
ffmpeg2.2.sh as ffmpeg2.1.sh except we put the -ss HH:MM:SS before the input file.
Now, let's run them again on the 138Mb file:
# the 138Mb mp4.mp4 file
ffmpeg2.1.sh 2m10.898s
ffmpeg2.2.sh 0m0.672s
187 times faster
And again, I re-ran this again against a bigger file that is 1.4Gb:
# the 1.4Gb mp4-1.44Gb.mp4 file
ffmpeg2.1.sh 10m1.143s
ffmpeg2.2.sh 0m1.428s
420 times faster
