27 July 2021 0 comments Python
We use Poetry in a GitHub project. There's a pyproject.toml file (and a poetry.lock file) which with the help of the executable poetry gets you a very reliable Python environment. The only problem is that adding the poetry
executable is slow. Like 10+ seconds slow. It might seem silly but in the project I'm working on, that 10+s delay is the slowest part of a GitHub Action workflow which needs to be fast because it's trying to post a comment on a pull request as soon as it possibly can.
First I tried caching $(pip cache dir) so that the underlying python -v pip install virtualenv -t $tmp_dir that install-poetry.py does would get a boost from avoiding network. The difference was negligible. I also didn't want to get too weird by overriding how the install-poetry.py works or even make my own hacky copy. I like being able to just rely on the snok/install-poetry
GitHub Action to do its thing (and its future thing).
The solution was to cache the whole $HOME/.local directory. It's as simple as this:
- name: Load cached $HOME/.local
uses: actions/cache@v2.1.6
with:
path: ~/.local
key: dotlocal-${{ runner.os }}-${{ hashFiles('.github/workflows/pr-deployer.yml') }}
The key is important. If you do copy-n-paste this block of YAML to speed up your GitHub Action, please remember to replace
.github/workflows/pr-deployer.yml with the name of your .yml file that uses this. It's important because otherwise, the cache might be overzealously hot when you make a change like:
- name: Install Python poetry
- uses: snok/install-poetry@v1.1.6
+ uses: snok/install-poetry@v1.1.7
with:
...for example.
Now, thankfully install-poetry.py (which is the recommended way to install poetry by the way) can notice that it's already been created and so it can omit a bunch of work. The result of this is as follows:
From 10+ seconds to 2 seconds
. And what's neat is that the optimization is very "unintrusive" because it doesn't mess with how the snok/install-poetry workflow works.
But wait, there's more!
If you dig up our code where we use poetry you might find that it does a bunch of other caching too. In particular, it caches .venv it creates too. That's relevant but ultimately unrelated. It basically caches the generated virtualenv from the poetry install command. It works like this:
- name: Load cached venv
id: cached-poetry-dependencies
uses: actions/cache@v2.1.6
with:
path: deployer/.venv
key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}-${{ hashFiles('.github/workflows/pr-deployer.yml') }}
...
- name: Install deployer
run: |
cd deployer
poetry install
if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
In this example, deployer is just the name of the directory, in the repository root, where we have all the Python code and the pyproject.toml etc. If you have yours at the root of the project you can just do: run: poetry install and in the caching step change it to: path: .venv.
Now, you get a really powerful complete caching strategy. When the caches are hot (i.e. no changes to the .yml, poetry.lock, or pyproject.toml files) you get the executable (so you can do poetry run ...) and all its dependencies in roughly 2 seconds. That'll be hard to beat!
23 June 2021 7 comments Python
tl;dr; To make 2 lists into 1 without mutating them use list1 + list2.
I'm blogging about this because today I accidentally complicated my own code. From now on, let's just focus on the right way.
Suppose you have something like this:
winners = [123, 503, 1001]
losers = [45, 812, 332]
combined = winners + losers
that will create a brand new list. To prove that it's immutable:
>>> combined.insert(0, 100)
>>> combined
[100, 123, 503, 1001, 45, 812, 332]
>>> winners
[123, 503, 1001]
>>> losers
[45, 812, 332]
What I originally did was:
winners = [123, 503, 1001]
losers = [45, 812, 332]
combined = [*winners, *losers]
This works the same and that syntax feels very JavaScript'y. E.g.
> var winners = [123, 503, 1001]
[ 123, 503, 1001 ]
> var losers = [45, 812, 332]
[ 45, 812, 332 ]
> var combined = [...winners, ...losers]
[ 123, 503, 1001, 45, 812, 332 ]
> combined.pop()
332
> losers
[ 45, 812, 332 ]
By the way, if you want to filter out duplicates, do this:
>>> a = [1, 2, 3]
>>> b = [2, 3, 4]
>>> list(dict.fromkeys(a + b))
[1, 2, 3, 4]
It's the most performant way to do it if the order is important.
And if you don't care about the order you can use this:
>>> a = [1, 2, 3]
>>> b = [2, 3, 4]
>>> list(set(a + b))
[1, 2, 3, 4]
>>> list(set(b + a))
[1, 2, 3, 4]
09 June 2021 0 comments Web development, MDN
tl;dr; git clone https://github.com/mdn/content.git && cd content && yarn install && yarn start && open http://localhost:5000/
will get you all of MDN Web Docs running on your laptop.
The MDN Web Docs is built from a git repository: github.com/mdn/content. It contains all you need to get all the content running locally. Including search. Embedded inside that repository is a package.json which helps you start a Yari server. Aka. the preview server. It's a static build of the github.com/mdn/yari project which handles client-side rendering, search, an just-in-time server-side rendering server.
Basics
All you need is the following:
▶ git clone https://github.com/mdn/content.git
▶ cd content
▶ yarn install
▶ yarn start
And now open http://localhost:5000 in your browser.
This will now run in "preview server" mode. It's meant for contributors (and core writers) to use when they're working on a git branch. Because of that, you'll see a "Writer's homepage" at the root URL. And when viewing each document, you get buttons about "flaws" and stuff. Looks like this:
Alternative ways to download
If you don't want to use git clone you can download the ZIP file. For example:
▶ wget https://github.com/mdn/content/archive/refs/heads/main.zip
▶ unzip main.zip
▶ cd content-main
▶ yarn install
▶ yarn start
At the time of writing, the downloaded Zip file is 86MB and unzipped the directory is 278MB on disk.
When you use git clone, by default it will download all the
git history. That can actually be useful. This way, when rendering each document, it can figure out from the git logs when each individual document was last modified. For example:
If you don't care about the "Last modified" date, you can do a "shallow git clone" instead. Replace the above-mentioned first command with:
▶ git clone --depth 1 https://github.com/mdn/content.git
At the time of writing the shallow cloned content folder becomes 234MB instead of (the deep clone) 302MB.
Just the raw rendered data
Every MDN Web Docs page has an index.json equivalent. Take any MDN page and add /index.json to the URL. For example /en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/slice/index.json
Essentially, this is the intermediate state that's used for server-side rendering the page. A glorified way of sandwiching the content in a header, a footer, and a sidebar to the side. These URLs work on localhost:5000 too. Try http://localhost:5000/en-US/docs/Web/API/Fetch_API/Using_Fetch/index.json for example.
The content for that index.json is built just in time. It also contains a bunch of extra metadata about "flaws"; a system used to highlight things that should be fixed that is somewhat easy to automate. So, it doesn't contain things like spelling mistakes or code snippets that are actually invalid.
But suppose you want all that raw (rendered) data, without any of the flaw detections, you can run this command:
▶ BUILD_FLAW_LEVELS="*:ignore" yarn build
It'll take a while (because it produces an index.html file too). But now you have all the index.json files for everything in the newly created ./build/ directory. It should have created a lot of files:
▶ find build -name index.json | wc -l
11649
If you just want a subtree of files you could have run it like this instead:
▶ BUILD_FOLDERSEARCH=web/javascript BUILD_FLAW_LEVELS="*:ignore" yarn build
Programmatic API access
The programmatic APIs are all about finding the source files. But you can use the sources to turn that into the built files you might need. Or just to get a list of URLs. To get started, create a file called find-files.js in the root:
const { Document } = require("@mdn/yari/content");
console.log(Document.findAll().count);
Now, run it like this:
▶ export CONTENT_ROOT=files
▶ node find-files.js
11649
Other things you can do with that findAll function:
const { Document } = require("@mdn/yari/content");
const found = Document.findAll({
folderSearch: "web/javascript/reference/statements/f",
});
for (const document of found.iter()) {
console.log(document.url);
}
Or, suppose you want to actually build each of these that you find:
const { Document } = require("@mdn/yari/content");
const { buildDocument } = require("@mdn/yari/build");
const found = Document.findAll({
folderSearch: "web/javascript/reference/statements/f",
});
Promise.all([...found.iter()].map((document) => buildDocument(document))).then(
(built) => {
for (const { doc } of built) {
console.log(doc.title.padEnd(20), doc.popularity);
}
}
);
That'll output something like this:
▶ node find-files.js
for 0.0143
for await...of 0.0129
for...in 0.0748
for...of 0.0531
function declaration 0.0088
function* 0.0122
All the HTML content in production-grade mode
In the most basic form, it will start the "preview server" which is tailored towards building just in time and has all those buttons at the top for writers/contributors. If you want the more "production-grade" version, you can't use the copy of @mdn/yari that is "included" in the mdn/content repo. To do this, you need to git clone mdn/yari and install that. Hang on, this is about to get a bit more advanced:
▶ git clone https://github.com/mdn/yari.git
▶ cd yari
▶ yarn install
▶ yarn build:client
▶ yarn build:ssr
▶ CONTENT_ROOT=../files REACT_APP_DISABLE_AUTH=true BUILD_FLAW_LEVELS="*:ignore" yarn build
▶ CONTENT_ROOT=../files node server/static.js
Now, if you go to something like http://localhost:5000/en-US/docs/Web/Guide/ you'll get the same thing as you get on https://developer.mozilla.org but all on your laptop. Should be pretty snappy.
Is it really entirely offline?
No, it leaks a little. For example, there are interactive examples that uses an iframe that's hardcoded to https://interactive-examples.mdn.mozilla.net/.
There are also external images for example. You might get a live sample that refers to sample images on https://mdn.mozillademos.org/files/.... So that'll fail if you're without WiFi in a spaceship.
Conclusion
Making all of MDN Web Docs available offline is, honestly, not a priority. The focus is on A) a secure production build, and B) a good environment for previewing content changes. But all the pieces are there. Search is a little bit tricky, as an example. When you're running it as a preview server you can't do a full-text search on all the content, but you get a useful autocomplete search widget for navigating between different titles. And the full-text search engine is a remote centralized server that you can't take with you offline.
But all the pieces are there. Somehow. It all depends on your use case and what you're willing to "compromise" on.
14 May 2021 0 comments Python
, MDN, Elasticsearch
This is how MDN Web Docs uses Elasticsearch. Daily, we build all the content and then upload it all using elasticsearch-dsl using aliases. Because there are no good complete guides to do this, I thought I'd write it down for the next person who needs to do something similar. Let's jump straight into the code. The reader will need a healthy dose of imagination to fill in their details.
Indexing
# models.py
from datetime.datetime import utcnow
from elasticsearch_dsl import Document
PREFIX = "myprefix"
class MyDocument(Document):
title = Text()
body = Text()
# ...
class Index:
name = (
f'{PREFIX}_{utcnow().strftime("%Y%m%d%H%M%S")}'
)
What's important to note here is that the MyDocument.Index.name is dynamically allocated every single time the module is imported. It's not very important exactly what it is called but it's important that it becomes unique each time.
This means that when you start using MyDocument it will automatically figure out which index to use. Now, it's time to create the index and bulk publish it.
# index.py
# Note! This example code skips over things like progress bars
# and verbose logging and misc sanity checks and stuff.
from elasticsearch.helpers import parallel_bulk
from elasticsearch_dsl import Index
from elasticsearch_dsl.connections import connections
from .models import MyDocument, PREFIX
def index(buildroot: Path, url: str, update=False):
"""
* 'buildroot' is where the files are we're going to read and index
* 'url' is the host URL for the Elasticsearch server
* 'update' is if just want to "cake on" a couple of documents
instead of starting over and doing a complete indexing.
"""
# Connect and stuff
connections.create_connection(hosts=[url], retry_on_timeout=True)
connection = connections.get_connection()
health = connection.cluster.health()
status = health["status"]
if status not in ("green", "yellow"):
raise Exception(f"status {status} not green or yellow")
if update:
for name in connection.indices.get_alias():
if name.startswith(f"{PREFIX}_"):
document_index = Index(name)
break
else:
raise IndexAliasError(
f"Unable to find an index called {PREFIX}_*"
)
else:
# Confusingly, `._index` is actually not a private API.
# It's the documented way you're supposed to reach it.
document_index = MyDocument._index
document_index.create()
def generator():
for doc in Path(buildroot):
# The reason for specifying the exact index name is that we might
# be doing an update and if you don't specify it, elasticsearch_dsl
# will fall back to using whatever Document._meta.Index automatically
# becomes in this moment.
yield to_search(doc, _index=document_index._name).to_dict(True)
for success, info in parallel_bulk(connection, generator()):
# 'success' is a boolean
# 'info' has stuff like:
# - info["index"]["error"]
# - info["index"]["_shards"]["successful"]
# - info["index"]["_shards"]["failed"]
pass
if update:
# When you do an update, Elasticsearch will internally delete the
# previous docs (based on the _id primary key we set).
# Normally, Elasticsearch will do this when you restart the cluster
# but that's not something we usually do.
# See https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html
document_index.forcemerge()
else:
# Now we're going to bundle the change to set the alias to point
# to the new index and delete all old indexes.
# The reason for doing this together in one update is to make it atomic.
alias_updates = [
{"add": {"index": document_index._name, "alias": PREFIX}}
]
for index_name in connection.indices.get_alias():
if index_name.startswith(f"{PREFIX}_"):
if index_name != document_index._name:
alias_updates.append({"remove_index": {"index": index_name}})
connection.indices.update_aliases({"actions": alias_updates})
print("All done!")
def to_search(file: Path, _index=None):
with open(file) as f:
data = json.load(f)
return MyDocument(
_index=_index,
_id=data["identifier"],
title=data["title"],
body=data["body"]
)
A lot is left to the reader as an exercise to fill in but these are the most important operations. It demonstrates how you can
- Correctly create indexes
- Atomically create an alias and clean up old indexes (and aliases)
- How you can add to an existing index
After you've run this you'll see something like this:
$ curl http://localhost:9200/_cat/indices?v
...
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open myprefix_20210514141421 vulVt5EKRW2MNV47j403Mw 1 1 11629 0 28.7mb 28.7mb
$ curl http://localhost:9200/_cat/aliases?v
...
alias index filter routing.index routing.search is_write_index
myprefix myprefix_20210514141421 - - - -
Searching
When it comes to using the index, well, it depends on where your code for that is. For example, on MDN Web Docs, the code that searches the index is in an entirely different code-base. It's incidentally Python (and elasticsearch-dsl) in both places but other than that they have nothing in common. So for the searching, you need to manually make sure you write down the name of the index (or name of the alias if you prefer) into the code that searches. For example:
from elasticsearch_dsl import Search
def search(params):
search_query = Search(index=settings.SEARCH_INDEX_NAME)
# Do stuff to 'search_query' based on 'params'
response = search_query.execute()
for hit in response:
# ...
If you're within the same code that has that models.MyDocument in the first example code above, you can simply do things like this:
from elasticsearch_dsl import Index
from elasticsearch_dsl.connections import connections
from .models import PREFIX
def analyze(
url: str,
text: str,
analyzer: str,
):
connections.create_connection(hosts=[url])
index = Index(PREFIX)
analysis = index.analyze(body={"text": text, "analyzer": analyzer})
# ...
07 May 2021 0 comments JavaScript, MDN
The list of stop words in Elasticsearch is:
a, an, and, are, as, at, be, but, by, for, if, in, into,
is, it, no, not, of, on, or, such, that, the, their,
then, there, these, they, this, to, was, will, with
The list of JavaScript reserved keywords is:
abstract, arguments, await, boolean, break, byte, case,
catch, char, class, const, continue, debugger, default,
delete, do, double, else, enum, eval, export, extends,
false, final, finally, float, for, function, goto, if,
implements, import, in, instanceof, int, interface, let,
long, native, new, null, package, private, protected,
public, return, short, static, super, switch, synchronized,
this, throw, throws, transient, true, try, typeof, var,
void, volatile, while, with, yield
That means that the overlap is:
for, if, in, this, with
And the remainder of the English stop words is:
a, an, and, are, as, at, be, but, by, into, is, it, no,
not, of, on, or, such, that, the, their, then, there,
these, they, to, was, will
Why does this matter? It matters when you're writing a search engine on English text that is about JavaScript. Such as, MDN Web Docs
. At the time of writing, you can search for this because there's a special case explicitly for that word. But you can't search for for which is unfortunate.
But there's more! I think we should consider certain prototype words to be considered "reserved" because they are important JavaScript words that should not be treated as stop words. For example...
23 April 2021 0 comments Web development, MDN
tl;dr; The old /favicon.ico was 15KB and due to bad caching was downloaded 24M times in the last month totaling ~350GB of server-to-client traffic which can almost all be avoided.
How to save the planet? Well, do something you can do, they say. Ok, what I can do is to reduce the amount of electricity consumed to browse the web. Mozilla MDN Web Docs, which I work on, has a lot of traffic from all over the world. In the last 30 days, we have roughly 70M pageviews across roughly 15M unique users.
A lot of these people come back to MDN more than once per month so good assets and good asset-caching matter.
I found out that somehow we had failed to optimize the /favicon.ico asset! It was 15,086 bytes when, with Optimage, I was quickly able to turn it down to 1,153 bytes. That's a 13x improvement! Here's what that looks like when zoomed in 4x:
The next challenge was the Cache-Control
. Our CDN is AWS Cloudfront and it respects whatever Cache-Control headers we set on the assets. Because favicon.ico doesn't have a unique hash in its name, the Cache-Control falls back to the default of 24 hours (max-age=86400) which isn't much. Especially for an asset that almost never changes and besides, if we do decide to change the image (but not the name) we'd have to wait a minimum of 24 hours until it's fully rolled out.
Another thing I did as part of this was to stop assuming the default URL of /favicon.ico and instead control it with the <link rel="shortcut icon" href="/favicon.323ad90c.ico" type="image/x-icon"> HTML meta tag. Now I can control the URL of the image that will be downloaded.
Our client-side code is based on create-react-app and it can't optimize the files in the client/public/ directory.
So I
wrote a script that post-processes the files in client/build/. In particular, it looks through the index.html template and replaces...
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
...with...
<link rel="shortcut icon" href="/favicon.323ad90c.ico" type="image/x-icon">
Plus it makes a copy of the file with this hash in it so that the old URL still resolves. But now can cache it much more aggressively. 1 year in fact.
In summary
Combined, we used to have ~350GB worth of data sent from our CDN(s) to people's browsers every month.
Just changing the image itself would turn that number to ~25GB instead.
The new Cache-Control hopefully means that all those returning users can skip the download on a daily basis which will reduce the amount of network usage even more, but it's hard to predict in advance.
