14 May 2021 0 comments Python, MDN, Elasticsearch
This is how MDN Web Docs uses Elasticsearch. Daily, we build all the content and then upload it all using elasticsearch-dsl
using aliases. Because there are no good complete guides to do this, I thought I'd write it down for the next person who needs to do something similar. Let's jump straight into the code. The reader will need a healthy dose of imagination to fill in their details.
Indexing
# models.py
from datetime.datetime import utcnow
from elasticsearch_dsl import Document
PREFIX = "myprefix"
class MyDocument(Document):
title = Text()
body = Text()
# ...
class Index:
name = (
f'{PREFIX}_{utcnow().strftime("%Y%m%d%H%M%S")}'
)
What's important to note here is that the MyDocument.Index.name is dynamically allocated every single time the module is imported. It's not very important exactly what it is called but it's important that it becomes unique each time.
This means that when you start using MyDocument it will automatically figure out which index to use. Now, it's time to create the index and bulk publish it.
# index.py
# Note! This example code skips over things like progress bars
# and verbose logging and misc sanity checks and stuff.
from elasticsearch.helpers import parallel_bulk
from elasticsearch_dsl import Index
from elasticsearch_dsl.connections import connections
from .models import MyDocument, PREFIX
def index(buildroot: Path, url: str, update=False):
"""
* 'buildroot' is where the files are we're going to read and index
* 'url' is the host URL for the Elasticsearch server
* 'update' is if just want to "cake on" a couple of documents
instead of starting over and doing a complete indexing.
"""
# Connect and stuff
connections.create_connection(hosts=[url], retry_on_timeout=True)
connection = connections.get_connection()
health = connection.cluster.health()
status = health["status"]
if status not in ("green", "yellow"):
raise Exception(f"status {status} not green or yellow")
if update:
for name in connection.indices.get_alias():
if name.startswith(f"{PREFIX}_"):
document_index = Index(name)
break
else:
raise IndexAliasError(
f"Unable to find an index called {PREFIX}_*"
)
else:
# Confusingly, `._index` is actually not a private API.
# It's the documented way you're supposed to reach it.
document_index = MyDocument._index
document_index.create()
def generator():
for doc in Path(buildroot):
# The reason for specifying the exact index name is that we might
# be doing an update and if you don't specify it, elasticsearch_dsl
# will fall back to using whatever Document._meta.Index automatically
# becomes in this moment.
yield to_search(doc, _index=document_index._name).to_dict(True)
for success, info in parallel_bulk(connection, generator()):
# 'success' is a boolean
# 'info' has stuff like:
# - info["index"]["error"]
# - info["index"]["_shards"]["successful"]
# - info["index"]["_shards"]["failed"]
pass
if update:
# When you do an update, Elasticsearch will internally delete the
# previous docs (based on the _id primary key we set).
# Normally, Elasticsearch will do this when you restart the cluster
# but that's not something we usually do.
# See https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html
document_index.forcemerge()
else:
# Now we're going to bundle the change to set the alias to point
# to the new index and delete all old indexes.
# The reason for doing this together in one update is to make it atomic.
alias_updates = [
{"add": {"index": document_index._name, "alias": PREFIX}}
]
for index_name in connection.indices.get_alias():
if index_name.startswith(f"{PREFIX}_"):
if index_name != document_index._name:
alias_updates.append({"remove_index": {"index": index_name}})
connection.indices.update_aliases({"actions": alias_updates})
print("All done!")
def to_search(file: Path, _index=None):
with open(file) as f:
data = json.load(f)
return MyDocument(
_index=_index,
_id=data["identifier"],
title=data["title"],
body=data["body"]
)
A lot is left to the reader as an exercise to fill in but these are the most important operations. It demonstrates how you can
- Correctly create indexes
- Atomically create an alias and clean up old indexes (and aliases)
- How you can add to an existing index
After you've run this you'll see something like this:
$ curl http://localhost:9200/_cat/indices?v
...
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open myprefix_20210514141421 vulVt5EKRW2MNV47j403Mw 1 1 11629 0 28.7mb 28.7mb
$ curl http://localhost:9200/_cat/aliases?v
...
alias index filter routing.index routing.search is_write_index
myprefix myprefix_20210514141421 - - - -
Searching
When it comes to using the index, well, it depends on where your code for that is. For example, on MDN Web Docs, the code that searches the index is in an entirely different code-base. It's incidentally Python (and elasticsearch-dsl) in both places but other than that they have nothing in common. So for the searching, you need to manually make sure you write down the name of the index (or name of the alias if you prefer) into the code that searches. For example:
from elasticsearch_dsl import Search
def search(params):
search_query = Search(index=settings.SEARCH_INDEX_NAME)
# Do stuff to 'search_query' based on 'params'
response = search_query.execute()
for hit in response:
# ...
If you're within the same code that has that models.MyDocument in the first example code above, you can simply do things like this:
from elasticsearch_dsl import Index
from elasticsearch_dsl.connections import connections
from .models import PREFIX
def analyze(
url: str,
text: str,
analyzer: str,
):
connections.create_connection(hosts=[url])
index = Index(PREFIX)
analysis = index.analyze(body={"text": text, "analyzer": analyzer})
# ...
07 May 2021 0 comments JavaScript, MDN
The list of stop words in Elasticsearch is:
a, an, and, are, as, at, be, but, by, for, if, in, into,
is, it, no, not, of, on, or, such, that, the, their,
then, there, these, they, this, to, was, will, with
The list of JavaScript reserved keywords is:
abstract, arguments, await, boolean, break, byte, case,
catch, char, class, const, continue, debugger, default,
delete, do, double, else, enum, eval, export, extends,
false, final, finally, float, for, function, goto, if,
implements, import, in, instanceof, int, interface, let,
long, native, new, null, package, private, protected,
public, return, short, static, super, switch, synchronized,
this, throw, throws, transient, true, try, typeof, var,
void, volatile, while, with, yield
That means that the overlap is:
for, if, in, this, with
And the remainder of the English stop words is:
a, an, and, are, as, at, be, but, by, into, is, it, no,
not, of, on, or, such, that, the, their, then, there,
these, they, to, was, will
Why does this matter? It matters when you're writing a search engine on English text that is about JavaScript. Such as, MDN Web Docs
. At the time of writing, you can search for this because there's a special case explicitly for that word. But you can't search for for which is unfortunate.
But there's more! I think we should consider certain prototype words to be considered "reserved" because they are important JavaScript words that should not be treated as stop words. For example...
23 April 2021 0 comments Web development, MDN
tl;dr; The old /favicon.ico was 15KB and due to bad caching was downloaded 24M times in the last month totaling ~350GB of server-to-client traffic which can almost all be avoided.
How to save the planet? Well, do something you can do, they say. Ok, what I can do is to reduce the amount of electricity consumed to browse the web. Mozilla MDN Web Docs, which I work on, has a lot of traffic from all over the world. In the last 30 days, we have roughly 70M pageviews across roughly 15M unique users.
A lot of these people come back to MDN more than once per month so good assets and good asset-caching matter.
I found out that somehow we had failed to optimize the /favicon.ico asset! It was 15,086 bytes when, with Optimage, I was quickly able to turn it down to 1,153 bytes. That's a 13x improvement! Here's what that looks like when zoomed in 4x:
The next challenge was the Cache-Control
. Our CDN is AWS Cloudfront and it respects whatever Cache-Control headers we set on the assets. Because favicon.ico doesn't have a unique hash in its name, the Cache-Control falls back to the default of 24 hours (max-age=86400) which isn't much. Especially for an asset that almost never changes and besides, if we do decide to change the image (but not the name) we'd have to wait a minimum of 24 hours until it's fully rolled out.
Another thing I did as part of this was to stop assuming the default URL of /favicon.ico and instead control it with the <link rel="shortcut icon" href="/favicon.323ad90c.ico" type="image/x-icon"> HTML meta tag. Now I can control the URL of the image that will be downloaded.
Our client-side code is based on create-react-app and it can't optimize the files in the client/public/ directory.
So I
wrote a script that post-processes the files in client/build/. In particular, it looks through the index.html template and replaces...
<link rel="shortcut icon" href="/favicon.ico" type="image/x-icon">
...with...
<link rel="shortcut icon" href="/favicon.323ad90c.ico" type="image/x-icon">
Plus it makes a copy of the file with this hash in it so that the old URL still resolves. But now can cache it much more aggressively. 1 year in fact.
In summary
Combined, we used to have ~350GB worth of data sent from our CDN(s) to people's browsers every month.
Just changing the image itself would turn that number to ~25GB instead.
The new Cache-Control hopefully means that all those returning users can skip the download on a daily basis which will reduce the amount of network usage even more, but it's hard to predict in advance.
25 March 2021 0 comments React, JavaScript
Suppose you have one of those React apps that lazy-load some chunk. It just basically means it injects a .js static asset URL into the DOM and once it's downloaded by the browser, it carries on the React rendering with the new code loaded. Well, what if the network is really slow? In local development, it can be hard to simulate this. You can mess with the browser's Devtools to try to slow down the network, but even that can be too fast sometimes.
What I often do is, I take this:
const SettingsApp = React.lazy(() => import("./app"));
...and change it to this:
const SettingsApp = React.lazy(() =>
import("./app").then((module) => {
return new Promise((resolve) => {
setTimeout(() => {
resolve(module as any);
}, 10000);
});
})
);
Now, it won't load that JS chunk until 10 seconds later. Only temporarily, in local development.
I know it's admittedly just a hack but it's nifty. Just don't forget to undo it when you're done simulating your snail-speed web app.
PS. That resolve(module as any); is for TypeScript. You can just change that to resolve(module); if it's regular JavaScript.
22 March 2021 0 comments Python, MacOSX
I edit a file called files/en-us/glossary/bézier_curve/index.html and then type git status and I get this:
▶ git status
...
Changes not staged for commit:
...
modified: "files/en-us/glossary/b\303\251zier_curve/index.html"
...What's that?! First of all, I actually had this wrapped in a Python script that uses GitPython to analyze the output of for change in repo.index.diff(None):. So I got...
FileNotFoundError: [Errno 2] No such file or directory: '"files/en-us/glossary/b\\303\\251zier_curve/index.html"'
What's that?!
At first, I thought it was something wrong with how I use GitPython and thought I could force some sort of conversion to UTF-8 with Python. That, and to strip the quotation parts with something like path = path[1:-1] if path.startwith('"') else path
After much googling and experimentation, what totally solved all my problems was to run:
▶ git config --global core.quotePath false
Now you get...:
▶ git status
...
Changes not staged for commit:
...
modified: files/en-us/glossary/bézier_curve/index.html
...And that also means it works perfectly fine with any GitPython code that does something with the repo.index.diff(None) or repo.index.diff(repo.head.commit).
Also, we I use the git-diff-action GitHub Action which would fail to spot files that contained umlauts but now I run this:
steps:
- uses: actions/checkout@v2
+
+ - name: Config git core.quotePath
+ run: git config --global core.quotePath false
+
- uses: technote-space/get-diff-action@v4.0.6
id: git_diff_content
with:
05 March 2021 0 comments Node, JavaScript
Sorry about the weird title of this blog post. Not sure what else to call it.
I have a function that recursively traverses the file system. You can iterate over this function to do something with each found file on disk. Silly example:
for (const filePath of walker("/lots/of/files/here")) {
count += filePath.length;
}
The implementation looks like this:
function* walker(root) {
const files = fs.readdirSync(root);
for (const name of files) {
const filepath = path.join(root, name);
const isDirectory = fs.statSync(filepath).isDirectory();
if (isDirectory) {
yield* walker(filepath);
} else {
yield filepath;
}
}
}
But I wondered; is it faster to not use a generator function since there might an overhead in swapping from the generator to whatever callback does something with each yielded thing. A pure big-array function looks like this:
function walker(root) {
const files = fs.readdirSync(root);
const all = [];
for (const name of files) {
const filepath = path.join(root, name);
const isDirectory = fs.statSync(filepath).isDirectory();
if (isDirectory) {
all.push(...walker(filepath));
} else {
all.push(filepath);
}
}
return all;
}
It gets the same result/outcome.
It's hard to measure this but I pointed it to some large directory with many files and did something silly with each one just to make sure it does something:
const label = "generator";
console.time(label);
let count = 0;
for (const filePath of walker(SEARCH_ROOT)) {
count += filePath.length;
}
console.timeEnd(label);
const heapBytes = process.memoryUsage().heapUsed;
console.log(`HEAP: ${(heapBytes / 1024.0).toFixed(1)}KB`);
I ran it a bunch of times. After a while, the numbers settle and you get:
- Generator function: (median time) 1.74s
- Big array function: (median time) 1.73s
In other words, no speed difference.
Obviously building up a massive array in memory will increase the heap memory usage. Taking a snapshot at the end of the run and printing it each time, you can see that...
- Generator function: (median heap memory) 4.9MB
- Big array function: (median heap memory) 13.9MB
Conclusion
The potential swap overhead for a Node generator function is absolutely minuscule. At least in contexts similar to mine.
It's not unexpected that the generator function bounds less heap memory because it doesn't build up a big array at all.
