22 March 2021 0 comments Python, MacOSX
I edit a file called files/en-us/glossary/bézier_curve/index.html and then type git status and I get this:
▶ git status
...
Changes not staged for commit:
...
modified: "files/en-us/glossary/b\303\251zier_curve/index.html"
...What's that?! First of all, I actually had this wrapped in a Python script that uses GitPython to analyze the output of for change in repo.index.diff(None):. So I got...
FileNotFoundError: [Errno 2] No such file or directory: '"files/en-us/glossary/b\\303\\251zier_curve/index.html"'
What's that?!
At first, I thought it was something wrong with how I use GitPython and thought I could force some sort of conversion to UTF-8 with Python. That, and to strip the quotation parts with something like path = path[1:-1] if path.startwith('"') else path
After much googling and experimentation, what totally solved all my problems was to run:
▶ git config --global core.quotePath false
Now you get...:
▶ git status
...
Changes not staged for commit:
...
modified: files/en-us/glossary/bézier_curve/index.html
...And that also means it works perfectly fine with any GitPython code that does something with the repo.index.diff(None) or repo.index.diff(repo.head.commit).
Also, we I use the git-diff-action GitHub Action which would fail to spot files that contained umlauts but now I run this:
steps:
- uses: actions/checkout@v2
+
+ - name: Config git core.quotePath
+ run: git config --global core.quotePath false
+
- uses: technote-space/get-diff-action@v4.0.6
id: git_diff_content
with:
26 February 2021 3 comments Web development, Django, Python, MDN
tl;dr; Periodically, the whole of MDN is built, by our Node code, in a GitHub Action. A Python script bulk-publishes this to Elasticsearch. Our Django server queries the same Elasticsearch via /api/v1/search. The site-search page is a static single-page app that sends XHR requests to the /api/v1/search endpoint. Search results' sort-order is determined by match and "popularity".
Jamstack'ing
The challenge with "Jamstack" websites is with data that is too vast and dynamic that it doesn't make sense to build statically. Search is one of those. For the record, as of Feb 2021, MDN consists of 11,619 documents (aka. articles) in English. Roughly another 40,000 translated documents. In English alone, there are 5.3 million words. So to build a good search experience we need to, as a static site build side-effect, index all of this in a full-text search database. And Elasticsearch is one such database and it's good. In particular, Elasticsearch is something MDN is already quite familiar with because it's what was used from within the Django app when MDN was a wiki.
Note: MDN gets about 20k site-searches per day from within the site.
Build
When we build the whole site, it's a script that basically loops over all the raw content, applies macros and fixes, dumps one index.html (via React server-side rendering) and one index.json. The index.json contains all the fully rendered text (as HTML!) in blocks of "prose". It looks something like this:
{
"doc": {
"title": "DOCUMENT TITLE",
"summary": "DOCUMENT SUMMARY",
"body": [
{
"type": "prose",
"value": {
"id": "introduction",
"title": "INTRODUCTION",
"content": "<p>FIRST BLOCK OF TEXTS</p>"
}
},
...
],
"popularity": 0.12345,
...
}
You can see one here: /en-US/docs/Web/index.json
Indexing
Next, after all the index.json files have been produced, a Python script takes over and it traverses all the index.json files and based on that structure it figures out the, title, summary, and the whole body (as HTML).
Next up, before sending this into the bulk-publisher in Elasticsearch it strips the HTML. It's a bit more than just turning <p>Some <em>cool</em> text.</p> to Some cool text. because it also cleans up things like <div class="hidden"> and certain <div class="notecard warning"> blocks.
One thing worth noting is that this whole thing runs roughly every 24 hours and then it builds everything. But what if, between two runs, a certain page has been removed (or moved), how do you remove what was previously added to Elasticsearch? The solution is simple: it deletes and re-creates the index from scratch every day. The whole bulk-publish takes a while so right after the index has been deleted, the searches won't be that great. Someone could be unlucky in that they're searching MDN a couple of seconds after the index was deleted and now waiting for it to build up again.
It's an unfortunate reality but it's a risk worth taking for the sake of simplicity. Also, most people are searching for things in English and specifically the Web/ tree so the bulk-publishing is done in a way the most popular content is bulk-published first and the rest was done after. Here's what the build output logs:
Found 50,461 (potential) documents to index
Deleting any possible existing index and creating a new one called mdn_docs
Took 3m 35s to index 50,362 documents. Approximately 234.1 docs/second
Counts per priority prefixes:
en-us/docs/web 9,056
*rest* 41,306So, yes, for 3m 35s there's stuff missing from the index and some unlucky few will get fewer search results than they should. But we can optimize this in the future.
Searching
The way you connect to Elasticsearch is simply by a URL it looks something like this:
https://USER:PASSWD@HASH.us-west-2.aws.found.io:9243
It's an Elasticsearch cluster managed by Elastic running inside AWS. Our job is to make sure that we put the exact same URL in our GitHub Action ("the writer") as we put it into our Django server ("the reader").
In fact, we have 3 Elastic clusters: Prod, Stage, Dev.
And we have 2 Django servers: Prod, Stage.
So we just need to carefully make sure the secrets are set correctly to match the right environment.
Now, in the Django server, we just need to convert a request like GET /api/v1/search?q=foo&locale=fr (for example) to a query to send to Elasticsearch. We have a simple Django view function that validates the query string parameters, does some rate-limiting, creates a query (using elasticsearch-dsl) and packages the Elasticsearch results back to JSON.
How we make that query is important. In here lies the most important feature of the search; how it sorts results.
In one simple explanation, the sort order is a combination of popularity and "matchness". The assumption is that most people want the popular content. I.e. they search for foreach and mean to go to /en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach not /en-US/docs/Web/API/NodeList/forEach both of which contains forEach in the title. The "popularity" is based on Google Analytics pageviews which we download periodically, normalize into a floating-point number between 1 and 0. At the of writing the scoring function does something like this:
rank = doc.popularity * 10 + search.score
This seems to produce pretty reasonable results.
But there's more to the "matchness" too. Elasticsearch has its own API for defining boosting and the way we apply is:
- match phrase in the
title: Boost = 10.0 - match phrase in the
body: Boost = 5.0 - match in
title: Boost = 2.0 -
match in
body: Boost = 1.0
This is then applied on top of whatever else Elasticsearch does such as "Term Frequency" and "Inverse Document Frequency" (tf and if). This article is a helpful introduction.
We're most likely not done with this. There's probably a lot more we can do to tune this myriad of knobs and sliders to get the best possible ranking of documents that match.
Web UI
The last piece of the puzzle is how we display all of this to the user. The way it works is that developer.mozilla.org/$locale/search returns a static page that is blank. As soon as the page has loaded, it lazy-loads JavaScript that can actually issue the XHR request to get and display search results. The code looks something like this:
function SearchResults() {
const [searchParams] = useSearchParams();
const sp = createSearchParams(searchParams);
// add defaults and stuff here
const fetchURL = `/api/v1/search?${sp.toString()}`;
const { data, error } = useSWR(
fetchURL,
async (url) => {
const response = await fetch(URL);
// various checks on the response.statusCode here
return await response.json();
}
);
// render 'data' or 'error' accordingly here
A lot of interesting details are omitted from this code snippet. You have to check it out for yourself to get a more up-to-date insight into how it actually works. But basically, the window.location (and pushState) query string drives the fetch() call and then all the component has to do is display the search results with some highlighting.
The /api/v1/search endpoint also runs a suggestion query as part of the main search query. This extracts out interest alternative search queries. These are filtered and scored and we issue "sub-queries" just to get a count for each. Now we can do one of those "Did you mean...". For example: search for
intersections.
In conclusion
There are a lot of interesting, important, and careful details that are glossed over here in this blog post. It's a constantly evolving system and we're constantly trying to improve and perfect the system in a way that it fits what users expect.
A lot of people reach MDN via a Google search (e.g. mdn array foreach) but despite that, nearly 5% of all traffic on MDN is the site-search functionality. The /$locale/search?... endpoint is the most frequently viewed page of all of MDN. And having a good search engine that's reliable is nevertheless important. By owning and controlling the whole pipeline allows us to do specific things that are unique to MDN that other websites don't need. For example, we index a lot of raw HTML (e.g. <video>) and we have code snippets that needs to be searchable.
Hopefully, the MDN site-search will elevate from being known to be very limited to something now that can genuinely help people get to the exact page better than Google can. Yes, it's worth aiming high!
08 January 2021 2 comments Python
tl;dr; selectolax is best for stripping HTML down to plain text.
The problem is that I have 10,000+ HTML snippets that I need to index into Elasticsearch as plain text. (Before you ask, yes I know Elasticsearch has a html_strip text filter but it's not what I want/need to use in this context).
Turns out, stripping the HTML into plain text was actually quite expensive at that scale. So what's the most performant way?
PyQuery
from pyquery import PyQuery as pq
text = pq(html).text()
selectolax
from selectolax.parser import HTMLParser
text = HTMLParser(html).text()
regular expression
import re
regex = re.compile(r'<.*?>')
text = clean_regex.sub('', html)
Results
I wrote a script that iterated through 10,000 files that contains HTML snippets. Note! The snippets aren't complete <html> documents (with a <head> and <body> etc) Just blobs of HTML. The average size is 10,314 bytes (5,138 bytes median).
pyquery
SUM: 18.61 seconds
MEAN: 1.8633 ms
MEDIAN: 1.0554 ms
selectolax
SUM: 3.08 seconds
MEAN: 0.3149 ms
MEDIAN: 0.1621 ms
regex
SUM: 1.64 seconds
MEAN: 0.1613 ms
MEDIAN: 0.0881 ms
I've run it a bunch of times. The results are pretty stable.
Point is: selectolax is ~7 times faster than PyQuery
Regex? Really?
No, I don't think I want to use that. It makes me nervous without even attempting to dig up some examples where it goes wrong. It might work just fine for the most basic blobs of HTML. Actually, if the HTML is <p>Foo & Bar</p>, I expect the plain text transformation should be Foo & Bar, not Foo & Bar.
More pressing, both PyQuery and selectolax
supports something very specific but important to my use case. I need to remove certain tags (and its content) before I proceed. For example:
<h4 class="warning">This should get stripped.</h4>
<p>Please keep.</p>
<div style="display: none">This should also get stripped.</div>
That can never be done with a regex.
Version 2.0
So my requirement will probably change but basically, I want to delete certain tags. E.g. <div class="warning"> and <div class="hidden"> and <div style="display: none">. So let's implement that:
PyQuery
from pyquery import PyQuery as pq
_display_none_regex = re.compile(r'display:\s*none')
doc = pq(html)
doc.remove('div.warning, div.hidden')
for div in doc('div[style]').items():
style_value = div.attr('style')
if _display_none_regex.search(style_value):
div.remove()
text = doc.text()
selectolax
from selectolax.parser import HTMLParser
_display_none_regex = re.compile(r'display:\s*none')
tree = HTMLParser(html)
for tag in tree.css('div.warning, div.hidden'):
tag.decompose()
for tag in tree.css('div[style]'):
style_value = tag.attributes['style']
if style_value and _display_none_regex.search(style_value):
tag.decompose()
text = tree.body.text()
This actually works. When I now run the same benchmark for 10,000 of these are the new results:
pyquery
SUM: 21.70 seconds
MEAN: 2.1701 ms
MEDIAN: 1.3989 ms
selectolax
SUM: 3.59 seconds
MEAN: 0.3589 ms
MEDIAN: 0.2184 ms
regex
Skip
Again, selectolax beats PyQuery by a factor of ~6.
Conclusion
Regular expressions are fast but weak in power. Makes sense.
This selectolax is very impressive.
I got the inspiration from this blog post which sets out to do something very similar to what I'm doing.
I hope this helps someone. Thank you Artem Golubin of selectolax and @lexborisov for Modest which
selectolax is built upon.
21 December 2020 1 comment Python
I love git on the command line and I actually never use a GUI to navigate git branches. But sometimes, I need scripting to make abstractions that make life more convenient. What often happens is that I need to go back to the "main" branch. I write main in quotation marks because it's not always called main. Sometimes it's called master. And it's tedious to have to remember which one is the default. So I wrote a script called Gcm.
#!/usr/bin/env python
import subprocess
def run(*args):
default_branch = get_default_branch()
current_branch = get_current_branch()
if default_branch != current_branch:
checkout_branch(default_branch)
else:
print(f"Already on {default_branch}")
return 1
def checkout_branch(branch_name):
subprocess.run(f"git checkout {branch_name}".split())
def get_default_branch():
origin_name = "origin"
res = subprocess.run(
f"git remote show {origin_name}".split(), check=True, capture_output=True,
)
for line in res.stdout.decode("utf-8").splitlines():
if line.strip().startswith("HEAD branch:"):
return line.replace("HEAD branch:", "").strip()
raise NotImplementedError(f"No remote called {origin_name!r}")
def get_current_branch():
res = subprocess.run("git branch --show-current".split(), capture_output=True)
for line in res.stdout.decode("utf-8").splitlines():
return line.strip()
raise NotImplementedError("Don't know what to do!")
if __name__ == "__main__":
import sys
sys.exit(run(*sys.argv[1:]))
It ain't pretty or a spiffy one-liner, but it works. It assumes that the repo has a remote called origin which doesn't matter if it's the upstream or your fork. Put this script into a file called ~/bin/Gcm and run chmox +x ~/bin/Gcm.
Now, whenever I want to go back to the main branch I type Gcm and it takes me there.
It might seem silly, and it might not be for you, but I love it and use it many times per day. Perhaps by sharing this tip, it'll inspire someone else to set up something similar for themselves.
Why it's spelled with an uppercase G
I have a pattern (or rule?) that all scripts that I write myself are always capitalized like that. It avoids clashes with stuff I install with brew or other bash/zsh aliases.
For example:
ls -l ~/bin/RemoteVSCodePeterbecom.sh
ls -l ~/bin/Cleanupfiles
ls -l ~/bin/RandomString.py
28 October 2020 1 comment Web development, Django, Python
tl;dr; <img src="/avatar.random.png" alt="Random avataaar"> generates this image:
(try reloading to get a random new one. funny aren't they?)
When you use Gravatar you can convert people's email addresses to their mugshot.
It works like this:
<img src="https://www.gravatar.com/avatar/$(md5(user.email))">
But most people don't have their mugshot on Gravatar.com unfortunately. But you still want to display an avatar that is distinct per user. Your best option is to generate one and just use the user's name or email as a seed (so it's always random but always deterministic for the same user). And you can also supply a fallback image to Gravatar that they use if the email doesn't match any email they have. That's where this blog post comes in.
I needed that so I shopped around and found avataaars generator which is available as a React component. But I need it to be server-side and in Python. And thankfully there's a great port called: py-avataaars.
It depends on CairoSVG to convert an SVG to a PNG but it's easy to install. Anyway, here's my hack to generate random "avataaars" from Django:
import io
import random
import py_avataaars
from django import http
from django.utils.cache import add_never_cache_headers, patch_cache_control
def avatar_image(request, seed=None):
if not seed:
seed = request.GET.get("seed") or "random"
if seed != "random":
random.seed(seed)
bytes = io.BytesIO()
def r(enum_):
return random.choice(list(enum_))
avatar = py_avataaars.PyAvataaar(
style=py_avataaars.AvatarStyle.CIRCLE,
# style=py_avataaars.AvatarStyle.TRANSPARENT,
skin_color=r(py_avataaars.SkinColor),
hair_color=r(py_avataaars.HairColor),
facial_hair_type=r(py_avataaars.FacialHairType),
facial_hair_color=r(py_avataaars.FacialHairColor),
top_type=r(py_avataaars.TopType),
hat_color=r(py_avataaars.ClotheColor),
mouth_type=r(py_avataaars.MouthType),
eye_type=r(py_avataaars.EyesType),
eyebrow_type=r(py_avataaars.EyebrowType),
nose_type=r(py_avataaars.NoseType),
accessories_type=r(py_avataaars.AccessoriesType),
clothe_type=r(py_avataaars.ClotheType),
clothe_color=r(py_avataaars.ClotheColor),
clothe_graphic_type=r(py_avataaars.ClotheGraphicType),
)
avatar.render_png_file(bytes)
response = http.HttpResponse(bytes.getvalue())
response["content-type"] = "image/png"
if seed == "random":
add_never_cache_headers(response)
else:
patch_cache_control(response, max_age=60, public=True)
return response
It's not perfect but it works. The URL to this endpoint is /avatar.<seed>.png and if you make the seed parameter random the response is always different.
To make the image not random, you replace the <seed> with any string. For example (use your imagination):
{% for comment in comments %}
<img src="/avatar.{{ comment.user.id }}.png" alt="{{ comment.user.name }}">
<blockquote>{{ comment.text }}</blockquote>
<i>{{ comment.date }}</i>
{% endfor %}
I've put together this test page if you want to see more funny avatar combinations instead of doing work :)
15 June 2020 0 comments Python
tl;dr hashin 0.15.0 makes package comparison agnostic to underscore or hyphens
See issue #116 for a fuller story. Basically, now it doesn't matter if you write...
hashin python_memcached
...or...
hashin python-memcached
And the same can be said about the contents of your requirements.txt file. Suppose
it already had something like this:
python_memcached==1.59 \
--hash=sha256:4dac64916871bd35502 \
--hash=sha256:a2e28637be13ee0bf1a8and you type hashin python-memcached it will do the version comparison on these independent of the underscore or hyphen.
Thank @caphrim007 who implemented this for the benefit of Renovate.
