A blog and website by Peter Bengtsson
25 February 2021 0 comments Node, JavaScript
tl;dr; polka is the lightest Node HTTP server package.
Highly unscientific but nevertheless worth writing down. Lightest here refers to the eventual weight added to the node_modules
directory which is a reflection of network and disk use.
When you write a serious web server in Node you probably don't care about which one is lightest. It's probably more important which ones are actively maintained, reliable, well documented, and generally "more familiar". However, I was interested in setting up a little Node HTTP server for the benefit of wrapping some HTTP endpoints for an integration test suite.
The test
In a fresh new directory, right after having run: yarn init -y run the yarn add ... and see how big the node_modules directory becomes afterward (du -sh node_modules).
The results
polka: 116Kkoa: 1.7Mexpress: 2.4M
fastify: 8.0M
Conclusion
polka is the lightest. But I'm not so sure it matters. But it could if this has to be installed a lot. For example, in CI where you run that yarn install a lot. Then it might save quite a bit of electricity for the planet.
13 February 2021 0 comments Node, JavaScript
There are a lot of 'rss' related NPM packages
but I think I've found a combination that is great for parsing RSS feeds. Something that takes up the minimal node_modules and works great. I think the killer combination is
The code impressively simple:
const got = require("got");
const parser = require("fast-xml-parser");
(async function main() {
const buffer = await got("https://hacks.mozilla.org/feed/", {
responseType: "buffer",
resolveBodyOnly: true,
timeout: 5000,
retry: 5,
});
var feed = parser.parse(buffer.toString());
for (const item of feed.rss.channel.item) {
console.log({ title: item.title, url: item.link });
break;
}
})();
// Outputs...
// {
// title: 'MDN localization update, February 2021',
// url: 'https://hacks.mozilla.org/2021/02/mdn-localization-update-february-2021/'
// }
I like about fast-xml-parser is that it has no dependencies. And it's tiny:
▶ du -sh node_modules/fast-xml-parser
104K node_modules/fast-xml-parser
The got package is quite a bit larger and has more dependencies. But I still love it. It's proven itself to be very reliable and very pleasant API. Both packages support TypeScript too.
A particular detail I like about fast-xml-parser is that it doesn't try to do the downloading part too. This way, I can use my own preferred library and I could potentially write my own caching code if I want to protect against flaky network.
03 February 2021 0 comments JavaScript
What do you think this code will print out?
function validateURL(url) {
if (url.includes("://")) {
const url = new URL(url);
return url.protocol === "https:";
} else {
return "dunno";
}
}
console.log(validateURL("http://www.peterbe.com"));
I'll give you a clue that isn't helpful,
▶ eslint --version
v7.19.0
▶ eslint code.js
▶ echo $?
0
OK, the answer is that it crashes:
▶ node code.js
/Users/peterbe/dev/JAVASCRIPT/catching_consts/code.js:3
const url = new URL(url);
^
ReferenceError: Cannot access 'url' before initialization
at validateURL (/Users/peterbe/dev/JAVASCRIPT/catching_consts/code.js:3:25)
at Object.<anonymous> (/Users/peterbe/dev/JAVASCRIPT/catching_consts/code.js:9:13)
...
▶ node --version
v15.2.1It's an honest and easy mistake to make. If the code was this:
function validateURL(url) {
const url = new URL(url);
return url.protocol === "https:";
}
// console.log(validateURL("http://www.peterbe.com"));
you'd get this error:
▶ node code2.js
/Users/peterbe/dev/JAVASCRIPT/catching_consts/code2.js:2
const url = new URL(url);
^
SyntaxError: Identifier 'url' has already been declaredwhich means node
refuses to even start it. But it can't with the original code because of the blocking scope that only happens in runtime.
Easiest solution
function validateURL(url) {
if (url.includes("://")) {
- const url = new URL(url);
+ const parsedURL = new URL(url);
- return url.protocol === "https:";
+ return parsedURL.protocol === "https:";
} else {
return "dunno";
}
}
console.log(validateURL("http://www.peterbe.com"));
Best solution
Switch to TypeScript.
▶ cat code.ts
function validateURL(url: string) {
if (url.includes('://')) {
const url = new URL(url);
return url.protocol === 'https:';
} else {
return "dunno";
}
}
console.log(validateURL('http://www.peterbe.com'));
▶ tsc --noEmit --lib es6,dom code.ts
code.ts:3:25 - error TS2448: Block-scoped variable 'url' used before its declaration.
3 const url = new URL(url);
~~~
code.ts:3:11
3 const url = new URL(url);
~~~
'url' is declared here.
Found 1 error.01 February 2021 0 comments React,
JavaScript
tl;dr; The useSearchParams hook from react-router is great as a hybrid state manager in React.
The wonderful react-router has a v6 release coming soon. At the time of writing, 6.0.0-beta.0 is the release to play with. It comes with a React hook called useSearchParams and it's fantastic. It's not a global state manager, but it can be used as one. It's not persistent, but it's semi-persistent in that state can be recovered/retained in browser refreshes.
Basically, instead of component state (e.g. React.useState()) you use:
import React from "react";
import { createSearchParams, useSearchParams } from "react-router-dom";
import "./styles.css";
export default function App() {
const [searchParams, setSearchParams] = useSearchParams();
const favoriteFruit = searchParams.get("fruit");
return (
<div className="App">
<h1>Favorite fruit</h1>
{favoriteFruit ? (
<p>
Your favorite fruit is <b>{favoriteFruit}</b>
</p>
) : (
<i>No favorite fruit selected yet.</i>
)}
{["🍒", "🍑", "🍎", "🍌"].map((fruit) => {
return (
<p key={fruit}>
<label htmlFor={`id_${fruit}`}>{fruit}</label>
<input
type="radio"
value={fruit}
checked={favoriteFruit === fruit}
onChange={(event) => {
setSearchParams(
createSearchParams({ fruit: event.target.value })
);
}}
/>
</p>
);
})}
</div>
);
}
See Codesandbox demo here
To get a feel for it, try the demo page in Codesandbox and note has it basically sets ?fruit=🍌 in the URL and if you refresh the page, it just continues as if the state had been persistent.
Basically, that's it. You never have a local component state but instead, you use the current URL as your store, and useSearchParams is your conduit for it. The advantages are:
- It's dead simple to use
- You get "shared state" across components without needing to manually inform them through prop drilling
- At any time, the current URL is a shareable snapshot of the state
The disadvantages are:
- It needs to be realistic to serialize it through the URLSearchParams web API
- The keys used need to be globally reserved for each distinct component that uses it
- You might not want the URL to change
That's all you need to know to get started. But let's dig into some more advanced examples, with some abstractions, to "workaround" the limitations.
To append or to reset
Suppose you have many different components, it's very likely that they don't really know or care about each other. Suppose, the current URL is /page?food=🍔 and if one component does: setSearchParams(createSearchParams({fruit: "🍑"})) what will happen is that the URL will "start over" and become /page?fruit=🍑. In other words, the food=🍔 was lost. Well, this might be a desired effect, but let's assume it's not, so we'll have to make it "append" instead. Here's one such solution:
function appendSearchParams(obj) {
const sp = createSearchParams(searchParams);
Object.entries(obj).forEach(([key, value]) => {
if (Array.isArray(value)) {
sp.delete(key);
value.forEach((v) => sp.append(key, v));
} else if (value === undefined) {
sp.delete(key);
} else {
sp.set(key, value);
}
});
return sp;
}
Now, you can do things like this:
onChange={(event) => {
setSearchParams(
- createSearchParams({ fruit: event.target.value })
+ appendSearchParams({ fruit: event.target.value })
);
}}
See Codesandbox demo here
Now, the two keys work independently of each other. It has a nice "just works feeling".
Note that this appendSearchParams() function implementation solves the case of arrays. You could now call it like this:
{/* Untested, but hopefully the point is demonstrated */}
<div>
<ul>
{(searchParams.getAll("languages") || []).map((language) => (
<li key={language}>{language}</li>
))}
</ul>
<button
type="button"
onClick={() => {
setSearchParams(
appendSearchParams({ languages: ["en-US", "sv-SE"] })
);
}}
>
Select 'both'
</button>
</div>
...and that will update the URL to become ?languages=en-US&languages=sv-SE.
Serialize it into links
The useSearchParams hook returns a callable setSearchParams() which is basically doing a redirect (uses the useNavigate() hook). But suppose you want to make a link that serializes a "future state". Here's a very basic example:
// Assumes 'import { Link } from "react-router-dom";'
<Link to={`?${appendSearchParams({fruit: "🍌"})}`}>Switch to 🍌</Link>
See Codesandbox demo here
Now, you get nice regular hyperlinks that uses can right-click and "Open in a new tab" and it'll just work.
Type conversion and protection
The above simple examples use strings and array of strings. But suppose you need to do more more advanced type conversions. For example: /tax-calculator?rate=3.14 where you might have something that needs to be deserialized and serialized as a floating point number. Basically, you have to wrap the deserializing in a more careful way. E.g.
function TaxYourImagination() {
const [searchParams, setSearchParams] = useSearchParams();
const taxRaw = searchParams.get("tax", DEFAULT_TAX_RATE);
let tax;
let taxError;
try {
tax = castAndCheck(taxRaw);
} catch (err) {
taxError = errl;
}
if (taxError) {
return (
<div className="error-alert">
The provided tax rate is invalid: <code>{taxError.toString()}</code>
</div>
);
}
return <DisplayTax value={tax} onUpdate={(newValue) => {
setSearchParams(
createSearchParams({ tax: newValue.toFixed(2) })
);
}}/>;
}
08 January 2021 2 comments Python
tl;dr; selectolax is best for stripping HTML down to plain text.
The problem is that I have 10,000+ HTML snippets that I need to index into Elasticsearch as plain text. (Before you ask, yes I know Elasticsearch has a html_strip text filter but it's not what I want/need to use in this context).
Turns out, stripping the HTML into plain text was actually quite expensive at that scale. So what's the most performant way?
PyQuery
from pyquery import PyQuery as pq
text = pq(html).text()
selectolax
from selectolax.parser import HTMLParser
text = HTMLParser(html).text()
regular expression
import re
regex = re.compile(r'<.*?>')
text = clean_regex.sub('', html)
Results
I wrote a script that iterated through 10,000 files that contains HTML snippets. Note! The snippets aren't complete <html> documents (with a <head> and <body> etc) Just blobs of HTML. The average size is 10,314 bytes (5,138 bytes median).
pyquery
SUM: 18.61 seconds
MEAN: 1.8633 ms
MEDIAN: 1.0554 ms
selectolax
SUM: 3.08 seconds
MEAN: 0.3149 ms
MEDIAN: 0.1621 ms
regex
SUM: 1.64 seconds
MEAN: 0.1613 ms
MEDIAN: 0.0881 ms
I've run it a bunch of times. The results are pretty stable.
Point is: selectolax is ~7 times faster than PyQuery
Regex? Really?
No, I don't think I want to use that. It makes me nervous without even attempting to dig up some examples where it goes wrong. It might work just fine for the most basic blobs of HTML. Actually, if the HTML is <p>Foo & Bar</p>, I expect the plain text transformation should be Foo & Bar, not Foo & Bar.
More pressing, both PyQuery and selectolax
supports something very specific but important to my use case. I need to remove certain tags (and its content) before I proceed. For example:
<h4 class="warning">This should get stripped.</h4>
<p>Please keep.</p>
<div style="display: none">This should also get stripped.</div>
That can never be done with a regex.
Version 2.0
So my requirement will probably change but basically, I want to delete certain tags. E.g. <div class="warning"> and <div class="hidden"> and <div style="display: none">. So let's implement that:
PyQuery
from pyquery import PyQuery as pq
_display_none_regex = re.compile(r'display:\s*none')
doc = pq(html)
doc.remove('div.warning, div.hidden')
for div in doc('div[style]').items():
style_value = div.attr('style')
if _display_none_regex.search(style_value):
div.remove()
text = doc.text()
selectolax
from selectolax.parser import HTMLParser
_display_none_regex = re.compile(r'display:\s*none')
tree = HTMLParser(html)
for tag in tree.css('div.warning, div.hidden'):
tag.decompose()
for tag in tree.css('div[style]'):
style_value = tag.attributes['style']
if style_value and _display_none_regex.search(style_value):
tag.decompose()
text = tree.body.text()
This actually works. When I now run the same benchmark for 10,000 of these are the new results:
pyquery
SUM: 21.70 seconds
MEAN: 2.1701 ms
MEDIAN: 1.3989 ms
selectolax
SUM: 3.59 seconds
MEAN: 0.3589 ms
MEDIAN: 0.2184 ms
regex
Skip
Again, selectolax beats PyQuery by a factor of ~6.
Conclusion
Regular expressions are fast but weak in power. Makes sense.
This selectolax is very impressive.
I got the inspiration from this blog post which sets out to do something very similar to what I'm doing.
I hope this helps someone. Thank you Artem Golubin of selectolax and @lexborisov for Modest which
selectolax is built upon.
21 December 2020 1 comment Python
I love git on the command line and I actually never use a GUI to navigate git branches. But sometimes, I need scripting to make abstractions that make life more convenient. What often happens is that I need to go back to the "main" branch. I write main in quotation marks because it's not always called main. Sometimes it's called master. And it's tedious to have to remember which one is the default. So I wrote a script called Gcm.
#!/usr/bin/env python
import subprocess
def run(*args):
default_branch = get_default_branch()
current_branch = get_current_branch()
if default_branch != current_branch:
checkout_branch(default_branch)
else:
print(f"Already on {default_branch}")
return 1
def checkout_branch(branch_name):
subprocess.run(f"git checkout {branch_name}".split())
def get_default_branch():
origin_name = "origin"
res = subprocess.run(
f"git remote show {origin_name}".split(), check=True, capture_output=True,
)
for line in res.stdout.decode("utf-8").splitlines():
if line.strip().startswith("HEAD branch:"):
return line.replace("HEAD branch:", "").strip()
raise NotImplementedError(f"No remote called {origin_name!r}")
def get_current_branch():
res = subprocess.run("git branch --show-current".split(), capture_output=True)
for line in res.stdout.decode("utf-8").splitlines():
return line.strip()
raise NotImplementedError("Don't know what to do!")
if __name__ == "__main__":
import sys
sys.exit(run(*sys.argv[1:]))
It ain't pretty or a spiffy one-liner, but it works. It assumes that the repo has a remote called origin which doesn't matter if it's the upstream or your fork. Put this script into a file called ~/bin/Gcm and run chmox +x ~/bin/Gcm.
Now, whenever I want to go back to the main branch I type Gcm and it takes me there.
It might seem silly, and it might not be for you, but I love it and use it many times per day. Perhaps by sharing this tip, it'll inspire someone else to set up something similar for themselves.
Why it's spelled with an uppercase G
I have a pattern (or rule?) that all scripts that I write myself are always capitalized like that. It avoids clashes with stuff I install with brew or other bash/zsh aliases.
For example:
ls -l ~/bin/RemoteVSCodePeterbecom.sh
ls -l ~/bin/Cleanupfiles
ls -l ~/bin/RandomString.py
