03 February 2020
0 comments
Node,
JavaScript
I'm working on a Node project that involves large transformations of large sets of data here and there. For example:
if (!Object.keys(this.allTitles).length) {
...
In my case, that this.allTitles is a plain object with about 30,000 key/value pairs. That particular line of code actually only runs 1 single time so if it's hundreds of milliseconds, it's really doesn't matter that much. However, that's not a guarantee! What if you had something like this:
for (const thing of things) {
if (!Object.keys(someObj).length) {
// mutate someObj
}
}
then, you'd potentially have a performance degradation once someObj becomes considerably large. And it gets particularly degraded if the length of things is considerably large as it would do the operation many times.
Actually, consider this:
const obj = {};
[...Array(30000)].forEach((_, i) => {
obj[i] = i;
});
console.time("Truthcheck obj");
[...Array(100)].forEach((_, i) => {
return !!Object.keys(obj).length;
});
console.timeEnd("Truthcheck obj");
On my macBook with Node 13.5, this outputs:
Truthcheck obj: 260.564ms
Maps
The MDN page on Map has a nice comparison, in terms of performance, between Map and regular object. Consider this super simple benchmark:
const obj = {};
const map = new Map();
[...Array(30000)].forEach((_, i) => {
obj[i] = i;
map.set(i, i);
});
console.time("Truthcheck obj");
[...Array(100)].forEach((_, i) => {
return !!Object.keys(obj).length;
});
console.timeEnd("Truthcheck obj");
console.time("Truthcheck map");
[...Array(100)].forEach((_, i) => {
return !!map.size;
});
console.timeEnd("Truthcheck map");
So, fill a Map instance and a plain object with 30,000 keys and values. Then, for each in turn, check if the thing is truthy 100 times. The output I get:
Truthcheck obj: 235.017ms
Truthcheck map: 0.029ms
That's not unexpected. The map instance maintains a size counter, which increments on .set (if the key is new), so doing that "truthy" check just takes O(1) seconds.
Conclusion
Don't run to rewrite everything to Maps!
In fact, I took the above mentioned little benchmark and changed the times to be a 3,000 item map and obj (instead of 30,000) and only did 10 iterations (instead of 100) and then the numbers are:
Truthcheck obj: 0.991ms
Truthcheck map: 0.044ms
These kinds of small numbers are very unlikely to matter in the scope of other things going on.
Anyway, consider using Map if you fear that you might be working with really reeeeally large mappings.
18 January 2020
0 comments
Python,
JavaScript
In Python
I'm sure it's been blogged about a buncha times before but, I couldn't find it, and I had to search too hard to find an example of this. Basically, what I'm trying to do is what Python does in this case, but in JavaScript:
def do_something(arg="notset", **kwargs):
print(f"arg='{arg.upper()}'")
do_something(arg="peter")
do_something(something="else")
do_something()
In Python, the output of all this is:
arg='PETER'
arg='NOTSET'
arg='NOTSET'
It could also have been implemented in a more verbose way:
def do_something(**kwargs):
arg = kwargs.get("arg", "notset")
print(f"arg='{arg.upper()}'")
This more verbose format has the disadvantage that you can't quickly skim it and see and what the default is. That thing (arg = kwargs.get("arg", "notset")) might happen far away deeper in the function, making it hard work to spot the default.
In JavaScript
Here's the equivalent in JavaScript (ES6?):
function doSomething({ arg = "notset", ...kwargs } = {}) {
return `arg='${arg.toUpperCase()}'`;
}
console.log(doSomething({ arg: "peter" }));
console.log(doSomething({ something: "else" }));
console.log(doSomething());
Same output as in Python:
arg='PETER'
arg='NOTSET'
arg='NOTSET'
Notes
I'm still not convinced I like this syntax. It feels a bit too "hip" and too one-liner'y. But it's also pretty useful.
Mind you, the examples here are contrived because they're so short in terms of the number of arguments used in the function.
A more realistic thing like be a function that lists, upfront, all the possible parameters and for some of them, it wants to point out some defaults. E.g.
function processFolder({
source,
destination = "/tmp",
quiet = false,
verbose = false
} = {}) {
console.log({ source, destination, quiet, verbose });
// outputs
// { source: '/user', destination: '/tmp', quiet: true, verbose: false }
}
console.log(processFolder({ source: "/user", quiet: true }));
One could maybe argue that arguments that don't have a default are expected to always be supplied so they can be regular arguments like:
function processFolder(source, {
destination = "/tmp",
quiet = false,
verbose = false
} = {}) {
console.log({ source, destination, quiet, verbose });
// outputs
// { source: '/user', destination: '/tmp', quiet: true, verbose: false }
}
console.log(processFolder("/user", { quiet: true }));
But, I quite like keeping all arguments in an object. It makes it easier to write wrapper functions and I find this:
setProfile(
"My biography here",
false,
193.5,
230,
["anders", "bengt"],
"South Carolina"
);
...harder to read than...
setProfile({
bio: "My biography here",
dead: false,
height: 193.5,
weight: 230,
middlenames: ["anders", "bengt"],
state: "South Carolina"
});
15 January 2020
0 comments
JavaScript
Suppose that you're working on ~/dev/my-cool-project and inside ~/dev/my-cool-project/package.json you might have something like this:
"dependencies": {
"that-cool-lib": "1.2.3",
...
But that that-cool-lib is one of your own projects. You're also working on that project and it's over at ~/dev/that-cool-lib. Within that-cool-lib you might be in a git branch or perhaps you're preparing a 2.0.0 release.
Now you're interested if that-cool-lib@2.0.0 is going to work here inside my-cool-project.
What you could do
First, you release this fancy that-cool-lib@2.0.0 to npmjs.com with that project's npm publish procedure. Then as soon as that's done and you can see that the release made it onto https://www.npmjs.com/package/that-cool-lib/v/2.0.0.
Then you go over to my-cool-project and start a new git branch to try the upgrade, npm install that-cool-project@2.0.0 --save so you have this:
"dependencies": {
- "that-cool-lib": "1.2.3",
+ "that-cool-lib": "2.0.0",
...
Now you can try it that new version of my-cool-project and if that-cool-lib had any of its own entry point executables or post/pre install steps, they'd be fully resolved.
What you should do
Instead, use install-local. Don't use npm link because it might not install entry point executables and I also don't like the fact that I need to go into that-cool-lib and install it (globally?) first (when you do cd that-cool-lib && npm link). Also, see "What's wrong with npm-link?".
Here's how you do it:
npx install-local ~/dev/that-cool-lib
and it acts pretty much exactly as if you had gotten it from npmjs.com the normal way.
Notes
I almost never use npm these days. Go yarn! So, perhaps I've misinterpreted something.
Also, I try my very hardest to never use npm install -g ... (or yarn global ... for that matter) now that we have npx. Perhaps if you'd install it locally it'd speed up the use of local-install by 1-3 seconds each time you run this. Again, my skillset of modern npm is fading so I don't think I understand why it takes me 14 seconds the first time I run npx install that-cool-lib and then it takes 14 seconds again when I run the exact same command again. Does it not benefit from any caching? How much of that time is spent on npmjs.com resolving other sub-dependencies that that-cool-lib requires?
Hopefully, this helps other people stuck in a similar boat.
03 January 2020
2 comments
Node,
JavaScript
cheerio is a great Node library for processing HTML. It's faster than JSDOM and years and years of jQuery usage makes the API feel yummily familiar.
What if you have a piece of HTML that you want to split up into multiple blocks? For example, you have this:
<div>Prelude</div>
<h2>First Header</h2>
<p>Paragraph <b>here</b>.</p>
<p>Another paragraph.</p>
<h2 id="second">Second Header</h2>
<ul>
<li>One</li>
<li>Two</li>
</ul>
<blockquote>End quote!</blockquote>
and you want to get this split by the <h2> tags so you end up with 3 (in this example) distinct blocks of HTML, like this:
first one
second one
<h2>First Header</h2>
<p>Paragraph <b>here</b>.</p>
<p>Another paragraph.</p>
third one
<h2 id="second">Second Header</h2>
<ul>
<li>One</li>
<li>Two</li>
</ul>
<blockquote>End quote!</blockquote>
You could try to cast the regex spell on that and try to, I don't know, split the string by the </h2>. But it's risky and error prone because (although a bit unlikely in this simple example) get caught up in <h2>...</h2> tags that are nested inside something else. Also, proper parsing almost always wins in the long run over regexes.
Use cheerio
This is how I solved it and hopefully A) you can copy and benefit, or B) someone tells me there's already a much better way.
What you do is walk the DOM root nodes, one by one, and keep filling a buffer and then yield individual new cheerio instances.
const html = `
<div>Prelude</div>
<h2>First Header</h2>
<p>Paragraph <b>here</b>.</p>
<p>Another paragraph.</p>
<!-- comment -->
<h2 id="second">Second Header</h2>
<ul>
<li>One</li>
<li>Two</li>
</ul>
<blockquote>End quote!</blockquote>
`;
// load the raw HTML
// it needs to all be wrapped in *one* big wrapper
const $ = cheerio.load(`<div id="_body">${html}</div>`);
// the end goal
const blocks = [];
// the buffer
const section = cheerio
.load("<div></div>", { decodeEntities: false })("div")
.eq(0);
const iterable = [...$("#_body")[0].childNodes];
let c = 0;
iterable.forEach(child => {
if (child.tagName === "h2") {
if (c) {
blocks.push(section.clone());
section.empty();
c = 0; // reset the counter
}
}
c++;
section.append(child);
});
if (c) {
// stragglers
blocks.push(section.clone());
}
// Test the result
const blocksAsStrings = blocks.map(block => block.html());
console.log(blocksAsStrings.length);
// 3
console.log(blocksAsStrings);
// [
// '\n<div>Prelude</div>\n\n',
// '<h2>First Header</h2>\n' +
// '<p>Paragraph <b>here</b>.</p>\n' +
// '<p>Another paragraph.</p>\n' +
// '<!-- comment -->\n' +
// '\n',
// '<h2 id="second">Second Header</h2>\n' +
// '<ul>\n' +
// ' <li>One</li>\n' +
// ' <li>Two</li>\n' +
// '</ul>\n' +
// '<blockquote>End quote!</blockquote>\n'
// ]
In this particular implementation the choice of splitting is by the every h2 tag. If you want to split by anything else, go ahead and adjust the conditional there where it's currently doing if (child.tagName === "h2") {.
Also, what you do with the blocks is up to you. Perhaps you need them as strings, then you use the blocks.map(block => block.html()). Otherwise, if it serves your needs they can remain as individual cheerio instances that you can do whatever with.
13 December 2019
2 comments
Web development,
Django,
Python,
Docker,
JavaScript
Heroku is great but it's sometimes painful when your app isn't just in one single language. What I have is a project where the backend is Python (Django) and the frontend is JavaScript (Preact). The folder structure looks like this:
/
- README.md
- manage.py
- requirements.txt
- my_django_app/
- settings.py
- asgi.py
- api/
- urls.py
- views.py
- frontend/
- package.json
- yarn.lock
- preact.config.js
- build/
...
- src/
...
A bunch of things omitted for brevity but people familiar with Django and preact-cli/create-create-app should be familiar.
The point is that the root is a Python app and the front-end is exclusively inside a sub folder.
When you do local development, you start two servers:
./manage.py runserver - starts http://localhost:8000cd frontend && yarn start - starts http://localhost:3000
The latter is what you open in your browser. That preact app will do things like:
const response = await fetch('/api/search');
and, in preact.config.js I have this:
export default (config, env, helpers) => {
if (config.devServer) {
config.devServer.proxy = [
{
path: "/api/**",
target: "http://localhost:8000"
}
];
}
};
...which is hopefully self-explanatory. So, calls like GET http://localhost:3000/api/search actually goes to http://localhost:8000/api/search.
That's when doing development. The interesting thing is going into production.
Before we get into Heroku, let's first "merge" the two systems into one and the trick used is Whitenoise. Basically, Django's web server will be responsibly not only for things like /api/search but also static assets such as / --> frontend/build/index.html and /bundle.17ae4.js --> frontend/build/bundle.17ae4.js.
This is basically all you need in settings.py to make that happen:
MIDDLEWARE = [
"django.middleware.security.SecurityMiddleware",
"whitenoise.middleware.WhiteNoiseMiddleware",
...
]
WHITENOISE_INDEX_FILE = True
STATIC_URL = "/"
STATIC_ROOT = BASE_DIR / "frontend" / "build"
However, this isn't quite enough because the preact app uses preact-router which uses pushState() and other code-splitting magic so you might have a URL, that users see, like this: https://myapp.example.com/that/thing/special and there's nothing about that in any of the Django urls.py files. Nor is there any file called frontend/build/that/thing/special/index.html or something like that.
So for URLs like that, we have to take a gamble on the Django side and basically hope that the preact-router config knows how to deal with it. So, to make that happen with Whitenoise we need to write a custom middleware that looks like this:
from whitenoise.middleware import WhiteNoiseMiddleware
class CustomWhiteNoiseMiddleware(WhiteNoiseMiddleware):
def process_request(self, request):
if self.autorefresh:
static_file = self.find_file(request.path_info)
else:
static_file = self.files.get(request.path_info)
# These two lines is the magic.
# Basically, the URL didn't lead to a file (e.g. `/manifest.json`)
# it's either a API path or it's a custom browser path that only
# makes sense within preact-router. If that's the case, we just don't
# know but we'll give the client-side preact-router code the benefit
# of the doubt and let it through.
if not static_file and not request.path_info.startswith("/api"):
static_file = self.files.get("/")
if static_file is not None:
return self.serve(static_file, request)
And in settings.py this change:
MIDDLEWARE = [
"django.middleware.security.SecurityMiddleware",
- "whitenoise.middleware.WhiteNoiseMiddleware",
+ "my_django_app.middleware.CustomWhiteNoiseMiddleware",
...
]
Now, all traffic goes through Django. Regular Django view functions, static assets, and everything else fall back to frontend/build/index.html.
Heroku
Heroku tries to make everything so simple for you. You basically, create the app (via the cli or the Heroku web app) and when you're ready you just do git push heroku master. However that won't be enough because there's more to this than Python.
Unfortunately, I didn't take notes of my hair-pulling excruciating journey of trying to add buildpacks and hacks and Procfiles and custom buildpacks. Nothing seemed to work. Perhaps the answer was somewhere in this issue: "Support running an app from a subdirectory" but I just couldn't figure it out. I still find buildpacks confusing when it's beyond Hello World. Also, I didn't want to run Node as a service, I just wanted it as part of the "build process".
Docker to the rescue
Finally I get a chance to try "Deploying with Docker" in Heroku which is a relatively new feature. And the only thing that scared me was that now I need to write a heroku.yml file which was confusing because all I had was a Dockerfile. We'll get back to that in a minute!
So here's how I made a Dockerfile that mixes Python and Node:
FROM node:12 as frontend
COPY . /app
WORKDIR /app
RUN cd frontend && yarn install && yarn build
FROM python:3.8-slim
WORKDIR /app
RUN groupadd --gid 10001 app && useradd -g app --uid 10001 --shell /usr/sbin/nologin app
RUN chown app:app /tmp
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y --no-install-recommends \
gcc apt-transport-https python-dev
# Gotta try moving this to poetry instead!
COPY ./requirements.txt /app/requirements.txt
RUN pip install --upgrade --no-cache-dir -r requirements.txt
COPY . /app
COPY --from=frontend /app/frontend/build /app/frontend/build
USER app
ENV PORT=8000
EXPOSE $PORT
CMD uvicorn gitbusy.asgi:application --host 0.0.0.0 --port $PORT
If you're not familiar with it, the critical trick is on the first line where it builds some Node with as frontend. That gives me a thing I can then copy from into the Python image with COPY --from=frontend /app/frontend/build /app/frontend/build.
Now, at the very end, it starts a uvicorn server with all the static .js, index.html, and favicon.ico etc. available to uvicorn which ultimately runs whitenoise.
To run and build:
docker build . -t my_app
docker run -t -i --rm --env-file .env -p 8000:8000 my_app
Now, opening http://localhost:8000/ is a production grade app that mixes Python (runtime) and JavaScript (static).
Heroku + Docker
Heroku says to create a heroku.yml file and that makes sense but what didn't make sense is why I would add cmd line in there when it's already in the Dockerfile. The solution is simple: omit it. Here's what my final heroku.yml file looks like:
build:
docker:
web: Dockerfile
Check in the heroku.yml file and git push heroku master and voila, it works!
To see a complete demo of all of this check out https://github.com/peterbe/gitbusy and https://gitbusy.herokuapp.com/
24 October 2019
0 comments
Node,
JavaScript
tl;dr; If you know that the only I/O you have is disk and the disk is SSD, then synchronous is probably more convenient, faster, and more memory lean.
I'm not a NodeJS expert so I could really do with some eyes on this.
There is little doubt in my mind that it's smart to use asynchronous ideas when your program has to wait for network I/O. Because network I/O is slow, it's better to let your program work on something else whilst waiting. But disk is actually fast. Especially if you have SSD disk.
The context
I'm working on a Node program that walks a large directory structure and looks for certain file patterns, reads those files, does some processing and then exits. It's a cli basically and it's supposed to work similar to jest where you tell it to go and process files and if everything worked, exit with 0 and if anything failed, exit with something >0. Also, it needs to be possible to run it so that it exits immediately on the first error encountered. This is similar to running jest --bail.
My program needs to process thousands of files and although there are thousands of files, they're all relatively small. So first I wrote a simple reference program: https://github.com/peterbe/megafileprocessing/blob/master/reference.js
What it does is that it walks a directory looking for certain .json files that have certain keys that it knows about. Then, just computes the size of the values and tallies that up. My real program will be very similar except it does a lot more with each .json file.
You run it like this:
▶ CHAOS_MONKEY=0.001 node reference.js ~/stumptown-content/kumadocs -q
Error: Chaos Monkey!
at processDoc (/Users/peterbe/dev/JAVASCRIPT/megafileprocessing/reference.js:37:11)
at /Users/peterbe/dev/JAVASCRIPT/megafileprocessing/reference.js:80:21
at Array.forEach (<anonymous>)
at main (/Users/peterbe/dev/JAVASCRIPT/megafileprocessing/reference.js:78:9)
at Object.<anonymous> (/Users/peterbe/dev/JAVASCRIPT/megafileprocessing/reference.js:99:20)
at Module._compile (internal/modules/cjs/loader.js:956:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)
at Module.load (internal/modules/cjs/loader.js:812:32)
at Function.Module._load (internal/modules/cjs/loader.js:724:14)
at Function.Module.runMain (internal/modules/cjs/loader.js:1025:10)
Total length for 4057 files is 153953645
1 files failed.
(The environment variable CHAOS_MONKEY=0.001 makes it so there's a 0.1% chance it throws an error)
It processed 4,057 files and one of those failed (thanks to the "chaos monkey").
In its current state that (on my MacBook) that takes about 1 second.
It's not perfect but it's a good skeleton. Everything is synchronous. E.g.
function main(args) {
// By default, don't exit if any error happens
const { bail, quiet, root } = parseArgs(args);
const files = walk(root, ".json");
let totalTotal = 0;
let errors = 0;
files.forEach(file => {
try {
const total = processDoc(file, quiet);
!quiet && console.log(`${file} is ${total}`);
totalTotal += total;
} catch (err) {
if (bail) {
throw err;
} else {
console.error(err);
errors++;
}
}
});
console.log(`Total length for ${files.length} files is ${totalTotal}`);
if (errors) {
console.warn(`${errors} files failed.`);
}
return errors ? 1 : 0;
}
And inside the processDoc function it used const content = fs.readFileSync(fspath, "utf8");.
I/Os compared
@amejiarosario has a great blog post called "What every programmer should know about Synchronous vs. Asynchronous Code". In it, he has this great bar chart:
If you compare "SSD I/O" with "Network SFO/NCY" the difference is that SSD I/O is 456 times "faster" than SFO-to-NYC network I/O. I.e. the latency is 456 times less.
Another important aspect when processing lots of files is garbage collection. When running synchronous, it can garbage collect as soon as it has processed one file before moving on to the next. If it was asynchronous, as soon as it yields to move on to the next file, it might hold on to memory from the first file. Why does this matter? Because if the memory-usage when processing many files asynchronously bloat so hard that it actually crashes with an out-of-memory error. So what matters is avoiding that. It's OK if the program can use lots of memory if it needs to, but it's really bad if it crashes.
One way to measure this is to use /usr/bin/time -l (at least that's what it's called on macOS). For example:
▶ /usr/bin/time -l node reference.js ~/stumptown-content/kumadocs -q
Total length for 4057 files is 153970749
0.75 real 0.58 user 0.23 sys
57221120 maximum resident set size
0 average shared memory size
0 average unshared data size
0 average unshared stack size
64160 page reclaims
0 page faults
0 swaps
0 block input operations
0 block output operations
0 messages sent
0 messages received
0 signals received
0 voluntary context switches
1074 involuntary context switches
Its maximum memory usage total was 57221120 bytes (55MB) in this example.
Introduce asynchronous file reading
Let's change the reference implementation to use const content = await fsPromises.readFile(fspath, "utf8");. We're still using files.forEach(file => { but within the loop the whole function is prefixed with async function main() { now. Like this:
async function main(args) {
// By default, don't exit if any error happens
const { bail, quiet, root } = parseArgs(args);
const files = walk(root, ".json");
let totalTotal = 0;
let errors = 0;
let total;
for (let file of files) {
try {
total = await processDoc(file, quiet);
!quiet && console.log(`${file} is ${total}`);
totalTotal += total;
} catch (err) {
if (bail) {
throw err;
} else {
console.error(err);
errors++;
}
}
}
console.log(`Total length for ${files.length} files is ${totalTotal}`);
if (errors) {
console.warn(`${errors} files failed.`);
}
return errors ? 1 : 0;
}
Let's see how it works:
▶ /usr/bin/time -l node async1.js ~/stumptown-content/kumadocs -q
Total length for 4057 files is 153970749
1.31 real 1.01 user 0.49 sys
68898816 maximum resident set size
0 average shared memory size
0 average unshared data size
0 average unshared stack size
68107 page reclaims
0 page faults
0 swaps
0 block input operations
0 block output operations
0 messages sent
0 messages received
0 signals received
0 voluntary context switches
62562 involuntary context switches
That means it maxed out at 68898816 bytes (65MB).
You can already see a difference. 0.79 seconds and 55MB for synchronous and 1.31 seconds and 65MB for asynchronous.
But to really measure this, I wrote a simple Python program that runs this repeatedly and reports a min/median on time and max on memory:
▶ python3 wrap_time.py /usr/bin/time -l node reference.js ~/stumptown-content/kumadocs -q
...
TIMES
BEST: 0.74s
WORST: 0.84s
MEAN: 0.78s
MEDIAN: 0.78s
MAX MEMORY
BEST: 53.5MB
WORST: 55.3MB
MEAN: 54.6MB
MEDIAN: 54.8MB
And for the asynchronous version:
▶ python3 wrap_time.py /usr/bin/time -l node async1.js ~/stumptown-content/kumadocs -q
...
TIMES
BEST: 1.28s
WORST: 1.82s
MEAN: 1.39s
MEDIAN: 1.31s
MAX MEMORY
BEST: 65.4MB
WORST: 67.7MB
MEAN: 66.7MB
MEDIAN: 66.9MB
Promise.all version
I don't know if the async1.js is realistic. More realistically you'll want to not wait for one file to be processed (asynchronously) but start them all at the same time. So I made a variation of the asynchronous version that looks like this instead:
async function main(args) {
// By default, don't exit if any error happens
const { bail, quiet, root } = parseArgs(args);
const files = walk(root, ".json");
let totalTotal = 0;
let errors = 0;
let values;
values = await Promise.all(
files.map(async file => {
try {
total = await processDoc(file, quiet);
!quiet && console.log(`${file} is ${total}`);
return total;
} catch (err) {
if (bail) {
console.error(err);
process.exit(1);
} else {
console.error(err);
errors++;
}
}
})
);
totalTotal = values.filter(n => n).reduce((a, b) => a + b);
console.log(`Total length for ${files.length} files is ${totalTotal}`);
if (errors) {
console.warn(`${errors} files failed.`);
throw new Error("More than 0 errors");
}
}
You can see the whole file here: async2.js
The key difference is that it uses await Promise.all(files.map(...)) instead of for (let file of files) {.
Also, to accomplish the ability to bail on the first possible error it needs to use process.exit(1); within the callbacks. Not sure if that's right but from the outside, you get the desired effect as a cli program. Let's measure it too:
▶ python3 wrap_time.py /usr/bin/time -l node async2.js ~/stumptown-content/kumadocs -q
...
TIMES
BEST: 1.44s
WORST: 1.61s
MEAN: 1.52s
MEDIAN: 1.52s
MAX MEMORY
BEST: 434.0MB
WORST: 460.2MB
MEAN: 453.4MB
MEDIAN: 456.4MB
Note how this uses almost 10x max. memory. That's dangerous if the processing is really memory hungry individually.
When asynchronous is right
In all of this, I'm assuming that the individual files are small. (Roughly, each file in my experiment is about 50KB)
What if the files it needs to read from disk are large?
As a simple experiment read /users/peterbe/Downloads/Keybase.dmg 20 times and just report its size:
for (let x = 0; x < 20; x++) {
fs.readFile("/users/peterbe/Downloads/Keybase.dmg", (err, data) => {
if (err) throw err;
console.log(`File size#${x}: ${Math.round(data.length / 1e6)} MB`);
});
}
See the simple-async.js here. Basically it's this:
for (let x = 0; x < 20; x++) {
fs.readFile("/users/peterbe/Downloads/Keybase.dmg", (err, data) => {
if (err) throw err;
console.log(`File size#${x}: ${Math.round(data.length / 1e6)} MB`);
});
}
Results are:
▶ python3 wrap_time.py /usr/bin/time -l node simple-async.js
...
TIMES
BEST: 0.84s
WORST: 4.32s
MEAN: 1.33s
MEDIAN: 0.97s
MAX MEMORY
BEST: 1851.1MB
WORST: 3079.3MB
MEAN: 2956.3MB
MEDIAN: 3079.1MB
And the equivalent synchronous simple-sync.js here.
for (let x = 0; x < 20; x++) {
const largeFile = fs.readFileSync("/users/peterbe/Downloads/Keybase.dmg");
console.log(`File size#${x}: ${Math.round(largeFile.length / 1e6)} MB`);
}
It performs like this:
▶ python3 wrap_time.py /usr/bin/time -l node simple-sync.js
...
TIMES
BEST: 1.97s
WORST: 2.74s
MEAN: 2.27s
MEDIAN: 2.18s
MAX MEMORY
BEST: 1089.2MB
WORST: 1089.7MB
MEAN: 1089.5MB
MEDIAN: 1089.5MB
So, almost 2x as slow but 3x as much max. memory.
Lastly, instead of an iterative loop, let's start 20 readers at the same time (simple-async2.js):
Promise.all(
[...Array(20).fill()].map((_, x) => {
return fs.readFile("/users/peterbe/Downloads/Keybase.dmg", (err, data) => {
if (err) throw err;
console.log(`File size#${x}: ${Math.round(data.length / 1e6)} MB`);
});
})
);
And it performs like this:
▶ python3 wrap_time.py /usr/bin/time -l node simple-async2.js
...
TIMES
BEST: 0.86s
WORST: 1.09s
MEAN: 0.96s
MEDIAN: 0.94s
MAX MEMORY
BEST: 3079.0MB
WORST: 3079.4MB
MEAN: 3079.2MB
MEDIAN: 3079.2MB
So quite naturally, the same total time as the simple async version but uses 3x max. memory every time.
Ergonomics
I'm starting to get pretty comfortable with using promises and async/await. But I definitely feel more comfortable without. Synchronous programs read better from an ergonomics point of view. The async/await stuff is just Promises under the hood and it's definitely an improvement but the synchronous versions just have a simpler "feeling" to it.
Conclusion
I don't think it's a surprise that the overhead of event switching adds more time than its worth when the individual waits aren't too painful.
A major flaw with synchronous programs is that they rely on the assumption that there's no really slow I/O. So what if the program grows and morphs so that it someday does depend on network I/O then your synchronous program is "screwed" since an asynchronous version would run circles around it.
The general conclusion is; if you know that the only I/O you have is disk and the disk is SSD, then synchronous is probably more convenient, faster, and more memory lean.
