Brotli compression quality comparison in the real world
December 1, 2021
2 comments Node, JavaScript
At work, we use Brotli (using the Node builtin zlib
) to compress these large .json
files to .json.br
files. When using zlib.brotliCompress
you can set options to override the quality number. Here's an example of it at quality 6:
import { promisify } from 'util'
import zlib from 'zlib'
const brotliCompress = promisify(zlib.brotliCompress)
const options = {
params: {
[zlib.constants.BROTLI_PARAM_MODE]: zlib.constants.BROTLI_MODE_TEXT,
[zlib.constants.BROTLI_PARAM_QUALITY]: 6,
},
}
export async function compress(data) {
return brotliCompress(data, options)
}
But what if you mess with that number. Surely, the files will become smaller, but at what cost? Well, I wrote a Node script that measured how long it would take to compress 6 large (~25MB each) .json
file synchronously. Then, I put them into a Google spreadsheet and voila:
Size
Time
Miles away from rocket science but I thought it was cute to visualize as a way of understanding the quality option.
How to string pad a string in Python with a variable
October 19, 2021
1 comment Python
I just have to write this down because that's the rule; if I find myself googling something basic like this more than once, it's worth blogging about.
Suppose you have a string and you want to pad with empty spaces. You have 2 options:
>>> s = "peter"
>>> s.ljust(10)
'peter '
>>> f"{s:<10}"
'peter '
The f-string notation is often more convenient because it can be combined with other formatting directives.
But, suppose the number 10
isn't hardcoded like that. Suppose it's a variable:
>>> s = "peter"
>>> width = 11
>>> s.ljust(width)
'peter '
>>> f"{s:<width}"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier
Well, the way you need to do it with f-string formatting, when it's a variable like that is this syntax:
>>> f"{s:<{width}}"
'peter '
How to bulk-insert Firestore documents in a Firebase Cloud function
September 23, 2021
1 comment Node, Firebase, JavaScript
You can't batch-add/bulk-insert documents in the Firebase Web SDK. But you can with the Firebase Admin Node SDK. Like, in a Firebase Cloud Function. Here's an example of how to do that:
const firestore = admin.firestore();
let batch = firestore.batch();
let counter = 0;
let totalCounter = 0;
const promises = [];
for (const thing of MANY_MANY_THINGS) {
counter++;
const docRef = firestore.collection("MY_COLLECTION").doc();
batch.set(docRef, {
foo: thing.foo,
bar: thing.bar,
favNumber: 0,
});
counter++;
if (counter >= 500) {
console.log(`Committing batch of ${counter}`);
promises.push(batch.commit());
totalCounter += counter;
counter = 0;
batch = firestore.batch();
}
}
if (counter) {
console.log(`Committing batch of ${counter}`);
promises.push(batch.commit());
totalCounter += counter;
}
await Promise.all(promises);
console.log(`Committed total of ${totalCounter}`);
I'm using this in a Cloud HTTP function where I can submit a large amount of data and have each one fill up a collection.
TypeScript generic async function wrapper function
September 12, 2021
0 comments JavaScript
I find this so fiddly! I love TypeScript and will continue to use it if there's a choice. But I just wanted to write a simple async function wrapper and I had to Google for it and nothing was quite right. Here's my simple solution, as an example:
function wrappedAsyncFunction<T>(
fn: (...args: any[]) => Promise<T>
): (...args: any[]) => Promise<T> {
return async function(...args: any[]) {
console.time("Took");
try {
return await fn(...args);
} catch(error) {
console.warn("FYI, an error happened:", error);
throw error;
} finally {
console.timeEnd("Took");
}
};
}
What I use it for is to wrap my Firebase Cloud Functions so that if any error happens, I can send that error to Rollbar. In particular, here's an example of it in use:
diff --git a/functions/src/cleanup-thumbnails.ts b/functions/src/cleanup-thumbnails.ts
index 46bdb34..a3e8d54 100644
--- a/functions/src/cleanup-thumbnails.ts
+++ b/functions/src/cleanup-thumbnails.ts
@@ -2,6 +2,8 @@ import * as admin from "firebase-admin";
import * as functions from "firebase-functions";
import { logger } from "firebase-functions";
+import { wrappedLogError } from "./rollbar-logger";
+
const OLD_DAYS = 30 * 6; // 6 months
// const ADDITIONAL_DAYS_BACK = 5;
// const ADDITIONAL_DAYS_BACK = 15;
@@ -9,7 +11,7 @@ const PREFIX = "thumbnails";
export const scheduledCleanupThumbnails = functions.pubsub
.schedule("every 24 hours")
- .onRun(async () => {
+ .onRun(wrappedLogError(async () => {
logger.debug("Running scheduledCleanupThumbnails");
...
And my wrappedLogError
looks like this:
export function wrappedLogError<T>(
fn: (...args: any[]) => Promise<T>
): (...args: any[]) => Promise<T> {
return async function(...args: any[]) {
try {
return await fn(...args);
} catch (error) {
logError(error);
throw error;
}
};
}
I'm not sure it's the best or correct way to do it, but it seems to work. Perhaps there's a more correct solution but for now I'll ship this because it seems to work fine.
TypeScript function keyword arguments like Python
September 8, 2021
0 comments Python, JavaScript
To do this in Python:
def print_person(name="peter", dob=1979):
print(f"name={name}\tdob={dob}")
print_person()
# prints: name=peter dob=1979
print_person(name="Tucker")
# prints: name=Tucker dob=1979
print_person(dob=2013)
# prints: name=peter dob=2013
print_person(sex="boy")
# TypeError: print_person() got an unexpected keyword argument 'sex'
...in TypeScript:
function printPerson({
name = "peter",
dob = 1979
}: { name?: string; dob?: number } = {}) {
console.log(`name=${name}\tdob=${dob}`);
}
printPerson();
// prints: name=peter dob=1979
printPerson({});
// prints: name=peter dob=1979
printPerson({ name: "Tucker" });
// prints: name=Tucker dob=1979
printPerson({ dob: 2013 });
// prints: name=peter dob=2013
printPerson({ gender: "boy" })
// Error: Object literal may only specify known properties, and 'gender'
Here's a Playground copy of it.
It's not a perfect "transpose" across the two languages but it's sufficiently similar.
The trick is that last = {}
at the end of the function signature in TypeScript which makes it possible to omit keys in the passed-in object.
By the way, the pure JavaScript version of this is:
function printPerson({ name = "peter", dob = 1979 } = {}) {
console.log(`name=${name}\tdob=${dob}`);
}
But, unlike Python and TypeScript, you get no warnings or errors if you'd do printPerson({ gender: "boy" });
with the JavaScript version.