How do you thousands-comma AND whitespace format a f-string in Python
March 17, 2024
0 comments Python
For some reason, I always forget how to do this. Tired of that. Let's blog about it so it sticks.
To format a number with thousand-commas you do:
>>> n = 1234567
>>> f"{n:,}"
'1,234,567'
To add whitespace to a string you do:
>>> name="peter"
>>> f"{name:<20}"
'peter '
How to combine these in one expression, you do:
>>> n = 1234567
>>> f"{n:<15,}"
'1,234,567 '
Leibniz formula for π in Python, JavaScript, and Ruby
March 14, 2024
0 comments Python, JavaScript
Officially, I'm one day behind, but here's how you can calculate the value of π using the Leibniz formula.
Python
import math
sum = 0
estimate = 0
i = 0
epsilon = 0.0001
while abs(estimate - math.pi) > epsilon:
sum += (-1) ** i / (2 * i + 1)
estimate = sum * 4
i += 1
print(
f"After {i} iterations, the estimate is {estimate} and the real pi is {math.pi} "
f"(difference of {abs(estimate - math.pi)})"
)
Outputs:
After 10000 iterations, the estimate is 3.1414926535900345 and the real pi is 3.141592653589793 (difference of 9.99999997586265e-05)
JavaScript
let sum = 0;
let estimate = 0;
let i = 0;
const epsilon = 0.0001;
while (Math.abs(estimate - Math.PI) > epsilon) {
sum += (-1) ** i / (2 * i + 1);
estimate = sum * 4;
i += 1;
}
console.log(
`After ${i} iterations, the estimate is ${estimate} and the real pi is ${Math.PI} ` +
`(difference of ${Math.abs(estimate - Math.PI)})`
);
Outputs
After 10000 iterations, the estimate is 3.1414926535900345 and the real pi is 3.141592653589793 (difference of 0.0000999999997586265)
Ruby
sum = 0
estimate = 0
i = 0
epsilon = 0.0001
while (estimate - Math::PI).abs > epsilon
sum += ((-1) ** i / (2.0 * i + 1))
estimate = sum * 4
i += 1
end
print(
"After #{i} iterations, the estimate is #{estimate} and the real pi is #{Math::PI} "+
"(difference of #{(estimate - Math::PI).abs})"
)
Outputs
After 10000 iterations, the estimate is 3.1414926535900345 and the real pi is 3.141592653589793 (difference of 9.99999997586265e-05)
Backwards
Technically, these little snippets are checking that it works since each language already has access to a value of π as a standard library constant.
If you don't have that, you can decide on a number of iterations, for example 1,000, and use that.
Python
sum = 0
for i in range(1000):
sum += (-1) ** i / (2 * i + 1)
print(sum * 4)
JavaScript
let sum = 0;
for (const i of [...Array(10000).keys()]) {
sum += (-1) ** i / (2 * i + 1);
}
console.log(sum * 4);
Ruby
sum = 0
for i in 0..10000
sum += ((-1) ** i / (2.0 * i + 1))
end
puts sum * 4
Performance test
Perhaps a bit silly but also a fun thing to play with. Pull out hyperfine
and compare Python 3.12, Node 20.11, Ruby 3.2, and Bun 1.0.30:
❯ hyperfine --warmup 10 "python3.12 ~/pi.py" "node ~/pi.js" "ruby ~/pi.rb" "bun run ~/pi.js"
Benchmark 1: python3.12 ~/pi.py
Time (mean ± σ): 53.4 ms ± 7.5 ms [User: 31.9 ms, System: 12.3 ms]
Range (min … max): 41.5 ms … 64.8 ms 44 runs
Benchmark 2: node ~/pi.js
Time (mean ± σ): 57.5 ms ± 10.6 ms [User: 43.3 ms, System: 11.0 ms]
Range (min … max): 46.2 ms … 82.6 ms 35 runs
Benchmark 3: ruby ~/pi.rb
Time (mean ± σ): 242.1 ms ± 11.6 ms [User: 68.4 ms, System: 37.2 ms]
Range (min … max): 227.3 ms … 265.3 ms 11 runs
Benchmark 4: bun run ~/pi.js
Time (mean ± σ): 32.9 ms ± 6.3 ms [User: 14.1 ms, System: 10.0 ms]
Range (min … max): 17.1 ms … 41.9 ms 60 runs
Summary
bun run ~/pi.js ran
1.62 ± 0.39 times faster than python3.12 ~/pi.py
1.75 ± 0.46 times faster than node ~/pi.js
7.35 ± 1.45 times faster than ruby ~/pi.rb
Comparing Pythons
Just because I have a couple of these installed:
❯ hyperfine --warmup 10 "python3.8 ~/pi.py" "python3.9 ~/pi.py" "python3.10 ~/pi.py" "python3.11 ~/pi.py" "python3.12 ~/pi.py"
Benchmark 1: python3.8 ~/pi.py
Time (mean ± σ): 54.6 ms ± 8.1 ms [User: 33.0 ms, System: 11.4 ms]
Range (min … max): 40.0 ms … 69.7 ms 56 runs
Benchmark 2: python3.9 ~/pi.py
Time (mean ± σ): 54.9 ms ± 8.0 ms [User: 32.2 ms, System: 12.3 ms]
Range (min … max): 42.3 ms … 70.1 ms 38 runs
Benchmark 3: python3.10 ~/pi.py
Time (mean ± σ): 54.7 ms ± 7.5 ms [User: 33.0 ms, System: 11.8 ms]
Range (min … max): 42.3 ms … 78.1 ms 44 runs
Benchmark 4: python3.11 ~/pi.py
Time (mean ± σ): 53.8 ms ± 6.0 ms [User: 32.7 ms, System: 13.0 ms]
Range (min … max): 44.8 ms … 70.3 ms 42 runs
Benchmark 5: python3.12 ~/pi.py
Time (mean ± σ): 53.0 ms ± 6.4 ms [User: 31.8 ms, System: 12.3 ms]
Range (min … max): 43.8 ms … 63.5 ms 42 runs
Summary
python3.12 ~/pi.py ran
1.02 ± 0.17 times faster than python3.11 ~/pi.py
1.03 ± 0.20 times faster than python3.8 ~/pi.py
1.03 ± 0.19 times faster than python3.10 ~/pi.py
1.04 ± 0.20 times faster than python3.9 ~/pi.py
Notes on porting a Next.js v14 app from Pages to App Router
March 2, 2024
0 comments React, JavaScript
Unfortunately, the app I ported from using the Pages Router to using App Router, is in a private repo. It's a Next.js static site SPA (Single Page App).
It's built with npm run build
and then exported so that the out/
directory is the only thing I need to ship to the CDN and it just works. There's a home page and a few dynamic routes whose slugs depend on an SQL query. So the SQL (PostgreSQL) connection, using knex
, has to be present when running npm run build
.
In no particular order, let's look at some differences
Build times
With caching
After running next build
a bunch of times, the rough averages are:
- Pages Router: 20.5 seconds
- App Router: 19.5 seconds
Without caching
After running rm -fr .next && next build
a bunch of times, the rough averages are:
- Pages Router: 28.5 seconds
- App Router: 31 seconds
Note
I have another SPA app that is built with vite
and wouter
and uses the heavy mantine
for the UI library. That SPA app does a LOT more in terms of components and pages etc. That one takes 9 seconds on average.
Static output
If you compare the generated out/_next/static/chunks
there's a strange difference.
Pages Router
360.0 KiB [##########################] /pages 268.0 KiB [################### ] 726-4194baf1eea221e4.js 160.0 KiB [########### ] ee8b1517-76391449d3636b6f.js 140.0 KiB [########## ] framework-5429a50ba5373c56.js 112.0 KiB [######## ] cdfd8999-a1782664caeaab31.js 108.0 KiB [######## ] main-930135e47dff83e9.js 92.0 KiB [###### ] polyfills-c67a75d1b6f99dc8.js 16.0 KiB [# ] 502-394e1f5415200700.js 8.0 KiB [ ] 0e226fb0-147f1e5268512885.js 4.0 KiB [ ] webpack-1b159842bd89504c.js
In total 1.2 MiB across 15 files.
App Router
428.0 KiB [##########################] 142-94b03af3aa9e6d6b.js 196.0 KiB [############ ] 975-62bfdeceb3fe8dd8.js 184.0 KiB [########### ] 25-aa44907f6a6c25aa.js 172.0 KiB [########## ] fd9d1056-e15083df91b81b75.js 164.0 KiB [########## ] ca377847-82e8fe2d92176afa.js 140.0 KiB [######## ] framework-aec844d2ccbe7592.js 116.0 KiB [####### ] a6eb9415-a86923c16860379a.js 112.0 KiB [####### ] 69-f28d58313be296c0.js 108.0 KiB [###### ] main-67e49f9e34a5900f.js 92.0 KiB [##### ] polyfills-c67a75d1b6f99dc8.js 44.0 KiB [## ] /app 24.0 KiB [# ] 1cc5f7f4-2f067a078d041167.js 24.0 KiB [# ] 250-47a2e67f72854c46.js 8.0 KiB [ ] /pages 4.0 KiB [ ] webpack-baa830a732d3dbbf.js 4.0 KiB [ ] main-app-f6b391c808310b44.js
In total 1.7 MiB across 27 files.
Notes
What makes the JS bundle large is most certainly due to using @primer/react
, @fullcalendar
, and react-chartjs-2
.
But why is the difference so large?
Dev start time
The way Next.js works, with npm run dev
, is that it starts a server at localhost:3000
and only when you request a URL does it compile something. It's essentially lazy and that's a good thing because in a bigger app, you might have too many different entries so it'd be silly to wait for all of them to compile if you might not use them all.
Pages Router
❯ npm run dev ... ✓ Ready in 1125ms ○ Compiling / ... ✓ Compiled / in 2.9s (495 modules)
App Router
❯ npm run dev ... ✓ Ready in 1201ms ○ Compiling / ... ✓ Compiled / in 3.7s (1023 modules)
Mind you, it almost always says "Ready in 1201ms" or but the other number, like "3.7s" in this example, that seems to fluctuate quite wildly. I don't know why.
Conclusion
Was it worth it? Yes and no.
I've never liked next/router
. With App Router you instead use next/navigation
which feels much more refined and simple. The old next/router
is still there which exposes a useRouter
hook which is still used for doing push
and replace
.
The getStaticPaths
and the getStaticProps
were not really that terrible in Pages Router.
I think the whole point of App Router is that you can get external data not only in getStaticProps
(or getServerSideProps
) but you can more freely go and get external data in places like layout.tsx
, which means less prop-drilling.
There are some nicer APIs with App Router. And it's the future of Next.js and how Vercel is pushing it forward.
How to avoid a count query in Django if you can
February 14, 2024
1 comment Django, Python
Suppose you have a complex Django QuerySet query that is somewhat costly (in other words slow). And suppose you want to return:
- The first N results
- A count of the total possible results
So your implementation might be something like this:
def get_results(queryset, fields, size):
count = queryset.count()
results = []
for record in queryset.values(*fields)[:size]
results.append(record)
return {"count": count, "results": results}
That'll work. If there are 1,234 rows in your database table that match those specific filters, what you might get back from this is:
>>> results = get_results(my_queryset, ("name", "age"), 5)
>>> results["count"]
1234
>>> len(results["results"])
5
Or, if the filters would only match 3 rows in your database table:
>>> results = get_results(my_queryset, ("name", "age"), 5)
>>> results["count"]
3
>>> len(results["results"])
3
Between your Python application and your database you'll see:
query 1: SELECT COUNT(*) FROM my_database WHERE ... query 2: SELECT name, age FROM my_database WHERE ... LIMIT 5
The problem with this is that, in the latter case, you had to send two database queries when all you needed was one.
If you knew it would only match a tiny amount of records, you could do this:
def get_results(queryset, fields, size):
- count = queryset.count()
results = []
for record in queryset.values(*fields)[:size]:
results.append(record)
+ count = len(results)
return {"count": count, "results": results}
But that is wrong. The count
would max out at whatever the size
is.
The solution is to try to avoid the potentially unnecessary .count()
query.
def get_results(queryset, fields, size):
count = 0
results = []
for i, record in enumerate(queryset.values(*fields)[: size + 1]):
if i == size:
# Alas, there are more records than the pagination
count = queryset.count()
break
count = i + 1
results.append(record)
return {"count": count, "results": results}
This way, you only incur one database query when there wasn't that much to find, but if there was more than what the pagination called for, you have to incur that extra database query.
Comparing different efforts with WebP in Sharp
October 5, 2023
0 comments Node, JavaScript
When you, in a Node program, use sharp
to convert an image buffer to a WebP buffer, you have an option of effort. The higher the number the longer it takes but the image it produces is smaller on disk.
I wanted to put some realistic numbers for this, so I wrote a benchmark, run on my Intel MacbookPro.
The benchmark
It looks like this:
async function e6() {
return await f("screenshot-1000.png", 6);
}
async function e5() {
return await f("screenshot-1000.png", 5);
}
async function e4() {
return await f("screenshot-1000.png", 4);
}
async function e3() {
return await f("screenshot-1000.png", 3);
}
async function e2() {
return await f("screenshot-1000.png", 2);
}
async function e1() {
return await f("screenshot-1000.png", 1);
}
async function e0() {
return await f("screenshot-1000.png", 0);
}
async function f(fp, effort) {
const originalBuffer = await fs.readFile(fp);
const image = sharp(originalBuffer);
const { width } = await image.metadata();
const buffer = await image.webp({ effort }).toBuffer();
return [buffer.length, width, { effort }];
}
Then, I ran each function in serial and measured how long it took. Then, do that whole thing 15 times. So, in total, each function is executed 15 times. The numbers are collected and the median (P50) is reported.
A 2000x2000 pixel PNG image
1. e0: 191ms 235KB 2. e1: 340.5ms 208KB 3. e2: 369ms 198KB 4. e3: 485.5ms 193KB 5. e4: 587ms 177KB 6. e5: 695.5ms 177KB 7. e6: 4811.5ms 142KB
What it means is that if you use {effort: 6}
the conversion of a 2000x2000 PNG took 4.8 seconds but the resulting WebP buffer became 142KB instead of the least effort which made it 235 KB.
This graph demonstrates how the (blue) time goes up the more effort you put in. And how the final size (red) goes down the more effort you put in.
A 1000x1000 pixel PNG image
1. e0: 54ms 70KB 2. e1: 60ms 66KB 3. e2: 65ms 61KB 4. e3: 96ms 59KB 5. e4: 169ms 53KB 6. e5: 193ms 53KB 7. e6: 1466ms 51KB
A 500x500 pixel PNG image
1. e0: 24ms 23KB 2. e1: 26ms 21KB 3. e2: 28ms 20KB 4. e3: 37ms 19KB 5. e4: 57ms 18KB 6. e5: 66ms 18KB 7. e6: 556ms 18KB
Conclusion
Up to you but clearly, {effort: 6}
is to be avoided if you're worried about it taking a huge amount of time to make the conversion.
Perhaps the takeaway is; that if you run these operations in the build step such that you don't have to ever do it again, it's worth the maximum effort. Beyond that, find a sweet spot for your particular environment and challenge.