A blog and website by Peter Bengtsson
You Should Watch, React, Firebase, JavaScript
I recently launched You Should Watch which is a mobile-friendly web app to have a to-watch list of movies and TV shows as well being able to quickly share the links if you want someone to "you should watch" it.
I'll be honest, much of the motivation of building that web app was to try a couple of newish technologies that I wanted to either improve on or just try for the first time. These are the interesting tech pillars that made it possible to launch this web app in what was maybe 20-30 hours of total time.
All the code for You Should Watch is here: https://github.com/peterbe/youshouldwatch-next
The cornerstone that made this app possible in the first place. The API is free for developers who don't intend to earn revenue on whatever project they build with it. More details in their FAQ.
The search functionality is important. The way it works is that you can do a "multisearch" which means it finds movies, TV shows, or people. Then, when you have each search result's id
and media_type
you can fetch a lot more information specifically. For example, that's how the page for a person displays things differently than the page for a movie.
In Next.js 13 you have a choice between regular pages
directory or an app
directory where every page (which becomes a URL) has to be called page.tsx
.
No judgment here. It was a bit cryptic to rewrap my brain on how this works. In particular, the head.tsx
is now different from the page.tsx
and since both, in server-side rendering, need some async data I have to duplicate the await getMediaData()
instead of being able to fetch it once and share with drop-drilling or context.
Wow! This was the most pleasant experience I've experienced in years. So polished and so much "just works". You sign in, with your GitHub auth, click to select the GitHub repo (that has a next.config.js
and package.json
etc) and you're done. That's it! Now, not only does every merged PR automatically (and fast!) get deployed, but you also get a preview deployment for every PR (which I didn't use).
I'm still using the free hobby tier but god forbid this app gets serious traffic, I'd just bump it up to $20/month which is cheap. Besides, the app is almost entirely CDN cacheable so only the search XHR backend would linearly increase its load with traffic I think.
Well done Vercel!
Not the first time I used Playwright but it was nice to return and start afresh. It definitely has improved in developer experience.
Previously I used npx
and the terminal to run tests, but this time I tried "Playwright Test for VSCode" which was just fantastic! There are some slightly annoying things in that I had to use the mouse cursor more than I'd hoped, but it genuinely helped me be productive. Playwright also has the ability to generate JS code based on me clicking around in a temporary incognito browser window. You do a couple of things in the browser then paste in the generated source code into tests/basics.spec.ts
and do some manual tidying up. To run the debugger like that, one simply types pnpm dlx playwright codegen
pnpm
It seems hip and a lot of people seem to recommend it. Kinda like yarn
was hip and often recommended over npm
(me included!).
Sure it works and it installs things fast but is it noticeable? Not really. Perhaps it's 4 seconds when it would have been 5 seconds with npm
. Apparently pnpm
does clever symlinking to avoid a disk-heavy node_modules/
but does it really matter ...much?
It's still large:
❯ du -sh node_modules
468M node_modules
A disadvantage with pnpm
is that GitHub Dependabot currently doesn't support it :(
An advantage with pnpm
is that pnpm up -i --latest
is great interactive CLI which works like yarn upgrade-interactive --latest
just
just
is like make
but written in Rust. Now I have a justfile
in the root of the repo and can type shortcut commands like just dev
or just emu[TAB]
(to tab autocomplete).
In hindsight, my justfile
ended up being just a list of pnpm run ...
commands but the idea is that just
would be for all and any command under one roof.
End of the day, it becomes a nifty little file of "recipes" of useful commands and you can easily string them together. For example just lint
is the combination of typing pnpm run prettier:check
and pnpm run tsc
and pnpm run lint
.
A gorgeously simple looking pure-CSS framework. Yes, it's very limited in components and I don't know how well it "tree shakes" but it's so small and so neat that it had everything I needed.
My favorite React component library is Mantine but I definitely love the piece of mind that Pico.css is just CSS so you're not A) stuck with React forever, and B) not unnecessary JS code that slows things down.
Good old Firebase. The bestest and easiest way to get a reliable and realtime database that is dirt cheap, simple, and has great documentation. I do regret not trying Supabase but I knew that getting the OAuth stuff to work with Google on a custom domain would be tricky so I stayed with Firebase.
react-lite-youtube-embed
A port of Paul Irish's Lite YouTube Embed which makes it easy to display YouTube thumbnails in a web performant way. All you have to do is:
import LiteYouTubeEmbed from "react-lite-youtube-embed";
<LiteYouTubeEmbed
id={youtubeVideo.id}
title={youtubeVideo.title} />
It's amazing how much time these tools saved compared to just years ago. I could build a fully working side-project with automation and high quality entirely thanks to great open source or well-tuned proprietary components, in just about one day if you sum up the hours.
At the time of writing, I don't know if this is the optimal way, but after some trial and error, I got it working.
This example demonstrates a hook that gives you the current value of the ?view=...
(or a default) and a function you can call to change it so that ?view=before
becomes ?view=after
.
In NextJS v13 with the pages
directory:
import { useRouter } from "next/router";
export function useNamesView() {
const KEY = "view";
const DEFAULT_NAMES_VIEW = "buttons";
const router = useRouter();
let namesView: Options = DEFAULT_NAMES_VIEW;
const raw = router.query[KEY];
const value = Array.isArray(raw) ? raw[0] : raw;
if (value === "buttons" || value === "table") {
namesView = value;
}
function setNamesView(value: Options) {
const [asPathRoot, asPathQuery = ""] = router.asPath.split("?");
const params = new URLSearchParams(asPathQuery);
params.set(KEY, value);
const asPath = `${asPathRoot}?${params.toString()}`;
router.replace(asPath, asPath, { shallow: true });
}
return { namesView, setNamesView };
}
In NextJS v13 with the app
directory.
import { useRouter, useSearchParams, usePathname } from "next/navigation";
type Options = "buttons" | "table";
export function useNamesView() {
const KEY = "view";
const DEFAULT_NAMES_VIEW = "buttons";
const router = useRouter();
const searchParams = useSearchParams();
const pathname = usePathname();
let namesView: Options = DEFAULT_NAMES_VIEW;
const value = searchParams.get(KEY);
if (value === "buttons" || value === "table") {
namesView = value;
}
function setNamesView(value: Options) {
const params = new URLSearchParams(searchParams);
params.set(KEY, value);
router.replace(`${pathname}?${params}`);
}
return { namesView, setNamesView };
}
The trick is that you only want to change 1 query string value and respect whatever was there before. So if the existing URL was /page?foo=bar
and you want that to become /page?foo=bar&and=also
you have to consume the existing query string and you do that with:
const searchParams = useSearchParams();
...
const params = new URLSearchParams(searchParams);
params.set('and', 'also')
React, Node, Nginx, JavaScript
UPDATE: Feb 21, 2022: The original blog post didn't mention the caching of custom headers. So warm cache hits would lose Cache-Control
from the cold cache misses. Code updated below.
I know I know. The title sounds ridiculous. But it's not untrue. I managed to make my NextJS 20x faster by allowing the Express server, which handles NextJS, to cache the output in memory. And cache invalidation is not a problem.
My personal blog is a stack of layers:
KeyCDN --> Nginx (on my server) -> Express (same server) -> NextJS (inside Express)
And inside the NextJS code, to get the actual data, it uses HTTP to talk to a local Django server to get JSON based on data stored in a PostgreSQL database.
The problems I have are as follows:
critters
which computes the critical CSS inline and lazy-loads the rest. I really like NextJS and it's a great developer experience. There are definitely many things I don't like about it, but that's more because my site isn't SPA'y enough to benefit from much of what NextJS has to offer. By the way, I blogged about rewriting my site in NextJS last year.
critters
If you're reading my blog right now in a desktop browser, right-click and view source and you'll find this:
<head>
<style>
*,:after,:before{box-sizing:inherit}html{box-sizing:border-box}inpu...
... about 19k of inline CSS...
</style>
<link rel="stylesheet" href="/_next/static/css/fdcd47c7ff7e10df.css" data-n-g="" media="print" onload="this.media='all'">
<noscript><link rel="stylesheet" href="/_next/static/css/fdcd47c7ff7e10df.css"></noscript>
...
</head>
It's great for web performance because a <link rel="stylesheet" href="css.css">
is a render-blocking thing and it makes the site feel slow on first load. I wish I didn't need this, but it comes from my lack of CSS styling skills to custom hand-code every bit of CSS and instead, I rely on a bloated CSS framework which comes as a massive kitchen sink.
To add critical CSS optimization in NextJS, you add:
experimental: { optimizeCss: true },
inside your next.config.js
. Easy enough, but it slows down my site by a factor of ~80ms to ~230ms on my Intel Macbook per page rendered.
So see, if it wasn't for this need of critical CSS inlining, NextJS would be about ~80ms per page and that includes getting all the data via HTTP JSON for each page too.
My server.mjs
looks like this (simplified):
import next from "next";
import renderCaching from "./middleware/render-caching.mjs";
const app = next({ dev });
const handle = app.getRequestHandler();
app
.prepare()
.then(() => {
const server = express();
// For Gzip and Brotli compression
server.use(shrinkRay());
server.use(renderCaching);
server.use(handle);
// Use the rollbar error handler to send exceptions to your rollbar account
if (rollbar) server.use(rollbar.errorHandler());
server.listen(port, (err) => {
if (err) throw err;
console.log(`> Ready on http://localhost:${port}`);
});
})
And the middleware/render-caching.mjs
looks like this:
import express from "express";
import QuickLRU from "quick-lru";
const router = express.Router();
const cache = new QuickLRU({ maxSize: 1000 });
router.get("/*", async function renderCaching(req, res, next) {
if (
req.path.startsWith("/_next/image") ||
req.path.startsWith("/_next/static") ||
req.path.startsWith("/search")
) {
return next();
}
const key = req.url;
if (cache.has(key)) {
res.setHeader("x-middleware-cache", "hit");
const [body, headers] = cache.get(key);
Object.entries(headers).forEach(([key, value]) => {
if (key !== "x-middleware-cache") res.setHeader(key, value);
});
return res.status(200).send(body);
} else {
res.setHeader("x-middleware-cache", "miss");
}
const originalEndFunc = res.end.bind(res);
res.end = function (body) {
if (body && res.statusCode === 200) {
cache.set(key, [body, res.getHeaders()]);
// console.log(
// `HEAP AFTER CACHING ${(
// process.memoryUsage().heapUsed /
// 1024 /
// 1024
// ).toFixed(1)}MB`
// );
}
return originalEndFunc(body);
};
next();
});
export default router;
It's far from perfect and I only just coded this yesterday afternoon. My server runs a single Node process so the max heap memory would theoretically be 1,000 x the average size of those response bodies. If you're worried about bloating your memory, just adjust the QuickLRU
to something smaller.
In my basic version, I chose this cache key:
const key = req.url;
but that means that http://localhost:3000/foo?a=1
is different from http://localhost:3000/foo?b=2
which might be a mistake if you're certain that no rendering ever depends on a query string.
But this is totally up to you! For example, suppose that you know your site depends on the darkmode
cookie, you can do something like this:
const key = `${req.path} ${req.cookies['darkmode']==='dark'} ${rec.headers['accept-language']}`
Or,
const key = req.path.startsWith('/search') ? req.url : req.path
As soon as I launched this code, I watched the log files, and voila!:
::ffff:127.0.0.1 [18/Feb/2022:12:59:36 +0000] GET /about HTTP/1.1 200 - - 422.356 ms ::ffff:127.0.0.1 [18/Feb/2022:12:59:43 +0000] GET /about HTTP/1.1 200 - - 1.133 ms
Cool. It works. But the problem with a simple LRU cache is that it's sticky. And it's stored inside a running process's memory. How is the Express server middleware supposed to know that the content has changed and needs a cache purge? It doesn't. It can't know. The only one that knows is my Django server which accepts the various write operations that I know are reasons to purge the cache. For example, if I approve a blog post comment or an edit to the page, it triggers the following (simplified) Python code:
import requests
def cache_purge(url):
if settings.PURGE_URL:
print(requests.get(settings.PURGE_URL, json={
pathnames: [url]
}, headers={
"Authorization": f"Bearer {settings.PURGE_SECRET}"
})
if settings.KEYCDN_API_KEY:
api = keycdn.Api(settings.KEYCDN_API_KEY)
print(api.delete(
f"zones/purgeurl/{settings.KEYCDN_ZONE_ID}.json",
{"urls": [url]}
))
Now, let's go back to the simplified middleware/render-caching.mjs
and look at how we can purge from the LRU over HTTP POST:
const cache = new QuickLRU({ maxSize: 1000 })
router.get("/*", async function renderCaching(req, res, next) {
// ... Same as above
});
router.post("/__purge__", async function purgeCache(req, res, next) {
const { body } = req;
const { pathnames } = body;
try {
validatePathnames(pathnames)
} catch (err) {
return res.status(400).send(err.toString());
}
const bearer = req.headers.authorization;
const token = bearer.replace("Bearer", "").trim();
if (token !== PURGE_SECRET) {
return res.status(403).send("Forbidden");
}
const purged = [];
for (const pathname of pathnames) {
for (const key of cache.keys()) {
if (
key === pathname ||
(key.startsWith("/_next/data/") && key.includes(`${pathname}.json`))
) {
cache.delete(key);
purged.push(key);
}
}
}
res.json({ purged });
});
What's cool about that is that it can purge both the regular HTML URL and it can also purge those _next/data/
URLs. Because when NextJS can hijack the <a>
click, it can just request the data in JSON form and use existing React components to re-render the page with the different data. So, in a sense, GET /_next/data/RzG7kh1I6ZEmOAPWpdA7g/en/plog/nextjs-faster-with-express-caching.json?oid=nextjs-faster-with-express-caching
is the same as GET /plog/nextjs-faster-with-express-caching
because of how NextJS works. But in terms of content, they're the same. But worth pointing out that the same piece of content can be represented in different URLs.
Another thing to point out is that this caching is specifically about individual pages. In my blog, for example, the homepage is a mix of the 10 latest entries. But I know this within my Django server so when a particular blog post has been updated, for some reason, I actually send out a bunch of different URLs to the purge where I know its content will be included. It's not perfect but it works pretty well.
The hardest part about caching is cache invalidation. It's usually the inner core of a crux. Sometimes, you're so desperate to survive a stampeding herd problem that you don't care about cache invalidation but as a compromise, you just set the caching time-to-live short.
But I think the most important tenant of good caching is: have full control over it. I.e. don't take it lightly. Build something where you can fully understand and change how it works exactly to your specific business needs.
This idea of letting Express cache responses in memory isn't new but I didn't find any decent third-party solution on NPMJS that I liked or felt fully comfortable with. And I needed to tailor exactly to my specific setup.
Go forth and try it out on your own site! Not all sites or apps need this at all, but if you do, I hope I have inspired a foundation of a solution.
So common now. You have focus inside a <textarea>
and you want to submit the form. By default, your only choice is to reach for the mouse and click the "Submit" button below, or use Tab to tab to the submit button with the keyboard.
Many sites these days support Command-Enter as an option. Here's how you implement that in React:
const textareaElement = textareaRef.current;
useEffect(() => {
const listener = (event: KeyboardEvent) => {
if (event.key === "Enter" && event.metaKey) {
formSubmit();
}
};
if (textareaElement) {
textareaElement.addEventListener("keydown", listener);
}
return () => {
if (textareaElement) {
textareaElement.removeEventListener("keydown", listener);
}
};
}, [textareaElement, formSubmit]);
The important bit is the event.key === "Enter" && event.metaKey
statement. I hope it helps.
React, Django, Node, JavaScript
My personal blog was a regular Django website with jQuery (later switched to Cash) for dynamic bits. In December 2021 I rewrote it in NextJS. It was a fun journey and NextJS is great but it's really not without some regrets.
Some flashpoints for note and comparison:
The way infinitely nested comments are rendered is isomorphic now. Before I had to code it once as a Jinja2 template thing and once as a Cash (a fork of jQuery) thing. That's the nice and the promise of JavaScript React and server-side rendering.
The total JS payload is now ~111KB in 16 files. It used to be ~36KB in 7 files. :(
Before
After
Like any website, the web pages are made up from A) getting the raw data from a database, B) rendering that data in HTML.
I didn't want to rewrite all the database queries in Node (inside getServerSideProps
).
What I did was I moved all the data gathering Django code and put them under a /api/v1/
prefix publishing simple JSON blobs. Then this is exposed on 127.0.0.1:3000
which the Node server fetches. And I wired up that that API endpoint so I can debug it via the web too. E.g. /api/v1/plog/sort-a-javascript-array-by-some-boolean-operation
Now, all I have to do is write some TypeScript interfaces that hopefully match the JSON that comes from Django. For example, here's the getServerSideProps
code for getting the data to this page:
const url = `${API_BASE}/api/v1/plog/`;
const response = await fetch(url);
if (!response.ok) {
throw new Error(`${response.status} on ${url}`);
}
const data: ServerData = await response.json();
const { groups } = data;
return {
props: {
groups,
},
};
I like this pattern! Yes, there are overheads and Node could talk directly to PostgreSQL but the upside is decoupling. And with good outside caching, performance never matters.
I considered full-blown static generation, but it's not an option. My little blog only has about 1,400 blog posts but you can also filter by tags and combinations of tags and pagination of combinations of tags. E.g. /oc-JavaScript/oc-Python/p3 So the total number of pages is probably in the tens of thousands.
So, server-side rendering it is. To accomplish that I set up a very simple Express server. It proxies some stuff over to the Django server (e.g. /rss.xml
) and then lets NextJS handle the rest.
import next from "next";
import express from "express";
const app = next();
const handle = app.getRequestHandler();
app
.prepare()
.then(() => {
const server = express();
server.use(handle);
server.listen(port, (err) => {
if (err) throw err;
console.log(`> Ready on http://localhost:${port}`);
});
})
Now, my site is behind a CDN. And technically, it's behind Nginx too where I do some proxy_pass
in-memory caching as a second line of defense.
Requests come in like this:
proxy_pass
) next().getRequestHandler()
And I set Cache-Control
in res.setHeader("Cache-Control", "public,max-age=86400")
from within the getServerSideProps
functions in the src/pages/**/*.tsx
files. And once that's set, the response will be cached both in Nginx and in the CDN.
Any caching is tricky when you need to do revalidation. Especially when you roll out a new central feature in the core bundle. But I quite like this pattern of a slow-rolling upgrade as individual pages eventually expire throughout the day.
This is a nasty bug with this and I don't yet know how to solve it. Client-side navigation is dependent of hashing. So loading this page, when done with client-side navigation, becomes /_next/data/2ps5rE-K6E39AoF4G6G-0/en/plog.json
(no, I don't know how that hashed URL is determined). But if a new deployment happens, the new URL becomes /_next/data/UhK9ANa6t5p5oFg3LZ5dy/en/plog.json
so you end up with a 404 because you started on a page based on an old JavaScript bundle, that is now invalid.
Thankfully, NextJS handles it quite gracefully by throwing an error on the 404 so it proceeds with a regular link redirect which takes you away from the old page.
Next has a built-in <Link>
component that you use like this:
import Link from "next/link";
...
<Link href={"/plog/" + post.oid}>
{post.title}
</Link>
Now, clicking any of those links will automatically enable client-side routing. Thankfully, it takes care of preloading the necessary JavaScript (and CSS) simply by hovering over the link, so that when you eventually click it just needs to do an XHR request to get the JSON necessary to be able to render the page within the loaded app (and then do the pushState
stuff to change the URL accordingly).
It sounds good in theory but it kinda sucks because unless you have a really good Internet connection (or could be you hit upon a CDN-cold URL), nothing happens when you click. This isn't NextJS's fault, but I wonder if it's actually horribly for users.
Yes, it sucks that a user clicks something but nothing happens. (I think it would be better if it was a button-press and not a link because buttons feel more like an app whereas links have deeply ingrained UX expectations). But most of the time, it's honestly very fast and when it works it's a nice experience. It's a great piece of functionality for more app'y sites, but less good for websites whose most of the traffic comes from direct links or Google searches.
Critical inline CSS is critical (pun intended) for web performance. Especially on my poor site where I depend on a bloated (and now ancient) CSS framework called Semantic-UI. Without inline CSS, the minified CSS file would become over 200KB.
In NextJS, to enable inline critical CSS loading you just need to add this to your next.config.js
:
experimental: { optimizeCss: true },
and you have to add critters
to your package.json
. I've found some bugs with it but nothing I can't work around.
I'm very familiar and experienced with React but NextJS is new to me. I've managed to miss it all these years. Until now. So there's still a lot to learn. With other frameworks, I've always been comfortable that I don't actually understand how Webpack and Babel work (internally) but at least I understood when and how I was calling/depending on it. Now, with NextJS there's a lot of abstracted magic that I don't quite understand. It's hard to let go of that. It's hard to get powerful tools that are complex and created by large groups of people and understand it all too. If you're desperate to understand exactly how something works, you inevitably have to scale back the amount of stuff you're leveraging. (Note, it might be different if it's absolute core to what you do for work and hack on for 8 hours a day)
The JavaScript bundles in NextJS lazy-load quite decently but it's definitely more bloat than it needs to be. It's up to me to fix it, partially, because much of the JS code on my site is for things that technically can wait such as the interactive commenting form and the auto-complete search.
But here's the rub; my site is not an app. Most traffic comes from people doing a Google search, clicking on my page, and then bugger off. It's quite static that way and who am I to assume that they'll stay and click around and reuse all that loaded JavaScript code.
With that said; I'm going to start an experiment to rewrite the site again in Remix.
Suppose you have one of those React apps that lazy-load some chunk. It just basically means it injects a .js
static asset URL into the DOM and once it's downloaded by the browser, it carries on the React rendering with the new code loaded. Well, what if the network is really slow? In local development, it can be hard to simulate this. You can mess with the browser's Devtools to try to slow down the network, but even that can be too fast sometimes.
What I often do is, I take this:
const SettingsApp = React.lazy(() => import("./app"));
...and change it to this:
const SettingsApp = React.lazy(() =>
import("./app").then((module) => {
return new Promise((resolve) => {
setTimeout(() => {
resolve(module as any);
}, 10000);
});
})
);
Now, it won't load that JS chunk until 10 seconds later. Only temporarily, in local development.
I know it's admittedly just a hack but it's nifty. Just don't forget to undo it when you're done simulating your snail-speed web app.
PS. That resolve(module as any);
is for TypeScript. You can just change that to resolve(module);
if it's regular JavaScript.