Page 3
How to run a GitHub Action workflow step if a file exists
April 24, 2023
2 comments GitHub
Suppose you have a GitHub Action workflow that does some computation and a possible outcome is that file comes into existence. How do you run a follow-up step based on whether a file was created?
tl;dr
- name: Is file created?
if: ${{ hashFiles('test.txt') != '' }}
run: echo "File exists"
The "wrong" way
Technically, there's no wrong way, but an alternative might be to rely on exit codes. This would work.
- name: Check if file was created
run: |
if [ -f test.txt ]; then
echo "File exists"
exit 1
else
echo "File does not exist"
fi
- name: Did the last step fail?
if: ${{ failure() }}
run: echo "Last step failed, so file must have maybe been created"
The problem with this is that not only leaves a red ❌ in the workflow logs, but it could also lead to false positives. For example, if the step that might create a file is non-trivial, you don't want to lump the creation of the file with a possible bug in your code.
A use case
What I needed this for was a complex script that was executed to find broken links in a web app. If there were broken links, only then do I want to file a new issue about that. If the script failed for some reason, you want to know that and work on fixing whatever its bug might be. It looked like this:
- name: Run broken link check
run: |
script/check-links.js broken_links.md
- name: Create issue from file
if: ${{ hashFiles('broken_links.md') != '' }}
uses: peter-evans/create-issue-from-file@433e51abf769039ee20ba1293a088ca19d573b7f
with:
token: ${{ env.GITHUB_TOKEN }}
title: More than one zero broken links found
content-filepath: ./broken_links.md
repository: ${{ env.REPORT_REPOSITORY }}
labels: ${{ env.REPORT_LABEL }}
That script/check-links.js
script is given an argument which is the name of the file to write to if it did indeed find any broken links. If there were any, it generates a snippet of Markdown about them which is the body of filed new issue.
Demo
To be confident this works, I created a dummy workflow in a test repo to test. It looks like this: .github/workflows/maybe-fail.yml
Automatically 'npm install'
April 6, 2023
0 comments Node, JavaScript
I implemented this at work recently and although it felt like a hack, I've come to like it and it's been very helpful to our many contributors.
As (Node) engineers, we know that you should keep your node_modules
up-to-date by running npm install
periodically or every time you git pull
from the upstream. It could be that some package got upgraded last night since you git pulled last time.
But not everyone remembers to run npm install
often enough. They might do git pull origin main && npm start
and now the code that starts up depends on some latest version that was upgraded in package.json
and package-lock.json
.
How we solved it was that we added this script:
node script/cmp-files.js package-lock.json .installed.package-lock.json || npm install && cp package-lock.json .installed.package-lock.json
And it's hooked up as a script in package.json
called prestart
:
"scripts": { ... "prestart": "node script/cmp-files.js ...", ... }
Now, every time you run npm start
to start up the local development server, it will run that piece of bash. No more having to remember to run npm install
after every git pull
.
A note on performance
The npm install
command is fast when all packages are already updated. You can see it with:
# First time
$ npm install
# Second time when nothing should happen
$ time npm install
...
2.53s user 0.37s system 134% cpu 2.166 total
So it only takes 2 seconds. Not bad.
$ time node script/cmp-files.js package-lock.json .installed.package-lock.json
...
0.08s user 0.03s system 100% cpu 0.110 total
But 0.08 seconds is better :)
The comparison script
The cmp-files.js
script looks like this:
#!/usr/bin/env node
// Given N files. Exit 0 if they all exist and are identical in content.
import fs from 'fs'
import { program } from 'commander'
program.description('Compare N files').arguments('[files...]', '').parse(process.argv)
main(program.args)
function main(files) {
if (files.length < 2) throw new Error('Must be at least 2 files')
try {
const contents = files.map((file) => fs.readFileSync(file, 'utf-8'))
if (new Set(contents).size > 1) {
process.exit(1)
}
} catch (error) {
if (error.code === 'ENOENT') {
process.exit(1)
} else {
throw error
}
}
}
The file .installed.package-lock.json
file is added to the repo's .gitignore
Note; given how well this works for running before npm start
we can probably add this to a post-checkout git
hook too.
Benchmarking npm install with or without audit
February 23, 2023
0 comments Node, JavaScript
By default, running npm install
will do a security audit of your installed packages. That audit is fast but it still takes a bit of time. To disable it you can either add --no-audit
or you can...:
❯ cat .npmrc
audit=false
But how much does the audit take when running npm install
? To find out, I wrote this:
import random
import statistics
import subprocess
import time
from collections import defaultdict
def f1():
subprocess.check_output("npm install".split())
def f2():
subprocess.check_output("npm install --no-audit".split())
functions = f1, f2
times = defaultdict(list)
for i in range(25):
f = random.choice(functions)
t0 = time.time()
f()
t1 = time.time()
times[f.__name__].append(t1 - t0)
time.sleep(5)
for f_name in sorted(times.keys()):
print(
f_name,
f"mean: {statistics.mean(times[f_name]):.1f}s".ljust(10),
f"median: {statistics.median(times[f_name]):.1f}s",
)
Note how it runs a lot of times in case there are network hiccups and it sleeps between each run just to spread out the experiment over a longer period of time. And the results are:
f1 mean: 2.81s median: 2.57s f2 mean: 2.25s median: 2.21s
Going by the median time, the --no-audit
makes the npm install
16% faster. If you look at the mean time dropping the --no-audit
can make it 25% faster.
How to intercept and react to non-zero exits in bash
February 23, 2023
2 comments Bash, GitHub
Inside a step in a GitHub Action, I want to run a script, and depending on the outcome of that, maybe do some more things. Essentially, if the script fails, I want to print some extra user-friendly messages, but the whole Action should still fail with the same exit code.
In pseudo-code, this is what I want to achieve:
exit_code = that_other_script() if exit_code > 0: print("Extra message if it failed") exit(exit_code)
So here's how to do that with bash
:
# If it's not the default, make it so that it proceeds even if
# any one line exits non-zero
set +e
./script/update-internal-links.js --check
exit_code=$?
if [ $exit_code != 0 ]; then
echo "Extra message here informing that the script failed"
exit $exit_code
fi
The origin, for me, at the moment, was that I had a GitHub Action where it calls another script that might fail. If it fails, I wanted to print out a verbose extra hint to whoever looks at the output. Steps in GitHub Action runs with set -e
by default I think, meaning that if anything goes wrong in the step it leaves the step and runs those steps with if: ${{ failure() }}
next.
The technology behind You Should Watch
January 28, 2023
0 comments You Should Watch, React, Firebase, JavaScript
I recently launched You Should Watch which is a mobile-friendly web app to have a to-watch list of movies and TV shows as well being able to quickly share the links if you want someone to "you should watch" it.
I'll be honest, much of the motivation of building that web app was to try a couple of newish technologies that I wanted to either improve on or just try for the first time. These are the interesting tech pillars that made it possible to launch this web app in what was maybe 20-30 hours of total time.
All the code for You Should Watch is here: https://github.com/peterbe/youshouldwatch-next
The Movie Database API
The cornerstone that made this app possible in the first place. The API is free for developers who don't intend to earn revenue on whatever project they build with it. More details in their FAQ.
The search functionality is important. The way it works is that you can do a "multisearch" which means it finds movies, TV shows, or people. Then, when you have each search result's id
and media_type
you can fetch a lot more information specifically. For example, that's how the page for a person displays things differently than the page for a movie.
Next.js and the new App dir
In Next.js 13 you have a choice between regular pages
directory or an app
directory where every page (which becomes a URL) has to be called page.tsx
.
No judgment here. It was a bit cryptic to rewrap my brain on how this works. In particular, the head.tsx
is now different from the page.tsx
and since both, in server-side rendering, need some async data I have to duplicate the await getMediaData()
instead of being able to fetch it once and share with drop-drilling or context.
Vercel deployment
Wow! This was the most pleasant experience I've experienced in years. So polished and so much "just works". You sign in, with your GitHub auth, click to select the GitHub repo (that has a next.config.js
and package.json
etc) and you're done. That's it! Now, not only does every merged PR automatically (and fast!) get deployed, but you also get a preview deployment for every PR (which I didn't use).
I'm still using the free hobby tier but god forbid this app gets serious traffic, I'd just bump it up to $20/month which is cheap. Besides, the app is almost entirely CDN cacheable so only the search XHR backend would linearly increase its load with traffic I think.
Well done Vercel!
Playwright and VS Code
Not the first time I used Playwright but it was nice to return and start afresh. It definitely has improved in developer experience.
Previously I used npx
and the terminal to run tests, but this time I tried "Playwright Test for VSCode" which was just fantastic! There are some slightly annoying things in that I had to use the mouse cursor more than I'd hoped, but it genuinely helped me be productive. Playwright also has the ability to generate JS code based on me clicking around in a temporary incognito browser window. You do a couple of things in the browser then paste in the generated source code into tests/basics.spec.ts
and do some manual tidying up. To run the debugger like that, one simply types pnpm dlx playwright codegen
pnpm
It seems hip and a lot of people seem to recommend it. Kinda like yarn
was hip and often recommended over npm
(me included!).
Sure it works and it installs things fast but is it noticeable? Not really. Perhaps it's 4 seconds when it would have been 5 seconds with npm
. Apparently pnpm
does clever symlinking to avoid a disk-heavy node_modules/
but does it really matter ...much?
It's still large:
❯ du -sh node_modules
468M node_modules
A disadvantage with pnpm
is that GitHub Dependabot currently doesn't support it :(
An advantage with pnpm
is that pnpm up -i --latest
is great interactive CLI which works like yarn upgrade-interactive --latest
just
just
is like make
but written in Rust. Now I have a justfile
in the root of the repo and can type shortcut commands like just dev
or just emu[TAB]
(to tab autocomplete).
In hindsight, my justfile
ended up being just a list of pnpm run ...
commands but the idea is that just
would be for all and any command under one roof.
End of the day, it becomes a nifty little file of "recipes" of useful commands and you can easily string them together. For example just lint
is the combination of typing pnpm run prettier:check
and pnpm run tsc
and pnpm run lint
.
Pico.css
A gorgeously simple looking pure-CSS framework. Yes, it's very limited in components and I don't know how well it "tree shakes" but it's so small and so neat that it had everything I needed.
My favorite React component library is Mantine but I definitely love the piece of mind that Pico.css is just CSS so you're not A) stuck with React forever, and B) not unnecessary JS code that slows things down.
Firebase
Good old Firebase. The bestest and easiest way to get a reliable and realtime database that is dirt cheap, simple, and has great documentation. I do regret not trying Supabase but I knew that getting the OAuth stuff to work with Google on a custom domain would be tricky so I stayed with Firebase.
react-lite-youtube-embed
A port of Paul Irish's Lite YouTube Embed which makes it easy to display YouTube thumbnails in a web performant way. All you have to do is:
import LiteYouTubeEmbed from "react-lite-youtube-embed";
<LiteYouTubeEmbed
id={youtubeVideo.id}
title={youtubeVideo.title} />
In conclusion
It's amazing how much time these tools saved compared to just years ago. I could build a fully working side-project with automation and high quality entirely thanks to great open source or well-tuned proprietary components, in just about one day if you sum up the hours.
Announcing: You Should Watch
January 27, 2023
3 comments You Should Watch
tl;dr You Should Watch is a mobile-friendly web app for remembering and sharing movies and TV shows you should watch. Like a to-do list to open when you finally sit down with the remote in your hand.
I built this in the holidays in some spare afternoons and evenings around the 2022 new years. The idea wasn't new in my head because it's an actual itch I've often had: what on earth should I watch now tonight?! Oftentimes you read or hear about great movies or TV shows but a lot of those memories are long gone by the time the kids have been put to sleep and you have control of the remote.
It's not rocket science. It's a neat little app that works great on your phone browser but also works as a website on your computer. You don't have to sign in but if you do, your list outlives the memory of your device. I.e. your selections get saved in the cloud and you can pick back up whenever you're on a different device. If you do decide to sign in, it currently uses Google for that. Your list is anonymized and even I, the owner of the database, can't tell which movies and TV shows you select.
If you do sign in to save your list, every time you check a movie or TV show off it goes into your archive. That can be useful for your next dinner party or date or cookout when you're scrambling to answer "Seen any good shows recently?".
One feature that I've personally enjoyed is the list of recommendations that each TV show or movie has. It's not a perfect list but it's fun and useful. Suppose you can't even think of what to watch and just want inspiration, start off by finding a movie like (but don't want to watch right now), then click on it and scroll down to the "Recommendations". Even if your next movie or TV show isn't on that list, perhaps clicking on something similar will take you to the next page where the right inspiration might be found. Try it.
It's free and will always be free. It was fun to build and thankfully free to run so it won't go away. I hope you enjoy it and get value from it. Please please share your thoughts and constructive criticism.
Try it on https://ushdwat.ch/
Some pictures
Your Watch list (mobile)
Search results
Navigating by "Recommendations"
"Add to Home Screen" on iOS Safari