hashin 0.15.0 now copes nicely with under_scores
June 15, 2020
0 comments Python
tl;dr hashin
0.15.0 makes package comparison agnostic to underscore or hyphens
See issue #116 for a fuller story. Basically, now it doesn't matter if you write...
hashin python_memcached
...or...
hashin python-memcached
And the same can be said about the contents of your requirements.txt
file. Suppose it already had something like this:
python_memcached==1.59 \ --hash=sha256:4dac64916871bd35502 \ --hash=sha256:a2e28637be13ee0bf1a8
and you type hashin python-memcached
it will do the version comparison on these independent of the underscore or hyphen.
Thank @caphrim007 who implemented this for the benefit of Renovate.
./bin/huey-isnt-running.sh - A bash script to prevent lurking ghosts
June 10, 2020
0 comments Python, Linux, Bash
tl;dr; Here's a useful bash script to avoid starting something when its already running as a ghost process.
Huey is a great little Python library for doing background tasks. It's like Celery but much lighter, faster, and easier to understand.
What cost me almost an hour of hair-tearing debugging today was that I didn't realize that a huey
daemon process had gotten stuck in the background with code that wasn't updating as I made changes to the tasks.py
file in my project. I just couldn't understand what was going on.
The way I start my project is with honcho which is a Python Foreman clone. The Procfile
looks something like this:
elasticsearch: cd /Users/peterbe/dev/PETERBECOM/elasticsearch-7.7.0 && ./bin/elasticsearch -q
web: ./bin/run.sh web
minimalcss: cd minimalcss && PORT=5000 yarn run start
huey: ./manage.py run_huey --flush-locks --huey-verbose
adminui: cd adminui && yarn start
pulse: cd pulse && yarn run dev
And you start that with simply typing:
honcho start
When you Ctrl-C, it kills all those processes but somehow somewhere it doesn't always kill everything. Restarting the computer isn't a fun alternative.
So, to prevent my sanity from draining I wrote this script:
#!/usr/bin/env bash
set -eo pipefail
# This is used to make sure that before you start huey,
# there isn't already one running the background.
# It has happened that huey gets lingering stuck as a
# ghost and it's hard to notice it sitting there
# lurking and being weird.
bad() {
echo "Huey is already running!"
exit 1
}
good() {
echo "Huey is NOT already running"
exit 0
}
ps aux | rg huey | rg -v 'rg huey' | rg -v 'huey-isnt-running.sh' && bad || good
(If you're wondering what rg
is; it's short for ripgrep
)
And I change my Procfile
accordingly:
-huey: ./manage.py run_huey --flush-locks --huey-verbose
+huey: ./bin/huey-isnt-running.sh && ./manage.py run_huey --flush-locks --huey-verbose
There really isn't much rocket science or brain surgery about this blog post but I hope it inspires someone who's been in similar trenches that a simple bash script can make all the difference.
Check your email addresses in Python, as a whole
May 22, 2020
0 comments Python, MDN
So recently, in MDN, we changed the setting WELCOME_EMAIL_FROM
. Seems harmless right? Wrong, it failed horribly in runtime and we didn't notice until it was in production. Here's the traceback:
SMTPSenderRefused: (552, b"5.1.7 The sender's address was syntactically invalid.\n5.1.7 see : http://support.socketlabs.com/kb/84 for more information.", '=?utf-8?q?Janet?=') (8 additional frame(s) were not displayed) ... File "newrelic/api/function_trace.py", line 151, in literal_wrapper return wrapped(*args, **kwargs) File "django/core/mail/message.py", line 291, in send return self.get_connection(fail_silently).send_messages([self]) File "django/core/mail/backends/smtp.py", line 110, in send_messages sent = self._send(message) File "django/core/mail/backends/smtp.py", line 126, in _send self.connection.sendmail(from_email, recipients, message.as_bytes(linesep='\r\n')) File "python3.8/smtplib.py", line 871, in sendmail raise SMTPSenderRefused(code, resp, from_addr) SMTPSenderRefused: (552, b"5.1.7 The sender's address was syntactically invalid.\n5.1.7 see : http://support.socketlabs.com/kb/84 for more information.", '=?utf-8?q?Janet?=')
Yikes!
So, to prevent this from happening every again we're putting this check in:
from email.utils import parseaddr
WELCOME_EMAIL_FROM = config("WELCOME_EMAIL_FROM", ...)
# If this fails, SMTP will probably also fail.
assert parseaddr(WELCOME_EMAIL_FROM)[1].count('@') == 1, parseaddr(WELCOME_EMAIL_FROM)
You could go to town even more on this. Perhaps use the email validator within django
but for now I'd call that overkill. This is just a decent check before anything gets a chance to go wrong.
Build pyenv Python versions on macOS Catalina 10.15
February 19, 2020
9 comments Python, MacOSX
UPDATE Mar 7, 2022: For OSX 12.2 Monterey
Here's what I needed to do in 2022 to get this to work:
SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.1.sdk \ MACOSX_DEPLOYMENT_TARGET=12.2 \ PYTHON_CONFIGURE_OPTS="--enable-framework" \ pyenv install 3.10.2
BELOW IS ORIGINAL BLOG POST
I'm still working on getting pyenv
in my bloodstream. It seems like totally the right tool for having different versions of Python available on macOS that don't suddenly break when you run brew upgrade
periodically. But every thing I tried failed with an error similar to this:
python-build: use openssl from homebrew python-build: use readline from homebrew Installing Python-3.7.0... python-build: use readline from homebrew BUILD FAILED (OS X 10.15.x using python-build 20XXXXXX) Inspect or clean up the working tree at /var/folders/mw/0ddksqyn4x18lbwftnc5dg0w0000gn/T/python-build.20190528163135.60751 Results logged to /var/folders/mw/0ddksqyn4x18lbwftnc5dg0w0000gn/T/python-build.20190528163135.60751.log Last 10 log lines: ./Modules/posixmodule.c:5924:9: warning: this function declaration is not a prototype [-Wstrict-prototypes] if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0) ^ ./Modules/posixmodule.c:6018:11: error: implicit declaration of function 'forkpty' is invalid in C99 [-Werror,-Wimplicit-function-declaration] pid = forkpty(&master_fd, NULL, NULL, NULL); ^ ./Modules/posixmodule.c:6018:11: warning: this function declaration is not a prototype [-Wstrict-prototypes] 2 warnings and 2 errors generated. make: *** [Modules/posixmodule.o] Error 1 make: *** Waiting for unfinished jobs....
I read through the Troubleshooting FAQ and the "Common build problems" documentation. xcode
was up to date and I had all the related brew
packages upgraded. Nothing seemed to work.
Until I saw this comment on an open pyenv
issue: "Unable to install any Python version on MacOS"
All I had to do was replace the 10.14
for 10.15
and now it finally worked here on Catalina 10.15. So, the magical line was this:
SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk \ MACOSX_DEPLOYMENT_TARGET=10.15 \ PYTHON_CONFIGURE_OPTS="--enable-framework" \ pyenv install -v 3.7.6
Hopefully, by blogging about it you'll find this from Googling and I'll remember the next time I need it because it did eat 2 hours of precious evening coding time.
redirect-chain - Getting a comfortable insight input URL redirects history
February 14, 2020
0 comments Python
You can accomplish the same with curl -L
but I've had this as a little personal hack script in my ~/bin
folder on my computer. Thought I'd make it a public tool. Also, from here, a lot more can be done to this script if you wanna help out with ideas.
▶ redirect-chain http://developer.mozilla.org/en-US/docs/xpcshell 0 http://developer.mozilla.org/en-US/docs/xpcshell 301 1 > https://developer.mozilla.org/en-US/docs/xpcshell 301 2 >> https://developer.mozilla.org/docs/en/XPConnect/xpcshell 302 3 >>> https://developer.mozilla.org/en-US/docs/en/XPConnect/xpcshell 301 4 >>>> https://developer.mozilla.org/en-US/docs/XPConnect/xpcshell 301 5 >>>>> https://developer.mozilla.org/en-US/docs/Mozilla/XPConnect/xpcshell 301 6 >>>>>> https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell 200
It basically gives you a pretty summary of redirects from a starting URL.
To install it on your system run:
pipx install redirect-chain
Happy Friday!
How to resolve a git conflict in poetry.lock
February 7, 2020
8 comments Python
We use poetry in MDN Kuma. That means there's a pyproject.toml
and a poetry.lock
file. To add or remove dependencies, you don't touch either file in an editor. For example, to add a package:
poetry add --dev black
It changes pyproject.toml
and poetry.lock
for you. (Same with yarn add somelib
which edits package.json
and yarn.lock
).
Suppose that you make a pull request to add a new dependency, but someone sneaks a new pull request in before you and have theirs landed in master
before. Well, that's how you end up in this place:
So how do you resolve that?
So, you go back to your branch and run something like:
git checkout master git pull origin master git checkout my-branch git merge master
Now you get this in git status
:
Unmerged paths: (use "git add <file>..." to mark resolution) both modified: poetry.lock
And the contents of poetry.lock
looks something like this:
I wish there just was a way poetry
itself could just figure fix this.
What you need to do is to run:
# Get poetry.lock to look like it does in master git checkout --theirs poetry.lock # Rewrite the lock file poetry lock --no-update
Now, your poetry.lock
file should correctly reflect the pyproject.toml
that has been merged from master
.
To finish up, resolve the conflict:
git add poetry.lock git commit -a -m "conflict resolved" # and most likely needed poetry install
content-hash
Inside the poetry.lock
file there's the lock file's hash. It looks like this:
[metadata] content-hash = "875b6a3628489658b323851ce6fe8dafacd5f69e5150d8bb92b8c53da954c1be"
So, as can be seen in my screenshot, when git conflicted on this it looks like this:
[metadata]
+<<<<<<< HEAD
+content-hash = "6658b1379d6153dd603bbc27d04668e5e93068212c50e76bd068e9f10c0bec59"
+=======
content-hash = "5c00dce18ddffd5d6f797dfa14e4d56bf32bbc3769d7b761a2b1b3ff14bce287"
+>>>>>>> master
Basically, the content-hash = "5c00dce1...
is what you'd find in master
and content-hash = "6658b137...
is what you would see in your branch before the conflict.
When you run that poetry lock
you can validate that the new locking worked because it should be a hash. One that is neither 5c00dce1...
or 6658b137...
.
Notes
I'm still new to poetry
and I'm learning. This was just some loud note-to-self so I can remember for next time.
I don't yet know what else can be automated if there's a conflict in pyproject.toml
too. And what do you do if there are serious underlying conflicts in Python packages, like they added a package that requires somelib<=0.99
and you added something that requires somelib>=1.11
.
Also, perhaps there are ongoing efforts within the poetry
project to help out with this.
UPDATE Feb 12, 2020
My colleague informed me that this change was actually NOT what I wanted. poetry lock
actually updates some dependencies as it makes a completely new lock file. I didn't immediately notice that in my case because the lock file is large. See this open issue which is about the ability to update the lock file without upgrading any other dependencies.
UPDATE June 24, 2021
To re-lock the file, use poetry lock --no-update
after you've run git checkout --theirs poetry.lock
.