19 February 2020 0 comments MacOSX, Python
I'm still working on getting pyenv in my bloodstream. It seems like totally the right tool for having different versions of Python available on macOS that don't suddenly break when you run brew upgrade
periodically. But every thing I tried failed with an error similar to this:
python-build: use openssl from homebrew
python-build: use readline from homebrew
Installing Python-3.7.0...
python-build: use readline from homebrew
BUILD FAILED (OS X 10.15.x using python-build 20XXXXXX)
Inspect or clean up the working tree at /var/folders/mw/0ddksqyn4x18lbwftnc5dg0w0000gn/T/python-build.20190528163135.60751
Results logged to /var/folders/mw/0ddksqyn4x18lbwftnc5dg0w0000gn/T/python-build.20190528163135.60751.log
Last 10 log lines:
./Modules/posixmodule.c:5924:9: warning: this function declaration is not a prototype [-Wstrict-prototypes]
if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0)
^
./Modules/posixmodule.c:6018:11: error: implicit declaration of function 'forkpty' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
pid = forkpty(&master_fd, NULL, NULL, NULL);
^
./Modules/posixmodule.c:6018:11: warning: this function declaration is not a prototype [-Wstrict-prototypes]
2 warnings and 2 errors generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
I read through the Troubleshooting FAQ and the "Common build problems" documentation. xcode was up to date and I had all the related brew packages upgraded. Nothing seemed to work.
Until I saw this comment on an open pyenv issue: "Unable to install any Python version on MacOS"
All I had to do was replace the 10.14 for 10.15 and now it finally worked here on Catalina 10.15. So, the magical line was this:
SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk \
MACOSX_DEPLOYMENT_TARGET=10.15 \
PYTHON_CONFIGURE_OPTS="--enable-framework" \
pyenv install -v 3.7.6
Hopefully, by blogging about it you'll find this from Googling and I'll remember the next time I need it because it did eat 2 hours of precious evening coding time.
14 February 2020 0 comments Python
You can accomplish the same with curl -L but I've had this as a little personal hack script in my ~/bin folder on my computer. Thought I'd make it a public tool. Also, from here, a lot more can be done to this script if you wanna help out with ideas.
▶ redirect-chain http://developer.mozilla.org/en-US/docs/xpcshell
0 http://developer.mozilla.org/en-US/docs/xpcshell 301
1 > https://developer.mozilla.org/en-US/docs/xpcshell 301
2 >> https://developer.mozilla.org/docs/en/XPConnect/xpcshell 302
3 >>> https://developer.mozilla.org/en-US/docs/en/XPConnect/xpcshell 301
4 >>>> https://developer.mozilla.org/en-US/docs/XPConnect/xpcshell 301
5 >>>>> https://developer.mozilla.org/en-US/docs/Mozilla/XPConnect/xpcshell 301
6 >>>>>> https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Language_bindings/XPConnect/xpcshell 200
It basically gives you a pretty summary of redirects from a starting URL.
To install it on your system run:
pipx install redirect-chain
Happy Friday!
07 February 2020 0 comments Python
We use poetry in MDN Kuma. That means there's a pyproject.toml and a poetry.lock file. To add or remove dependencies, you don't touch either file in an editor. For example, to add a package:
poetry add --dev black
It changes pyproject.toml and poetry.lock for you. (Same with yarn add somelib which edits package.json and yarn.lock).
Suppose that you make a pull request to add a new dependency, but someone sneaks a new pull request in before you and have theirs landed in master before. Well, that's how you end up in this place:
So how do you resolve that?
So, you go back to your branch and run something like:
git checkout master
git pull origin master
git checkout my-branch
git merge master
Now you get this in git status:
Unmerged paths:
(use "git add <file>..." to mark resolution)
both modified: poetry.lockAnd the contents of poetry.lock looks something like this:
I wish there just was a way poetry itself could just figure fix this.
What you need to do is to run:
# Get poetry.lock to look like it does in master
git checkout --theirs poetry.lock
# Rewrite the lock file
poetry lock
Now, your poetry.lock file should correctly reflect the pyproject.toml that has been merged from master.
To finish up, resolve the conflict:
git add poetry.lock
git commit -a -m "conflict resolved"
# and most likely needed
poetry install
content-hash
Inside the poetry.lock file there's the lock file's hash. It looks like this:
[metadata]
content-hash = "875b6a3628489658b323851ce6fe8dafacd5f69e5150d8bb92b8c53da954c1be"
So, as can be seen in my screenshot, when git conflicted on this it looks like this:
[metadata]
+<<<<<<< HEAD
+content-hash = "6658b1379d6153dd603bbc27d04668e5e93068212c50e76bd068e9f10c0bec59"
+=======
content-hash = "5c00dce18ddffd5d6f797dfa14e4d56bf32bbc3769d7b761a2b1b3ff14bce287"
+>>>>>>> master
Basically, the content-hash = "5c00dce1... is what you'd find in master and content-hash = "6658b137... is what you would see in your branch before the conflict.
When you run that poetry lock you can validate that the new locking worked because it should be a hash. One that is neither 5c00dce1... or 6658b137....
Notes
I'm still new to poetry and I'm learning. This was just some loud note-to-self so I can remember for next time.
I don't yet know what else can be automated if there's a conflict in pyproject.toml too. And what do you do if there are serious underlying conflicts in Python packages, like they added a package that requires somelib<=0.99 and you added something that requires somelib>=1.11.
Also, perhaps there are ongoing efforts within the poetry project to help out with this.
UPDATE Feb 12, 2020
My colleague informed me that this change was actually NOT what I wanted. poetry lock actually updates some dependencies as it makes a completely new lock file. I didn't immediately notice that in my case because the lock file is large. See this open issue which is about the ability to update the lock file without upgrading any other dependencies.
05 February 2020 0 comments MacOSX, Python
I don't know how many times I've encountered this but by blogging about it, hopefully, next time it'll help me, and you!, find this sooner.
If you get this:
clang -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/peterbe/.pyenv/versions/3.8.0/lib -L/opt/boxen/homebrew/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/peterbe/.pyenv/versions/3.8.0/lib -L/opt/boxen/homebrew/lib -L/opt/boxen/homebrew/lib -I/opt/boxen/homebrew/include build/temp.macosx-10.14-x86_64-3.8/MySQLdb/_mysql.o -L/usr/local/Cellar/mysql/8.0.18_1/lib -lmysqlclient -lssl -lcrypto -o build/lib.macosx-10.14-x86_64-3.8/MySQLdb/_mysql.cpython-38-darwin.so
ld: library not found for -lssl
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang' failed with exit status 1(The most important line is the ld: library not found for -lssl)
On most macOS systems, when trying to install a Python package that requires a binary compile step based on the system openssl
(which I think comes from the OS), you'll get this.
The solution is simple, run this first:
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
Depending on your install of things, you might need to adjust this accordingly. For me, I have:
▶ ls -l /usr/local/opt/openssl/
total 1272
-rw-r--r-- 1 peterbe staff 717 Sep 10 09:13 AUTHORS
-rw-r--r-- 1 peterbe staff 582924 Dec 19 11:32 CHANGES
-rw-r--r-- 1 peterbe staff 743 Dec 19 11:32 INSTALL_RECEIPT.json
-rw-r--r-- 1 peterbe staff 6121 Sep 10 09:13 LICENSE
-rw-r--r-- 1 peterbe staff 42183 Sep 10 09:13 NEWS
-rw-r--r-- 1 peterbe staff 3158 Sep 10 09:13 README
drwxr-xr-x 4 peterbe staff 128 Dec 19 11:32 bin
drwxr-xr-x 3 peterbe staff 96 Sep 10 09:13 include
drwxr-xr-x 10 peterbe staff 320 Sep 10 09:13 lib
drwxr-xr-x 4 peterbe staff 128 Sep 10 09:13 share
Now, with those things set you should hopefully be able to do things like:
pip install mysqlclient
03 February 2020 0 comments Javascript, Node
I'm working on a Node project that involves large transformations of large sets of data here and there. For example:
if (!Object.keys(this.allTitles).length) {
...
In my case, that this.allTitles is a plain object with about 30,000 key/value pairs. That particular line of code actually only runs 1 single time so if it's hundreds of milliseconds, it's really doesn't matter that much. However, that's not a guarantee! What if you had something like this:
for (const thing of things) {
if (!Object.keys(someObj).length) {
// mutate someObj
}
}
then, you'd potentially have a performance degradation once someObj becomes considerably large. And it gets particularly degraded if the length of things is considerably large as it would do the operation many times.
Actually, consider this:
const obj = {};
[...Array(30000)].forEach((_, i) => {
obj[i] = i;
});
console.time("Truthcheck obj");
[...Array(100)].forEach((_, i) => {
return !!Object.keys(obj).length;
});
console.timeEnd("Truthcheck obj");
On my macBook with Node 13.5, this outputs:
Truthcheck obj: 260.564ms
Maps
The MDN page on Map has a nice comparison, in terms of performance, between Map and regular object. Consider this super simple benchmark:
const obj = {};
const map = new Map();
[...Array(30000)].forEach((_, i) => {
obj[i] = i;
map.set(i, i);
});
console.time("Truthcheck obj");
[...Array(100)].forEach((_, i) => {
return !!Object.keys(obj).length;
});
console.timeEnd("Truthcheck obj");
console.time("Truthcheck map");
[...Array(100)].forEach((_, i) => {
return !!map.size;
});
console.timeEnd("Truthcheck map");
So, fill a Map instance and a plain object with 30,000 keys and values. Then, for each in turn, check if the thing is truthy 100 times. The output I get:
Truthcheck obj: 235.017ms
Truthcheck map: 0.029ms
That's not unexpected. The map instance maintains a size counter, which increments on .set (if the key is new), so doing that "truthy" check just takes O(1) seconds.
Conclusion
Don't run to rewrite everything to Maps!
In fact, I took the above mentioned little benchmark and changed the times to be a 3,000 item map and obj (instead of 30,000) and only did 10 iterations (instead of 100) and then the numbers are:
Truthcheck obj: 0.991ms
Truthcheck map: 0.044ms
These kinds of small numbers are very unlikely to matter in the scope of other things going on.
Anyway, consider using Map if you fear that you might be working with really reeeeally large mappings.
24 January 2020 8 comments Python
I often find myself Googling for this. Always a little bit embarrassed that I can't remember the incantation (syntax).
Suppose you have a string mystr that you want to fill with with spaces so it's 10 characters wide:
>>> mystr = 'peter'
>>> mystr.ljust(10)
'peter '
>>> mystr.rjust(10)
' peter'
Now, with "f-strings" you do:
>>> mystr = 'peter'
>>> f'{mystr:<10}'
'peter '
>>> f'{mystr:>10}'
' peter'
What also trips me up is, suppose that the number 10 is variable. I.e. it's not hardcoded into the f-string but a variable from somewhere else. Here's how you do it:
>>> width = 10
>>> f'{mystr:<{width}}'
'peter '
>>> f'{mystr:>{width}}'
' peter'
What I haven't figured out yet, is how you specify a different character than a simple single whitespace. I.e. does anybody know how to do this, but with f-strings:
>>> width = 10
>>> mystr.ljust(width, '*')
'peter*****'
UPDATE
First of all, I left two questions unanswered. One was how do you make the filler something other than ' '. The answer is:
>>> f'{"peter":*<10}'
'peter*****'
The question question was, what if you don't know what the filler character should be. In the above example, *
was hardcoded inside the f-string. The solution is stunningly simple actually.
>>> width = 10
>>> filler = '*'
>>> f'{"peter":{filler}<{width}}'
'peter*****'
But note, it has to be a single length string. This is what happens if you try to make it a longer string:
>>> filler = 'xxx'
>>> f'{"peter":{filler}<{width}}'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier
