A blog and website by Peter Bengtsson
Web development, Django, Python, Docker, JavaScript
Heroku is great but it's sometimes painful when your app isn't just in one single language. What I have is a project where the backend is Python (Django) and the frontend is JavaScript (Preact). The folder structure looks like this:
/ - README.md - manage.py - requirements.txt - my_django_app/ - settings.py - asgi.py - api/ - urls.py - views.py - frontend/ - package.json - yarn.lock - preact.config.js - build/ ... - src/ ...
A bunch of things omitted for brevity but people familiar with Django and preact-cli/create-create-app should be familiar.
The point is that the root is a Python app and the front-end is exclusively inside a sub folder.
When you do local development, you start two servers:
./manage.py runserver
- starts http://localhost:8000
cd frontend && yarn start
- starts http://localhost:3000
The latter is what you open in your browser. That preact
app will do things like:
const response = await fetch('/api/search');
and, in preact.config.js
I have this:
export default (config, env, helpers) => {
if (config.devServer) {
config.devServer.proxy = [
{
path: "/api/**",
target: "http://localhost:8000"
}
];
}
};
...which is hopefully self-explanatory. So, calls like GET http://localhost:3000/api/search
actually goes to http://localhost:8000/api/search
.
That's when doing development. The interesting thing is going into production.
Before we get into Heroku, let's first "merge" the two systems into one and the trick used is Whitenoise. Basically, Django's web server will be responsibly not only for things like /api/search
but also static assets such as / --> frontend/build/index.html
and /bundle.17ae4.js --> frontend/build/bundle.17ae4.js
.
This is basically all you need in settings.py
to make that happen:
MIDDLEWARE = [
"django.middleware.security.SecurityMiddleware",
"whitenoise.middleware.WhiteNoiseMiddleware",
...
]
WHITENOISE_INDEX_FILE = True
STATIC_URL = "/"
STATIC_ROOT = BASE_DIR / "frontend" / "build"
However, this isn't quite enough because the preact
app uses preact-router
which uses pushState()
and other code-splitting magic so you might have a URL, that users see, like this: https://myapp.example.com/that/thing/special
and there's nothing about that in any of the Django urls.py
files. Nor is there any file called frontend/build/that/thing/special/index.html
or something like that.
So for URLs like that, we have to take a gamble on the Django side and basically hope that the preact-router
config knows how to deal with it. So, to make that happen with Whitenoise we need to write a custom middleware that looks like this:
from whitenoise.middleware import WhiteNoiseMiddleware
class CustomWhiteNoiseMiddleware(WhiteNoiseMiddleware):
def process_request(self, request):
if self.autorefresh:
static_file = self.find_file(request.path_info)
else:
static_file = self.files.get(request.path_info)
# These two lines is the magic.
# Basically, the URL didn't lead to a file (e.g. `/manifest.json`)
# it's either a API path or it's a custom browser path that only
# makes sense within preact-router. If that's the case, we just don't
# know but we'll give the client-side preact-router code the benefit
# of the doubt and let it through.
if not static_file and not request.path_info.startswith("/api"):
static_file = self.files.get("/")
if static_file is not None:
return self.serve(static_file, request)
And in settings.py
this change:
MIDDLEWARE = [
"django.middleware.security.SecurityMiddleware",
- "whitenoise.middleware.WhiteNoiseMiddleware",
+ "my_django_app.middleware.CustomWhiteNoiseMiddleware",
...
]
Now, all traffic goes through Django. Regular Django view functions, static assets, and everything else fall back to frontend/build/index.html
.
Heroku tries to make everything so simple for you. You basically, create the app (via the cli or the Heroku web app) and when you're ready you just do git push heroku master
. However that won't be enough because there's more to this than Python.
Unfortunately, I didn't take notes of my hair-pulling excruciating journey of trying to add buildpacks and hacks and Procfile
s and custom buildpacks. Nothing seemed to work. Perhaps the answer was somewhere in this issue: "Support running an app from a subdirectory" but I just couldn't figure it out. I still find buildpacks confusing when it's beyond Hello World. Also, I didn't want to run Node as a service, I just wanted it as part of the "build process".
Finally I get a chance to try "Deploying with Docker" in Heroku which is a relatively new feature. And the only thing that scared me was that now I need to write a heroku.yml
file which was confusing because all I had was a Dockerfile
. We'll get back to that in a minute!
So here's how I made a Dockerfile
that mixes Python and Node:
FROM node:12 as frontend
COPY . /app
WORKDIR /app
RUN cd frontend && yarn install && yarn build
FROM python:3.8-slim
WORKDIR /app
RUN groupadd --gid 10001 app && useradd -g app --uid 10001 --shell /usr/sbin/nologin app
RUN chown app:app /tmp
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y --no-install-recommends \
gcc apt-transport-https python-dev
# Gotta try moving this to poetry instead!
COPY ./requirements.txt /app/requirements.txt
RUN pip install --upgrade --no-cache-dir -r requirements.txt
COPY . /app
COPY --from=frontend /app/frontend/build /app/frontend/build
USER app
ENV PORT=8000
EXPOSE $PORT
CMD uvicorn gitbusy.asgi:application --host 0.0.0.0 --port $PORT
If you're not familiar with it, the critical trick is on the first line where it builds some Node with as frontend
. That gives me a thing I can then copy from into the Python image with COPY --from=frontend /app/frontend/build /app/frontend/build
.
Now, at the very end, it starts a uvicorn
server with all the static .js
, index.html
, and favicon.ico
etc. available to uvicorn
which ultimately runs whitenoise
.
To run and build:
docker build . -t my_app docker run -t -i --rm --env-file .env -p 8000:8000 my_app
Now, opening http://localhost:8000/
is a production grade app that mixes Python (runtime) and JavaScript (static).
Heroku says to create a heroku.yml
file and that makes sense but what didn't make sense is why I would add cmd
line in there when it's already in the Dockerfile
. The solution is simple: omit it. Here's what my final heroku.yml
file looks like:
build:
docker:
web: Dockerfile
Check in the heroku.yml
file and git push heroku master
and voila, it works!
To see a complete demo of all of this check out https://github.com/peterbe/gitbusy and https://gitbusy.herokuapp.com/
tl;dr; Watch out for .dockerignore
causing no such file or directory
when building Docker images
First I tried to use docker-compose
:
ā¶ docker-compose build ui Building ui Step 1/8 : FROM node:9 ---> 29831ba76d93 Step 2/8 : ADD ui/package.json /package.json ERROR: Service 'ui' failed to build: ADD failed: stat /var/lib/docker/tmp/docker-builder079614651/ui/package.json: no such file or directory
What the heck? Did I typo the name in the ui/Dockerfile
?
The docker-compose.yml
directive for this was:
yaml ui: build: context: . dockerfile: ui/Dockerfile environment: - NODE_ENV=development ports: - "3000:3000" - "35729:35729" volumes: - $PWD/ui:/app command: start
I don't know if it's the awesomest way to do docker-compose
but I did copy this exactly from a different (working!) project. That other project called the web stuff frontend
instead of ui
in this project.
The Dockerfile
looked like this:
FROM node:9 ADD ./ui/package.json ./ ADD ./ui/package-lock.json ./ RUN npm install EXPOSE 3000 EXPOSE 35729 CMD [ "npm", "start" ]
Let's try without docker-compose
. Or rather, do with docker
what docker-compose
does for me automatically.
ā¶ docker build ui -f ui/Dockerfile Sending build context to Docker daemon 158.2MB Step 1/8 : FROM node:9 ---> 29831ba76d93 Step 2/8 : ADD ui/package.json /package.json ADD failed: stat /var/lib/docker/tmp/docker-builder001494654/ui/package.json: no such file or directory
So I thought I perhaps have misunderstood how relative paths worked. I tried EVERYTHING!
I tried changing to docker build . -f ui/Dockerfile
. No luck.
I tried all sorts of combinations of this with also changing the line to ADD ui/package.json ...
or ADD /ui/package.json ...
or ADD ./package.json ...
or ADD package.json ...
. Nothing worked.
Finally I got it to work by doing
ā¶ cd ui ā¶ docker build . -f Dockerfile
but for that to work I had to remove all references of the directory name ui
in the Dockerfile
. Nice, but this is not going to work in docker-compose.yml
since that starts outside the directory ./ui/
.
Sigh!
So then I learned about contexts in docker. Well, I skimmed the docs rapidly. To keep things clean it's a good idea to do things within the directory that matters. So I made this change in the docker-compose.yml
:
ui: build: context: ui dockerfile: Dockerfile environment: - NODE_ENV=development ports: - "3000:3000" - "35729:35729" volumes: - $PWD/ui:/app command: start
(Note the build.context:
and build.dockerfile:
)
Still doesn't work! š« Still various variations of no such file or directory
.
Turns out, in projectroot/.dockerignore
it had ui/
as a line entry!!!
I believe this project used to do some of the Python stuff with Docker and the React web app was done "locally" on the host. And since the ui/node_modules
directory is so huge someone decided it was smart of avoid Docker mounting that.
Now the .dockerignore
has .ui/node_modules
and now everything works. I can build it with plain docker
and docker-compose
from outside the directory.
Perhaps I should have spent the time I spent writing this blog post to instead file a structured GitHub issue on Docker itself somewhere. I.e. that it should have warned be better. Any takers?
Web development, Django, MacOSX, Docker
I have a side-project that is basically a React frontend, a Django API server and a Node universal React renderer. The killer feature is its Elasticsearch database that searches almost 2.5M large texts and 200K named objects. All the data is stored in a PostgreSQL and there's some Python code that copies that stuff over to Elasticsearch for indexing.
The PostgreSQL database is about 10GB and the Elasticsearch (version 6.1.0) indices are about 6GB. It's moderately big and even though individual searches take, on average ~75ms (in production) it's hefty. At least for a side-project.
On my MacBook Pro, laptop I use Docker to do development. Docker makes it really easy to run one command that starts memcached, Django, a AWS Product API Node app, create-react-app for the search and a separate create-react-app for the stats web app.
At first I tried to also run PostgreSQL and Elasticsearch in Docker too, but after many attempts I had to just give up. It was too slow. Elasticsearch would keep crashing even though I extended my memory in Docker to 4GB.
This very blog (www.peterbe.com) has a similar stack. Redis, PostgreSQL, Elasticsearch all running in Docker. It works great. One single docker-compose up web
starts everything I need. But when it comes to much larger databases, I found my macOS host to be much more performant.
So the dark side of this is that I have remember to do more things when starting work on this project. My PostgreSQL was installed with Homebrew and is always running on my laptop. For Elasticsearch I have to open a dedicated terminal and go to a specific location to start the Elasticsearch for this project (e.g. make start-elasticsearch
).
The way I do this is that I have this in my Django projects settings.py
:
import dj_database_url
from decouple import config
DATABASES = {
'default': config(
'DATABASE_URL',
# Hostname 'docker.for.mac.host.internal' assumes
# you have at least Docker 17.12.
# For older versions of Docker use 'docker.for.mac.localhost'
default='postgresql://peterbe@docker.for.mac.host.internal/songsearch',
cast=dj_database_url.parse
)
}
ES_HOSTS = config('ES_HOSTS', default='docker.for.mac.host.internal:9200', cast=Csv())
(Actually, in reality the defaults in the settings.py
code is localhost
and I use docker-compose.yml
environment variables to override this, but the point is hopefully still there.)
And that's basically it. Now I get Docker to do what various virtualenv
s and terminal scripts used to do but the performance of running the big databases on the host.
Python, Web development, Mozilla, Docker
tl;dr; Whatsdeployed.io is an impressively simple web app to help web developers and web ops people quickly see what GitHub commits have made it into your Dev, Stage or Prod environment. Today it got a facelift.
The code is now more than 5 years old and has served me well. It's weird to talk too positively about the app because I actually wrote it but because it's so simple in terms of design and effort it feels less personal to talk about it.
requests.get()
that the Flask app does, fails. Also includes which URL it failed on. README.md
Please let me know if there's anything broken or missing.
Linux, Web development, JavaScript, React, Docker
Why would you want to use Docker to do React app work? Isn't Docker for server-side stuff like Python and Golang etc? No, all the benefits of Docker apply to JavaScript client-side work too.
So there are three main things you want to do with create-react-app
; dev server, running tests and creating build artifacts. Let's look at all three but using Docker.
If you haven't already, install create-react-app
globally:
ā¶ yarn global add create-react-app
And, once installed, create a new project:
ā¶ create-react-app docker-create-react-app ...lots of output... ā¶ cd docker-create-react-app ā¶ ls README.md node_modules package.json public src yarn.lock
We won't need the node_modules
here in the project directory. Instead, when building the image we're going let node_modules
stay inside the image. So you can go ahead and... rm -fr node_modules
.
Dockerfile
Let's just dive in. This Dockerfile
is the minimum:
FROM node:8 ADD yarn.lock /yarn.lock ADD package.json /package.json ENV NODE_PATH=/node_modules ENV PATH=$PATH:/node_modules/.bin RUN yarn WORKDIR /app ADD . /app EXPOSE 3000 EXPOSE 35729 ENTRYPOINT ["/bin/bash", "/app/run.sh"] CMD ["start"]
A couple of things to notice here.
First of all we're basing this on the official Node v8 repository on Docker Hub. That gives you a Node and Yarn by default.
Note how the NODE_PATH
environment variable puts the node_modules
in the root of the container. That's so that it doesn't get added in "here" (i.e. the current working directory). If you didn't do this, the node_modules
directory would be part of the mounted volume which not only slows down Docker (since there are so many files) it also isn't necessary to see those files.
Note how the ENTRYPOINT
points to run.sh
. That's a file we need to create too, alongside the Dockerfile
file.
#!/usr/bin/env bash set -eo pipefail case $1 in start) # The '| cat' is to trick Node that this is an non-TTY terminal # then react-scripts won't clear the console. yarn start | cat ;; build) yarn build ;; test) yarn test $@ ;; *) exec "$@" ;; esac
Lastly, as a point of convenience, note that the default CMD
is "start"
. That's so that when you simply run the container the default thing it does is to run yarn start
.
Now let's build it:
ā¶ docker image build -t react:app .
The -t react:app
is up to you. It doesn't matter so much what it is unless you're going to upload your container the a registry. Then you probably want the repository to be something unique.
Let's check that the build is there:
ā¶ docker image ls react:app REPOSITORY TAG IMAGE ID CREATED SIZE react app 3ee5c7596f57 13 minutes ago 996MB
996MB! The base Node image is about ~700MB and the node_modules
directory (for a clean new create-react-app
) is ~160MB (at the time of writing). What the remaining difference is, I'm not sure. But it's empty calories and easy to lose. When you blow away the built image (docker image rmi react:app
) your hard drive gets all that back and no actual code is lost.
Before we run it, lets go inside and see what was created:
ā¶ docker container run -it react:app bash root@996e708a30c4:/app# ls Dockerfile README.md package.json public run.sh src yarn.lock root@996e708a30c4:/app# du -sh /node_modules/ 148M /node_modules/ root@996e708a30c4:/app# sw-precache Total precache size is about 355 kB for 14 resources. service-worker.js has been generated with the service worker contents.
The last command (sw-precache
) was just to show that executables in /node_modules/.bin
are indeed on the $PATH
and can be run.
Now to run it:
ā¶ docker container run -it -p 3000:3000 react:app yarn run v1.3.2 $ react-scripts start Starting the development server... Compiled successfully! You can now view docker-create-react-app in the browser. Local: http://localhost:3000/ On Your Network: http://172.17.0.2:3000/ Note that the development build is not optimized. To create a production build, use yarn build.
Pretty good. Open http://localhost:3000
in your browser and you should see the default create-react-app
app.
create-react-app
does not support hot reloading of components. But it does support web page reloading. As soon as a local file is changed, it sends a signal to the browser (using WebSockets) to tell it to... document.location.reload()
.
To make this work, we need to do two things:
1) Mount the current working directory into the Docker container
2) Expose the WebSocket port
The WebSocket thing is set up by exposing port 35729 to the host (-p 35729:35729
).
Below is an example running this with a volume mount and both ports exposed.
ā¶ docker container run -it -p 3000:3000 -p 35729:35729 -v $(pwd):/app react:app yarn run v1.3.2 $ react-scripts start Starting the development server... Compiled successfully! You can now view docker-create-react-app in the browser. Local: http://localhost:3000/ On Your Network: http://172.17.0.2:3000/ Note that the development build is not optimized. To create a production build, use yarn build. Compiling... Compiled successfully! Compiling... Compiled with warnings. ./src/App.js Line 7: 'neverused' is assigned a value but never used no-unused-vars Search for the keywords to learn more about each warning. To ignore, add // eslint-disable-next-line to the line before. Compiling... Failed to compile. ./src/App.js Module not found: Can't resolve './Apps.css' in '/app/src'
In the about example output. First I make a harmless save in the src/App.js
file just to see that the dev server notices and that my browser reloads when I did that. That's where it says
Compiling... Compiled successfully!
Secondly, I make an edit that triggers a warning. That's where it says:
Compiling... Compiled with warnings. ./src/App.js Line 7: 'neverused' is assigned a value but never used no-unused-vars Search for the keywords to learn more about each warning. To ignore, add // eslint-disable-next-line to the line before.
And lastly I make an edit by messing with the import line
Compiling... Failed to compile. ./src/App.js Module not found: Can't resolve './Apps.css' in '/app/src'
This is great! Isn't create-react-app
wonderful?
There are many things you can do with the code you're building. Let's pretend that the intention is to build a single-page-app and then take the static assets (including the index.html
) and upload them to a public CDN or something. To do that we need to generate the build
directory.
The trick here is to run this with a volume mount so that when it creates /app/build
(from the perspective) of the container, that directory effectively becomes visible in the host.
ā¶ docker container run -it -v $(pwd):/app react:app build yarn run v1.3.2 $ react-scripts build Creating an optimized production build... Compiled successfully. File sizes after gzip: 35.59 KB build/static/js/main.591fd843.js 299 B build/static/css/main.c17080f1.css The project was built assuming it is hosted at the server root. To override this, specify the homepage in your package.json. For example, add this to build it for GitHub Pages: "homepage" : "http://myname.github.io/myapp", The build folder is ready to be deployed. You may serve it with a static server: yarn global add serve serve -s build Done in 5.95s.
Now, on the host:
ā¶ tree build build āāā asset-manifest.json āāā favicon.ico āāā index.html āāā manifest.json āāā service-worker.js āāā static āāā css āĀ Ā āāā main.c17080f1.css āĀ Ā āāā main.c17080f1.css.map āāā js āĀ Ā āāā main.591fd843.js āĀ Ā āāā main.591fd843.js.map āāā media āāā logo.5d5d9eef.svg 4 directories, 10 files
The contents of that file you can now upload to a CDN some public Nginx server that points to this as the root directory.
This one is so easy and obvious now.
ā¶ docker container run -it -v $(pwd):/app react:app test
Note the that we're setting up a volume mount here again. Since the test runner is interactive it sits and waits for file changes and re-runs tests immediately, it's important to do the mount now.
All regular jest options work too. For example:
ā¶ docker container run -it -v $(pwd):/app react:app test --coverage ā¶ docker container run -it -v $(pwd):/app react:app test --help
node_modules
First of all, when I say "debugging the node_modules
", in this context, I'm referring to messing with node_modules
whilst running tests or running the dev server.
One way to debug the node_modules
used is to enter a bash shell and literally mess with the files inside it. First, start the dev server (or start the test runner) and give the container a name:
ā¶ docker container run -it -p 3000:3000 -p 35729:35729 -v $(pwd):/app --name mydebugging react:app
Now, in a separate terminal start bash
in the container:
ā¶ docker exec -it mydebugging bash
Once you're in you can install an editor and start editing files:
root@2bf8c877f788:/app# apt-get update && apt-get install jed root@2bf8c877f788:/app# jed /node_modules/react/index.js
As soon as you make changes to any of the files, the dev server should notice and reload.
When you stop the container all your changes will be reset. So if you had to sprinkle the node_modules
with console.log('WHAT THE HECK!')
all of those disappear when the container is stopped.
This'll come as no surprise by now. You basically run bash
and you're there:
ā¶ docker container run -it -v $(pwd):/app react:app bash root@2a21e8206a1f:/app# node > [] + 1 '1'
When I look back at all the commands above, I can definitely see how it's pretty intimidating and daunting. So many things to remember and it's got that nasty feeling where you feel like your controlling your development environment through unwieldy levers rather than your own hands.
But think of the fundamental advantages too! It's all encapsulated now. What you're working on will be based on the exact same version of everything as your teammate, your dev server and your production server are using.
Pros:
node_modules
directory gets out of your hair.docker-compose
magic you can have it all running with one command without needing to run the frontend in a separate terminal.Cons:
In my (Mozilla Services) work, the projects I work on, I actually use docker-compose
for all things. And I have a Makefile
to help me remember all the various docker-compose
commands (thanks Jannis & Will!). One definitely neat thing you can do with docker-compose
is start multiple containers. Then you can, with one command, start a Django server and the create-react-app
dev server with one command. Perhaps a blog post for another day.
tl;dr; To build once and run Docker containers with different files use a volume mount. If that's not an option, like in CircleCI, avoid volume mount and rely on container build every time.
Laugh all you like but after almost year of using Docker I'm still learning the basics. Apparently. This, now, feels laughable but there's a small chance someone else stumbles like I did and they might appreciate this.
If you have a volume mounted for a service in your docker-compose.yml
it will basically take whatever you mount and lay that on top of what was in the Docker container. Doing a volume mount into the same working directory as your container is totally common. When you do that the files on the host (the files/directories mounted) get used between each run. If you don't do that, you're stuck with the files, from your host, from the last time you built.
Consider...:
# Dockerfile FROM python:3.6-slim LABEL maintainer="mail@peterbe.com" COPY . /app WORKDIR /app CMD ["python", "run.py"]
and...:
#!/usr/bin/env python
if __name__ == '__main__':
print("hello!")
Let's build it:
$ docker image build -t test:latest . Sending build context to Docker daemon 5.12kB Step 1/5 : FROM python:3.6-slim ---> 0f1dc0ba8e7b Step 2/5 : LABEL maintainer "mail@peterbe.com" ---> Using cache ---> 70cf25f7396c Step 3/5 : COPY . /app ---> 2e95935cbd52 Step 4/5 : WORKDIR /app ---> bc5be932c905 Removing intermediate container a66e27ecaab3 Step 5/5 : CMD python run.py ---> Running in d0cf9c546fee ---> ad930ce66a45 Removing intermediate container d0cf9c546fee Successfully built ad930ce66a45 Successfully tagged test:latest
And run it:
$ docker container run test:latest hello!
So basically my little run.py
got copied into the container by the Dockerfile
. Let's change the file:
$ sed -i.bak s/hello/allo/g run.py $ python run.py allo!
But it won't run like that if we run the container again:
$ docker container run test:latest hello!
So, the container is now built based on a Python file from back of the time the container was built. Two options:
1) Rebuild, or
2) Volume mount in the host directory
This is it! That this is your choice.
Rebuild might take time. So, let's mount the current directory from the host:
$ docker container run -v `pwd`:/app test:latest allo!
So yay! Now it runs the container with the latest file from my host directory.
So, if it's more convenient to "refresh the files in the container" with a volume mount instead of container rebuild, why not always do it for everything?
For one thing, there might be files built inside the container that cease to be visible if you override that workspace with your own volume mount.
The other crucial thing I learned the hard way (seems to obvious now!) is that there isn't always a host directory to mount. In particular, in tecken we use a base ubuntu image and in the run parts of the CircleCI configuration we were using docker-compose run ...
with directives (in the docker-compose.yml
file) that uses volume mounts. So, the rather cryptic effect was that the files mounted into the container was not the files checked out from the git branch.
The resolution in this case, was to be explicit when running Docker commands in CircleCI to only do build followed by run without a volume mount. In particular, to us it meant changing from docker-compose run frontend lint
to docker-compose run frontend-ci lint
. Basically, it's a separate directive in the docker-compose.yml
file that is exclusive to CI.
I feel dumb for not seeing this clearly before.
The mistake that triggered me was that when I ran docker-compose run test test
(first test
is the docker compose directive, the second test
is the of the script sent to CMD
) it didn't change the outputs when I edit the files in my editor. Adding a volume mount to that directive solved it for me locally on my laptop but didn't work in CircleCI for reasons (I can't remember how it errored).
So now we have this:
# In docker-compose.yml frontend: build: context: . dockerfile: Dockerfile.frontend environment: - NODE_ENV=development ports: - "3000:3000" - "35729:35729" volumes: - $PWD/frontend:/app command: start # Same as 'frontend' but no volumes or command frontend-ci: build: context: . dockerfile: Dockerfile.frontend