Building and Running Go Apps in Docker

Posted by Nathan Osman on July 18, 2017

With CGO_ENABLED=0 set, the Go compiler produces binaries with no runtime dependencies, not even libc. This greatly simplifies deploying the application in Docker since no base image is required. However, there are a few caveats and pitfalls that I would like to address in this article.

I am going to walk you through the process of creating caddy-docker. This project provides a single binary that embeds the Caddy HTTP server. In addition, it also watches the Docker daemon for containers being started and stopped. When these events occur, the configuration is reloaded on-the-fly and TLS certificates are automatically obtained. I won’t go into extensive detail on the application itself. Instead, I would like to focus on how the application was built.

My first goal was to set up the build environment in such a way that the Go compiler didn’t actually need to be installed. Docker Hub provides an image that includes everything needed to compile a Go application: golang. In order to build a Go application using the container, the source tree must be mounted as a volume. Because the Go compiler places binaries in the container’s /go/bin directory, a second volume is also needed.

The Docker invocation ends up looking something like this:

PKG=github.com/nathan-osman/caddy-docker
CMD=caddy-docker

docker run \
    --rm \
    -v `pwd`/dist:/go/bin \
    -v `pwd`:/go/src/$PKG \
    -w /go/src/$PKG \
    golang \
    go get ./...

The first problem with this command is that the resulting binary is dynamically-linked, requiring libc to run:

$ ldd caddy-docker
    linux-vdso.so.1 =>  (0x00007ffce25ad000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbb0b8eb000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbb0b523000)
    /lib64/ld-linux-x86-64.so.2 (0x0000562dfe37b000)

We can fix that by adding the following argument to the Docker command:

-e CGO_ENABLED=0

In order to keep the build process simple, we can use a Makefile. Since make only builds targets that are out of date, we can avoid running the Docker command altogether if none of the source files have changed. We can also implement a clean target to clear the cache and remove the binaries.

CWD = $(shell pwd)
PKG = github.com/nathan-osman/caddy-docker
CMD = caddy-docker

SOURCES = $(shell find -type f -name '*.go')

all: dist/${CMD}

dist/${CMD}: ${SOURCES} | dist
    docker run \
        --rm \
        -e CGO_ENABLED=0 \
        -v ${CWD}/dist:/go/bin \
        -v ${CWD}:/go/src/${PKG} \
        -w /go/src/${PKG} \
        golang \
        go get ./...

dist:
    @mkdir dist

clean:
    @rm -rf dist

.PHONY: clean

There is a lot going on here, so I’ll highlight a couple important points:

  • The SOURCES variable is initialized to a list of all files in the project with the .go extension. If the modification time of any Go source file is newer than the binary, the binary is rebuilt.

  • The dist directory needs to be created before the Docker command is run. This ensures that the current user is the owner. Otherwise, the directory would be owned by root.

Now that we have a Makefile, we can build the project with a single command:

make

Having to fetch all of the packages that the application imports every time the binary is rebuilt seems like a waste. A third volume could be used for persisting the files in /go/src. This requires the following modification to the Makefile:

  • ! -path './cache/*' added to the end of the SOURCES command
  • cache added as a dependency of dist/${CMD}
  • -v ${CWD}/cache/lib:/go/lib and -v ${CWD}/cache/src:/go/src added to the Docker command
  • another target named cache that executes @mkdir cache
  • cache added to the @rm command

We now have a new problem. The make clean command will instantly fail. Even though the directory is owned by the current user, all of the subdirectories and their contents are owned by root:

$ make clean
rm: cannot remove 'cache/github.com/sirupsen/logrus/logger_bench_test.go': Permission denied
rm: cannot remove 'cache/github.com/sirupsen/logrus/alt_exit.go': Permission denied
...

The only way to avoid this problem is to have the commands in the container run as a different user — and not just any user, but one with the same user and group ID. At this point, I decided to create a new image based off golang that tackled this problem: bettergo.

The heart of this image is a script named bettergo.sh that wraps all of the commands:

#!/bin/bash

# Create a new group and user with the correct values
groupadd -g $GID $USER
useradd -mu $UID -g $GID $USER

# Ensure /go is owned by the user
chown -R $UID:$GID /go

# Switch to the specified user's account and run the command
sudo -EHu $USER env "PATH=$PATH" "$@"

Notice that the script uses the values of $UID and $GID for the user ID and group ID of the new user. These are environment variables that will be passed to the script via the Docker command. sudo doesn’t preserve the environment by default, so we need to use the -E flag to override this behavior. Even with this flag present, $PATH must still explicitly be added.

The next step is to modify the Makefile to use the bettergo image:

CWD = $(shell pwd)
PKG = github.com/nathan-osman/caddy-docker
CMD = caddy-docker

UID = $(shell id -u)
GID = $(shell id -g)

SOURCES = $(shell find -type f -name '*.go' ! -path './cache/*')

all: dist/${CMD}

dist/${CMD}: ${SOURCES} | cache dist
    docker run \
        --rm \
        -e CGO_ENABLED=0 \
        -e UID=${UID} \
        -e GID=${GID} \
        -v ${CWD}/cache/lib:/go/lib \
        -v ${CWD}/cache/src:/go/src \
        -v ${CWD}/dist:/go/bin \
        -v ${CWD}:/go/src/${PKG} \
        -w /go/src/${PKG} \
        nathanosman/bettergo \
        go get ./...

cache:
    @mkdir cache

dist:
    @mkdir dist

clean:
    @rm -rf cache dist

.PHONY: clean

Once again, we have another problem. Because we specified CGO_ENABLED=0, the Go compiler will rebuild the standard library and attempt to write it to disk. This means we need to modify our Docker command once again to do things a little bit differently:

docker run \
    --rm \
    -e CGO_ENABLED=0 \
    -e UID=${UID} \
    -e GID=${GID} \
    -v ${CWD}/cache/lib:/go/lib \
    -v ${CWD}/cache/src:/go/src \
    -v ${CWD}/dist:/go/bin \
    -v ${CWD}:/go/src/${PKG} \
    nathanosman/bettergo \
    go get -pkgdir /go/lib ${PKG}/cmd/${CMD}

Before we move on, there is one more item we need to add to our Makefile. caddy-docker uses fileb0x for embedding static content for the HTTP server into the binary. fileb0x reads a configuration file (b0x.yaml) and produces a Go source file (server/ab0x.go) that includes the content of the static files. Thus, a new variable and two new targets must be added to the Makefile:

BINDATA = $(shell find server/static)

server/ab0x.go: ${BINDATA} | dist/fileb0x
    dist/fileb0x b0x.yaml

dist/fileb0x: | dist
    docker run \
        --rm \
        -e CGO_ENABLED=0 \
        -e UID=${UID} \
        -e GID=${GID} \
        -v ${CWD}/cache/lib:/go/lib \
        -v ${CWD}/cache/src:/go/src \
        -v ${CWD}/dist:/go/bin \
        nathanosman/bettergo \
        go get -pkgdir /go/lib github.com/UnnoTed/fileb0x

The last target creates the fileb0x binary which is used to produce the server/ab0x.go file. We need to add the file to the dist/${CMD} target:

dist/${CMD}: ${SOURCES} server/ab0x.go | cache dist

We also need to make sure the server/ab0x.go file is removed when the clean target is built:

@rm -f server/ab0x.go

We now have a Makefile that allows the entire application to be built with a single command. We can now focus on getting the application running within a container. Since the Go application was built with CGO_ENABLED=0, we don’t need to worry about library dependencies. The Dockerfile begins with the following lines:

FROM scratch
MAINTAINER Nathan Osman <nathan@quickmediasolutions.com>

# Add the binary
ADD dist/caddy-docker /usr/local/bin/

Because caddy-docker uses Let’s Encrypt for TLS certificates, we need to add some root CAs to the container. curl conveniently provides these for us and we can add them directly to the container from the web:

# Add the root CAs
ADD https://curl.haxx.se/ca/cacert.pem /etc/ssl/certs/

The rest of the Dockerfile ensures the correct ports are exposed, a volume is created for storing TLS certificates and private keys, and the entrypoint for launching the application is specified:

# Expose ports 80 and 443
EXPOSE 80 443

# Create a volume for the TLS files
VOLUME /var/lib/caddy-docker

# Tell Caddy to use the volume
ENV CADDYPATH=/var/lib/caddy-docker

# No arguments are needed for running the app
ENTRYPOINT ["/usr/local/bin/caddy-docker"]

The application itself and the Docker container can now be built with:

make
docker build -t nathanosman/caddy-docker .

This concludes the article. If you want to learn more about bettergo, you can do so here. Information about caddy-docker can be found here.