Running Go Programs in Docker Scratch Containers

Docker’s one of those weird bits of technology where, depending on how you start using it, you can easily miss some important fundamentals of how it works. When I started working with Docker it was in contexts where “no, it’s not a virtualization technology but also yeah it basically is a virtualization technology just run this command and bam you’re magically running some Linux software” was the norm.

If the past few years I’ve had a chance to reconsider some Docker fundamentals, and I’m starting to get more and more comfortable with the idea that Docker, and containers more generally, are a Linux process technology. Today we’re going to take a quick spin on the different ways you might run a compiled go program in a Docker container and why you might not want a base Linux container image to do it.

Our Program

We’ll start with a hello world program written in Go

// File: main.go
package main
import "fmt"
func main() {
    fmt.Println("hello world")
}

We can run this program directly

% go run main.go
hello world

However, one of the powers of go is the ability to compile a program down to (almost always) statically linked native code. So we can build the program like this.

% go build main.go

Then examine the executable binary we just created

% ls -lh main
-rwxr-xr-x  1 astorm  staff   1.9M Dec 21 11:48 main

% file main
main: Mach-O 64-bit executable x86_64 # ha, I forgot I'm still running a
                                      # rosetta terminal

And then run the program as we would any other.

% ./main
hello world

Inside a Container

I’m working on a Mac running macOS — so this means my program is compiled specifically for my Mac. If I try to run this program on a computer running Linux or in a container I’ll have problems.

% docker run -v `pwd`:/test -i -t ubuntu bash
root@787c551869fb:/# cd /test
root@787c551869fb:/test# ./main
bash: ./main: cannot execute binary file: Exec format error

Trying to run the executable in a container gives us the error cannot execute binary file: Exec format error.

What Just Happened

The docker run command tells Docker you want to execute a program. The -v option tells Docker to mount a folder from your local computer as a folder the container can also see. The

-v `pwd`:/test

uses some shell back tick magic to tell Docker to mount your current working directory at /test “inside” the container. I also could have also used

-v /Users/astorm/Documents/bare-continer:/test

The -t option tells Docker we want to run the command in the container name/tagged ubuntu. If you don’t have a container named/tagged ubuntu on your computer, Docker will download it from Docker Hub. A Docker container’s full name is separated into two segments — a short name and a tag (short-name:tag). If you omit the :tag, Docker will assume a tag of :latest. Culturally “tag” and “name” are often used interchangeably.

The final bash in our invocation tells Docker “don’t run the command that the ubuntu:latest image is configured to run, instead run the program bash inside the container”. The -i option tells Docker we’ll be running a program that’s an interactive shell — without it weird stuff will happen to standard out and standard error.

Once in the container, we navigate to that mounted /test, and then try to execute our program, main.

root@787c551869fb:/# cd /test
root@787c551869fb:/test# ./main
bash: ./main: cannot execute binary file: Exec format error

The computer yells at us because we compiled the program for macOS and not for Linux.

Compiling for Linux

This is a (relatively!) easy program to fix. Back on my Mac we can tell go we want to build an executable for linux by setting the GOOS environment variable.

% env GOOS=linux go build main.go

If we try to run this new binary locally we’ll get an error

% ./main
zsh: exec format error: ./main

This is expected. We compiled this program for Linux, not for MacOS

However, if we hop into the container

% docker run -v `pwd`:/test -i -t ubuntu bash
root@f0e4f05789fe:/# /test/main
hello world

we can now run the program.

Building a Container for Reuse

While Docker’s useful for running containers locally, at some point you’l want to package up a container for reuse elsewhere. Maybe to distribute an open source project, maybe to run a private service at your company.

To do this, you need to create a container image. A container image is made up of the files you need to run your program, the program itself, and the specific CLI invocation of your command. Docker has an entire build language to help you build these container images. These build programs are put in files most commonly named Dockerfile.

To build our container, create a file that looks like this (in the same location as your go program)

# File: Dockerfile
FROM ubuntu
ADD ./main /main
CMD ["/main"]

and then type the following to build your container

% docker build -t my-container-name .

You’ll then be able to use the following to run your command/container

$ docker run my-container-name

What Just Happened

In your Dockerfile, the FROM ubuntu line tells Docker which container image to use as a base image file. We chose ubuntu because that’s what we were using to run our program before.

The ADD ./main /main command adds files from the build machine to the container image. Here we’re adding our built executable file main to the container’s root folder.

Finally, CMD ["/main"] tells Docker which command should be the default command for our container.

When we ran the build command

% docker build -t my-container-name .

we were telling Docker to build a container image named my-container-name, and that it should use the Dockerfile in the current directory (.). You can see the container images available on your machine by running docker images.

 % docker images
REPOSITORY          TAG       IMAGE ID       CREATED          SIZE
my-container-name   latest    5b57c7630cdd   10 minutes ago   71.1MB
//...

Where Docker stores these image files is a complicated story. They are not, unfortunately, just disk images on your machine and the storage format will vary depending on what version of Docker you’re using and how its configured. For example, here’s what the container image storage folder looks like on my Mac

% find ~/Library/Containers/com.docker.docker/Data/vms/0
/Users/astorm/Library/Containers/com.docker.docker/Data/vms/0
/Users/astorm/Library/Containers/com.docker.docker/Data/vms/0/00000002.00001003
/Users/astorm/Library/Containers/com.docker.docker/Data/vms/0/console.sock
/Users/astorm/Library/Containers/com.docker.docker/Data/vms/0/log
/Users/astorm/Library/Containers/com.docker.docker/Data/vms/0/data
/Users/astorm/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw

Finally, when you pass docker run a single container name as an argument Docker will find the container with that name and execute the container’s command.

Big Image

Let’s take a look at the images on our machine again

% docker images
REPOSITORY          TAG       IMAGE ID       CREATED          SIZE
my-container-name   latest    5b57c7630cdd   19 minutes ago   71.1MB
ubuntu              latest    4c2c87c6c36e   12 days ago      69.2MB
// ...

It looks like our image takes up 71.1MB of disk space — which is pretty big. Most of that size comes from our container’s parent image (the ubuntu image). Although we intended to distribute our program with a container, every file in the ubuntu image has come along for the ride.

In some ways this is great — it means we can use our container to run any file that’s part of the ubuntu image. This includes an interactive shell like bash.

% docker run -it my-container-name:latest bash
root@4d1b37d5072f:/#

This is super useful for debugging if something’s going wrong inside a container.

In other ways it’s less great — 71.1 MB is a lot of disk space when the program we want to run is just under 2 MB

% ls -lh main
-rwxr-xr-x  1 astorm  staff   1.8M Dec 21 12:10 main

At the scale of a tutorial? Not a big deal? At a scale where there are hundred or thousands of unique container images? Then it starts to become a problem.

While container technology is often thought of and discussed as though it was a virtualization technology, at its core container technology is about running programs that are isolated from other programs on your computer. While the base Linux images that Docker provides (ubuntu, alpine, etc.) are incredibly useful, they’re not required for running programs that aren’t making calls to dynamically linked libraries.

Put another way, while a python program will (almost) always require a container with a full Linux distribution, a statically compiled go program doesn’t share this requirement.

Scratch Containers

So how can we get rid of that 69 MB of dead weight? We’ll just need to change the FROM line in our container’s build file

FROM scratch
ADD ./main /main
CMD ["/main"]

Here we’re using FROM scatch instead of FROM ubuntu. The scratch keyword doesn’t point to a base container image — instead it’s telling Docker that we don’t want any base container image.

If we build a new container image with this file (changing its name)

% docker build -t my-small-container .

we’ll see we can run it the same as we normally would

% docker run my-small-container
hello world

but its total file size is dramatically smaller than our first.

% docker images
REPOSITORY           TAG       IMAGE ID       CREATED         SIZE
my-small-container   latest    7f0fbcb6ec1b   3 minutes ago   1.94MB
my-container-name    latest    5b57c7630cdd   3 hours ago     71.1MB
#...

Wrap Up

Using scratch containers for your statically compiled program has a number of benefits. It can dramatically reduce the size of your container images, you sidestep any security issues with that base Linux distro, and if you’re writing tutorials it’s a much better demonstration of a container’s true identity as a stand-alone Linux process.

All that said it’s something I would approach with caution. While there’s a certain elegance to this approach, without the standard tools of a Linux distribution like ubuntu, alpine, etc. you may be putting up serious roadblocks to users who will need to debug containers that are running in production.