I must admit that this setup took longer then expected and the suggested solutions were not really cutting it for me. I'm still not sure whether this setup is perfect, but hey, it works. And today we're going to link:

Why should I bother?

First of all, the setup requires no manual configuration except creation of user/pass for registry authentication and telling nginx that 3GB POST requests are fine. The benefits of nginx-proxy are described below. Apart from that a companion container, called, docker-letsencrypt-nginx-proxy-companion (sic!) generates and renews automatically all certs based on the domain you pass as a variable. The registry allows you to have a private space for your images and with hosted alternatives being a bit pricey, I think it's worth the hassle.

NOTE: All this can be done with single docker-compose file which I'll do a separate post about. This gets into more details on what's happening:

Set-up nginx-proxy

Go!

docker run -d -p 80:80 -p 443:443 \  
-v /var/run/docker.sock:/tmp/docker.sock:ro \
--name nginx-proxy \
--label com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy \
-v /some/path/on/host:/etc/nginx/vhost.d \
-v /usr/share/nginx/html \
-v /some/path/on/host/certs:/etc/nginx/certs:ro \
--net="bridge" \
jwilder/nginx-proxy  

If you're interested, read below, else just skip to next section.

  • -d - deamonizes the process instead of running in a foreground. You know it already.
  • -v /var/run/docker.sock:/tmp/docker.sock:ro - this ensures that the magic works by listening on docker socket and checking if a new container has been spawned
  • -v /some/path/on/host:/etc/nginx/vhost.d - this is where the virtual host config regenerated after any addition is updated. And custom configs are put if necessary.
  • -v /usr/share/nginx/html - static dir
  • -v /some/path/on/host/certs:/etc/nginx/certs:ro - this is quite important. This is the place where our companion container will create certificates. :ro means that nginx-proxy is not going to tinker with the certs.
  • --label com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy - this label is used by companion container to tell which nginx-proxy to serve.
  • --net="bridge" - for whatever reason some versions need explicit specification of the network. I personally had problems after restarts when running with this.

Nginx-proxy is a great docker image and a go-to-solution for any kind of container orchestration, where you plan to load balance containers for instance via Swarm. It automagically routes traffic to containers based on environmental variables passed in "target" container. For my private usage, I simply have one proxy that routes to this blog, Gogs, Jenkins, Nextcloud and couple other applications including a private Docker registry each with it's own subdomain a certificate (no SAN).

Setup docker-letsencrypt companion container

The image is specifically created to handle nginx-proxy but it really saves a lot of time.

docker run -d --name nginx-letsencrypt \  
-v /some/path/on/host/certs:/etc/nginx/certs:rw \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
--volumes-from nginx-proxy \
jrcs/letsencrypt-nginx-proxy-companion  

Again, for some more elaborate description (skipping -d, --name and binding the socket):

  • -v /some/path/on/host/certs:/etc/nginx/certs:rw - we again bind our cert dir with /etc/nginx/certs but this time with :rw - the main purpose of this container is to take your certs and write or renew them in this dir.
  • --volumes-from nginx-proxy - this ensures that the companion is using the volumes from our nginx-proxy container.

Setup private docker registry

Here we have some prerequisites. To access the registry form outside, you need to have a dns resolvable domain name. The instructions will also show you how to use it locally (useful for instance for managing the containers with e.g. portainer.io

This one is quite long but it also shows how cool the setup is:

docker run -d -p 127.0.0.1:5000:5000 \  
--restart=always \
-v /another/path/on/host/data:/var/lib/registry \
-v /some/path/on/host/certs:/certs \
-v /another/path/on/host/auth:/auth \
-e VIRTUAL_HOST=xyz.yourdomain.com \
-e VIRTUAL_PORT=5000 \
-e LETSENCRYPT_HOST=xyz.yourdomain.com \
-e LETSENCRYPT_EMAIL=yourmail@mail.com \
-e REGISTRY_AUTH=htpasswd \
-e "REGISTRY_AUTH_HTPASSWD_REALM=My Super Realm" \
-e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
--name pdr
registry:2  
  • -p 127.0.0.1:5000:5000 - this allows to authenticate locally from the host itself. Useful if you also host Jenkins and want to push images locally.
  • --restart=always - this is essential as the registry will die couple of times before the certs are created.
  • -v /some/path/on/host/certs:/certs - we again bind the certs. This is specific to PDR, you don't have to do it for other applications. This is because Docker expects PDR to be secured.
  • -v /another/path/on/host/data:/var/lib/registry - just for persisting the registry data
  • -v /another/path/on/host/auth:/auth - here we're going to maintain our htpasswd. Steps for that are in the next section.

Variables for talking to nginx-proxy:

  • -e VIRTUAL_HOST=xyz.yourdomain.com - tells nginx-proxy that this container should be served when xyz.yourdomain.com is called.
  • -e VIRTUAL_PORT=5000 - this tells nginx-proxy to bind upstream to this port. --expose <port> can be used alternatively.

Variables for talking to docker-letsencrypt-nginx-proxy-companion (ugh...)

  • -e LETSENCRYPT_HOST=xyz.yourdomain.com - again, domain for which certificate should be created. You can use SAN, but for that follow readme
  • -e LETSENCRYPT_EMAIL=yourmail@mail.com - mail for notifications from Letsencrypt.

Registry variables

  • -e REGISTRY_AUTH=htpasswd - method of authentication - in our case htpasswd
  • -e "REGISTRY_AUTH_HTPASSWD_REALM=My Super Realm" - basically a string of the authentication realm
  • -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd - htpasswd file which we'll generate shortly

That's it. PDR should now be running and restarting itself until it finally gets the certs. You can look at the whole process in docker logs <container_name>. In general, nginx-proxy should inform you about regeneration of the config, companion should successfully create .crt and .key with your subdomain's name and registry should just stay alive 😎.

NOTE: This approach has a caveat if you want to, e.g. expose your registry for pulling to everyone and restrict pushing to authenticated users. There are two solutions, either you would need perhaps another container that would inspect the traffic and force authentication on pushing, or just create another registry container, use --with volumes pdr working on different port, with a different subdomain like pdr-public.yourdomain.com but without configuring the htpasswd part. You essentially create a read-only endpoint.

Setup htpasswd and custom config

This snippet is almost straight from Docker docs:

docker run --entrypoint htpasswd registry:2 -Bbn username password > /another/path/on/host/auth/htpasswd

This is the path that we've used as a volume for PDR: -v /another/path/on/host/auth:/auth and then in respective variables on the container side.

The last thing to do is to put the custom config for PDR. This is not needed for most applications, but you're not doing POSTs with 1GB filesizes with most of the applications, am I right?

Just like with maven - convention > configuration, nginx-proxy expects config to be passed in a certain manner. In this example, I'm using per-host settings rather than globals. I want only PDR to accept huge uploads:
echo 'client_max_body_size 3000m;' > /opt/docker/nginx/vhost.d/<domain>_location so in our case: /opt/docker/nginx/vhost.d/xyz.yourdomain.com_location

Testing

Easiest way to test is to simply try pushing an image. If you don't have anything handy, use something from Dockerhub. I used redis to see if default body size has been overwritten corretly (it's couple of MBs by default)

  1. Firstly try logging to the registry:
    docker login -u username -p password pdr.yourdomain.com (or to avoid exposing the password cat ~/secret.txt | docker login -u username --password-stdin to read from standard input). You should see Login Succeeded in stdout.
  2. Pull an image from Dockerhub:
    docker pull redis
  3. Tag the image:
    docker tag $(docker images redis --format="{{.ID}}") pdr.yoursite.com/redis
  4. Puuuuush:
    docker push pdr.yoursite.com/redis

Other examples

Our setup for registry was much more complicated than it is for most containers. As an example consider Nextcloud. Once you have nginx-proxy and letsencrypt companion running adding Nextcloud is a breeze:
docker run -d --expose 80 -e "VIRTUAL_HOST=somedomain.com" -e "LETSENCRYPT_HOST=somedomain.com" -e "LETSENCRYPT_EMAIL=you@mail.com" nextcloud

Done. You have a Nextcloud container proxied by Nginx, and secured with Letsencrypt cert. No setup for the proxy, fpm, no cert renewals. Persist the data volume though ;).

I hope you liked the post, it's lenghty as I wanted to dwell on the details. The next one, the errata of sorts, will show a MUCH easier way to do it with docker-compose.

Tags: