- Docker Repository
- Docker Registry-mirrors Authentication
- Docker Registry Mirror Multiple
- Docker Registry Mirrors For Sale
- Docker Registry Mirrors China
- Docker Registry Mirror Gcr.io
As part of SUSE Hackweek 17 I decided to work on afully fledged docker registry mirror.
Nov 26, 2020 Docker Registry is a centralized application that can be used to store your images and share them with other users. The private registry gives you full control to protect your images. You can also use Docker Hub to store your images, but these images will be public and anyone can access them. Docker Registry Mirror Helm Chart. This directory contains a Kubernetes chart to deploy a private Docker Registry Mirror that will run the registry as a 'pull through cache' and cache the requests to Docker hub. The Docker daemon checks the Container Registry cache and fetches the images if it exists. If your daemon configuration includes other Docker mirrors, the daemon checks each one in order for a. Dec 15, 2016 I expect docker to use the registry specified in the registry-mirrors daemon config when retrieving images with a pull command or build of a Dockerfile. We're using a nexus registry as a pass-thru cache of dockerhub. Actual behavior. Docker ignores the registry mirror config and goes straight to dockerhub registry. Sep 28, 2016 Mirror 是一种代理中转服务，我们(指daocloud)提供的 Mirror 服务，直接对接 Docker Hub 的官方 Registry。 Docker Hub 上有数以十万计的各类 Docker 镜像。 在使用 Private Registry 时，需要在 Docker Pull 或 Dockerfile 中直接键入 Private Registry 的地址，通常这样会导致与 Private Registry.
You might wonder why this is needed, after all it’s already possible to run adocker distribution (aka registry) instance as apull-through cache. Whilethat’s true, this solution doesn’t address the needs of more “sophisticated”users.
Based on the feedback we got from a lot of SUSE customers it’s clear that a simpleregistry configured to act as a pull-through cache isn’t enough.
Let’s go step by step to understand the requirements we have.
On-premise cache of container images
First of all it should be possible to have a mirror of certain container imageslocally. This is useful to save time and bandwidth. For example there’s noreason to download the same image over an over on each node of a Kubernetescluster.
A docker registry configured to act as a pull-through cache can help with that.There’s still need to warm the cache, this can be left to the organic pullof images done by the cluster or could be done artificially by some scriptrun by an operator.
Unfortunately a pull-through cache is not going to solve this problem fornodes running inside of an air-gapped environment. Nodes operated in such anenvironment are located into a completely segregated network, that would make itimpossible for the pull-through registry to reach the external registry.
Retain control over the contents of the mirror
Cluster operators want to have control of the images available inside of thelocal mirror.
For example, assuming we are mirroring the Docker Hub, an operator might befine with having the
library/mariadb image but not the
When operating a registry configured as pull-through cache, all the images ofthe upstream registry are at the reach of all the users of the cluster. It’sjust a matter of doing a simple
docker pull to get the image cached intothe local pull-through cache and sneak it into all the nodes.
If you run across a battery that is so dead it won't even kick on your battery charger, I discovered a little trick. All you need is a somewhat good battery. Definition of dead battery.: a battery that has no more electricity. You forgot any electric consumers on. One of the reasons for an empty battery is human error. Iphone dead battery. No dome light when opening the door or no door chime with keys inserted.
Moreover some operators want to grant the privilege of adding images to the localmirror only to trusted users.
There’s life outside of the Docker Hub
The Docker Hub is certainly the most known container registry. However there arealso other registries being used: SUSE operates its own registry, there’s Quay.io,Google Container Registry (aka
gcr) and there are even user operated ones.
A docker registry configured to act as a pull-through cache can mirror only oneregistry. Which means that, if you are interested in mirroring both the DockerHub and Quay.io, you will have to run two instances of docker registrypull-through caches: one for the Docker Hub, the other for Quay.io.
This is just overhead for the final operator.
A better solution
During the last week I worked to build a PoC to demonstrate we can create a dockerregistry mirror solution that can satisfy all the requirements above.
I wanted to have a single box running the entire solution and I wanted all thedifferent pieces of it to be containerized. I hence resorted to use anode powered by openSUSE Kubic.
I didn’t need all the different pieces of Kubernetes, I just needed
kubelet sothat I could run it in disconnected mode. Disconnected means the
kubeletprocess is not connected to a Kubernetes API server, instead it reads PODsmanifest files straight from a local directory.
The all-in-one box
I created an openSUSE Kubic node and then I started by deploying a standarddocker registry.This instance is not configured to act as a pull-through cache. However itis configured to use an external authorization service. This is needed to allowthe operator to have full control of who can push/pull/delete images.
I configured the registry POD to store the registry data to a directory on themachine by using a Kubernetes hostPathvolume.
On the very same node I deployed the authorization service needed by thedocker registry. I choose Portus, an open source solutioncreated at SUSEa long time ago.
Portus needs a database, hence I deployed a containerized instance of MariaDBon the same node. Again I used a Kubernetes
hostPath to ensure the persistenceof the database contents. I placed both Portus and its MariaDB instance into thesame POD. I configured MariaDB to listen only to localhost, making it reachableonly by the Portus instance (that’s because they are in the sameKubernetes POD).
I configured both the registry and Portus to bind to a local unix socket,then I deployed a container running HAProxy to expose both of them tothe world.
The HAProxy is the only container that uses the host network. Meaning it’sactually listening on port 80 and port 443 of the openSUSE Kubic node.
I went ahead and created two new DNS entries inside of my local network:
registry.kube.lan: this is the FQDN of the registry
portus.kube.lan: this is the FQDN of portus
I configured both the names to be resolved with the IP address of my containerhost.
I then used cfssl to generate a CA andthen a pair of certificates and keys for
Finally I configured HAProxy to:
- Listen on port 80 and 443.
- Automatically redirect traffic from port 80 to port 443.
- Perform TLS termination for registry and Portus.
- Load balance requests against the right unix socket usingthe Server Name Indication (SNI).
By having dedicated FQDN for the registry and Portus and by using HAProxy’s SNIbased load balancing, we can leave the registry listening on a standard port(
443) instead of using a different one (eg:
5000). In my opinion that’s a bigwin, based on my personal experience having the registry listen on a non standardport makes things more confusing both for the operators and the end users.
Once I was over with these steps I was able to log into
https://portus.kube.lanand perform the usual setup wizard of Portus.
We now have to mirror images from multiple registries into the local one, buthow can we do that?
Sometimes ago I stumbled over this tool,which can be used to copy images from multiple registries into a single one.While doing that it can change the namespace of the image to put it all theimages coming from a certain registry into a specific namespace.
I wanted to use this tool, but I realized it relies on the docker open-sourceengine to perform the pull and push operations. That’s a blocking issue for mebecause I wanted to run the mirroring tool into a container without doing nastytricks like mounting the docker socket of the host into a container.
Basically I wanted the mirroring tool to not rely on the docker open sourceengine.
At SUSE we are already using and contributing toskopeo, an amazing toolthat allows interactions with container images and container registries withoutrequiring any docker daemon.
The solution was clear: extend skopeo to provide mirroring capabilities.
I drafted a design proposal with my colleague Marco Vedovati,started coding and then ended up with this pull request.
While working on that I also uncovered a small glitchinside of the
containers/image library used by skopeo.
Using a patched skopeo binary (which include both the patches above) I thenmirrored a bunch of images into my local registry:
The first command mirrored only the
busybox:musl container image from theDocker Hub to my local registry, while the second command mirrored all the
coreos/etcd images from the quay.io registry to my local registry.
Since the local registry is protected by Portus I had to specify my credentialswhile performing the sync operation.
sync commands is not really practical, that’s why we addeda
source-file flag. That allows an operator to write a configuration fileindicating the images to mirror. More on that on a dedicated blog post.
At this point my local registry had the following images:
- … more
As you can see the namespace of the mirrored images is changed toinclude the FQDN of the registry from which they have been downloaded.This avoids clashes between the images and makes easier to track theirorigin.
Mirroring on air-gapped environments
As I mentioned above I wanted to provide a solution that could be used alsoto run mirrors inside of air-gapped environments.
The only tricky part for such a scenario is how to get the images from theupstream registries into the local one.
This can be done in two steps by using the
skopeo sync command.
We start by downloading the images on a machine that is connected to the internet.But instead of storing the images into a local registry we put them on a localdirectory:
This is going to copy all the versions of the
quay.io/coreos/etcd image intoa local directory
/media/usb-disk is the mount point of an external USB drive.We can then unmount the USB drive, scan its contents with some tool, andplug it into computer of the air-gapped network. From this computer wecan populate the local registry mirror by using the following command:
This will automatically import all the images that have been previously downloadedto the external USB drive.
Pulling the images
Now that we have all our images mirrored it’s time to start consuming them.
It might be tempting to just update all our
Dockerfile(s), Kubernetesmanifests, Helm charts, automation scripts, …to reference the images from
registry.kube.lan/<upstream registry FQDN>/<image>:<tag>.This however would be tedious and unpractical.
As you might know the docker open source engine has a
--registry-mirror.Unfortunately the docker open source engine can only be configured to mirror theDocker Hub, other external registries are not handled.
This annoying limitation lead me and Valentin Rothbergto create this pull requestagainst the Moby project.
Valentin is also porting the patch against libpod,that will allow to have the same feature also inside ofCRI-O and
During my experiments I figured somelittle bitswere missing from the original PR.
I built a docker engine with the full patchapplied and I created this
/etc/docker/daemon.json configuration file:
Then, on this node, I was able to issue commands like:
That resulted in the image being downloaded from
registry.kube.lan/quay.io/coreos/etcd:v3.1,no communication was done against
What about unpatched docker engines/other container engines?
Everything is working fine on nodes that are running this not-yet merged patch,but what about vanilla versions of docker or other container engines?
I think I have a solution for them as well, I’m going to experiment a bitwith that during the next week and then provide an update.
Show me the code!
This is a really long blog post. I’ll create a new one with all the configurationfiles and instructions of the steps I performed. Stay tuned!
In the meantime I would like to thankMarco Vedovati,Valentin Rothberg for their help with skopeo andthe docker mirroring patch, plus Miquel Sabaté Solàfor his help with Portus.
to create a local pull-through registry to speed up image pulling in a Kind cluster, run:
you can’t use this pull-through proxy registry to push your own images (e.g. to speed up Tilt builds), but you can create two registries (one for caching, the other for local images). See this section for more context; the lines are:
in case you often create & delete Kind clusters, using a local registry that serves as a proxy avoids redundant downloads
KIND_EXPERIMENTAL_DOCKER_NETWORKis useful but remember that the default network (
bridge) doesn’t have DNS resolution for container hostnames
the Docker default network (
bridge) has limitations as detailed by Docker.
If you play with ClusterAPI with its Docker provider, you might not be able to use a local registry due to the clusters being created on the default network, which means the “proxy” hostname won’t be resolved (but we could work around that).
Kind is an awesome tool that allows you to spin up local Kubernetes clusters locally in seconds. It is perfect for Kubernetes developers or anyone who wants to play with controllers.
One thing I hate about Kind is that images are not cached between two Kind containers. Even worse: when deleting and re-creating a cluster, all the downloaded images disappear.
In this post, I detail my discoveries around local registries and why the default Docker network is a trap.
Kind has no image caching mechanism
Whenever I re-create a Kind cluster and try to install ClusterAPI, all the (quite heavy) images have to be re-downloaded. Just take a look at all the images that get re-downloaded:
That’s a total of 418 MB that get re-downloaded every time I restart both clusters!
Unfortunately, there is no way to re-use the image registry built into your default Docker engine (both on Linux and on macOS). One solution to this problem is to spin up an intermediary Docker registry in a side container; as long as this container exists, all the images that have already been downloaded once can be served from cache.
Creating a caching proxy registry
We want to create a registry with a simple Kind cluster; let’s start with the registry:
--net kindis required because Kind creates its containers in a separate network; it does that the because the “bridge” has limitations and doesn’t allow you to use container names as DNS names:
By default, a container inherits the DNS settings of the host, as defined in the
/etc/resolv.confconfiguration file. Containers that use the default bridge network get a copy of this file, whereas containers that use a custom network use Docker’s embedded DNS server, which forwards external DNS lookups to the DNS servers configured on the host.
which means that the container runtime (containerd) that runs our Kind cluster won’t be able to resove the address
REGISTRY_PROXY_REMOTEURLis required due to the fact that by default, the registry won’t forward requests. It simply tries to find the image in
/var/lib/registry/docker/registry/v2/repositoriesand returns 404 if it doesn’t find it.
Using the pull-through feature (I call it “caching proxy”), the registry will proxy all requests coming from all mirror prefixes and cache the blobs and manifests locally. To enable this feature, we set
Other interesting bit about
REGISTRY_PROXY_REMOTEURL: this environement variable name is mapped from the registry YAML config API. The variable
is equivalent to the following YAML config:
⚠️ The registry can’t be both in normal mode (“local proxy”) and in caching proxy mode at the same time, see below.
Creating a Kind cluster that knows about this caching proxy registry
The second step is to create a Kind cluster and tell the container runtime to use a specific registry; here is the command to create it:
containerdConfigPatches is a way to semantically patch
/etc/containerd/config.conf. By default, this file looks like:
Note 2: the mirror prefix (
docker.io) can be omitted for images stored on Docker Hub. For other registries such as
gcr.io, this mirror prefix has to be given. Here is a table with some examples of image names that are first prepended with “docker.io” if the mirror prefix is not present, and we get the final address by mapping these mirror prefixes with mirror entries:
|image name||“actual” image name||registry address w.r.t. mirrors|
Check that the caching proxy registry works
Let’s see if the proxy registry works by running a pod:
We can also see through the registry logs that everything is going well:
Docker proxy vs. local registry
A bit later, I discovered that you can’t push to a proxy registry. Tilt is a tool I use to ease the process of developping in a containerized environment (and it works best with Kubernetes); it relies on a local registry in order to cache build containers even when restarting the Kind cluster.
Docker Registry-mirrors Authentication
Either the registry is used as a “local registry” (where you can push images), or it is used as a pull-through proxy. So instead of configuring one single “proxy” registry, I configure two registries: one for local images, one for caching.
Note that we do use a port-forwarding proxy (
-p 5000:5000) so that we can push images “from the host”, e.g.:
If you use Tilt, you might also want to tell Tilt that it can use the local registry. I find it a bit weird to have to set an annotation (hidden Tilt API?) but whatever. If you set this:
then Tilt will use
docker push localhost:5000/you-image (from your host, not from the cluster container) in order to speed up things. Note that there is a proposal (KEP 1755) that aims at standardizing the discovery of local registries using a configmap. Tilt already supports it, so you may use it!
Improving the ClusterAPI docker provider to use a given network
Docker Registry Mirror Multiple
When I play with ClusterAPI, I usually use the CAPD provider (ClusterAPI Provider Docker). This provider is kept in-tree inside the cluster-api projet.
I want to use the caching mechanism presented above. But to do that, I need to make sure the clusters created by CAPD are not created on the default network (current implementation creates CAPD clusters on the default “bridge” network).
Docker Registry Mirrors For Sale
I want to be able to customize the network on which the CAPD provider creates the container that make up the cluster. Imagine that we could pass the network name as part of a DockerMachineTemplate (the content of the
spec is defined in code here):
Docker Registry Mirrors China
Update 26 July 2020: added a section about local registry vs. caching proxy. Reworked the whole post (less noise, more useful information).