Configuring a GitLab Runner to Use a Paid Docker Hub Account

CI/CD at Fenris

At Fenris, as stated in other blogs, we adhere to a specific set of software engineering best practices. These best practices allow us to iterate rapidly on our core software and services. One of these best practices is known as Continuous Integration/ Continuous Deployment (CI/CD). There are entire courses and books dedicated to this topic. This post is intended as a useful resource for engineers who may encounter a specific challenge we encountered that wasn’t well-documented online.

Our CI/CD Setup

  • Version Control System: Gitlab
  • CI/CD System: Self-hosted Gitlab Runner application on Linux instance
  • Docker Image Registries:
    • Docker Hub
      • Docker Hub is the default image registry for all Docker users.
      • Fenris has a paid account with Docker, allowing us to host/access our own private registry and pull thousands of images a day as part of our CI/CD pipelines.

Problem

While many teams use runners which are provided and hosted by Gitlab, our specific needs require us to use runners which we host and control on our AWS infrastructure. This has served us well, and in the past we have had our runner pulling standard images anonymously from Docker Hub, as well as both public and private images from AWS ECR. We wanted to configure our Gitlab Runner to use the Fenris Docker Hub account, allowing us to avoid limitations like rate-limiting when using Docker anonymously. We were surprised to find that this wasn’t nearly as simple as expected. There’s a lot of info about connecting to private repositories out there, but none that describe this problem.

Background

If you run a Docker executor in your GitLab Runner, it will connect to Docker Hub in order to pull images to be used by the jobs in the pipelines. An anonymous login can be problematic, as that limits the number of pulls to 100 in a 6 hour period, and limiting your CI/CD jobs is never a good idea. In addition, it’s often desirable to create your own private images and use those instead of public images which may contain unknown security risks. In order to do this, we need to set the DOCKER_AUTH_CONFIG variable for the pipeline. There’s good documentation here about the different ways to set that variable for the different scopes in your system, but very little info about what needs to go into the value.

Getting Started

This first step is optional, but it’s a Very Good Idea. To get the GitLab CI Runner connected to Docker Hub, you’re going to need to give the system a set of credentials. You don’t really want to use the password for the account. This is what User Access Tokens are for. Log into your Docker Hub account, select “Account Settings” from the dropdown on the right under your profile picture and then select the “Security” tab. Follow the directions to create a user access token and copy it to a secure location.

Configuring the the CI Runner

As we mentioned before, you need to put your Docker Hub credentials into the DOCKER_AUTH_CONFIG variable. The information you need is stored in the config.json file that Docker creates in its .docker directory when you log into a Docker registry. You want to create that file with your Docker Hub credentials so you can use it in the GitLab CI system.

Linux – the easy way

The easiest way to get the content you need is to log into a Linux instance (preferably your GitLab Runner if you’re rolling your own) with Docker installed. You can also use WSL if you’ve installed an image that includes docker. If you regularly use this machine with Docker, you may want to save your existing ~/.docker/config.json file by renaming it and restoring it when you are done. If you can’t work on the actual instance that is running the GitLab CI Runner, try to make sure that the version of Docker installed is as close to that of the Runner as possible, or a number of tricky or difficult to catch issues could arise due to differences in behavior between Docker versions.

Enter docker logout at the command prompt to make sure there aren’t any existing connections. Now enter docker login and follow the prompts.

Important: Use the user access key that you created above when prompted for your password.

Now look in the .docker directory in your home directory. You should see a file at the path .docker/config.json You should see the contents of the file are something like

{
    "auths": {
            "https://index.docker.io/v1/": {
                "auth": "bkksfofakiklefngalfYWNjrajiILKeQ=="
             }
    }
}

If you see this

{
    "auths": {
            "https://index.docker.io/v1/": {}
    },
    "credsStore": "desktop"
}

do the following

  1. 1. Log out of Docker Hub: docker logout
  2. 2. Edit the config.json file, removing the entry for the credsStore. Don’t forget to remove the comma on the preceding line.
  3. 3. Repeat the Docker Hub login. Now the file should look like the one above. If it doesn’t, you’ll have to follow the “Hard Way” instructions below.

If your config.json file contains other objects in addition to the auths object, you can remove them. Make sure you have valid JSON.

Copy the contents of the config.json file into the DOCKER_AUTH_CONFIG variable in the GitLab CI system, and run a pipeline. If everything is working, you’ll see a line like

Authenticating with credentials from \$DOCKER_AUTH_CONFIG

near the top of the CI Console for the job.

The Hard Way

Modern desktop systems have secure credential stores, and Docker logically puts the sensitive authorization information there. Most of the time, using a credentials store is the right thing to do, but in this case, we want to get the raw data that Docker uses to authenticate in order to use it to configure the GitLab CI system’s DOCKER_AUTH_CONFIG variable. Luckily we can work around this using a few simple commands on the command line.

You will want to use the following snippet of JSON as a starting point:

{
    "auths": {
            "https://index.docker.io/v1/": {
                "auth": "<key_will_go_here>"
             }
     }
}

 Next, you will want to generate the auth key, using your username and access token. Run the following command, replacing the username and password strings with the actual values. echo -n "username:password" | base64 This command works on either Linux or Mac. When run in the terminal, it should output an encoded string which ends with an equals sign.

IMPORTANT: the -n part of the command needs to be present to prevent the echo command from adding a newline to the string before passing it to be encoded. If you don’t use this, the encoded string will be incorrect and prevent Docker from authenticating.

BONUS TIP: If you don’t have access to a *NIX shell, you can use this online Base64 encoder to do this step.

  1. 1. Copy the encoded string returned by the previous command, and carefully paste it as the value for the “auth” key in the above JSON.
  2. 2. Once this value is placed into the JSON above as the “auth” value, paste the entire JSON into the DOCKER_AUTH_CONFIG variable in the GitLab CI system, and run a pipeline to test as above.

Why it works

Docker stores its authentication information in a file called config.json in the .docker directory. This file can contain other information, but it’s primarily used for authentication to Docker repositories like Docker Hub. The auths object can have several entries, each keyed by the URL of the registry it will connect to. You can find out more about this file here. We’ve followed these steps in order to find the current URL for Docker Hub https://index.docker.io/v1/ and to find the Basic Authorization string used to connect to the repository. We couldn’t find any documentation of that URL, so you should assume it may change over time, and verify it using the steps above.

A note about security

The auth string that we created is a Base64 encoding of the username and access code used to connect to Docker Hub. Base64 is a reversible encoding, so that means that anyone that sees that string will be able to extract your credentials. This is one reason we recommend using an access token above. Access tokens for Docker Hub are configured to allow the account owner to provide granular permissions, ensuring that if an access token is found and used by a nefarious actor, they won’t be able to use the token to access the account and do further damage.

Posted in