CircleCI’s machine runner installation with Docker

1+ year ago4 min read
Last updated • Read time
Server v4.x
This document is applicable to CircleCI Server v4.x
Server v3.x
This document is applicable to CircleCI Server v3.x

This page describes how to install CircleCI’s machine runner with the Docker executor on server. If you are looking to set up self-hosted runners in a private Kubernetes cluster, please visit the Container runner page.

The container runner is the recommended approach for running containerized jobs on self-hosted runners. Container runner offers the ability to seamlessly define, publish, and use custom Docker images during job execution, as well as the ability to easily manage dependencies or libraries through custom Docker images instead of enumerating dependencies as part of steps in the .circleci/config.yml file.

Machine-based approach with Docker

Runner terms

Before you can install self-hosted runners through the web app, you will need to agree to the CircleCI Runner Terms. To be able to gain access to the Self-Hosted Runners section of the CircleCI web app or your CircleCI server app, an admin in your organization needs to navigate to Organization Settings  Self-Hosted Runners, and agree to the terms.

Runner terms and conditions

Once the terms have been accepted, Self-Hosted Runners will appear permanently in the side navigation.

CircleCI mirrors VCS permissions for organizations. If you are an admin on your organization’s VCS, you are an admin on CircleCI. If you are unsure, check the admin permissions on your VCS.

Prerequisites

Additional prerequisites with Docker

The host needs to have Docker installed. Once the runner container is started, the container will immediately attempt to start running jobs. The container will be reused to run more jobs indefinitely until it is stopped.

The number of containers running in parallel on the host is constrained by the host’s available resources and your jobs' performance requirements.

1. Create namespace and resource class

In order to install self-hosted runners, you will need to create a namespace and resource class token. To create resource classes and tokens, you need to be an organization admin in the VCS provider. You can read about namespaces and resource classes on the Concepts page.

You can view your installed runners on the inventory page, by clicking Self-Hosted Runners on the left navigation.

  1. On the CircleCI web app, navigate to Self-Hosted Runners.

    Runner set up
  2. If this is your organization’s first time using self-hosted runners, you will need to create a namespace as the first part of this step. Each organization can only create a single namespace. While not required, we suggest using a lowercase representation of your CircleCI account name. CircleCI will populate your org name as the suggested namespace by default in the UI.

    If your organization already uses orbs, do not create a new namespace. Instead, enter the namespace your organization uses for orbs.

  3. The second input field is to name your self-hosted runner resource class.

    Runner set up
  4. Copy and save the resource class token. Self-hosted runners use this token to claim work for the associated resource class.

    Runner set up
  5. Select the Machine tab for installation instructions specific to setting up self-hosted runners on Linux, macOS, or Windows. These platform-specific instructions are also included in the next section of this installation guide.

    Machine runner in the web app
    Figure 1. Install machine runner through the web app
  6. You will be presented with configuration code for your new resource class. If you have an existing .circleci/config.yml file, add your new resource class below the existing jobs section, or add the whole jobs section if you do not yet have any configured. If you do not have an existing .circleci/config.yml file, you can create the file and copy all of the code over.

    Runner set up

2. Create a Dockerfile that extends the machine runner image

Create a Dockerfile.runner.extended file. In this example, Python 3 is installed on top of the base image.

FROM circleci/runner:launch-agent
RUN sudo apt-get update; \
    sudo apt-get install --no-install-recommends -y \
        python3

3. Build the Docker image

docker build --file ./Dockerfile.runner.extended .

4. Start the Docker container

CIRCLECI_RESOURCE_CLASS=<resource-class> CIRCLECI_API_TOKEN=<runner-token> docker run --env CIRCLECI_API_TOKEN --env CIRCLECI_RESOURCE_CLASS --name <container-name> <image-id-from-previous-step>

When the container starts, it will immediately attempt to start running jobs.

Start the Docker container on server

When starting the docker container on server, the agent_version and LAUNCH_AGENT_API_URL environment variables will need to be passed in using the --env flag.

CIRCLECI_RESOURCE_CLASS=<resource-class> CIRCLECI_API_TOKEN=<runner-token> agent_version=<agent_version_for_server> LAUNCH_AGENT_API_URL=<server_host_name> docker run --env agent_version --env LAUNCH_AGENT_API_URL --env CIRCLECI_API_TOKEN --env CIRCLECI_RESOURCE_CLASS --name <container-name> <image-id-from-previous-step>

Stopping the Docker container

docker stop <container-name>

Remove the Docker container

In some cases you might need to fully remove a stopped machine runner container from the system, such as when recreating a container using the same name.

docker stop <container-name>; docker rm <container-name>;