Gitlab CI/CD

CI/CD stands for Continuous Integration/Continuous Deployment. The concept of finishing a computer application is obsolete. We deliver software in small manageable chunks. To support the process of writing, testing, and deploying these applications, we use Gitlab. The important parts of a Gitlab installation are:

  • Version control and update management using Git
  • Automated tests of these updates
  • Automatic building and compiling of the software into runnable images
  • Deploying the software to a test environment
  • Given the green light, deploying the software to a Production environment

To do this, we will build the following resources:

  1. A Gitlab server that manages the version control aspects and automatically starts automated CI/CD procedures when new updates are committed to the repository
  2. A set of runners based on Docker that can execute the CI/CD jobs
  3. A registry where we can store Docker container images for later deployment

We will have a specific entry point for this named gitlab.nerdhole.me.uk. Apart from standard Docker images and the software itself, we will keep all the resources inside our own environment and not rely on rented or free resources on the Internet.

Virtual hardware

We have the following machines:

Function Class Data(GB) Hostname
Gitlab server heavy 40 labo105.nerdhole.me.uk
Docker server for runners basic 40 labo106.nerdhole.me.uk
Docker server for applications basic 10 labo107.nerdhole.me.uk

The labo105 virtual machine has an alias in DNS: gitlab.nerdhole.me.uk, which we have configured by hand and installed using the ipman role in N-SCHOOL.

Topology

GitlabCICD-01 topology

Components

  • Internet - Gitlab and many other CI/CD tools rely on servers on the Internet to download and install resources.
    • Install script - This script will check your local machine and determine which RPM repository you need
    • Gitlab RPM repo - This is a repository from which you can install Gitlab onto your Linux machine.
  • Gitlab server - The machine that hosts the Gitlab web GUI and all the project information. tasks to run.
    • Gitlab data - A file system mounted on /var/opt/gitlab that contains all the Gitlab information.
  • Gitlab runners - The machine that hosts the Docker containers that execute Gitlab tasks for CI/CD, such as testing, building, or deploying.
    • Runners - A process running on a Linux¹ host that can be given jobs to run by Gitlab, such as building a docker image, running a test, deploying applications, and so on.
  • App servers - The machines that host the actual applications that you develop for the Greater Good.

Note: ¹ They can run on other operating systems, but that is beyond our scope.

Users and groups

Some processes will run as root, but for some processes such as SSH access to Git repositories there is a user git with group git. When Git is installed, it will get a UID/GID from the administrative users pool.

User Group Home Shell Info
git git /var/opt/gitlab /bin/sh Gitlab application user
gitlab-prometheus gitlab-prometheus /var/opt/gitlab/prometheus nologin Event monitoring and alerting user
gitlab-redis gitlab-redis /var/opt/gitlab/redis nologin In-core database user
gitlab-www gitlab-www /var/opt/gitlab/nginx nologin Gitlab NGINX web server
registry registry /var/opt/gitlab/registry nologin Gitlab registry owner
git git /var/opt/gitlab /bin/sh Gitlab application user
gitlab-runner gitlab-runner /var/opt/gitlab/runner /bin/sh Gitlab runner user created separately

Software

We need the following software on the Gitlab server as a prerequisite:

Gitlab itself will be installed from the official Gitlab repositories. First, we download a script called script.rpm.sh to set up the Gitlab repositories. Then, we install the package gitlab-ee using DNF. That package contains all we need to set up the Gitlab server.

For the runner machines, we will need the gitlab-runner package as well as Docker.

Installation of the Gitlab server

We will automate as much of the procedure as we can using Ansible. We will create a role under N-SCHOOL called gitlab. We will use a configuration that sets up the Gitlab server, a Git repository, and a Gitlab container registry.

Configuration file

When you get rid of all the documentation in the comments, the configuration file for Gitlab (/etc/gitlab/gitlab.rb) is refreshingly simple. This is the template:

#########################################
## Nerdhole GitLab configuration settings
#########################################
##! Original annotated config file: /opt/gitlab/etc/gitlab.rb.template
##! We will name our Gitlab server gitlab.main-domain.

letsencrypt['enable'] = false
external_url "https://gitlab.{{nschool.environment.main_domain}}"
registry_external_url "https://gitlab.{{nschool.environment.main_domain}}:4567"

This sets up the Gitlab server on HTTPS, without its Let's Encrypt integration feature. We run the Gitlab container registry for docker on port 4567 on the Gitlab server itself. All other values are left at their defaults.

First: automated steps

This is the first procedure that will take a machine from an empty Linux host to a running Gitlab server, but without the facilities needed for a CI/CD pipeline.

  1. Install Git and OpenSSL if not already there.
  2. Configure the Gitlab repository (/etc/yum.repos.d/gitlab_gitlab-ee.repo)
  3. Install the latest version of gitlab-ee
  4. Create a Firewalld service file for Gitlab. Ports:
    • 4567/tcp - Port for the Container Registry.
    • 5000/tcp (May not be necessary)
  5. Open up http, https, and gitlab on the firewall
  6. Obtain an OpenSSL certificate for gitlab using our certificate_signed role.
  7. Copy the new OpenSSL certificate to the following locations:
    • /etc/docker/certs.d/gitlab.nerdhole.me.uk:5000/ca.crt - For Docker.
    • /etc/gitlab-runner/certs/gitlab.nerdhole.me.crt - For the Gitlab runner
    • /etc/pki/ca-trust/source/anchors/gitlab.nerdhole.me.uk.crt - For the OS, like curl.
  8. Generate the Gitlab configuration
    • Generate the /etc/gitlab/gitlab.rb configuration file
    • Disable Let's Encrypt for Gitlab - that is only for the webserver.
    • Set the external URL to "https://gitlab.nerdhole.me.uk"
    • Set the registry's URL to "https://gitlab.nerdhole.me.uk:4567"
  9. Run the gitlab-ctl reconfigure command to set up Gitlab.

This will set up Gitlab on the server. At this point it can already serve Git repositories, but we need additional steps to enable a CI/CD pipeline to be created.

Second: manual steps

Next, we execute these manual actions:

  1. Find the initial root password in /etc/gitlab/initial_root_password on the Gitlab server.
  2. Log in on https://gitlab.nerdhole.me.uk as root with the password you just found. You will be shown the Web GUI welcome screen with a few actions to perform now.
  3. Disable the user signup as directed by the to-do item
    • "Your GitLab instance allows anyone to register for an account, which is a security risk on public-facing GitLab instances. You should deactivate new sign ups if public users aren't expected to register for an account."
  4. Disable single origin fallback. It is a security risk.
    • "Your GitLab instance serves VS Code static assets if the Web IDE extension host domain is unreachable. To prevent high-severity security risks, disable the single origin fallback and ensure that the extension host domain is accessible."
  5. Create the first group.
    • We will name this group "pioneers"
    • We leave the group URL unchanged: https://gitlab.nerdhole.me.uk/pioneers.
    • The visibility level will be "internal" so that you need to log into Gitlab as any user to see it.
    • "Who will be using this group" is set to "My company or team".
    • "What will you use this group for?" will be set to "Explore Gitlab..."
    • Since we have no users yet, we do not invite any members.
    • Create the group.
  6. Add your account to the Gitlab server as an administrator.
    • Name: Your name
    • Username: A username of your choice.
    • Email: An email address. Emails are off by default.
    • Under Access, leave "Project limit", "Can create Top-level group", and "Private profile" as is.
    • Set "User type" to "Administrator"
    • Set your profile (Avatar, social links, website) as you will.
    • Save the user.
    • Edit the user and set a password on it.
    • Save the user again.
  7. Add the new user to the pioneers group.
    • Under the "Overview" menu, go to "Groups".
    • Select the "pioneers" group.
    • In the "Group members" section, click "Manage access"
    • Click "Invite members"
    • Click on "username, name, or email address" and select your own user account.
    • Set the role to "Owner"
    • Click "Invite"
  8. Log in as the new user
    • From the top right profile picture menu, click "Sign out"
    • Sign in as your new user. You will be asked to change your password. Do so.
    • Add an SSH key to clone projects and submit changes. This assumes you already have an SSH keypair.
      • Click your avatar and select "Edit profile"
      • In the left side bar, select "SSH keys" from "Access"
      • Click "Add new key"
      • Under "Add an SSH key", paste the contents of your public keyfile into "Key".
      • Leave all else at its default.
      • Click "Add key"
  9. Create the first project
    • click on "Create a new project." Select "Create blank project" for the first project.
    • Name the first project "landinsight".
    • Set the project URL to "https://gitlab.nerdhole.me.uk/pioneers/landinsight"
    • Set "Visibility level" to "Internal"
    • Leave the "Project Configuration" as-is.
    • Click "Create project"

You now have one server with one group, one user, and one project.

Setup of the Gitlab runners

We have a dedicated virtual machine that can run Gitlab runners of various sorts. In our environment, it is a standard basic CentOS Stream 9 virtual machine configured as a Docker server by the N-SCHOOL installation procedures.

Automated steps

We will create a new role gitlab-runner that will install the Gitlab runner on an N-SCHOOL CentOS server as a service and start it. The role will:

  1. Create the gitlab-runner group
  2. Create the gitlab-runner user
  3. Install the Gitlab server's OpenSSL certificate
    • Use the openssl command to download the Gitlab server's certificate
    • Install it into /etc/gitlab-runner/certs/gitlab.nerdhole.me.uk.crt
  4. Install the gitlab runner from the Gitlab repository
    • Download the script that enables the gitlab_gitlab-ce repo from gitlab.com.
    • Run the script.
    • Install gitlab-runner from the gitlab_gitlab-ce repo. This also installs gitlab-runner-helper-images as a dependency.
    • Set up the gitlab runner as a service into Systemd

Manual steps

To allow Gitlab to use the runner, we need to register it. This is a manual action to be performed from the Gitlab web GUI. The WebGUI will then give us the commands to run on the Gitlab runner.

Setup of the Gitlab container registry

We use Gitlab's docker container registry to store the Docker images we create ourselves. Populating this registry is part of the Gitlab CI/CD pipeline. We will be hosting the registry on the gitlab server itself, and we will be using our Gitlab credentials to log in using the docker login command. This is all standard Gitlab functionality. All we need to do is enable it.

References