Redhat-like Operating System V10

This is the upgraded version of the design for a Main Server, this time using Rocky Linux v10.2. In general, we want to build a network fit for a small company of a home office to work on - we aim for professional-grade quality here. The main concern for such an environment is confidence. We must be confident that our setup will work, and if not that we can get it to work. The basis of confidence is knowledge. This document provides that knowledge.

Part of any design is security. We will design our environment to take advantage of most of the security features available to Linux. The firewall will be on, SELinux will be enabled, and all network traffic containing user data will be encrypted. We will distinguish between admin users and everyone else - only administrators will be able to work on the environment. The defining mark of an administrator is wheel group membership. We will not at this time support temporary elevations to admin, but will keep our options open to implement it.

Rocky Linux v10.2 has been out for a while now, and our current setup only runs CentOS v9. Rocky 10 has a few interesting features, such as an image mode that greatly speeds up installations based on immutable images. Since we're using that at work, We need to acquaint ourselves with it. We will install Rocky 10 on Paya, an Intel NUC. Once we are satisfied that we can automatically install a Main server, we will upgrade Sypha.

While we can at this moment manually install several different versions of Linux, we want to expand the automated installs so we can automatically install both Debian-like operating systems such as Linux Mint and Redhat-like operating systems such as Rocky Linux and CentOS Stream.

We will try to automate as much as we can using Ansible. Logging in on and hand-hacking systems is not something you want to do in the middle of the night.

We want to make our setups more structured. At the moment, adding a new host to the system is a manual operation that we also want to automate. We want to make the inevitable upgrade to V11 more smooth.

We need to upgrade service management so that we can install, move, and redesign services at will, including runnning Kubernetes clusters, Docker, and other cloud features. Simply reinstalling is not enough. We want to specify and build servers from a single YAML file. We also want a fully fledged CI/CD pipeline based on Gitlab.

Flatpak applications are becoming more and more common these days and we want to support them as widely as possible. For that reason, Flatpak support will be a standard feature of any N-SCHOOL installation.

At the moment, system updates and backups are mostly a manual operation. We will add procedures to run regular backups in an automated and scheduled fashion.

We will try to keep the digital footprint of N-SCHOOL as small as possible, meaning we will prefer services that do not require subscriptions over ones that do.

Things we won't work with

N-SCHOOL will not include support for the following:

• Public cloud systems such as Azure, Google Cloud, Amazon Web Services, or Azure. We run all of our services on our own hardware, using only open-source software.
• Any version of Windows. This is Nerdhole Small Company or Home Office On Linux after all.

Building to-do list

This is what we need to build:

• MUST: N-SCHOOL bootstrap scripts and playbooks
• NICE: Machine creation, update and removal scripts
• NICE: Maintenance scripts (run updates, reconfigure)
• MUST: Updated documentation

The machines

The machines involved in this project are:

Paya - main server

• Intel NUC
• 4 CPUs, Intel(R) Core(TM) i3-8109U CPU @ 3.00GHz
• 8GB RAM
• 1x 1Gb Ethernet adapter
• 250GB NvME boot drive
• 400GB HDD data drive for boot/install services.

Emerald - Workstation/KVM host

• Acer PC
• 8 CPUs AMD
• 12GB RAM
• 1x Gb Ethernet adapter

Rayla - Lenovo Thinkpad laptop.

• Lenovo Thinkpad t750 laptop
• 6 CPUs
• 16GB RAM
• 1x 100Mb Ethernet adapter

The software

This is the software involved in this project:

Operating systems

These are the operating systems:

Operating System	Type	Auto	Mirror	Purpose
Rocky Linux v10.2	Redhat	Yes	Yes	Main server operating system
Rocky Linux v9.8	Redhat	Yes	Yes	Compatibility testing
Linux Mint Debian Edition v7	Debian	Maybe	Maybe	Laptops, special workstations
Ubuntu	Debian	No	No	Special testing
Omarchy	Arch	No	No	Experimental installs

Also considering adding Debian itself to the mix. We won't be adding Redhat because that requires subscriptions.

Rocky Linux v10.2 - main operating system

We will upgrade the main server to Rocky Linux. We will install the main server using a normal Anaconda-based procedure with Kickstart. The seed image will be on a USB stick together with needed files such as ISO images. Subsequent installations will optionally be in image mode or package mode. Rocky 10 will be the default installation for a new machine, virtual or otherwise.

We will maintain a local Rocky 10 mirror to speed up our installs and ease the dependance on Internet resources.

Since Rocky 10, the ISC DHCP server is no longer included, and has been replaced by the KEA DHCP server. We will need to update our IP-man DHCP script to take into account the new configuration file format.

Rocky Linux v9.8 - compatibility testing

In some cases we may want a Rocky 9 installation for compatibility testing or for building packages against the V9 libraries. We will try to keep the installation procedures as similar to Rocky 10 as possible.

Likewise, we will maintain a local Rocky 9 mirror for fast installs.

Linux Mint Debian Edition v7 - laptops or workstations

It is essential that we can support Linux distributions not only based on Redhat but also on Debian. The main differences are that Debian uses a different package manager. Linux Mint is also better positioned for end users, in that user applications are more likely to exist for Mint than for Redhat. If time allows, we will support local workstations based on Mint as well as on Rocky. We will set up a local mirror.

Ubuntu - curiosity

We will download and network-boot enable Ubuntu, but not make any efforts to automate that installation.

Omarchy Linux

This is an interesting distribution that aims to radically change the way we work with computers. It is highly dependent on local Artificial Intelligence and uses a tiling window manager by default. We will download and network-boot an image, but not automate anything.

Support software

This is the software that we use as a base for our infrastructure. Some of these products are a combination of packages.

FreeIPA 1.16 - User authentication and authorisation

We will continue to use FreeIPA to manage user accounts on most of our Linux systems. We will also use it to manage SSH host keys and the Sudo configuration. While we could run FreeIPA in a container, we will wait to implement this and keep the current setup.

Ansible v3.

Our current version of Ansible (2.14) is starting to complain about being too old to be supported by certain community packages. We will change this to using the most recent version available.

We will be using Ansible extensively to solidify our installation procedures. This is an opportunity to upgrade our setup such that it will be able to support multiple teams within the Nerdhole.

Docker Community Edition v29.5

Containers are becoming more and more common in today's computing environment, and this very website comes to you through a Docker container. We will continue to support this.

Gitlab v18

We want to support the popular paradigm of Continuous Integration and Continuous Deployment (CI/CD). Since Gitlab is aiming to become the solution for this, we will use that for support.

Facilities

The facilities available to N-SCHOOL based architectures will include the following:

• One shared file tree
• One home directory on all local machines
• One mobile home directory on mobile devices
• Regular backups, initially using rsync, but eventually using a more advanced product.
• Access to printer queues from all workstations
• Automated procedures to support machine life cycles
• NFS shared file systems
• CIFS shared file systems (with limitations)
• Local web server

Shared file tree

Path names should be the same on all machines, no matter which operating system they run. We will standardise as follows:

Path name	Contents
/local/	All locally mounted volumes.
/remote/	All file systems mounted from other machines by any means.
/data/	Data stores to be shared between users

System setup for main server

The basis for the server is the Rocky 10.2 minimal image ISO. This will set up a basic shell-based server that is accessible using Secure Shell. It installs some 450 packages at initial install.

Disk setup

Our main servers will all have an M.2 SSD card to use as their operating system disk, and either an SSD or an HDD for data storage. We will be using Logical Volume Manager to manage storage for flexibility.

The operating system VG by default is named "rlm_paya" for a host named paya. This makes sense when you have multiple installations on the machine, with different hostnames. We don't do that. We will change this in the initial kickstart file, and name it rootvg. All other local physical volumes will be put in a volume group named "datavg." If at any time we have shared storage (perhaps for clusters), we will put those disks in a VG named clustervg. If for some reason we need multiple datavgs or clustervgs, they will be named datavg01 or clustervg01.

In a virtual machine host, we will put the virtual disks on the datavg, and name them paya_rootvg_01 for the rootvg, paya_datavg_02 for data volume groups (for multiple datavgs, we use paya_datavg02_01). Clustervgs will be named after the cluster since they are used by multiple machines: dbclu_clustervg01_02.

By default, the /home file system is in the rootvg. We will move that to /local/home and place it in the datavg. We will NFS-export /local/home and mount it on all machines in the net, including the main server. Thus, home directory access is the same everywhere.

Where we desire extra robustness and safety, we will use RAID volumes - mirroring or RAID-5 or RAID-6.

Disk selection

The traditional disk names such as /dev/sda, /dev/vda, and so on are no longer used to uniquely identify disks. What used to be sda during install, can easily become sdb afterwards, or when we add a disk to the system. We need to use a different set of names provided by Linux, based on their location in the system. These device names are in the directory /dev/disk/by-path, and include both the physical disks and their partitions, for example:

pci-0000:04:00.0-nvme-1 -> ../../nvme0n1
pci-0000:04:00.0-nvme-1-part1 -> ../../nvme0n1p1
pci-0000:04:00.0-nvme-1-part2 -> ../../nvme0n1p2

In this example, we would specify /dev/disks/by-path/pci-0000:04:00.0-nvme-1 as the installation disk, and we can be confident that as long as we do not move the installation disk from one place to another, we will always be referencing the same disk.

Kickstart installation

We want to install a new main server without spending too much thought on the standard pre-defined things. Hence we will add an nschool.ks kickstart file on the installation media itself. To keep things fast, we use the text install.

The purpose of the Kickstart file is to provide a basic Redhat-like Linux host that can be managed by Ansible. When Kickstart completes, we will have the following:

1. A running Linux System.
2. A root account with a password, but not allowed to log in remotely.
3. A builder account with a known password Ansible can use to log in and switch to the root account using sudo.
4. The system disk, using Logical Volume Manager and partitioned like this:
   • /boot - 1024 MB partition for the boot image
   • /boot/efi - 600 MB partition for the EFI data
   • pv.1 - Logical Volume Manager volume for rootvg, using the rest of the disk. In the rootvg:
     • swap - 8GB no matter the size of RAM
     • root - Rest of available space for the / file system.
5. The following packages installed:
   • Minimal Install - The Basic functionality filesets needed to run a Linux System
   • Standard - Common set of utilities that extend the minimal installation.
   • epel-release - Extra Packages for Enterprise Linux repository configuration
   • libselinux-utils - SELinux utilities
6. The following packages removed:
   • cockpit and supporting packages. Cockpit itself isn't bad, but we use Ansible, not Cockpit.
   • gnome-initial-setup - We do not need to be taught about Linux every time we reinstall a workstation.
7. A network configuration based on DHCP. Select machines will have a fixed IP address, but we configure that after the minimal install.

This will result in a bare-bones system that we can customise as needed using Ansible.

Ansible roles

We will create a new role named main_server, which will automatically turn a machine from bare metal into a machine capable of supporting the network with DHCP, DNS, NFS, etc.

The main_server role will set up the following:

• The /local file system on the datavg
• The Bind name server
• The KEA DHCP server

The boot/install server will be configured using the bis role.

Configuration variables in Ansible

Note: Move this chapter into the Ansible design in due course.

Since Ansible is the most popular configuration management tool at this time, we will be using that to do everything beyond the most basic of OS installations. While our current setup works, it is now time to optimise it for usability and clarity, so that it will be easier to read and update. Topics include:

• Environmental information - Environmental information such as DNS domains, passwords.
• User information - Initial set of users to be created. Group memberships and such.
• Machine information - Identity and configuration of a machine
• Application information - Configuration items needed by specific applications

Variable management

We use three places to define variables:

1. The inventory (in /local/nschool/inventory)
2. Group variables (in /local/nschool/ansible/group_vars/groupname/99-xxx)
3. Host variables (in /local/nschool/ansible/host_variables/hostname.yml)

Inventory variables

The inventory consists of variables, stored in INI files in the directory /local/nschool/inventory/. We only put simple key="value" entries here, more complicated variables go into YAML files. These variables are per-host or per-group variables. They live at the top level of the namespace.

Variable	Meaning	For	Example Value
description	Description of the host	all	Experimental Docker host
printer	Name of the printer queue	workstations	nero
vm_extra_swap	Extra swapspace to assign for a KVM host	kvmhosts	256G
install_drive	Device name of the disk containing rootvg	kvmguests¹	/dev/vda
kvmhost	KVM host where the virtual machine lives	kvmguests	algernon.nerdhole.me.uk
host_boot_install_server	Which machine serves boot and install images	os_linux	paya.nerdhole.me.uk
host_os	Identifier for the host's operating system	all²	rocky-10.2
main_ip	IP address of the host's installation interface	all	10.12.0.60
local_pxefile	Filename of a PXE configuration file	all	0A0C003C
macaddress	MAC Address of the host's installation interface	all	74:56:3c:3c:bd:bd

¹ Every Linux host has an install drive, but for KVM guests, this will always be /dev/vda, because we get to set that at installation time.
² This variable is a filename-safe value that is the same as the directory name on the various services.

Group variables

Group variables are stored in the Ansible group_vars directory, and use the YAML format. We have one top-level dictionary variable per application, with "nschool" being the variable for the main server and any of its dependents. We have two varieties, one for plain text variables, one for encrypted values using Ansivle Vault.

We have the following group variables and files:

Filename	Variable	Contents
all/00-nschool.yml	nschool	Environment for an N-SCHOOL main server
all/00-nschool-vault.yml	nschool_vault	Encrypted variables for an N-SCHOOL main server
all/01-bis.yml	bis	Information needed to build a boot/install server
all/02-certificate_authority.yml	certificate_authority	Information for the certificate authorities
all/02-kvmguest.yml	kvmguests	Specifications of the KVM guests
all/02-kvmhost.yml	kvmhosts	Configuration information for KVM hosts
all/02-openshift.yml	okd	OpenShift information
all/02-openshift-vault.yml	okd_vault	Encrypted OpenShift information
docker_hosts/00-docker.yml	docker	Information for a machine that hosts docker containers
docker_hosts/01-gitlab.yml	gitlab, gitlab_runner	Information for a Gitlab server and runner

Nschool and nschool_vault

The nschool and nschool_vault variables contain the configuration information for an N-SCHOOL main server and any of its dependents.

The nschool variable is stored in the file 00-nschool.yml in /local/nschool/ansible/group_vars/all/. It contains the following sub-variables:

• fs - A list of file systems needed by the Main server, in our case mostly the /local file system in the datavg.
• environment - Identifying strings for this environment:
  • name: Name of the enterprise
  • locality: Main location for the environment ("Medway")
  • state: This is used in various OpenSSL configurations, we say "Kent"
  • organizationalunit: Also an OpenSSL variable, I have put in "N-SCHOOL"
  • commonname: The name of the head administrator/owner: "Menno Willemse"
  • emailaddress: My email address. "abuse@whitehouse.gov"
  • main_domain: The top domain for this enterprise, which in my case is "nerdhole.me.uk"
  • main_server_fqdn: The name of the main server in case any install procedure needs it.
  • main_server_ip: IP address of the main server.
  • main_server_os: Which operating system the main server runs, in this case "rocky-10.2".
  • nschool_home: The directory under which we place all our N-SCHOOL files: /local/nschool.
  • first_user_id: This is used when installing FreeIPA, and it determines where the "normal" userIDs start.
• aliases - Used by the IP Address Manager to add CNAME aliases to hosts, such as "git" for the Git server. It is a dictionary of lists, indexed by the fully qualified domain name of the server, followed by a list of the aliases.
  • One special value is OPENSHIFT_WILDCARD. This will expand to *.apps, a value used in OpenShift DNS configurations.

These can be accessed from Ansible or Jinja templates as, for instance, {{nschool.environment.main_domain}}.

The nschool_vault variable is stored in a file encrypted with Ansible Vault, with an undocumented password that we store in ~/.ansible/vault_password.txt. Readable only to the user. There are more advanced ways of doing this, and we can even simply type it in every time it is needed, but life is too short. The nschool_vault dictionary contains the following:

• secrets - Passwords.
  • directory_manager_password: Used when installing FreeIPA
  • admin_password: Password of the FreeIPA admin user
  • kerberos_password: Password used by Kerberos.
  • ca_privkey_passphrase: Used by the local OpenSSL certificate authority.
  • initial_password: The initial password given to a new user. When adding users for real, find a better mechanism.
• org - Groups and users per organisation, used when setting up the main server. In case we will be creating more organisations within the same enterprise. It is indexed by the domain name of the organisation ("nerdhole.me.uk") and contains the following:
  • name: Name of the organisation ("Nerdhole Enterprises").
  • groups: Dictionary of groups indexed by group name
    • gname: Name of the group
    • gid: numerical group ID
    • comments: A description of the group for documents
    • users: a list of usernames who are the initial members
  • users: Dictionary of the users indexed by username.
    • givenname: First name of the user ("Terry")
    • surname: Surname of the user ("Pratchett")
    • uid: Numerical user ID of the user. Stored here in case we want to recreate the main server with the original storage.
    • group: Primary group of the user.

If sufficiently authenticated, we can access these variables as {{nschool_vault.secret.admin_password}} in playbooks or Jinja templates.

bis: Boot/Install Server

The Boot/Install Server is a facility that allows us to boot machines over the network using DHCP and PXE, and usually start an installation procedure. It is

Machine information

This is the information about specific machines, from their existence to their configuration.

Identity of the machine

Before we can manage a machine using Ansible, we need to add it to our Ansible inventory. The authoritative place to define that a machine exists is in the inventory file 00-all which is in the /local/nschool/inventory directory. Each machine has a line like this:

[all]
paya.nerdhole.me.uk main_ip=10.12.0.3 local_pxefile=0A0C0003 macaddress=1c:69:7a:66:cd:f4

We always refer to a machine by its fully qualified domain name. The main IP address is on the first network interface that we use for basic logins and installations. The local PXE file is the IP address in hexadecimal, which is easier to include here than to calculate from Ansible. The MAC address is used by DHCP to assign an IP address. For physical machines, this is taken from the primary interface. For virtual machines, it is assigned a value to be configured.

Group membership of the machine

We configure operating systems and applications on machines by putting them in groups. For instance, Paya has the following entries:

[os_rocky_10_2]
paya.nerdhole.me.uk

[main]
paya.nerdhole.me.uk

[bis]
paya.nerdhole.me.uk

[docker_hosts]
paya.nerdhole.me.uk

[certificate_authority]
paya.nerdhole.me.uk

Configuration of the machine

description: "Development for V10 main server"
install_drive: /dev/disk/by-path/pci-0000:00:17.0-ata-2

local_storage:
  vgs: # Always specify local volume groups here if you have them
    datavg:
      label: datavg
      description: "Local disks used for data"
      size_gb: 0  # A size of 0 means as large as the disks
      disks:
      - /dev/disk/by-path/pci-0000:00:17.0-ata-3

docker_apps:
- name: www-nerdhole

How we support applications

We will be using Ansible to automate the installation of all our applications, which can be the following:

• Servers running on virtual or real machines such as KVM, FreeIPA, web apps and so on.
• Infrastructure services such as the Boot/Install server, file servers, DNS, DHCP etcetera.
• Docker containers running on Docker server; any machine can become a docker server.