Nerdhole Network Master

I want to move my main network server away from CentOS, and onto Rocky Linux. To do that, I have purchased a new piece of hardware and will install it as my main server. This machine is called Sypha. Once Sypha is capable of supporting the entire network on her own, I will re-spot Paya as a secondary server.

Well bother and double bother.I am going to have to stick with CentOS for now for a Stupid Reason. My KVM switch. Sypha is a Dell Optiplex 5050 mini computer. I have it hooked up to my new shiny KVM switch that lets me switch between Algernon, Paya, Sypha, and my work laptop. When using Rocky, when I switch away from it, the display doesn't come back. The machine stays up and running, the keyboard is fine, but Rocky has forgotten all about the monitor it used to have and only a reboot will bring it back. Not acceptable. I want to use the same OS on all my workstations and if one of them misbehaves, I can't trust any others.

So CentOS Stream 9 it is...

Network architecture

Network architecture

Specifications

The new server will need the following specs:

  • CPU - Reasonably fast, quad core or more.
  • Memory - 8GB RAM, anything more is a bonus.
  • Disk Storage
    • 1x M.2 SSD 256GB for the operating system
    • 1x HDD, minimum of 2TB (Sadly, will only fit one)
  • Network - One gigabit Ethernet interface

What does it need to do?

The netmaster will have the following jobs in the Nerdhole network:

  • Ansible - Collection of useful playbooks
  • DNS (using Bind)
  • DHCP (Normal dhcpd, maybe IPV6 as well?)
  • LDAP login/authentication server (using IPA)
  • NFS (NFS V4 with full encryption this time but still support V3)
  • Web server (Apache)
  • Git server (Using the SSH protocol, keep it simple)
  • TFTP - For serving up boot images
  • PXE boot services (a matter of putting files where they need to be)
  • Samba server

It will hold the following data:

  • /var/named/ - DNS database (mostly generated).
  • /local/bis/ - Boot/install resources for various Linux versions
  • /local/git/ - Various Git repositories
  • /local/home/ - Everybody's home directory
  • /local/data/ - Various data stores (files, sw, git, www)

These data stores can optionally be exported using NFS, Samba, or HTTP.

This time round, I will write the entire installation process into an Ansible playbook so I can speedily build more netmasters for any kind of place.

Ansible

I will not go the AAP way just yet. My Ansible environment will be a git-maintained directory living on the netmaster. I will base the configurations on group membership. There will be three main playbooks:

  • REINSTALL.yml - Complete rebuild of one of the machines
  • RECONFIGURE.yml - Everything that happens after the OS reinstall
  • REPORT.yml - Generate a report on the machine config on the web server.

These playbooks will contain plays for all the features of the Nerdhole, so that if ever I need to reinstall something, I can run them on any machine.

All the hard work in my Ansible environment will be done in roles. The main playbooks consist of a list of plays, each to be executed on a specific group of machines.

I will also enable Ansible playbooks to be run as a normal Nerdhole user using the become: yes directive. The Ansible playbook directory will be owned by the nerdhole group so all the nerds can run playbooks.

Collections...

It has come to my attention that the ansible-core repository is not advanced enough to do things like make logical volumes. We need to download from the Internet the Ansible Galaxy collections ansible.posix and community.general. This is done with the following command:

ansible-galaxy collection install \
    ansible.posix community.general \
    -p /usr/share/ansible/collections/ansible_collections

The Ansible inventory

I will maintain a collection of YAML files in the Ansible setup that I can use to generate things like DNS records, the DHCP configuration, and other network related files. This will be an improvement on the current NerdHole setup that requires too much fiddling. Maybe I'll put everything in a SQLite database. My current Ansible environment is too much of a "beat on it till it works" affair, and now I want to redesign it using all the knowledge gathered from there. I will have the following inventory files:

  • 00-all - The list of all hosts known to the Nerdhole. Contains:
    • Group name all.
    • Fully qualified domain name
    • Primary IP address
    • PXE file (too hard to calculate in Ansible)
    • MAC address
  • 01-groups - Where we register group membership of machines. Contains:
    • Group names
    • FQDNs of the machines
    • Group specific identifiers (such as okd_type = bootstrap, master etc.)

There will also be group_vars files that contain information for specific groups. For instance KVM guests' KVM host. We will also maintain host_vars files that contain host specific data such as the installation disks, whether it is a UEFI host, which disks are in its datavg, any additional IP addresses, and so on.

I will create a tool that can generate DNS databases and DHCP configurations from the 00-all inventory file. This will mean that I maintain my host info in one place and one place alone.

Domain Name Services (DNS)

I will try to make my name server as professional and secure as I can. It will be generated from the Ansible inventory.

The name server must be able to support DNSSEC, have a primary and a secondary (paya) webserver, be able to serve multiple domains (nerdhole.me.uk and example.com) and be generated from the Ansible inventory. I see some scripting in my future. We will have the following domains:

Domain IP subnets Contents
nerdhole.me.uk 10.12.0.0/24 Main Nerdhole front end network (frontnet)
okd.nerdhole.me.uk 10.12.2.0/24 OKD Openshift network (shiftnet)
example.com 10.12.3.0/24 Example network for experiments

DHCP

I will maintain the DHCP database from the Ansible inventory. Entries for clients should be automatically generated from the inventory. In order for the PXE boot process to work, we need to provide a header with the needed options. This is the first part of /etc/dhcp/dhcpd.conf:

# Nerdhole configuration file
#----------------------------------------------------------------------
option domain-name "nerdhole.me.uk.";
option domain-name-servers 10.12.0.2;
use-host-decl-names on;
default-lease-time 600;
max-lease-time 7200;
authoritative;

# New Centos8 magic...
option space pxelinux;
option pxelinux.magic code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;
option architecture-type code 93 = unsigned integer 16;

# Send PCs or VMs to the Pixie-boot
class "PCs" {
   match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
   next-server 10.12.0.2;
   if option architecture-type = 00:07 {
      filename = "efi/shimx64.efi";
   }
   else {
      filename = "pxelinux/pxelinux.0";
   }
}

subnet 10.12.0.0 netmask 255.255.255.0 {
 option routers 10.12.0.1;
 range 10.12.0.200 10.12.0.215;
}

The first set of options declare which DNS domain we are serving and who the name servers are. As shown, this is Sypha's IP address. The next block of options I admittedly don't fully understand but it's needed to make PXE boot work. Next we decide which boot file to send a machine when it asks for one. Architecture type 00:07 is an EFI client, while anything else is a "Legacy" client. Shimx64 and pxelinux azre boot loaders that will download their configuration from TFTP, then download the OS that we are going to use to install the machine.

Following that are the entries for the individual hosts that we want to have a constant IP address:

host labo100 { hardware ethernet 52:54:ba:be:02:64; fixed-address 10.12.0.100; }
host labo101 { hardware ethernet 52:54:ba:be:02:65; fixed-address 10.12.0.101; }
host labo102 { hardware ethernet 52:54:ba:be:02:66; fixed-address 10.12.0.102; }
host labo103 { hardware ethernet 52:54:ba:be:02:67; fixed-address 10.12.0.103; }
host labo104 { hardware ethernet 52:54:ba:be:02:68; fixed-address 10.12.0.104; }
host labo105 { hardware ethernet 52:54:ba:be:02:69; fixed-address 10.12.0.105; }

These entries simply couple a known MAC address to an IP address. They are used whenever the machine starts, not just when we install it.

Web server

Because FreeIPA requires it, I will be using Apache as our web server. Maybe I will use NGINX for another web server. I will have to make sure that nobody in the Great Outdoors ever accesses my IPA server.

After a little thought, I have decided to say goodbye to WordPress and replace it with something a little simpler: mkdocs. Mkdocs is a Python script that will turn a directory of Markdown files into a browsable web site. Disadvantages are: It is written in Python, and "It works in my venv" effects are in full effect. You install it not with yum/rpm, but with pip. You have to install it everywhere you want to build the website. So the netmaster will have to be set up accordingly.

Boot/install server configuration

Refer to it as: bis.nerdhole.me.uk, and can be either Paya of Sypha.

I want the Nerdhole to be able to install any version of Linux for which we have an installation CD. This is how we install a machine over the network using PXE. To run an installer, we need three components:

  • Kernel (vmlinuz)
  • Initial RAM file system file (initrd)
  • A repository (Either a copy of the Linux repos or a CD)

When we boot a machine from a USB stick, all these are on there. All we need is to "burn" the ISO onto a USB drive and stick that in the machine. Then we need to boot the machine off it and away we go!

When we boot the machine from the network, it depends on whether the machine is a "Legacy" machine (most of my VMs are) or a new UEFI machine. In either case, the kernel and the initrd are downloaded using TFTP. The installation repository lives on the web server, and the installer will use it to install itself into RAM then execute the installation procedure.

Grubby config files

UEFI systems use the shim64 bootloader to start up their installers. Shim64 uses a grub configuration file named grub.cfg. If we want to initiate a hands-off reinstall for a specific machine, we create a file named grub.cfg-0A0C0002 where the 0A0C0002 is the machine's IP address in hexadecimal, just like we used to do with isolinux. And the content will be like:

set default=0
set timeout=3

menuentry  'Kickstart sypha.nerdhole.me.uk' --class fedora --class gnu-linux --class gnu --class os {
   linuxefi efi/images/CentOS-Stream-9-latest-x86_64-dvd1/vmlinuz inst.stage2=http://www.nerdhole.me.uk/cd-iso/CentOS-Stream-9-latest-x86_64-dvd1 inst.ks=http://www.nerdhole.me.uk/pub/kickstart/sypha.nerdhole.me.uk.ks quiet
   initrdefi efi/images/CentOS-Stream-9-latest-x86_64-dvd1/initrd.img
}

This gives you three ohnoseconds to interrupt, and then the machine will be reinstalled. All we need to do is generate the /var/lib/tftpboot/efi/grub.cfg-0A0C0002 file using a Jinja template, generate the kickstart file to include the proper installation disks (as /dev/disk/by-path/....) and maybe a few more variables, and then reboot the client from the net. Simples!

Web based installers

Most of the RedHat style Linuxes use this method. The idea is to mount the filesystem off a CD image into your web server. The installer can then download the files it needs. We download the installation CD from the web, and store the ISO file on the BIS in /local/bis/cd-iso/xxxxx.iso. Then, we add an entry to /etc/fstab like this:

/local/bis/cd-iso/Rocky-9.3-x86_64-minimal.iso /var/www/html/cd-iso/Rocky-9.3-x86_64-minimal iso9660 ro 0 0

Once we have that mounted, we can copy the installer bootstrap files vmlinuz and initrd.img from After that, we can add an entry to tftpboot/efi/grub.cfg like this:

menuentry 'Install Rocky Linux 9.3' --class fedora --class gnu-linux --class gnu --class os {
  linuxefi efi/images/Rocky-9.3-x86_64-minimal/vmlinuz inst.stage2=http://www.nerdhole.me.uk/cd-iso/Rocky-9.3-x86_64-minimal/ quiet
  initrdefi efi/images/Rocky-9.3-x86_64-minimal/initrd.img
}

Then, finally, we can boot the clients off the network, select the menu entry and off the install goes!

Ubuntu Studio NFS installer

Ubuntu Studio works slightly differently from Redhat, CentOS and Rocky in that it uses a live CD to boot from that allows one to try out the software before you install it. It is also part of the Debian family, which uses the APT form of installations rather than RPM. The Ubuntu installer wants to use NFS to access its installation CD, and we accomodate that as follows:

Download the iso file, store it in /local/bis/cd-iso/ubuntustudio-23.10-dvd-amd64.iso. Mount it on /var/www/html/cd-iso/ubuntustudio-23.10-dvd-amd64. Then copy the contents of that directory to /local/bis/cd-nfs/ubuntustudio-23.10-dvd-amd64. The parent directory of that is already exported using NFS V3 (because PXE clients are stupid). Copy the vmlinuz and initrd files into /var/lib/tftpboot/efi/images/ubuntustudio-23.10-dvd-amd64/ where the PXE boot can access it. Then create an entry iin grub.cfg like this:

menuentry  'Manually Install Ubuntu Studio 23.10' --class fedora --class gnu-linux --class gnu --class os {
   linuxefi efi/images/ubuntustudio-23.10-dvd-amd64/vmlinuz ip=dhcp netboot=nfs nfsroot=10.12.0.65:/local/cd-nfs/ubuntustudio-23.10-dvd-amd64 quiet
   initrdefi efi/images/ubuntustudio-23.10-dvd-amd64/initrd
}

You need to specify the IP address of the boot/install server because PXE clients are stupid. Then, you can boot the EFI client off the network, and the Ubuntu Studio live CD and Installer will start.

Configuration of Sypha

I have now removed Sypha as a workstation, and will now install her into her Final Form, which is as the Master of the Network.

Operating system

Network installation of Rocky Linux 9.3. Installed as a Server with GUI. Root only. no user configured as we will do that using IPA.

Storage

We have two disks: the 128GB SSD and the 2TB local disk. We will of course be using LVM to store our data.

Oh boy... It seems that names like /dev/sda and /dev/sdb are no longer persistent across reboots and which volume gets which name is anyone's guess. I am Not Pleased.

Rootvg

The operating system will be installed on a VG named rootvg. It will hold mostly the operating system. We will be using the xfs file system for all our volumes as is standard. These are the volumes:

| Volume | Mount point | Size | Owner | Comments | |---|---|---|---| | /dev/mapper/rootvg-root | / | 70GB | root:root | Root file system | | /dev/mapper/rootvg-swap | Swap | 8GB | root:root | Swap space (100% RAM) | | /dev/sda1 (Partition) | /boot/efi | 600MB | root:root | EFI boot information | | /dev/sda2 (Partition) | /boot | 1GB | root:root | Kernel and initrd |