Nerdhole SCHOOL environment

Nerdhole Small Company or Home Office On Linux

I want the Nerdhole to be a professional grade office environment suitable to a small company. This is with an eye to doing the same thing in other places. We start simple with one main server and one workstation. Then we can expand to multiple workstations and a secondary server that can take over should the main server fail.

Nerdhole Environment

This document describes:

  • The main server - The core of the network that runs all the net's essential services
  • The secondary server - A fallback option for when the main server fails
  • Workstations - The machines we are doing our work on that may run extra services.
  • Laptops - NSCHOOL machines on the road
  • Printer - An HP laserjet printer/scanner.
  • Data stores - File systems or simple directories where NSCHOOL data is kept

Design principles

These are the design principles behind NSCHOOL:

  • What we care about:
    • Users care that it works now.
    • Operators care that it works after a reboot.
    • Sysadmins care that it works after a reinstall.
    • Architects care that it works after a redesign.
    • We are architects.
  • Nerdhole SCHOOL is a school of thought, not a script you can install and run.
  • Keep this environment as simple as possible, but not simpler.
  • No concessions to security parameters only because this is a home network.
    • The firewall stays on. Ports are opened as required.
    • SELinux stays on in enforcing mode, as is a common baseline in corporate world.
    • No [CENSORED] world writable files or directories except /tmp.
    • Passwords need to be good and changed every three months.
    • No passwordless sudo
  • The hardest part of any design is organisation.

Terms

  • Main Domain - The top level domain used by most of the machines in the network. This is the home domain for the main server. In Ansible, it is accessible as nschool.environment.main_domain.
  • Personal user - A user account that corresponds to a human person, as distinguished from a functional user.
  • Functional user - A user account that is not owned by an actual human, but is put in specifically to own processes or files for an application, such as apache for the web server or builder for NSCHOOL administrative uses.

Networks

NSCHOOL needs to support multiple networks. There is one main network, and there may be several other networks, either real (an extra Ethernet switch) or virtual. At the time of writing, we anticipate two networks:

  • Frontnet - The main network in which the main server, the workstations, Internet routers, and other devices live. Its IP subnet is 10.12.0.0/24.
  • Shiftnet - A separate virtual network where we install an OpenShift cluster. It will connect to Frontnet through a reverse proxy that accepts client requests and passes them on to the worker nodes in the Openshift cluster. Its IP subnet is 10.12.2.0/24, but more IP subnets may be used in Shiftnet.

Users and groups

NSCHOOL uses a central authentication server on an LDAP/Kerberos basis using IPA. In principle, we have three classes of user:

  1. Admins
  2. Users
  3. Guests

The Admins have the ability to use root access using sudo, and use other administrative features. The users are able to use the network facilities such as file shares, a home directory, printers, the Internet, and what have you. Guests are users of devices that are not in the NSCHOOL configuration. They are basically anonymous and have no write access to anything, though they will be able to connect to the Great Outdoors.

We have the following Unix groups to support this, with a few examples:

Group GID Comments
users 1000 Everybody's primary group. For the most basic of permissions
sysadm 1001 Administrative users with system privileges.
admins tbd* IPA group containing people with admin permissions on IPA
nerdhole 10001 The main organisational group, meant for anyone "in the family"
rsync 10002 Group for the rsync functional user used for backup facilities
openshift 10003 Openshift operators (Example group)
webmaster 10004 Able to manage the contents of NSCHOOL websites.

(*) The groupID for the admins group is determined randomly at IPA installation time unless we specify the --idstart parameter with ipa-server-install. In order to keep our known user IDs on a reinstall, we set this in an Ansible variable named nschool.environment.first_user_id.

We will use IPA to assign the needed privileges to each of the groups. Most NSCHOOL client machines will use this. Every user should be able to do their normal work without elevated permissions.

Every regular user of the NSCHOOL environment will get an account on the IPA server, with the following features:

  • Username (uname) - The name used to log in on the server
  • User ID (uid) - The numerical user ID for the users.
  • Full name (gecos) - Full first and last name of the user.
  • Home directory (home) - Where the users keep their personal data. (/home/user by default).
  • Primary group (gid) - Number of the user's primary group. By default, 1000 for users. We do not create personal groups for every user.
  • Additional groups (groups) - Extra groups conferring extra privileges on the user.

Every user gets one home directory in the NSCHOOL environment, which is normally on the Main server, but if we should outgrow that, we can move home directories to dedicated file servers. If they are laptop users, there will also be a home directory on the laptop.

Sudo

Users in the sysadm group will be given permission to execute anything they want using the sudo command, though they will have to enter their password after five minutes of inactivity to prevent unwanted escalation if the admin were to leave a session open. This is the sudo default. We can bestow additional sudo privileges on specific groups, for instance so that webmasters can restart the web server. We will manage these privileges in IPA using Sudo commands, Sudo command groups, and Sudo rules. These rules will be automatically applied to any of the resident local Nerdhole machines, physical or virtual.

Managing users and groups

FreeIPA does not offer the functionality of supporting users' home directories when those are stored on an NFS server. We will manage the users from Ansible, using the IPA Ansible modules:

  • community.general.ipa_user
  • community.general.ipa_group
  • community.general.ipa_sudocmd
  • community.general.ipa_sudocmdgroup
  • community.general.ipa_sudorule

User management is achieved by updating the source files, then running the appropriate "Update Users" playbook.

Home directories

Every user has a home directory. System users such as root have a local home directory. All personal users take their home directory from the main server using NFS V4. The permissions on anyone's home directory should be: drwx------ or 0700. Out of respect for people's privacy, we do not look in their home directories unless we have a good reason to.

Shared data

When we want more than one user to collaborate on some project or other, we will create a data store for that project where the users can store the project data. That data store will be under the group ownership of a project group. Multiple data stores can be owned by the same group. We will mount data stores under /data using NFS V4. They can physically reside on any machine we desire.

Main server

To all intents and purposes, the main server is the environment. It will hold all the IP and authentication information, all the file stores, replicas of used operating systems and more. The main server has the following functions:

  • DNS server. It should be able to serve a number of local domains (nerdhole.me.uk, okd.nerdhole.me.uk, example.com) and forward any other domains to the Great Outdoors preferably using DNSSEC.
  • DHCP server. Assign the IP addresses to the MAC addresses of the machines in the domains.
  • Ansible host. Run Ansible playbooks on every machine in the network.
  • Authentication server. Using IPA, provide password authentication using LDAP.
  • NFS Server. Network file systems sharing people's home directories and various file stores.
  • Web server. Using Apache, serve up documentation pages, the www.nerdhole.me.uk home page, the local mirrors of the repository and anything else.
  • Boot/install server.
    • TFTP Server. For PXE boots, serve up Kernels and Initrds.
    • PXE Boot services. Support PCs booting from the network, mostly for installations.
    • Kickstart. Provides installation parameters to PXE installations.
  • Git server. Store Git repositories and access them using SSH.

The required software is installed in three stages:

  1. The operating system is installed using any available means. Currently a PXE boot off the existing Main Server with a kickstart file that configures disks and operating system.
  2. The kickstart file includes a file named /root/bin/environment_bootstrap that installs the EPEL repository and Ansible, including the Community and Posix collections from Galaxy.
  3. The /local/nschool/ansible/environment_bootstrap.yml configures the server as an NSCHOOL main server and boot/install server.

Secondary server

I had an instance recently where my main data disk gave up the ghost. I have also had an instance of an Intel NUC bricking itself. To allow me to continue nerding I want to build a secondary server that can if needed take over the duties of net boss. It will have the same facilities as the main server, but most of them will be disabled until needed. A regular sync from main to secondary will make sure that my data is backed up.

  • DNS server. The secondary will be a running replica of the main server.
  • DHCP server. The configuration will be synced regularly, but the server will be off.
  • Ansible host. Both main and secondary will be able to run playbooks all over the net.
  • Authentication server. Configure the secondary server as an IPA secondary.
  • NFS Server. Same shares on the secondary as on the main server.
  • Web server. Same configuration on secondary as on the main server, but www alias will be on the main server.
  • Boot/install server. Files will be synced from main to secondary, but services will be off.

Workstations

The workstations in the net are PCs that the users are supposed to do their work on. Workstations are expected to be permanent fixtures. They take their users and their data from the network. They will also have access to the printer. Other than that, they are simply CentOS machines with the Workstation group installed, which includes Libre Office, Skype, and more.

Laptops

Laptops by their nature are more independent spirits. They are meant to be carried around and should not rely on the network for their basic operations. They will have all the Workstation software installed for work on the road. We will use Git to carry copies of work on the home network.

Printer

The NerdHole printer is a network connected HP multi-function printer. We support this printer using Cups. We have obtained the PPD file for this printer, and we have automated the creation of a printer queue.

Data stores

This is a list of the data stores in the Nerdhole:

Server(s) Mountpoint Owner Size Shared Comments
fs(1) /local/ root:root 1TB No(4) General use storage tank
main /local/nschool root:root - No NSCHOOL home directory (5)
bis(2) /local/bis/ root:root 500GB NFS,HTTP Boot/Install server data
kvmhosts /local/kvm/ qemu:qemu 500GB No Storage for KVM guests
backup(3) /local/backup/ root:root 660GB No Backups of other hosts' file systems
algernon /local/okd/ qemu:qemu 180GB No OpenShift rootvgs on fast disks

(1): fs is an alias for the main server. We may make extra file servers.
(2): bis is an alias for either Sypha or Paya depending on who's in charge.
(3): backup is an alias for Algernon.
(4): /local itself is not shared in its entirety, but its subdirectories may be.
(5): This store should be kept in sync between main and secondary servers.

NSCHOOL Directory layout

We will be developing several scripts in NSCHOOL to parse configurations, generate files, perform administrative functions and so on. They will all be placed in /local/nschool. This is referred to as the NSCHOOL home directory. We will add /local/nschool/bin/ to all admin users' paths, for now using that user's .profile. In the future we may want to rename or move NSCHOOL, so the anchor point for any script is the nschool/bin directory. Remove /bin to find the NSCHOOL home. The data in the NSCHOOL home will serve as input for the system files needed run the network. The NSCHOOL home will contain the following directories:

  • .git/ - Git information for /local/nschool.
  • bin/ - The place for commands to be run by admin users.
  • lib/ - The place to put NSCHOOL Perl and maybe even Python libraries.
  • ansible/ - Playbooks, and roles for Ansible.
  • ansible/group_vars/ - Ansible group variables
  • ansible/host_vars/ - Ansible host variables
  • ansible/files/ - Files that can be used by multiple roles
  • ansible/templates/ - Templates usable by multiple roles
  • ansible/roles - Ansible roles
  • inventory/ - Ansible inventory, which may on occasion be read by scripts.
  • docs/ - MkDocs home for Markdown documents.

Output directories

The NSCHOOL home, while crucial to system administration, is not essential to running the system. The NSCHOOL scripts and playbooks will put configuration files and data in standard system places from where they can be used even if the NSCHOOL home directory is absent. The output directories are:

  • /var/named/prime domain - This is where the nschool scripts and playbooks put and expect the DNS zone databases.
  • /etc/dhcp/ - This is where nschool puts its DHCP configuration file dhcpd.conf.
  • /etc/httpd/conf.d - This is where we put configuration files for nschool-related web resources.
  • /local/www/ - The root directory for the NSCHOOL main mkdocs website. This will be given the appropriate SELinux file context of httpd_sys_rw_content_t. This directory is wiped clean on every mkdocs build.
  • /local/ca/ - This is where we store the files pertaining to the OpenSSL Certificate Authority. The private subdirectory contains private keys, the public subdirectory contains public information.
  • /etc/pki/ca-trust/source/anchors/ - Destination for trusted root certificates.

System administration facilities

NSCHOOL concerns itself with the management of systems. In the pet/cattle division of things, we want most machines to be cattle. If something bad happens to it, we wipe the entire machine and reinstall it from our NSCHOOL facilities. We need to manage several things:

  • Installation media. ISO images downloaded from the Internet or ripped off actual media. We need to make these available to the network when first we obtain them, update things on them, and finally retire them.
  • Applications. - We need to design them, install them, automate that installation, install any vendor or local fixes, and finally get rid of them when they are no longer needed.
  • Computers. - We need to introduce them to the net, install the OS on them, install their applications, keep them up to date, and finally retire them when they are no longer needed.
  • Other hardware. - Hardware like printers and anything else with an IP address such as tablets or smart TVs.

Installation media

by "Installation media," we mean downloaded ISO images. Tar files, downloaded packages and disk images fall under a different category. They are mostly used for Linux installation CDs using the iso9660 file format. In the future, we may also support the Windows Joliet file system. When we decide to use Windoiws for something. The ISO files live on the Main server, in its Boot/install server file system (bis/iso/). From there, they can be copied out to secondary boot/install servers if needed. The iso files are managed by the bis role. Each iso file has an image label (like centos-stream-9), specified in the "os" stanza with the filename and other OS related information. The bis role will mount the ISO file onto bis/label/iso/ and make this directory available through NFS and web services. With that done, Linux' various installation programs can use them when booted from PXE.

xxxToDo: Implement a way of getting rid of old images. Maybe nuke all and rebuild only the ones specified.

Applications

Our installation procedures must be able to produce a working application in a fully automated way - once we are finished designing the application. Assigning an application to a machine is then a simple matter of adding the machine to the application group and running the reconfigure playbook. An application is only finished when we can automatically install it from the information given in the NSCHOOL inventory and variables. Wherever possible, we assign one role to an application for managing it. In some cases, a single application may need multiple roles, for instance: one to install the application server and one to install its clients.

Any application role should be called as follows:

roles:
- role: bacula_server
  action: install

- role: bacula_client
  action: reinstall
  bacula_server: {{local_backup_server}}

Every application role should look at its action parameter to see what it needs to do, which can be one of the following:

  • install - Install the needed packages from the repos, configure firewall, SELinux, local users, storage, and so on, enable any services, and start the application. When the role comes back, the application is up and running.
  • uninstall - Stop and disable the services, remove any configuration, but leave the application storage so we can re-install the application at a later time with the data still there.
  • purge - Uninstall the application (see above), and remove any traces of it including application storage so that the machine looks like the application was never installed.
  • reinstall - Uninstall the application and then install it again. This is the way of updating an application and its configuration to the latest level.
  • reconfigure - Remove any application configuration and re-apply it. This is the way to apply changes in the configuration files.
  • report - Generate a report with all the pertinent information on the application as installed.
  • query - Add all the information for the role such as file systems, users and so on to the main nschool_info dictionary.

xxxToDo: Design a way to specify an environment with multiple servers (web, database, clusters, load balancers, proxies...)

Computers

We assume that a new computer's disks are empty, or that any data on them can be discarded. Our computers are able to boot from the network using DHCP and PXE, using the UEFI platform. Moving from this empty state to a fully installed system is done in the following steps:

  1. Provisioning - For a physical machine, this means connectng the machine to the network using a known Ethernet adapter. For a virtual machine, this means to erase any existing storage from its VM host, remove the machine's definitions, and recreate them.
  2. OS Installation - The machine is booted from a boot/install server, and installs its designated operating system. This can be done either manually from the graphical installer of the chosen OS, or using a kickstart file that is automatically used from the machine's custom boot file (grub.cfg-XXXXXXXX). When this completes, the machine can be managed through Secure Shell using the builder account.
  3. Authentication - This allows personal users to log into the system, either using the main server's authentication services or as local users on the machine. A machine can have local users or network users, but not both for organisation reasons. To enable this, we have the following procedures:
    • Local admins - This creates a set of local groups and users who can be administrators and have the appropriate entries in /etc/sudoers.d/99-nschool.
    • Local users - For non-IPA clients, this adds the normal users and groups. These users will not get any sudo permissions, but can use all the normal facilities.
    • IPA Client - This configures the machine as an authentication client of the main server's IPA installation, and also configures the machine's /home and /data directories to be mounted via NFS. This covers both administrators and non-admin users.
  4. Configuration - After OS installation, any member of sysadm can install the machine's software and configure the appropriate system resources.
    • Workstation - Installs a Gnome desktop and office productivity applications. Also customises the background picture of the machine's login screen.
    • KVM Host - Hosts virtual servers for experiments.
    • Backup host - Stores file backups of other machines using either rsync or Bacula.
    • Other applications - All applications supported in the NSCHOOL environment.

We will have two main playbooks for configuring our machines: Rebuild (nschool/ansible/rebuild.yml) that takes a machine from bare metal to full functionality, and Reconfigure (nschool/ansible/reconfigure.yml), which will reset the configuration of a machine's applications, including storage, firewall, SELinux etcetera. We will make plays for every machine group. The hard work will be done in roles.

Supporting commands

In order to start any procedures such as Ansible playbooks, backups, and more, we will have one command: nschool. This command will take additional parameters such as backup or refresh website that will start needed procedures, and where required, use sudo to gain needed privileges. It will be a script that is installed into /usr/local/bin. Commands include the following:

  • sync - Uses rsync to copy data from the local machine to the backup server.
  • update ip - Run on the main server to update DNS maps and DHCP configuration from the inventory and other sources.
  • update website - On the main server, pulls the latest updates from Git and runs mkdocs to update the main website.
  • reinstall - On the main server, reinstalls the operating system and any automated applications on the target machine.