NerdHole News

This is the Nerdhole system documentation where I consolidate the knowledge I gain from my nerdish activities.

Current projects:

• Nerdhole Small Company or Home Office On Linux or N-SCHOOL.
• Medway Little Theatre to the 21st Century
• The Beast From The South - Flagship writing project

2025 Blog Entries

2025-04-10 11:18 Find something to do

I am about to become the company's leading PowerBroker expert. I have all the installation manuals and I have examples of the installation playbooks used last time. High Up has been trying to get rid of Powerbroker for ages, but they don't know what exactly it does, and in addition the cherished replacement solution has not been shown to actually, like, work. Hope this sets me up for the years until I retire.

Meanwhile, to get my creative fix, I am installing one of my lab boxes as a file server, the likes of which I shall then install at Medway Little Theatre.

Keep my brain in motion...

2025-03-23 15:53 Fixin' the Blues...

Work is not really helping my mental health at the moment, so to convince myself I still have the mojo to do heavy lifting, I've reinstalled my not-in-use workstations Alucard and (soon) Paya. Being CentOS Stream 9, the installation of various third party video codecs is proving a bit challenging. I need to figure it out, and then I need to automate it into the Workstation role.

Meanwhile, I think Alucard would be better of not being a boot/install server so I think I'll re-spot it in a while.

At the theatre, things are progressing - we now have a new sound setup based on a Polycom 12x12 matrix mixer - sound goes from anywhere to anywhere. I am going to have to design MLT a new fileserver where we can store all of our show files, configurations, and what have you. This machine will be well and truly headless.

So now Alucard is running, on to Paya.

2025-02-06 21:50 Reality check

I'm sitting in the lobby of the Indigo hotel in Brussels (for Work), sipping a Leffe blonde, typing away on Rayla. I have a connection to my main server at home and while iut's not fast enough to run an email client remotely, I can run git updates just fine.

Internetting like mad!

2025-02-05 16:50 I have a console!

So basically, what you need to do is just wait an hour or so between Bootstrap and Masters, and between Masters and Workers, and then the thing will come alive.

It is a bit of a performance pig, though. I was wise to stuff as much memory in Algernon as will fit. This is the performance of an empty cluster! Good thing I'm not going to put any actual applications on them or I'd need a much bigger box.

So now I have two months to grok OpenShift, and then I need to reinstall the sucker. It's DOCUMENTING TIME!!!

2025-02-05 09:30 Get the damn thing to run!

I tried reinstalling the cluster yesterday and it failed doing its usual OIpenShift thing of sitting still sulking. I think this is due to various files being from earlier installs. I am now going to remove everything to do with the cluster from everywhere and retry.

On Algernon, remove:

• /local/kvm/xml/bootstrap.ign
• /local/kvm/xml/master.ign
• /local/kvm/xml/worker.ign

From my home directory, remove: - ~/okd/

On the load balancer, remove: - /root/okd/

And now I shall run all the plays. If this works, I will add cleanup plays to all relevant playbooks.

• ansible-playbook -Kk --tags openshift_host test.yml - OK.
• ansible-playbook -Kk --tags openshift_load_balancer test.yml - OK.
• ansible-playbook -Kk --tags openshift_bootstrap test.yml
  • Start of the installation is OK
  • RHCOS install went OK.
  • Cluster bootstrap not looking good...
• ansible-playbook -Kk --tags openshift_masters test.yml
  • Start of the installation is OK
  • RHCOS install is OK
  • Bootstrapping still not complete
  • Network load high as masters are downloading their software.
  • Installation seems to be complete

Well. Several hours later, and bootstrap is complete. Install is not complete yet, but I'm kicking off the workers install anyway. Maybe it will be flagged complete when the workers are up.

• ansible-playbook -Kk --tags openshift_bootstrap test.yml
  • Start of the installation is OK
  • RHCOS install went OK.

Debugging commands:

• curl -k https://localhost:6443/livez?verbose
• curl -k https://localhost:6443/readyz?verbose
• curl -k https://localhost:6443/healthz?verbose

On both load balancer and bootstrap, all the checks are passing. Still wait-for bootstrap-complete does not continue. Why is this? I'm assuming that this is because the masters aren't starting. So I will start those now.

2025-02-01 09:52 I have a running cluster!

OpenShift did its thing again... It sat there for hours apparently doing nothing, and then it sudenly completed and started giving me node lists with the oc commands and everything else. So Openshift is now sitting here eating up Algernon's CPU cycles and scratching on its NVME disk. It is most unsatisfying. The last things I did were:

• Disable the firewall on the load balancer. I may just keep it like that because life is too short.

• Wait for the bootstrap to complete with:

  • openshift-install --dir ./okd/install/ wait-for bootstrap-complete

This completed in seconds, but I haven't accurately measured the time it took. I may just have to repeat the entire exercise and give it a ludicrous time-out.

• Wait for the installation to complete with:
  • openshift-install --dir ./okd/install/ wait-for install-complete I was likewise late to the party with that, and so I don't know how long this is likely to take.

So the secret ingredient seems to be time. I am still not sure what exactly the hold-up is. This is an AMD Ryzen 5 running at 4GHz, 128GB of RAM and an SSD disk for all important things. Its little CPUs are now 25% busy doing nothing! I guess in the age of cloud computing we have CPU capacity to burn and efficiency is Somebody Else's Problem.

So what comes out of this?

• OpenShift is not going to be a permanent fixture in the Nerdhole. It is a study object and nothing more.
• I am going to disable the firewall on the OpenShift load balancer. I could painstakingly figure out which ports to open, but basically screw that.
• I will need to turn my ramblings and braindumps into a proper design for Openshift that I can just follow and have a running cluster afterwards.

So... On with that. As soon as the Young Man finishes his homework.

Is it Polish Beer Time? I think when I nuke and recreate my cluster, it will be.

2025-01-31 10:25 Now or never...

Last day, and not even a full day. Tonight I pick up the Young Man from his mum, and then it's no more hacking.

Out of nowhere, installs started throwing up messages like:

Get "https://localhost:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-01-31T10:18:45Z is after 2025-01-30T15:06:21Z

The installer is clearly using old certificates. I'd like it to stop doing that please. Maybe somewhere in the plays, the certificates don't get deleted.

2025-01-30 07:30 Another day, another doesn't work...

Here we go again. Tried to reboot the bootstrap machine after the installation "Completed" but that's not led to happiness. So. How can I tell that the bootstrap server is done cooking?

I found a promising troubleshooting guideline.

Right then. Time for a start-to-finish disembuggerance.

Are my config files correct?

For "Compute," I corrected my "worker" stanza to having zero replicas. Not sure if that fixes anything, but I know you should not be specifying replicas at this stage. You add the things later. Gods, if that is it...

A YouTube video on the various networks

2025-01-29 09:09 If this is madness, there's method to it.

So where am I now? I have the auitomation to set off the installation of the bootstrap node, but I still have no way of telling if this is actually working. RHCOS helpfully shows you how to watch the log files as it goes along. So what to do next?

Redhat says that the sequence is as follows:

1. Provision and reboot the bootstrap machine
2. Provision and reboot the master machine
3. Bootstrap the master machine
4. Shut down the bootstrap machine
5. Provision and reboot the worker machine
6. Worker machine joins the OpenShift cluster

Which is nice, but lacking in details. So off to Google we go to get more details.

Firstly, I will switch my rhcos image to the one referenced by OpenShift itself: openshift-install coreos print-stream-json. This will get you a URL where you can dwnload a RHCOS image. Maybe that will work better.

Hmm... It looks like the KVM dnsmasq installation isn't working properly. It does not give us fully qualified domain names. Let's see if I can't improve on that. Found promising info at the libvirt.org website. I have now added a DNS domain to the config, like so:

<domain name="{{cluster.name}}.{{cluster.basedomain}}" localOnly="no" register="no"/>

So time to rebuild the whole damn lot...

And maybe again. I may have to populate the Shiftnet name server like this:

<dns>
  <txt name="example" value="example value"/>
  <forwarder addr="8.8.8.8"/>
  <forwarder domain='example.com' addr="8.8.4.4"/>
  <forwarder domain='www.example.com'/>
  <srv service='name' protocol='tcp' domain='test-domain-name' target='.'
    port='1024' priority='10' weight='10'/>
  <host ip='192.168.122.2'>
    <hostname>myhost</hostname>
    <hostname>myhostalias</hostname>
  </host>
</dns>

Maybe I can use this construction to point all the OKD nodes at the main name server, which already has all the correct entries. This is just a side hustle.

Well, I just reinstalled the bootstrap machine, and it is strapping its boots at the moment and it has its fully qualified domain name. Hopefully, this is why it didn't work. If DNS is shot, most other things are too.

Well, I have found how to get oc to work on the load balancer: you need the file kubeconfig and then set an environment variable like:

export KUBECONFIG=/root/okd/install/auth/kubeconfig

Not that I'm getting much actual data, but it's a start.

When I start the master nodes, they keep shooting off message like:

GET error: Get "https://api-int.okd.nerdhole.me.uk:22623/config/master": dial tcp 10.12.2.100:22623: connect: no route to host

This port is indeed not open on the bootstrap node, so the question is why not.

2025-01-28 14:41 Back to business...

Well, OKDLB is running again and from now on I will use sudo virt-manager to manage hosts. Time to re-run my setup playbooks: ansible-playbook -Kk --tags openshift_host,openshift_load_balancer test.yml.

Got a syntax error on my shiftnet.xml definition file. How the hell did I not notice that sooner?!

The /bin/qemu-img resize command will (reasonably) fail if the size is smaller than what it already is. Turns out I missed the "G" behind the size. I don't think RHCOS will fit in 120 bytes. In fact, I will not resize the rhcos qcow file after copying it over, but when I install the OKD node. That way Ansible will not overwrite it every time.

Are we nearly there yet? I can now install the bootstrap server using the qcow file and feed it the proper ignition file, but I think I am missing something. What I've seen up to now:

• Done: Generate /root/cluster/install-config.yaml
• Done: openshift-install create ignition-configs
• Not done: openshift-install create manifests

Openshift-install requires an installation directory. For our okd cluster, the installation directory will be /root/okd/install/. Since openshift-install has the nasty habit of deleting your install-config.yaml, we will be copying that from the directory above every time we invoke it.

Right then, I think I understand this. The command openshift-install create manifests generates some intermediate files that in some cases need to be changed before the ignition files are created using openshift-install create ignition-configs. If desired, create ignition-configs can also go straight from the YAML file. What comes rolling out are the ignition files, which you feed to RHCOS' fw-cfg parameter so RHCOS can set itself up as a bootstrap node, a master node, or a worker node. I have seen this work on the bootstrap node, but the messages it spits out are far from clear and there's a lot of them. I honestly can't tell whether it's OK or not. And I still don't have a working config for the oc command.

Info from freekb on the subject is here.

We then need to transfer these ignition files to the KVM server so we can put them into our new virtual machine. They need an SELinux type of svirt_home_t or KVM won't be able to touch them.

2025-01-28 12:18 Well that was fun...

The reason I could not run my VMs turned out to be rather banale: libvirtd wasn't running despite being enabled to run at boot. Also from now on as soon as I start virt-manager, all my VMs crash. This is not the quality I like from my systems! But in order to keep going, I'll avoid running virt-manager as myself and run it as root on Algernon.

Hmm... On Emerald, libvirtd is not running either, and I can start Ariciel just fine. So the poison seems to be virt-manager exclusively. I am not pleased.

Anyway, time for lunch and then we go back to Openshift hacking.

2025-01-28 09:18 Tuesday

No social at MLT tonight, so I'll have the whole day to hack OpenShift... yay!

I need to figure out two things: First, how to bootstrap OpenShift using RHCOS. I'll hack that into the openshift_nodes role for now, but I think at a later date I'm going to add a "qcow2 copy" installation method to the BIS repertoire. Second, how to make the kubectl or oc command available to the users at large so we can do openshift stuff with it. Mostly a matter of figuring out where to get the config files from and where to put them.

But first, Sypha is long overdue a yummy update. So let's do that first.

Oh yes... Juuust run a quick yum update on Algernon. And now my virtual machines won't start. So now I am reinstalling Algernon from scratch. Hope that fixs the problem. If not, I am royally screwed. Thanks CentOS!

2025-01-27 13:50 Late start...

I procrastinated. I have turned off all the equipmenty upstairs - no TV, no Youtube. Time to get serious about this. Maybe I'll take today to write a proper design for the Nerdhole OpenShift installation. I'll start with rebuilding the load balancer. I'll also download a fresh version of the installer and Kubernetes client because I can. I find I have not included entries in my playbook to install the installer and client, so that's next I suppose.

2025-01-24 13:56 Just one weekend, and then...

Another glorious week of home network hacking is about to start. The main theme is going to be to build an OpenShift cluster on Algernon. Why this has to be so hard is beyond me, but it is. I am still trying to figure out how best to set up the various RedHat CoreOS (RHCOS) images. The first thing I tried was to copy the RHCOS image to the KVM server in QCOW2 format and feeding it an ignition file using the fw-config mechanism. This works perfectly for bare-bones RHCOS installs, but for some bizarre reason OpenShift won't have it. The second way is to provision a new machine, assign it the RHCOS install CD and somehow feed it an ignition file. Research suggests that it needs to be on a web server somewhere, but details are scarce and vague.

This if coursae leaves open the quesation of what I'll do with it once I have it. I have the premier application environment in my very own home, now deploy something on it! But first I need to see the OpenShift console. Then, I can start playing with the environment.

In other news, Rayla is really enjoying her new batteries. She can keep going for hours! Tipping over is but a minor inconvenience.

It'll be very good to be away from Work for a while. I spend most of my days being annoyed, anxious, angry, and disappointed. If I were young and employable, I'd be out of here. As it is, I'm stuck.

2025-01-11 09:35 Time for another hacking session

Right. I have the infrastructure, I have the information. What I do not yet have is a syntactically correct OpenShift install-config.yaml. Like all modern software, messages on the failures are next to useless. "I'm having trouble unmarshalling your JSON, but I won't tell you what's wrong." This is my current template:

---
apiVersion: v1
baseDomain: {{cluster.basedomain}}

metadata:
  name: {{cluster.name}}

compute:
  - name: worker
    hyperthreading: Enabled
    platform: {}
    replicas: 0

controlPlane:
  hyperthreading: Enabled
  name: master
  platform: {}
  replicas: 3

networking:
  networkType: OVNKubernetes
  machineNetwork:
    - cidr: 10.12.2.0/24
  clusterNetwork:
    - cidr: 10.254.0.0/16
      hostPrefix: 23
  serviceNetwork:
    - cidr: 10.255.0.0/16

platform:
  none: {}

pullSecret: '{{okd_vault.pull_secret}}'
sshKey: '{{okd_vault.ssh_pubkey}}'

hostPrefix: The subnet prefix length to assign to each individual node. For example, if hostPrefix is set to 23, then each node is assigned a /23 subnet out of the given cidr, which allows for 510 (2^(32 - 23) - 2) pod IP addresses. If you are required to provide access to nodes from an external network, configure load balancers and routers to manage the traffic.

I also have a somewhat working example that at least produced a set of ignition files:

apiVersion: v1
baseDomain: nerdhole.me.uk

metadata:
  name: okd

compute:
- name: worker
  hyperthreading: Enabled
  replicas: 2

controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3

networking:
  networkType: OpenShiftSDN
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  machineNetwork:
  - cidr: 10.12.2.0/24
  serviceNetwork:
  - 172.30.0.0/16

platform:
  none: {}

fips: false

pullSecret: '{{okd_vault.pull_secret}}'
sshKey: '{{okd_vault.ssh_pubkey}}'

So now to find the right combinations. I am removing the platform lines from the compute and controlPlane stanzas see if that works better. The network needs to be OVNKubernetes because openshift-install tells me so.

And still it does not work... Doing the magical cut-and-paste. See if that helps.

OK, now at least I have ignition files... Okay. is it stumbling on the cidr: in my serviceNetwork stanza?! Consistency truly is the last resource of the unimaginative. Time for a commit.

2024 Blog Entries

2024-12-31 11:57 Cows are not supported?!

Right. One step forward, two steps back. I had my infra set up to install a cluster using the method of copying a QCOW2 image to the KVM host and then booting that. Simple! Sadly this is not supported, and I need to install the cluster nodes from an ISO image. Right then... Down we load the iso.

Looking at this, it's actually not too bad. The virt-install command supports a CD as an install source and then it's just a matter of specifying the correct startup parameters.

2024-12-30 09:51 Final steps on OpenShift

I think I have my KVM environment set up to the specifications needed by OpenShift. Separate virtual network (Shiftnet) with a separate IP subnet (10.12.2.0/24), DNS entries for the Openshift nodes defined, special DNS wildcard for *.apps.okd.nerdhole.me.uk, a template for the YAML file from which to generate an ignition file, a RHCOS image on the main server to be copied out to the KVM host. Now all I need to do is to get the actual OpenShift software on there. If memory serves, that is a matter of providing an installation file, and feeding that to the bootstrap server. So let's see how we can do that.

Upwards and onwards as they say.

2024-12-27 08:50 Not quite yet...

While I have made some progress on the provisioning of the OpenShift nodes, I'm not quite there yet. I now know how to set up an OpenShift node and feed it an ignition file, so I'll have to Ansible that up. And then I can finally get on with the actual OpenShift installation. If Work leaves me a moment, I'll see about inserting that into the playbooks.

2024-12-25 18:53 Well well...

Okay, so I have made some progress on the OpenShift front, but I hit a snag. The Ansible playbooks provided by my benefactor assume that you install the thing with PXE - they have you download a kernel, initramfs, and rootfs, and take it from there. So what I have up to now is a KVM config for Shiftnet, with the following features:

• A separate NAT network named shiftnet, that seems to be working well.
• A running load balancer okdlb with one Ethernet in Frontnet and another in Shiftnet. Waiting to have the OC client installed on it.
• Some commands to build a fresh machine in KVM.

I have also built a bootstrap node that runs Fedora CoreOS, but due to this and that, it won't let me in. Still it has its proper IP addresses so that is something.

So now I need to get the setup instructions from some other place.

2024-12-23 14:35 OpenShift or bust

"So do you have any plans for Christmas?"

Why yes, yes I do. I want my openshift cluster. I will use the latest instructions I found, renew my subscriptions with Redhat is I need to, and get the damn thing running on Algernon. I have two days. I am going full on Autistic mode, and not do anything else those 48 hours.

So that's what I'm planning.

2024-12-16 09:03 And it's a wrap...

It's a Wonderful Life just went through its last performance this Saturday. This is the first time I actually had some doubts as to whether we'd pull it off, but we did! Sterling effort by all concerned. I am now attempting to burn the show onto a DVD. I've had to cycle through a few machines and pieces of software to get this to happen. Strangely, I was able to squeeze a 11GB file onto a 4.7GB DVD. Must be some form of compression going on. And I've now replaced the fuse on my BluRay player so I won't have to miss Alita Battle Angle after all. Will put the DVD players back in their original setup soon.

Work wise, we're in the last few weeks of the year. We have a change freeze, lots of us are on holidays, but I'm working through. I hope I'll be able to fix a few things to the working environment and come to terms with more obstacles put in our way by the unsophisticated. Joy. If I don't see you, Merry Christmas!

2024-12-12 12:17 I bought a thing

Well, two things actually. One is a tiny bit of string that goes between Rayla's Mini DisplayPort and an HDMI monitor. This will enable me to hook her up to Flippin' Big Monitors or projectors and run presentations. The other thing is a tiny 12-channel DMX512 controller that I intend to use for MLT's house lights. I've been programming scenes on the Big Desk while people were in the building, and surprisingly, they want to see where they are going. And what happens then is that you forget to turn the house lights off and they get added to the scene you are working on. So. Separate set of Slidey Things, and that will not happen anymore.

On a mental note: I just subscribed to Bread on Penguins on Youtube, and she makes content on mid-to entry level Linux. And puts in the occasional piece of wisdom having to do with hacking your mind. I have no idea why, but she's inspired me to claw out of my current rather dismal mental state by reminding me that I have control over my mental processes! Amazing no? So. From now on, I'll be counting blessings, concentrating on positive, and try to suppress my ongoing anger at the world at large. Let's see how that works out. Just writing it down turns it into a commitment of sorts, even if nobody reads it.

2024-12-04 11:12 Well then... some videos will run!

I just reinstalled Alucard and the Workstation role still chokes on the installation of RPM Fusion's video codecs. But I have at least some progress. The poison package is compat-ffmpeg4, and if you simply do not install that, the process completes. However, some codecs do not get installed. But Youtube now works, and certain educational sites now work as well. So I can wait for the RPM Fusion people to recompile their offerings against libav 7 rather than 6.

2024-11-25 08:56 Rsync is rsyncing

I just downloaded the latest and greatest from Centos, and then of course I had to back that up to Algernon. The new disk is doing what it should, and the rsync process can now be started unattended from a Cron job or instead from a playbook. Don't exactly know which I'll choose. So the first half of the backup facilities are done, documented and automated.

So what next? I think I may have to start putting an actual user interface on NSCHOOL - meaning commands to be run on clients and servers. I don't really want to go on a trek through my documentation every time I update my hosts.

Oh, and I have to make sure that Thunderbird is at the very same level everywhere. Because it throws a strop if you access the mail using anything older than the latest. So that's the next job I guess.

Looking at my collection of hardware... Where do I want my machines to live? I think it would ber more aesthetically pleasing to have a stack of Optiplexes sitting under my TV. But that would mean I have three PCs attached to my TV, and a DVD player. My TV only has three HDMI connections. Not that I would use them that often.

I think I am going to put Paya back into her role as Little Server. So I'd have to put her somewhere I can have her on all the time. Decisions, decisions. Will put that off till later. Meanwhile I have theatricals to consider. Premiere night for "It's A Wonderful Life" is closer than we think.

Ugh. Repos are not being cooperative. We have a version clash between RPM Fusion and EPEL. This is explained in a Reddit post here. So we don't upgrade any workstations just now...

2024-11-22 06:24 Theatrics

The production of "It's a wonderful life" is gearing up. I'm doing the lights, and as it turns out some of the sound. Digging through the script as I normally do, I find more sound cues than light cues. The director also wantds images projected on the back of the stage, which Chris is doing. We have a Fuck-off Big Projector for the purpose that has been securely hung up above the stage. The only problem is that it casts a shadow when I turn on the big floods. So let's not do that then. I have other lights. Today is my last day going into the office for this month. I have one day left on my weekly seasons ticket, so maybe I'll take the Boy out to London. Been wondering what he wants for Christmas and it turns out to be... video games. No surprise there. I'll have to think about that, there may be more useful presents.

Meanwhile, the Nerdhole rsync facilities are designed, and now I have only to build them. I'm expecting some trouble with SELinux. But then I always do. It'll be good to have. Having just one copy of my important stuff on one disk worries me. Though of course I do have git repos all over the place.

After last month's strop, Rayla is now behaving properly. The new battetries are great. The main tank has never run empty yet, so I don't have to lug around a charger. Just plug her in when I get home. Nice. She's turning into quite a capable machine.

And as a periodic note to self: Get OpenShift running on Algernon while it can still run it.

2024-11-20 06:29 I'm onna train...

It's that week of the month again, where Work want me to be in the office. So having outfitted Rayla with a fresh pair of batteries, I'm using the time to update my documentation. Needless to say I don't do that on the work laptop so I am now lugging around two laptops! Went to Theatre yesterday and hooked up Rayla to the DMX network. Tried out the QLC+ Sequence feature. I can now put waves of colour all over the stage. I'm still not doing any real stage plays with this, but next time I have to busk a music performance, I may well give it a go.

Now, on with the backup server design. Thinking on rsync, the client will have to run as root in order to back up every file on the client. How I start this, I'm not sure yet. I see rsync as more of a sysadmin facility, whereas Bacula users need to back up and restore files without gaining root access. Sysadmins can put up with having to kick of a chmod or chown or two. Office users don't want to.

2024-11-17 10:26 Get my back up

I am really going to have to get the backup system running on Algernon and its big 12TB disk. Twelve commercial terabytes, that is. Whoever came up with Kibibytes? While I figure out Bacula, this will be done with a simple rsync. The main problem is moving things from one machine to the other. How to do that in a way that a small company would find cryptographically sound?

Secure shell with public and private keys is of course the first answer to that. However, we may have to prepare for the event that one of these keys will fall into the wrong hands. Who will be initiating that contact? From the user perspective, it is most convenient to have the client initiate the backups. To be able to back up literally everything, the backup needs to run as root. It needs to be started as a local user.

So. We need a script (say, /usr/local/bin/rsync-backup) on every backup client that starts up an rsync to the backup server.

So. Time to generate some SSH keys. Root at Sypha will use a passwordless ed25519 key. It is specifically meant for unattended automated processes, so putting a password on it is not desirable. Then, we need to make sure that this key only gets used for rsync. Needless to say, this will still let root@sypha read any file from the target host,

2024-11-13 09:00 Proof of Concept

So yesterday I took my little USB-to-DMX box to the Theatre, hooked it up to the Universe, and found I could actually control the lights from Rayla using QLC+. This is progress. It means I could run an entire show all from Linux Mint Debian Edition. Now all I need to do is familiarise myself more with QLC+, and then I can cook up some stuff at home for use in theatre.

2024-11-10 08:41 Minecraft and Nerdhole-craft

So it's back to work tomorrow. My restorative week off is over. Sitting here watching my son watcing Minecraft videos. Ye gods. People are building entire cathedrals in there! So today we chill, then we lunch, then we go out for an Activity. Must have an Activity. On a related note, the phrase "Without further Adieu" is really starting to grind my gears.

I'm going to have to take stock of my Nerdcrafting here and see where I need to add. My North Star is: OpenShift. I need to get the software again, re-subscribe to the appropriate accounts, and turn Algernon into an Openshift cluster.

2024-11-08 10:21 Final day...

This is the last day of unfettered Linux hacking. I have actually achieved quite a lot. Sypha is now the unchallenged boss of the network, and Paya is taking a well deserved rest. All my workstations (Algernon, Alucard, Paya) are now at the latest stable release of CentOS, and lots of small things have improved about the environment. Remote X displays are faster for one. I can manually install several different flavours of Linux including Linux Mint Debian Edition. At some point I want to add Debian itself to the mix as well. I am writing this on Rayla, using both of her shiny new batteries, in a coffee shop in Chatham, watching the people walk by. I am much restored after the last months of stress. Monday will be my next day at work, but let's not think about that now.

So. Now comes the time to go over my documentation and let the world know just how clever I've been.

I also need to split up the nschool role into its constituent parts so I can, for instance, reconfigure my DNS/DHCP servers. Maybe when I get home.

2024-11-07 11:23 Automatic vdisks are a go

As expected, the precise syntax of a dynamic results loop variable took a while to figure out, but I have it now. So now the storage role will create the new storage volumes on the KVM host and then attach them to the host. The filename is: labo100-datavg-vdc.qcow2. Short hostname, volume group name, disk name. I have to admit that the role is not as idempotent as could be desired, but I want to get on with the rest of the installation procedure. So. Keep this as a todo item. It's almost lunch time, so I think Pie and Mash are in order.

ETA: And now with the storage good enough to be getting on with, I will be constructing my Rebuild playbook that should be able to rebuild anything except maybe the main server.

2024-11-07 09:23 Last building day

Today is the last day I have genuinely free from 09:00 till 18:00. I need to add a few more functions:

• Make the storage role add virtual data disks to a virtual machine.
• Create the Rebuild.yml and Reconfigure.yml playbooks where I can put my app roles.
• Add a wrapper script nschool for them so I can explain things to the user about which password to use.

To do nested loops in Ansible is (surprise) a bit of a nightmare, but a Redittor named onefst250r has a way of flattening nested data into a single list of items. For example:

- name: debug
  debug:
    msg: "{{ item.source }} {{ item.dest }} {{ item.port }}"
  vars:
    csv_data:
      - [ "sourcehost1", "destinationhost1", "22 3000 8096" ]
      - [ "sourcehost1", "destinationhost2", "22 3000" ]
      - [ "sourcehost2", "destinationhost1", "22 80 443" ]
  loop: >-
    {%- set results = [] -%}
    {%- for item in csv_data -%}
    {%- for port in item[2].split(' ') -%}
    {%- set _ = results.append({
      "source": item[0],
      "dest": item[1],
      "port": port
    }) %}
    {%- endfor -%}
    {%- endfor -%}
    {{ results }}

I'm sure I can adapt this technique to my needs.

2024-11-06 09:00 Partial success.

Yesterday, the installation of a VM went reasonably smooth. I am using the same pxeinstall role for putting the OS on both physical and virtual machines. I am generating XML files for defining the VMs on the KVM hosts, and of course I forgot to put in a few variables. I'll check out the template one more time. There was only one problem that was vaguely perplexing. A VM stores its non volatile RAM in a file, and since I had not put the hostname in that file, I could only start one VM at a time. The only reason that file even interests me is that it contains the setting for safe boot, which for the nonce I want off until I find a way to support signed boot images.

Meanwhile Rayla is enjoying her new battery. I briefly feared I'd already broken it by completely discharging it for the statistics, but it's charging now. I hope it, with the other new battery, will extend Rayla's stamina on the road. I intend to take her to a cafe when time comes to document all that I have done.

So today will mostly be spent debugging the kvmguest role. When Emerald can reliably install a VM, I will reinstall the Beast of Algernon with KVM and get all my labo VMs up and running again. And then if I have time left, I will put in the Reinstall/Reconfigure playbooks.

But first... COFFEE!!!

ETA: Ansible and the default sftp server don't like each other. Set transfer_method to true in ansible.cfg to make Ansible use scp only and stop this silliness.

2024-11-05 10:23 And now we go virtual!

Right. Time for the big part of this week - restoring the Nerdhole's ability to auto-install virtual machine hosts and guests. We start with Emerald, and then we do Algernon.

2024-11-05 08:39 And so begins the second day

I've made some progress. The boot/install server role will now use the general storage and filesystems roles. I have tested it on Alucard, and while I can't test it of course, all the exports seem to be there. The "Restart services" bit is still giving me some trouble though, so will have to look into that. Change them from handlers to simple tasks, most likely. They didn't get triggered. ETA:* I've rebuilt Alucard now, and am running the bis role on him with the ISO images already in place. So now, the appropriate handlers should be triggered. ETATA:** All except the web server. Have added that now. Do I really want to work on the uninstall and purge options now?

BIS is important because it is an essential part of my recovery strategy should Sypha (gods forbid) catastrophically fail.

So today I will look into why this might be the case. And then I'll start on the KVM host and KVM guest roles, which is the whole point of this exercise. I want to have my lab up and running again.

All in all, this should be an improvement. Upgrading to CentOS 9 has already benefited me. The remote X sessions are a lot faster now, videos have improved. The idfea is of course that once I finish (haha) this, I will be able to apply the playbooks and knowledge gained to the Medway Little Theatre. Which is an entire new round of design and building.

I also need to find a way to initially transport installation media to the boot/install server. If they are not there, then the ISO mounts and the NFS exports will fail and I have to redo them. The ISO images at the moment total about 70GB of storage. Should fit on a reasonably large USB key. It should easily fit on a reasonably sized laptop. Maybe I should build a laptop based environment bootstrap environment. I'm not risking Rayla at the moment, but I could definitely try to press Yang into that kind of service. She's got a 1TB SSD if memory serves.

2024-11-04 08:47 Monday monday

Yesterday I managed to get a "filesystems" role running. It is meant to be a utility for other roles that want separate volumes for their data. It uses my standard application variables layout and creates all the file systems. The volume groups will be created nby a relative, named "storage" which I will be developing shortly.

Now that I am depending on Sypha for my daily drivings, I can't reinstall her willy-nilly, so I may have to use either Alucard or Paya as a test main server. Well, I have coffee, I have music, I have nothing keeping me back. Upwards and onwards!

2024-11-03 Holidays!

I have thought on this and on reflection I don't like an hourly job to update the NSCHOOL website. Most of the time it won't be necessary and it should simply be triggered from the fact I am pushing an update to Sypha. I will have to put something in the sudoers file on Sypha that will allow members of sysadm to update. Will have a coffee and see how to implement this.

Let the N-SCHOOLing begin!

2024-11-02 15:29 Am I procrastinating?

I have now added a Git Log report to the NSCHOOL website. I have also put in a script called nschool_refresh that will git-pull the nschool repository from the local git directory (this adds the requirement for the main server to also host a git repo). I intend to run this from cron.hourly so that my info will always be up to date. I will also run sync_repos.sh every week to download all the fixes. Having done this, I'll also need to run a yum update everywhere afterwards.

No more excuses now! Get busy on the playbooks!

2024-11-02 14:44 Paya is now upgraded.

Since she wasn't doing anything anymore, I have taken down Paya, and re-installed her as a workstation as was her original purpose. She has served me well as an emergency main server on CentOS 8. I think I will be using her henceforth as a web server and SSH server for the Great Outdoors, but for now, she can have a rest.

I am lurking on the Work chat groups, and maybe I'll be able to start my validations early. But maybe not.

So this is a bit of a confused start to my week of rest, relaxation, and work. I need to do thye following this week:

• Reorganise my installation playbooks into the provisioning-os install-authentication-configuration workflow.
• Build the provisioning playbook for KVM guests. (KVM may be playing a bigger role in my future so... yay?)
• Rebuild the WoW KVM guests and the labo hosts.
• Organise at least an rsync backup unto Algernon.
• Think of how to put my fictions onto the website.

So... Busy. Also, I need to go into Theatre tonight and film the Youth Extravaganza there. Let's see how much of this I can get done.

2024-10-31 15:43 A week of rest...

I'm taking next week off. I've dealt with Covid, all kinds of corporate unrecovery, and at times I was too tired to move my brain from one thought to the next. I want to spend at least a week operating a Linux network with a strict No Stupid Policy. At the end of which I hope to have all my machines, virtual or otherwise, up and running again using the current version of CentOS. I also want to have a proper backup running on Algernon using Bacula. ANd get some proper sleep. These 5am starts are taking it out of me, as is the whole day sitting still and quiet in an open plan office.

I'm currently reading through my documentation, and adjusting it to new insights. I don't trust myself yet with the actual playbooks and config files - that will come on Monday.

2024-10-30 06:27 Bob the builder

Just to collect some thoughts, what do I want to do with my builder account? Once a machine is in the IPA environment, you don't need it anymore. My own account can become root as necessary. But maybe not all machines will be IPA clients - laptops spring to mind.

For normal experimental use, I want to be using my personal account. So installing a web server, building a cluster, making a file server - that should all be done using the sysadmin accounts. The watershed for that is (for IPA clients) after the IPA configuration and for laptops and for non-IPA machines once the end user account is created. So I may need another layer in my system provisioning system: Authentication.

2024-10-28 07:38 Reorganisation of the plays

I've been thinking about how to organise the installation of my machines, both physical and virtual, and I have come to the conclusion that I need to reorganise. I need to order my playbooks into three stages:

1. Provisioning - building a machine.
   • For a physical box, that means hooking the thing up, and configuring the BIOS to boot in UEFI mode, normally from the hard drive and when you mash F12 at boot time, from PXE.
   • For a KVM box, it means to delete (undefine) the entire machine including its disks, then recreating it from its XML definition file.
   • Once this is done, the machine can be installed the same way whether it is physical or virtual.
2. OS Install - Booting installation media and setting up the OS.
   • Manually - Delete any grub.cfg files so that when booted from the network, it will present the menu of Linux flavours.
   • Automatically - Generate a grub.cfg file for the machine that specifies a kickstart file that enters the configuration for you and starts the OS installation. Hands-off. This is for custom jobs or pets.
   • Once this is done, the machine has a builder account that we can ssh into, from which we can configure the machine further.
3. Configuration - Now that we have a known OS image on the machine, we can configure it further. We have a choice of the following:
   • IPA client - Set up the authentication from the main server. It also configures home directories and data directories auto-mounted through NFS.
   • Workstation - Puts on the Gnome desktop and a collection of useful office applications. It also puts a nice login picture on the machine if there is one.
   • KVM Host - Prepares the machine for hosting kernel virtual machines.
   • Backup host - Runs a backup server and also provides storage for rsyncs of data that we need no incremental backups for.
   • Backup client - Useful for the main server and any laptops we may have.

I think I will need a "Machine hardware" variable for my hosts, with the following options:

• pc - For the workstations and laptops
• kvm - For KVM hosted virtual machines.
• lpar - For AIX machines, if I choose to resurrect my Power5 box.

2024-10-21 08:42 Rayla's back...

I have re-installed Rayla using Linux Mint Debian Edition this time. Look ma! I'm running debian! So now I get to reinstall all those little aps I've been accumulating. The big ones (QLC, Openshot, Linux Show Player and the like) are back, but Git wasn't so I installed that this morning. Needless to say, I am not impressed. But I like Cinnamon enough to keep using it for now. Also the mainstream applications are better with LMDE than with Centos, which is more aimed at the corporate world.

So hey ho, off we go.

2024-10-20 11:17 Well crap!

Linux Mint disappointed me yesterday by bricking Rayla after a kernel update. Apparently the new and improved version doesn't understand something about my Thinlpad T450's BIOS. It also stopped working with my PXE boot Mint version. So I have now added Linux Mint Debian Edition to the Nerdhole Supported Distros. That shits itself on boot, so I may have to tweak the linuxefi lines.

Damn it! I was enjoying Linux Mint, but if this sort of thing keeps happening I may have to start looking for another distro.

2024-10-17 21:23 KVM Clients next

As it turns out, KVM is a breeze to install. The most difficult part is to move the KVM host's IP address from the Ethernet intertface to its Bridge interface, and the script I wrote to do that still works. All I need is to re-add the extra swap space. But I'll do that when I find I need it.

Next are the KVM Clients. These may be slightly more challenging, because when you create the things with the Virt-manager, it turns on safe boot by default, and that does not seem to work well with shimx86. I think I have found where in the XML to change that, so I'll try that next.

2024-10-13 13:23 On to the next...

Well, I think the "Workstation" role is done for now. Next comes a complicated one: the KVM host. I'll have to copy that one over from Paya and adapt it to NSCHOOL sensibilities.

2024-10-13 11:06 Quality of life

I am now using Alucard to do most of my work. Just to get used to the environment. I am becoming a little disenchanted with CentOS as a working environment I have to say. On Rayla, I've been using Cinnamon and that feels a lot more eager to please for want of a better term. There is no more "Places" menu in Gnome. Also Rhythmbox has disappeared and I am now using Audacious to listen to Mary and Eva.

I have "Solved" my certificate problem by ignoring it for now. At some point, I will have to solve the whole certificate situation because it is a major pain in the proverbials and won't look good if I ever install someone else's network. But be that as it may... onwards and upwards.

The only app I actually use that isn't installed yet is Skype. Microsoft seem to have changd its packaging from RPM to something not-RPM called Snap. Snap requires me to install support. I'll do it by hand for now and see where it leads me.

ETA: Well, it seems Skype is joining Discord in being a browser-only application. Snaps do not work with my setup unless I allow root to access NFS-based file systems. It also doesn't seem to work well with SELinux, which I am also not turning off. I need to put in a call with my parents on Skype. Give them the news that I'm doing Covid again... Better shave first.

2024-10-12 21:21 Certifiably insane...

I'm having trouble with my certificates. Again. When I try to access Sypha's website from Alucard, I get the error SEC_ERROR_REUSED_ISSUER_AND_SERIAL. This is because both Paya and Sypha claim to be the owner of www.nerdhole.me.uk. I have in fact tried several times to remove the nerdhole certificate from my Firefox, but like a fkn vampire it comes back every time! I my just have to nuke my entire firefox configuration and start all over again. Builder can actually access Sypha's www.

However, there, I have a different problem. IPA has not included www.nerdhole.me.uk in its list of host that the nerdhole certificate applies to. So it says "I'm not gonna trust this one." I really need to get a better grip on how these damn certificates actually work. I could probably get this to work, but I( want to do this properly. Part of the problem of course is that I'm using the same box for both IPA and WWW, which is not advisable as soon as you grow past a single server. Keep your IPA in its tight little corner and use a different box or VM for your web services.

Imma sleep on it. Mr. Tesco just brought me four cans of Doom Bar.

Oh. I tested Wednesday, and I have the coofs! I hope there will be no stripe on the tester come Monday or I am not going to Krakow.

2024-10-12 13:48 Getting complicated quickly...

I'm having some trouble with users. I want my reinstall playbooks to be runnable by any sysadmin. However, at the time of PXE install, my account isn't on the target system yet so I need to use the builder account, with its ability to become root. There are two problems I ran into:

• The wait_for module is shite.
• The target host gets a shiny new SSH key on each reinstall.

The wait_for module does weird things with sudo. I use the wait_for module to determine that the target (physical) host is down, and then that it has started its Anaconda installer by testing for port 111 open on the target. Once it is reinstalling, I can go ahead and delete all of the target's SSH keys from both IPA using the appropriate module and from everybody's known_hosts. Then I wait until the machine shows signs of life on port 22, and then I run an ssh-keyscan on it. So... Who needs that key? Where it ends up, is in root's known hosts, where it is no use at all. Root will in due course put it in a central location managed by IPA, where everybody can see it. The only user who needs it is the system administrator running the playbook, until such time as it has an IPA record. IPA's feature of managing SSH keys is brilliant. Solves a lot of problems. No more WARNINGS THAT SOMEONE MAY BE DOING SOMETHING NASTY OMG!!!

Going to have a cup of coffee and think about the problem.

Right. The Solution That Works after many tries:

- name: "Wait 10 minutes for user to restart the system from the network"
  local_action:
    module: ansible.builtin.wait_for
    host: "{{inventory_hostname}}"
    port: 22
    timeout: 600
    state: stopped
    msg: "Host {{inventory_hostname}} is still running SSH!"
  become: no

The "become: no" clause will stop Ansible from using sudo to switch to root and getting confused about the sudo password prompts. This is a local action, so I am almost sure that it won't try to ssh into localhost for this. Once ssh is gone, a similar stanza will wait until Anaconda on the target opens up port 111, and then we will know the installation is underway and we can relinquish control to the calling process, which will remove the host from IPA and the calling user's known_hosts file. So that works now, all that remains is to properly document it because it is not at all obvious.

2024-10-08 11:02 Well that seems to have worked

Adding the CRB repo to the Nerdhole seems to have done the trick. A yum install of openshot now shows all green and we will be able to edit videos in the Nerdhole. Woot. As a side benefit I now also have ffmpeg installed, which is good for sites bearing videos. A lot of apps like Skype use these libraries, so that should be easier to install as well.

2024-10-07 16:34 It's been that long...

Well folks, I have not been doing much about NSCHOOL for almost a month, but that is going to change! I am done with Lion in Winter thank COBOL and Algol, so now I have my evenings free for reinstalling Alucard over and over again.

I tried to install Openshot (a video editor), and it refused with a lot of failed dependencies such as libffmpeg and the like. Turns out these are in a repository named CRB which is roughly the same idea as EPEL, but different. Don't know, don't care. So I have just finished syncing that repo from the Internet, and now it is time to reinstall Alucard yet again and see if I can install cool applications with the CRB repo added. The idea is still that I can install the entire machine from my local resources.

In other news I have bought and installed into Algernon a big fat twelve-terabyte disk for the express purpose of storing backups from all the other machines. I will consider adding an external USB disk for offsite storage later. I think I'll be using a combination of direct (rsync) copies of the local files for things I download off the Internet, and Bacula for things where I might want to store multiple versions of the same files and directories. In addition, I'll create a big storage tank for things like video files downloaded off other people's cameras and the like. You need some place to store them.

I think I'll go and find something to cook and eat it now.

2024-09-11 22:27 Chickens and eggs

Just got home from another rehearsal. I managed to get all the DMX information for an Elumen8 Virtuoso 600 RGBAL profile spot into QLC Plus. I can't test it of course, because I have no little bit between Rayla and the MLT DMX universe. I think I'll buy one. The provisional DMX cable I ran to get a light in the window has developed a fault, and breaks the whole rest of the universe. Need to fix that.

Also there is a problem with Project Belnades: chicken and egg problem. I always intended the builder user to be a temporary fixture, to be removed once the authentication services are up and running. So now I have IPA, I have my boot/install server, and now I want to install a client. That client also has a local builder user that I need to install the IPA client. For reasons of SELinux, I cannot use public-key authentication for that, and the rest of the bare metal install runs under my own account, with a well-aimed become stanza.

I am not known on the client before IPA is up and running. I am known on the BIS server, so I can prepare all the installation files there as root. And while builder is still there now, I need to remove that from Sypha before long, so I can't use builder to run the install from.

Options:

• Keep builder on as a permanent fixture. I can run the bare metal install as Builder, with the well-known password. Which is not what I want, but I can justify it if I use builder only to build new machines.
• Install IPA from Kickstart instead of Ansible so all the users will be there right out of the gate. But laptops don't get IPA, so I would have to make a separate kickstart file for the laptops. Don't want to.
• Fix the SELinux problem with Builder's home directory so I can use private key authentication to ssh to the client from my own account to builder. I'd need builder's private key in my SSH keyring.

What do? Time for a nap.

2024-09-04 10:55 Pulp!

I'm just putting the name here so I don't forget it. Pulp is an application that manages local copies of remote mirrors. A bit of a shell round reposync. Since I intend to have several machines on various versions of Linux, this may be of interest.

2024-09-02 17:51 Subscription manager?!

Apparently RedHat, our upstream benefactor to CentOS, is installing a subscription manager. This gives us errors. So put this on the todo list:

In the file: /etc/dnf/plugins/subscription-manager.conf

Change the line enabled=1 to enabled=0. Really people? I use CentOS specifically so I don't need a RedHat subscription. Oh well. One stanza in pxeinstall's tasks/main.yml should fix it.

2024-08-30 08:29 So bloody tired

Why am I bothering building this entire environment? Isn't a simple PC with a web browser enough for me? Why am I doing this? The answer is simple. Therapy. I geek Linux professionally in some indstitution in London. Or at least I try to. In fact I spend most of my time filling out forms, where I have to beg for access to the systems I need and explain to a variety of people what it is I am planning to do.

This place is my sanctuary. I have full control over it. I can design and build it exactly to my own specifications. I can prove to myself that I still can wrangle Linux, because I get SFA opportunity to do it at work. If anyone else reads it, if anyone else benefits from my ramblings, that is a bonus. Gravy. A point of light in what is rapidly becoming an unending whirlwind of crap. Right then. got this off my chest, time for more meetings.

2024-08-28 11:04 Lunch break at work

Rather than take a potentially expensive walk outside, I think I'll work a little on my NSCHOOL documentation. Take the opportunity to do some genuine Linux work. I'm afraid I'm on the downswing again. I really really need to get some result, any result. On a positive note, I can now auto-install a minimal image of CentOS Stream 9 using Kickstart. All the important things are in variables. Now all I need to do is port my IPA Client and Workstation role over from Paya and then I'll be able to build a working Linux office environment fully automated. Even the main server is now installed with a playbook. Maybe I'll re-spot Sypha one more time to verify that everything does in fact work.

I can now install seven different flavours of Linux from my Linux server!

I need to remind myself occasionally that I do have my moments. With all the crap at work, it's easy to forget that I am in fact a capable Linux admin. Soon, I will be able to auto-install whole virtual machine environments from my main server, and not only that, I know how to set one up from scratch. I can build an entire office out of Linux machines! And this will start with the Nerdhole itself.

2024-08-18 10:13 feels like the final push

Well then. I have solved the login problem, and I have downloaded the CentOS Stream 9 repos to Sypha. The CentOS 8 Stream repos seem to have gone a bit screwy with alternate "Vault" URLs. I'll figure those out later. I am now writing this on Sypha itself, under my own username. So I need to add a few stanzas to put the correct SELinux file context on /local/nschool/bis/repos so that Apache can have it. Then I need to figure out the proper URLs for the NerdHole CentOS repos and configure those into Yum on the clients. Sypha will never use her own repositories, but download straight from the Internet so she'll be the most up-to-date machine. Will need a little thought on how to set up a regular job for downloading updates for the rest of the Nerdhole.

So. Let's get on with it.

Hmm... I keep getting "Unreachable" errors for Sypha when comingh from Sypha half way through the playbooks. This is disconcerting.

2024-08-16 17:22 Good ideas?

I would like to add my many stories to the Nerdhole website under a separate heading. So I'll have to produce a prul script to convert tales to Markdown. I tried inserting HTML and it did not go well.

Mkdocs is annoying. It tends to get broken by updates to the system and then I have to reinstall it with force flags. Loth as I am to do it, I think I'll need to put it in a Virtual Environment to keep it from breaking. Because Pythonistas are idiots who think backwards compatibility is for wimps.

2024-08-16 10:40 Almost weekend

This weekend I will divide my time between doing Tech at MLT and finishing up the user authentication and logins. I will also need to replicate the CentOS Stream 9.x repos to Sypha. Which will take a while, so I'll start it up on the Saturday, go sweat away at the theatre and hopefully find a complete copy of the repos when I get back. Then I need to copy over and redesign the Kickstart file. I'll have one kickstart file template for each version of Linux I support. Those kickstart files will be available on http://sypha.nerdhole.me.uk/bis/ks/hostname.nerdhole.me.uk.ks. When I can reinstall Alucard without fail using Sypha's resources, then I can start copying over the data from Paya to Sypha and start using it.

For now, Paya is the only machine to run Thunderbird. I did that back in the day because version differences in the Thunderbird client caused problems. I will have to keep all my Thunderbirds on the same level to avoid stuff like this.

Also, I am ditching the Zoom and Discord clients on Linux. They are genuinely more trouble than they're worth. I spend more time updating Discord than actually using it. Someone please teach these people the joys of RPMs and software that doesn't mainly live in the users' home directories thank you very much!

On with the show!

2024-08-14 16:13 Found it!

Well, I think I have found the login problem. I was creating the users with the same UID as they now have on Paya. However, when IPA server gets installed, it picks a random 20000 size range for its users, and if you define users outside of that range, you can't log in.

So I'll have to use a flag on ipa-server-install --idstart=IDSTART to set that range to what it is on Paya. And re-install Sypha again. See if it works now.

2024-08-14 11:58 Ippa!

So I didn't get round to doing my FreeIPA user creation debugging. It's a bit weird. When I use the Ansible IPA module to create a user, it gets created. You can su to it from root, but as soon as a password gets involved, it worketh not. I create another usrer from the WebGUI, and it works fine. So the pythonistic snake fuckers have messed up? Am I missing something?

I think I'll try to create a new user using the IPA command and see how that fares.

2024-07-30 08:25 Tired, tired...

Why is it such a bloody pain to get anything done at work?

Anyway, I need to debug my FreeIPA installation. As far as I can tell, it's the Kerberos part.

2024-07-19 07:11 Re-work of the "os" data structure

I made a little design mistake in the bis.os data structure. It is currently a list, like so:

os:
    - label: centos-stream-9
      name: CentOS Stream 9
      iso: CentOS-Stream-9-latest-x86_64-dvd1.iso
      initrd_file: images/pxeboot/initrd.img
      vmlinuz_file: images/pxeboot/vmlinuz
...
    - label: linuxmint-21.3-cinnamon
      name: Linux Mint 21.3 Cinnamon
      iso: linuxmint-21.3-cinnamon-64bit.iso
      initrd_file: casper/initrd.lz
      vmlinuz_file: casper/vmlinuz
...

This means if I need to access the OS for a given machine, I'll have to loop through this list rather than just jumping to the one I want. I need to rework this into a dictionary so that I can address the OS record of a host as {{bis.os["centos-stream-9"]}}. So I need to change the template that generates the default grub.cfg to use this and maybe the other places. Baby's first refactor. This is what it should look like:

os:
  "centos-stream-9":
    label: centos-stream-9
    name: CentOS Stream 9
    iso: CentOS-Stream-9-latest-x86_64-dvd1.iso
    initrd_file: images/pxeboot/initrd.img
    vmlinuz_file: images/pxeboot/vmlinuz
...
  "linuxmint-21.3-cinnamon":
    label: linuxmint-21.3-cinnamon
    name: Linux Mint 21.3 Cinnamon
    iso: linuxmint-21.3-cinnamon-64bit.iso
    initrd_file: casper/initrd.lz
    vmlinuz_file: casper/vmlinuz
...

I am repeating the label inside thye record so that I can pass the entire record to a role and address it as: {{item.label}}.

Currently the documents that refer to bis.os are:

• ansible/roles/bis/tasks/main.yml
• ansible/templates/efi_grub.cfg.j2
• docs/designs/Ansible.md

And of course, now, the home page. Maybe I'll find some time to do that today.

2024-07-18 06:20 Bit rot already...

It's one of those things. One of the big rules in IT. Store information in only one place. Becauee as soon as you store the same information in two places, those places will go out of sync. Update one and you need to update the other. Even in something as mundane and simple as a set of Markdown files! So today i will mostly be reading my documents and checking them for doubles and inconsistencies.

2024-07-17 06:22 In the train in the train...

I'm being forced into the office. This means I have to get up around 5am, spend an hour in transport, and then sit at a desk all day. Monday was a complete washout for human contact, nobody I knew was there. Yesterday, I was sitting next to one colleague but since we don't really do the same things, we hardly spoke a word. Today, a few more people are expected to be in. Let's see how that works out. This is a complete and utter waste of time and money!

But it does give me the chance to update my NSCHOOL docs under way. The reorganisation of my Git repo takes the strain out of bringing all my stuff on the road. So what's next? I've written up my group design which is pretty simple. So now I need to do work on my workstation installation scripts. I already have a library of Ansible playbooks for making physical and virtual machines, so I need to adapt those to CentOS 9. Let's see... I think I'll document it in my BIS design

TO DO: Bring the BIS braindump directory structure in sync with what is on the machine...

2024-07-15 12:23 Download all the things!

So I now have a main server that can support clients - IP address management with DNS and DHCP, authentication with FreeIPA, web services with Apache and so on. I have also got the manual installations to work. So now I need to start installing things automatically, using Kickstart or the Debianoid equivalent. I will continue the current setup in that Kickstart is only used to make the machine Ansibleable using SSH. From there, everything is done by Ansible. I will probably have lots of VMs running CentOS 8, as well as CentOS 9. So those repositories at least I want to have locally mirrored using Reposync. I'll reinstate the cron job that downloads the latest updates. I do want to start supporting Rocky Linux, so maybe I'll mirror that one as well. Earlier CentOS versions, Fedora, Debian, I am not that fussed about. My laptops work wonderfully with Linux Mint, but I don't anticipate reinstalling them all that often. Laptops will likely devolve into pets at some point. So they can be installed from the Internets.

At this point I would like to welcome my second Dell Optiplex called Alucard. I have defenestrated it by ripping out its SSD MD.2 card and replacing it with another. Alucard will likely spend its first months being repeatedly reinstalled with various versions of Linux. After that, I may actually turn it into the Nerdhole Secondary. It has more CPU power than Sypha. I'll see how it performs connected to my TV.

So next step: Get local mirrors. Then, make sure I can install a CentOS 9 workstation. After that, get KVM running. I'll appropriate Emerald for that.

I am not a gamer. My computer game is Infrastructure!

2024-07-15 06:22 A good night for Reading of the Will

Our little ad hoc drama group Nerdhole Dramatics did well at the Duncan Rand one-act play festival! Our play "Reading of the Will" We got:

• A Certificate of Merit on the whole group
• A Best Original Play award
• A second place in the "Best play" category for all of us
• An Adjudicator's Award for Lisa Rouselle

I am beyond happy with this. I am not a noob writer anymore. I've been writing stories in the World of Warcraft universe, Legend of Zelda, RWBY, The Dragon Prince. I stick those up on Archive Of Our Own and I usually get a few hundred hits and 10% kudos, which is nice. I also have my own series of stories called The Algernon Expeditions. An adventure into Steampunk. My main claim to fame on that is that Evan Wright of Generation Kill fame called it "Imaginative".

This is a whole new level. Three very experienced actors: Fiona, Lisa, and Chris P, not only read it, not only liked it enopugh to pour their tiume into, but memorised it, rehearsed it, and performed it in front of a nearly full house. Walking in the theatre afterwards, I got nice comments from people, and had a lovely chat with Amanda the Adjudicator. My story of two women who start out in a classic Wife/Mistress situation, only to find that they both got screwed by the Deceased, and the Lawyer who desperately tries to lead them onto the path to unimaginable riches must have struck something.

Thank you Fiona. Thank you Chris. Thank you Lisa. Let's do it again some time soon. There's a radio play coming up, and in radio, you can let your fantasy run wild. Maybe I'll take the audience on a flight on the Airship Lady I.

2024-07-11 10:20 I guess I can live with that

Sypha.nerdhole.me.uk is no longer mine. It is irrevocably tied to IPA. But luckily www.nerdhole.me.uk will still show the Nerdhole docs as I intended. I guess I can live with that. Now tonight is Duncan Rand Evening. The play wot I wrote will be performed at 7:30pm or thereabouts. I will be in theatre as soon as I get off work to set the stage. We will perform. We will be Adjudicated. And then we will WIN!!!

In other news, I have also fixed the problem with my Pixie-boot. Basically the syntax of the grub.cfg files was off. So I fixed that and now I can start the manual installers of the Linux distros that I have copied onto Sypha. Except for Fedora CoreOS, because I know not of the correct incantations to get that to install. Let's see if Google will be my friend. Mind you, I need to install CoreOS into Shiftnet, which is an isolated net. So more fun to be had there.

2024-07-10 07:14 FreeIPA's a bit intrusive...

Hit a snag that should clear itself up when Sypha takes over from Paya. When I try to access Sypha's HTTPS from Algernon, it throws a wobbly because Sypha's certificates are not the same as Paya's.

Also, it should not be claiming https://sypha.nerdhole.me.uk as its own because I expressly asked it not to. And yet my sysdocs are now taken over by IPA. Screw this. This takes hours of my time that I wanted to spend on designing the group layout for the new Nerdhole. Forty minutes left till Work happens.

When I am Emperor of the World, all software must have a button saying "Don't try to be fucking clever."

2024-07-08 07:28 Thirty minutes to go...

Well, the holiday is over. I have just installed Sypha on my dining room table and disconnected her from the rest of the net. DNS works, DHCP works, I can actually log in from another workstation. This is good. I then tried to boot Rayla the Laptop from Sypha, and sadly I have made a few mistakes in the boot/install server configuration. So that is the next thing to fix. After that comes a reinstall of Emerald as a Sypha client. The big challenge there is KVM. I will need to port my KVM configuration to CentOS Stream 9.

Endless fun in the Nerdhole. I really did do a lot of work on my little home network. My home office is awesome!

2024-07-07 09:41 Lazing on a Sunday afternoon

My IPA installation has a few issues. First, the web UI isn't working. This is because I have assigned the root directory to the system documentation, and I will not change that. So I'll have to add a few aliases so that IPA will work. I have now disabled my hard-fought CA block, allowing IPA to take over CA management.

Time to fix this. But first: Coffee. Sing for me, Mary.

2024-07-05 08:00 Final day of the holidays

Well then. Today, I really need to get IPA and NFS going, and then test to see if Sypha can actually support machines on the network. And then reinstall and see if my playbook results in a functional Main Server.

I have hit a snag with my certificates. I started out making my own self-signed root certificate, but then IPA made its own and now I have two root certificates that hate each other. IPA actually has a nice interface for making additional certificates, so maybe I'll ditch the original one and use IPA to manage my CA.

I'm also debating with myself whether to expose my main server's web server to the Great Unwashed. I could set Paya up as a dedicated webserver for outside business, but I don't really want a growth of always-on machines in the Nerdhole. One is enough. So maybe I can install KVM on Sypha and have her run a separate web server for the outdoors activities? Or just go the hell with it and make IPA listen only on 10.12? Think think think....

For now I'll see if I can reinstall Sypha without the local CA and use IPA for the purpose.

2024-07-04 12:11 Flagging a little... still need my IPA

It is now thursday. Yesterday was fairly productive in that I can now auto-configure a CA for web purposes using OpenSSL. This lets me configure HTTPS on Apache. Now, I need to get my user authentication going. After that, configuring NFS is fairly simple.

So. I've added ipa-client and ipa-server to the "Software to install" section - and by Darwin's Beard there's a lot of pre-requisites attached to that. But be that as it may, the configuration is done with a single command: ipa-server-install. I do not want it to start asking questions, so I need to feed it all the information. See you in the Braindup section!

2024-07-03 08:30 Midway point

Wednesday... Middle day of the week. Completed the Ansible vault and documented how I am doing it. I think today I need to get Sypha set up to authenticate users. So installation time it is. Thing is, this one is a bit hard to test because there is no easy way to erase every trace of FreeIPA. Oh well...

I'm expecting my Stinky Ink toner cartridges some time today, so I can print out scripts with wild abandon, and and my power supply sometime tomorrow. And then I'll have a workstation to attach to Sypha in her very own network. I'll call him Treffy? God no, that's terrible. Alucard. That's the thing.

2024-07-02 07:20 Day two of Project Belnades

Having slept on this, I think I am not going to go all out on the security angle. I will create an Ansible vault so I can store my FreeIPA passwords in an encrypted fashion. So that is Step One for today. With that done, I'll develop playbook tasks for FreeIPA and put them into my Environment Bootstrap playbook. I think my focus right now is to get things up and running. Later I will divide up the bootstrap playbook into smaller parts that I can re-use.

2024-07-01 07:35 Project Belnades Continues!

I want to have Sypha capable of controlling her own network sometime today. Things standing in the way of that are:

• Fixed IP address for Sypha
• FreeIPA authentication
• NFS export of /local/home

Once I have that, I can rip Sypha out of the Nerdhole network, hook her up to her own Nerdhole network and... Start testing.

2024-06-28 07:17 A week of freedom!

I have next week off, mostly by accident but very welcome. I'm going to use it for Theatre work (Lights! Music!) I need to paint my gravestone for Reading of the Will in a fetching grey.

2024-06-27 07:09 Busy busy...

Today and tomorrow are going to be busy. We will be having the third couple of run-throughs of the Reading of the Will. Then tomorrow I am picking up the Boy and also pointing the stage lights at MLT. Picnic on Saturday with friends. So not much time for hacking. I am OK with that. Next week, Duncan Rand starts for the Yoof. They have been practicing, and it should be good. Week after is the Adult section. My cast have been absolutely nailing it from almost the first day.

Next hacking project: the NFS server for home directories and the authentication server. This time, I want to be sure that my NFS file systems are fully encrypted as they go over the wire.

A very good question is: Does FreeIPA only work with Apache, or can I use nginx as well? I am severely tempted to go Command Line Only or even Ansible Only with IPA.

 httpd                                  x86_64    2.4.57-8.el9              appstream     48 k
 httpd-core                             x86_64    2.4.57-8.el9              appstream    1.5 M
 httpd-filesystem                       noarch    2.4.57-8.el9              appstream     13 k
 httpd-tools                            x86_64    2.4.57-8.el9              appstream     83 k

Well, crap. I suppose NGINX is going out of the window as FreeIPA depends on Apache. Internet Denizens are making a very good point that it is unwise to expose my FreeIPA server to the Great Outdoors, but my current Main Servers do not have the resources to run a VM for an outside webserver. Eight GB is not enough. And I don't want to have a Big Computer always on. That's why I bought small form factor PCs. What do? I suppose I'll have to retain a separate Little Computer as a dedicated webserver.

I'm not going to give that new computer to MLT, am I?

2024-06-26 10:02 DNS and DHCP Done, Dusted, and Documented.

Well, all I have to do now to specify a new machine is enter it in the Ansible inventory. That took a lot of time, but I've been thinking about doing this for a long time and now I have an automatic name server. Onwards and upwards. I think I'll re-spot Sypha after this and see if the automation really works.

2024-06-25 06:42 Ansible crisis contained... for now.

Well then, I've added the Galaxy to my environment_bootstrap script that gets installed with Kickstart. I need to schedule a full wipe/reinstall of Sypha at some point to see if this will make the playbook run. I'm fairly certain I need to install additional galaxies to manage KVM, but we'll burn those bridges when we need to.

Now, I am going to slot the mkdnsserver script into the Environment_bootstrap playbook.

2024-06-24 07:57 Fuck you too, Ansible!

So. Today I learnt.The Ansible-core you get with Centos Stream 9 is braindead. I had already found out that adding a directory to your perl library path is insufficiently core to include in the basic install and you need to install additional Perl packages to enable that niche functionality.

Ansible is worse.

I tried running my environment bootstrap playbook on sypha, and it turns out that adding a logical volume to a volume group is not core enough. You need to install additional software to do it.

It gets worse.

That module is in a "Collection". That collection is not available as an rpm. You need to download it from god-knows-where. The Internet. The Internet that at installation bootstrap time may not be available.

But it gets worse.

You put in the magic words ansible-galaxy collection install ansible.posix and ansible-galaxy collection install community.general. It squirrels away those collections in /root/.ansible/collections/ansible_collections! In the .ansible directory that I throw away every time Shit Stops Working.

Screw you, Ansible. Screw you for thinking like all snake fuckers that backwards compatibility is for wimps. Screw you for thinking everybody's machines are fucking pets.

It's workaround time again!

Ansible, like Python, is firmly in the "Not because I want to, but because I have to" category of software.

Right then. For future reference:

ansible-galaxy collection install \
    ansible.posix community.general \
    -p /usr/share/ansible/collections/ansible_collections

2024-06-23 09:18 Separation of Data and Software

I've come to a decision point - I need to decide on what is what. Given that I want to apply the NSCHOOL principles to the network of anyone foolish enough to ask me to, I may have to keep the Ansible inventory separate from the rest of NSCHOOL. What is NSCHOOL?

NSCHOOL is ultimately a way in which you can run your small company or home network, and I am not in the business of supporting other people's computers. Yet. I'll be developing scripts and playbooks for whatever tickles my fancy. Those things will be under Git control.

So. I need to produce something that will create an environment bootstrap. It'll take the most recent versions of scripts and playbooks and stick them all on a USB stick. With that in hand, I can go out and install a new network.

What this leads to: /local/nschool is a single project. I will put /local/nschool in my Git repo, and if I need to go somewhere else and NSCHOOL up their environment, then I will use a bootstrapping script to create a clean envioronment to install.

There. Decided.

2024-06-22 08:25 Layout of the directories

I want to start working on Sypha... on Sypha. I have most of the boot/install server features set to the point where I have to test it to see if it works. Now, I want to start working on the actual nschool features that turn an empty machine into a Nerdhole main server. I have for now decided on /local/nschool as the home directory for NSCHOOL, and if ever I want to move it, thanks to Perl's FindBin and Python abspath, I won't have to rewrite my whole library of scripts.

So now I want to bring all of this goodness under Git control so I can work on it wherever I am. I think I'll use /local/nschool as the Git home. I'm sure there is a clever Git way to preserve the entire history, but I find I don't care. I'll remove the Git info and re-initialise it.

So. On with the design, and then on with the implementation!

2024-06-19 07:47 Medway Little Theatre

MLT is my new spiritual home. I'm one of the techies there. There is a small group of grey-haired veterans who have been in the place for decades sometimes. The place is a prime example of Organic Growth. Things get added, used, sometimes replaced, sometimes removed. The result is a contol room that looks like an explosion in a spaghetti factory. But this will change. At least three of the geeks are opposed to the state of affairs and are determined to drag the place into the eighteenth century.

I've started by writing a document on the place detailing all the current and currently-used facilities. The LEDs, the dimmer banks feeding the lush warm Tungsten lights, the sound system with its large and small speakers and its sound desk. It was fairly well received. Cool. The next big thing that will happen at MLT is the Duncan Rand one act play festival where we compete fiercely for the privilege of having Duncan Rand's head in our homes till next year. I have a script, a plan for the tech, and three fine actors to make it happen.

After Duncan Rand, there will be a bit of a recess, and we will use that time to rip every last cable out of the control room and re-do it properly. No more fookin loose boxes everywhere - all important things rack mounted. I have Thoughts. Things I really want are:

• A file server where we can store all the files that will come out of cameras and audio recording devices, documentation, standard files for light desk and so on and so forth. It will also do DHCP, DNS and so on. Not quite the full NSCHOOL setup but useful.
• A proper network. Several segments for the payment machines, ART-net, the security cameras, maybe network access in the dressing room, the rehearsal room. No more makeshift duct-taped network boxes.
• In connection with which: CAT-6 cables everywhere. CAT-6 cables are very versatile. You can use them for DMX, for audio, for HDMI video... and oh yes. Ethernet. I want patch panels and labelled connections that tell you where everything is going.
• A replacement for that sound effect laptop. The damn thing doesn't even have an Ethernet connection!

We do actually have some nice equipment. There are WiFi access points that use Power Over Ethernet so you need only one cable to put Wifi in the house. We have modern LED lights. There's a bunch of smart people in the place. Now all we need to do is get organised. Also we need to get some young'uns in! Theatre tech is a fun place to be in. But we need to teach them.

2024-06-19 07:42 Homing from work

So I'm writing this on Rayla while sitting in the London office. It would have been great if I'd got the most recent version of my playbooks on Rayla, but that I will do tonight when I get home. I still think generating an /etc/hosts file from the inventory and using that for aliases is the best option. Which means that the same host/ip can appear multiple times in the IP address collection and I have to merge them rather than replacing or ignoring them. Duplicates within a host source are still a no-no though.

I'm supposed to start work by 9 today, so I'll keep geeking till 8:30 and then I'll see what's in the queue for me.

2024-06-17 16:47 On the road again

Looks like I'm heading to the trains again for tomorrow. So now I have put my Nerdhole SCHOOL docs under Git control so I can download them onto Rayla and take them on the road with me. So this week, I will be mostly theorising. Which is not necessarily bad.

So for tomorrow what I need to do in my hopefully uneventful train travels:

• Design the variable tree for NSCHOOL and BIS
• Design the way to feed this into my DNS generation script.

Especially the host aliases are important. My gut says I should put the aliases in group variables so that I can include the aliases with the applications they belong to. That is a bit hard to access from Perl though. So I may have to write a task that generates a go-between file that my Perl script can pick up. I'll have to resurrect my hostfile generator template and add the aliases to it. I'm determined that I don't want teh same information in several places, so where do I put host aliases?

2024-06-15 15:47 Made a mistake

I've made a slight organisational mistake. Most of the playbook I've just written is for the boot/install server, and it is in the environment bootstrap role. I intend to support multiple BISes, so that needs to change. The BIS is important for installing the rest of the machine park, so a well educated Main server needs to be a BIS, but not all BISes are Main servers. So. Roles:

1. nschool- Environment bootstrap from nothing
2. bis - Boot/Install server according to the Gospel of NSCHOOL.

Reorganisation will now commence.

2024-06-14 08:21 Focus, focus!

Right. I find my brain jumping all over the place, and if I don't stop it, I will never finish Project Belnades. So. The priority list:

1. Establish Sypha as the network master and install her permanently on the downstairs desk

Sypha still needs:

• Ansible (Install, configure automatically)
• DNS and DHCP servers (80-90% done)
• Authentication services (Automatically by environment_bootstrap.yml)
• Mirrors of the most important Linux distros (kept up to date with Cron)
• File shares
• Data from Paya

2024-06-12 08:11 I have DNS files!

OK, so now I have the zone files and a named.conf.local file. Can I stick the named.conf.local in the same place as the database files? I think I can, and it keeps all the Nerdhole files tidily together. Ripe for rm -rf and regenerate. And now, after work, I shall see if these files actually work on Sypha. Just a tiiiiny detail...

2024-06-11 10:48 Bespoke Virtual Machines

I need to acommodate different kinds of virtual machine in my KVM installation. Sometimes I need just a simple warm body to run a few simple servers, but at other times (I'm looking at you OpenShift!) I need something on very fast disks with a lot of RAM. So how to do?

Some VMs need graphics, others don't. Some need access to USB storage, others do not. I think I need to have different types of machine, and within those types I need to set some parameters like RAM size, number of CPUs and whether it needs datavg storage or maybe a CD/DVD mounted. I need to name the various machine types. Then, I need to assign every VM one of these types with a default.

It would be nice if the machines could figure out for themselves how much datavg storage they need based on which applications are installed. This however requires me to do maths in Ansible to gather up all the applications and their storage needs. I'm going to have to make a bloody manifest!

Time for a little soul searching. I still have to complete my auto-setup of the Enterprise Network, but after that, I will make the most luxurious lab VM environment you ever saw!

2024-06-10 06:50 I think I just automated DNS

This weekend, I got the spirit and wrote a Perl script that can generate your DNS forward and reverse zones for you on the basis of the N-SCHOOL Ansible inventory file. So far it can read the information from the Ansible inventory file I am using to define all the machines.

I have gone the OO way, and have created classes for:

• DNS Hosts - The actual host entry.
• DNS Host sources - A base class that allows the script to read host info from various sources.
  • Ansible Inventory - The file that is described in the Braindumps that I have to raise to the status of "Design".
  • Hostfile - Thought Of. Read an /etc/host type file into DNS.
• DNS Zone - Represents one of the DNS zonefiles
  • DNS forward zone - Translation table for hostnames to IP addresses.
  • DNS reverse zone - Translate IP addresses to hostnames.
• *DNSServer - The DNS server itself.

Things I still need to figure out:

• How to administer host aliases like www, git, ldap and the like. I think I'll create a groupvar for that and generate a config file from it.
• The named.conf.local file - it ties all the zones together.

I think I'll create a directory in /local/bis or /opt/nschool to contain all the scripts I will need. Will think on this and let you know.

2024-06-07 06:13 That which we call a Rose...

I need a name for this architecture. So... This is an architecture for small offices and home offices on Linux. It's for small organisations to start with, but with an eye to growth later. It embodies my personal philosophy on network organisation. It is also not something I just want people to download and run - this is to promote infrastructure thinking and autonomy. I don't want to get this out and spend the rest of my days holding people's hands! So.

Small Company and Home Office On Linux.

The SCHOOL architecture. The Nerdhole SCHOOL architecture to prevent name clashes. N-SCHOOL.

Vampire Sisters! We have a Scheme!

2024-06-05 18:57 Farewell to legacy

OK, decision time. I am not going to bother with legacy boot anymore. It's just a doubling of labour for the PXE server. KVM guests are also capable of running as "UEFI" so that should not be a problem.

2024-06-03 08:24 Things left to do

The goal is to get Sypha up and running as the main network server. What I still need to do:

• Set up the main website using mkdocs and NGINX.
• Automate generation of DNS zone files and the main configuration.
• Automate the generation of kickstart files.
• Automate the generation of grub.cfg files - main grub config and per-machine.
• Automate the population if /var/lib/tftpboot
• Automate the synchronisation of often-used repositories

So... a lot then. Add to that all the things I'm doing at Medway Little Theatre, and I am a busy boy.

2024-05-31 11:10 Well that was a bit of a scare...

So. I was playing around with Sypha and rebooted her. By accident, I left the tick box "Install patches" on. So she started updating herself. I looked at the progress for a moment, thought "O sod it, I need to reinstall anyway" and power-cycled her. And she did not boot anymore - no POST, no logo, no screen output, no nothing.

Shit.

So I assumed that the final steps of the update process were updating the BIOS settings, and my powercycle buggered it up. Well.. I opened up the case, pulled out the button cell, waited ten minutes and put it back in. Then, reboot. I got a pretty HUMONGOUS logo now, so assumed that the BIOS had reverted to something ancient. Downloaded the latest firmware for a Dell Omniplex 5050, installed it, and now Sypha is at the latest firmware level. So far, I like Dell. I could flash the BIOS even without a functioning operating system, and Sypha is now at the very latest firmware level. Cool!

Now to see if I can get Linux Mint to run, which was the whole point of this exercise.

2024-05-25 11:40 Building DNS

Had a spot of bother with my name server the other day. It stopped resolving anything outside the Nerdhole because the root server file had gone mouldy. So I suppose it's as good a time as any to start on generating DNS configurations.

Every host (even printers and switches) has a FQDN and an IP address in the inventory, so it shouldn't be impossible to make the needed BIND databases. I will not be doing that using Ansible itself though. I suppose these days you need a Python script. So... see you in the DNS section of the braindumps department.

2024-05-16 09:08 So much easier in Perl...

So I now have a data structure in Ansible that contains the information on all the Linux distributions I want to support. And of course I need to iterate through that and make a bunch of directories underneath all of them. And of course the pure Ansible method of doing that looks like it was optimised by psychopaths. Screw it. I'm using shell to do it. Much faster, much easier. Let Ansible bitch at me about the beautiful file module they have. It's inadequate. Stop whining.

2024-05-14 13:29 Do I want a /local file system?

Paya now has a single 1TB file system named /local, which contains everybody's data. This is basically zero maintenance, and zero chance of running out of storage anywhere until the whole disk is full. But not all servers have a single /local - Algernon has separate file systems for everything. KVM, backups etc. The latest design has a /local/data for all kinds of data, a /local/bis for boot install server data, a /local/home for people's home directories and so on. My disk space is not infinite, but maybe I'll want to build a separate BIS server.

Right then. I am going to give the main servers a single 1TB /local file system, but I am still going to create separate /local/xxx file systems for BIS data. Decision made.

2024-05-14 10:30 Growing slowly

I've just come up with a directory structure for a boot/install server. It's funny how much of a design is making directories. A place for everything, everything in its place. Anyway - it now has Braindump status, not yet Design status. The data is in Boot Install Server.

2024-05-13 14:17 Pet or Cattle?

I'm still thinking about how I want to set up Sypha. Paya was, is, and was always going to be a pet. How much of cattle do I want Sypha to be? I am starting to sound like a vampire...

The main concern is something breaking and then having to redo the whole thing from memory. I am countering this by having two potential leaders in the network and being able to reinstall one from the other. So. I think I will create a separate logical volume for my environmental configuration files and then replicate this between Paya and Sypha and also onto a USB key. Maybe I'll install one of my laptops as an emergency backup boot/install server. A three-headed dragon!

I am still going to have a lot of file generation to do though. Automating the generation of DNS and DHCP will require shifting lots of numbers around. Which will take time to design and build.

2024-05-12 18:15 Bootstrap files

So, now I can install Sypha from Paya using a kickstart file, and when she's done there is a GUI-capable server with Ansible installed. And now what? I need to store a rather sizable set of Ansible playbooks and associated files. I can't stuff those all into the kickstart file. I think I may have arrived at the stage where I need to have some alternative storage for my config. A bootstrap USB key maybe? It is possible to add files to a USB stick after you've burnt it using an install ISO. And these things are more than large enough. Something like a mksysb? Hah!

2024-05-12 12:57 SSH with time-based certificates.

We'll be using SSH Certificates soon at work, so maybe it would be a good idea to implement it here. Same thing goes for Systemd-based application startups.

2024-05-12 10:17 Bob the builder

I needed a local user for the server-with-gui installation, because you really don't want to run Gnome as root. So for a name, after a little soul searching, I've decided on "builder". Bob the builder. As long as you need to do stuff on the console, you use the builder account. Once the scaffolding comes off, you give Bob a pint and send him packing.

2024-05-12 09:29 Project Belnades continues!

Today I will continue Project Belnades - Sypha's journey towards being the perfect office environment server. When I installed Paya, I did try to keep things neat and tidy, but new insights breed hacks. Getting stuff to work now hardly ever leads to sound designs that are supposed to survive the ages. Given that, I am going to put Sypha under strict change control just like real. Nothing gets done on her unless there is a piece of design saying what's what.

First, Douwe Egberts coffee. Then, more designing.

2024-05-11 10:26 Journal of sorts

Okay then. Mkdocs does not have a native journalling feature (that I am aware of) but I have now added the "Add date" hotkey to Gedit. So this is my blog now.