Certificate Authority

This is a design for the certificate authority for N-School. We will eventually have this local certificate authenticated to the Internet at large using the Let's Encrypt service. This is how we deal with certificates within N-SCHOOL.

A certificate is cryptographical proof that the machine you are talking to is actually the machine you think it is. We don't trust just any certificate - any certificate is in turn authenticated by another certificate higher up the chain until you arrive at a certificate that you trust implicitly. Your web browser contains a few hundred of those certificates provided by various authorities such as Let's Encrypt, DigiCert, or other Authorities. This is called a root certificate. You can also make and add your own root certificate, which will allow your own machines to trust your own servers. This is called a self-signed certificate, and only you and any trusting fools you convince to add it to their machines will use it.

There will be only one certificate authority per environment no matter how large the environment gets. One set of certificates will rule them all. In our network (nerdhole.me.uk), the machine containing the certificate authority will be called: ca.nerdhole.me.uk, which will be an alias to the actual machine.

We will only create a certificate authority for the main domain and its subdomains (nerdhole.me.uk and anyting.nerdhole.me.uk). If you have another domain (example.com) you will have to create a certificate authority for that domain.

A certificate needs to include the URLs, names, and/or IP addresses of the services that it applies to. The certificate always has one subject, which is a string including the location, country, organisation, and DNS name of the service, and one or more Subject Alternative Names in case you have multiple names to refer to the machine. If the name is not in the certificate, it will not be used.

Useful information

Anatomy of a certificate authority

A certificate authority in essence is an organised directory of files in the X.509 file format. Usually these files are encrypted using the SHA or RSA algorithm and not readable by humans. The file types involved are:

  • Key file - Used to either encrypt or sign pieces of plaintext.
  • Certificate file - a public file that anyone can use to verify the authenticity of a given file.
  • Key signing request - A file that is sent to a Certificate Authority so they can add their own key to testify that this file is indeed coming from where it says.
  • Extension files - A file containing information about a key or certificate in a more convenient format than entering it all on the command line.
  • Passphrase files - Very very temporary files containing the passphrases to keys. These are created to avoid specifying passwords on the command line for non-interactive procedures.

Directory layout

Before we do anything, we need to think of SELinux of course. Certification files have a context of system_u:object_r:cert_t:s0. The path of least resistance would be to place our CA files in /etc/ssl. I have tested this hypothesis, and in fact the CA will work fine outside the /etc/ssl directory. I have therefore moved the CA files to /local/crypto/ca. That way the CA files will be included in the backup regimen. The /local/crypto could be a separate file system.

We have the following directories and files:

  • /local/crypto/ca - The home directory for the Certificate Authority
    • cacert.pem - The Cartificate Authority certificate
    • serial - Serial number for the current configuration
    • index.txt - Database index file
    • private/ - Files that are meant to be kept secret. Permissions to 0700.
      • cakey.pem - Private root key of the domain.
    • certs/ - Certificates you have issued
      • labo107.nerdhole.me.uk-my-service.crt
      • labo108.nerdhole.me.uk-my-service.crt
      • labo109.nerdhole.me.uk-my-service.crt
      • ...
    • crl/ - Certificates you have revoked
    • csr/ - Certificate signing requests you have issued for a service
      • labo107.nerdhole.me.uk-my-service.csr
      • labo108.nerdhole.me.uk-my-service.csr
      • labo109.nerdhole.me.uk-my-service.csr
      • ...
    • newcerts/ - Newly created certificates before they are enabled.
      • 1000.pem
      • 1001.pem
      • 1002.pem
      • 1003.pem
      • ...

Building the certificate authority

The steps are:

  1. Create the directory structure.
  2. Create the /etc/ssl/ca/openssl.cnf file
  3. Create a temporary passphrase file
  4. Create the initial serial file (contents: 1000)
  5. Create an empty database index.txt.
  6. Generate the CA key using openssl genrsa - generates cakey.pem.
  7. Sign the key with itself - generates cacert.pem.
  8. Remove the temporary passphrase file

You now have a Certificate Authority.

Creating the directory structure

This is achieved with the following Ansible stanza:

- name: Create the {{certificate_authority.home_directory}} directory structure
  ansible.builtin.file:
    path: "{{ item.path }}"
    state: directory
    owner: "{{ item.owner | default('root') }}"
    group: "{{ item.group | default('root') }}"
    mode: "{{ item.mode | default('0755') }}"
  loop:
    - { path: "{{certificate_authority.home_directory}}" }
    - { path: "{{certificate_authority.home_directory}}/private", mode: '0700' }
    - { path: "{{certificate_authority.home_directory}}/certs" }
    - { path: "{{certificate_authority.home_directory}}/crl" }
    - { path: "{{certificate_authority.home_directory}}/csr" }
    - { path: "{{certificate_authority.home_directory}}/newcerts" }

This will create all the directories with the correct permissions.

Creating the configuration file

This is done with a template that inserts the N-SCHOOL environment data into a config file stripped of all the explanatory comments. Some of the modifications to the template are:

[ req_distinguished_name ]
countryName                     = UK
countryName_default             = UK
countryName_min                 = 2
countryName_max                 = 2
stateOrProvinceName             = {{nschool.environment.state}}
#stateOrProvinceName_default    = {{nschool.environment.state}}
localityName                    = {{nschool.environment.locality}}
localityName_default            = {{nschool.environment.locality}}
0.organizationName              = {{nschool.environment.name}}
0.organizationName_default      = {{nschool.environment.name}}
organizationalUnitName          = Organizational Unit Name (eg, section)
#organizationalUnitName_default =
commonName                      = {{nschool.environment.main_domain}}
commonName_max                  = 64
emailAddress                    = Email Address
emailAddress_max                = 64

This provides some defaults to the request creation commands based on the nschool environment.

[ CA_default ]
dir             = {{certificate_authority.home_directory}} # Where everything is kept

... snipped for brevity ...

unique_subject  = no                    # allow the same subject line
copy_extensions = copy                  # Copy the SANs into the certificate

The dir line points OpenSSL at the place where we have configured the Certificate Authority files. The unique_subject line allows openssl to create multiple certificates for the same subject, so that we can re-run the certificate_signed role and refresh the certificates every time. The copy_extensions line lets us copy any extensions such as the Subject Alternative Names from the request into the certificate. This is seen as a risk on the Internet, but not for us as we create these requests ourselves.

Interesting information: Stackoverflow post about copying the subjectAltNames into the certificate. The solution was to add the copy_extensions line as directed in one of the answers.

[ tsa_config1 ]
dir = {{certificate_authority.home_directory}} # TSA root directory

I am not sure whether we actually need the tsa config, but if it is there, then the directory must be correct.

Creating a temporary passphrase file

This is done with a copy stanza that puts the vault variable nschool_vault.secrets.ca_privkey_passphrase into a file /etc/ssl/ca/private/passphrase.tmp. Passphrases in plaintext are frowned upon, but if you let someone gain root access to your CA server, you have more urgent matters to attend to.

Create the initial serial number and database files

A simple copy stanza will put the value "1000" into the serial number file. A similar stanza will empty out the index.txt file. This only happens if the files don't exist already to avoid overwriting important information. In case a complete wipe is desired, there is a "purge" operation that will delete the entire CA home directory.

Generate the Certificate Authority key

This is a shell stanza that runs an openssl genrsa command:

- name: Generate the CA private key
 shell: >
   /usr/bin/openssl genrsa
   -aes256
   -out /etc/ssl/ca/private/cakey.pem
   -passout file:/etc/ssl/ca/private/passphrase.tmp
   4096
 args:
   creates: /etc/ssl/ca/private/cakey.pem

Sign the Certificate Authority key with itself

This is a shell stanza that runs an openssl req command:

- name: Sign the CA private key with itself
  shell: > 
    /usr/bin/openssl req
    -config /etc/ssl/ca/openssl.cnf
    -x509 -new -nodes
    -key /etc/ssl/ca/private/cakey.pem
    -sha256
    -days 3650
    -out /etc/ssl/ca/cacert.pem
    -passin file:/etc/ssl/ca/private/passphrase.tmp
    -subj "/C=UK/ST={{nschool.environment.state}}/L={{nschool.environment.locality}}/O={{nschool.environment.name}}/CN={{nschool.environment.main_domain}}"
  args:
    creates: /etc/ssl/ca/cacert.pem

Remove the temporary password file

That is a simple matter of a file stanza that absents the file.

Using the Certificate Authority

Given that we want to automate the installation of all software packages, they should be able to get a certificate using an Ansible role. We will call this role certificate_signed, and we will call it like this:

roles:
- role: certificate_signed
  service:
    keyname: my-service
    key_directory: /local/mykeys
    certificate_directory: /local/mycerts
    subjectAltNames:
    - fs.nerdhole.me.uk
    - git.nerdhole.me.uk
    scope: local

This role when used will generate a key and have it signed by the local certificate authority. The keyname identifies the service we want to generate a certificate for, and is used in various filenames. The scope is either local (to this environment, with among other things a longer lifetime) or internet for a service that needs to be visible to the Internet at large. The role will do the following:

  1. Generate a keyfile /local/mykeys/{keyname}.pem.
  2. Generate a certificate signing request /local/mycerts/{keyname}.csr.
  3. Transport this request to the certificate authority into /etc/ssl/ca/csr/{hostname}-{keyname}.csr. The hostname is included so that we can have the same service on multiple hosts.
  4. Have the authority sign the request creating the certificate /etc/ssl/ca/newcerts/{hostname}-{keyname}.crt.
  5. Transport this certificate back to the original machine into /local/mycerts/{keyname}.crt as specified in the certificate_directory parameter for use by the service that needs it.

The service's installation playbook can then configure the service to use the just generated certificate and key.

Generating the keyfile

We will generate a key file per service. So if we have a file server and a web server running on the same machine, we will have one keyfile for each. This is done using a simple openssl genrsa command, which gives us a key named my-service.pem in the key directory configured in the service parameter. We will not put a password on this key, because we want to use it non-interactively. If we did put in a password, we would have to enter that password every time we start the service, even at boot, which is not desirable.

Generating the Certificate Signing Request

This procedure generates a file named my-service.csr in the certificate directory specified in the service parameter. This is a slightly more complicated openssl command, generated by the following stanza:

  - name: "Create a Certificate Signing request for {{service.keyname}}.pem"
    shell: >
      /usr/bin/openssl req
      -new
      -key {{service.key_directory}}/{{service.keyname}}.pem
      -out {{service.certificate_directory}}/{{service.keyname}}.csr
      -subj "/C=UK/ST={{nschool.environment.state}}/L={{nschool.environment.locality}}/O={{nschool.environment.name}}/CN={{inventory_hostname}}"
      {% if service.subjectAltNames is defined and service.subjectAltNames|length > 0 %}
      -addext subjectAltName={{'"'}}{%- for item in service.subjectAltNames -%}
      DNS.{{loop.index}}:{{item}}{{ "," if not loop.last else "" }}
      {%- endfor -%}{{'"'}}
      {% endif %}
    register: req_out
    # No Creates, this is also used to refresh an existing certificate.
  - debug: var=req_out

Line by line:

  • openssl req - the command to generate a request.
  • key - The name of the key we just generated.
  • out - The name of the request file (my-service.csr).
  • subj The subject of the request file. Web browsers will recognise the certificate by this.
  • addext - A certificate can apply to multiple names on the network. For instance, a web server can be called both paya.nerdhole.me.uk and www.nerdhole.me.uk, and the certificate must include all these names or it won't be trusted.

This command uses a slightly more complicated Jinja2 template. The command that is generated is as follows, with newlines inserted for readability:

/usr/bin/openssl req -new \
-key /local/mykeys/my-service.pem \
-out /local/mycerts/my-service.csr \
-subj "/C=UK/ST=Kent/L=Medway/O=Nerdhole Enterprises/CN=labo107.nerdhole.me.uk" \
-addext subjectAltName="DNS.1:fs.nerdhole.me.uk,DNS.2:git.nerdhole.me.uk"

The addext subjectAltName part is the complicated part. The if service.subjectAltNames is defined... line makes sure that the addext parameter is only added if alternate names are specified and if the subjectAltNames list actually contains elements. The {{'"'}} parts are there to get actual quotes in the command. The {%- for item in ... -} syntax (with the hyphen after the percent sign) will generate output without any separators, which is what we want here. Leve out the hyphen and you will get spaces between the items.

The part {{ "," if not loop.last else "" }} prevents a trailing comma, and is one of the many things that would have been so much easier in Perl.

The result is a file named my-service.csr in the Cert directory on the client. At the end of the procedure, we fetch the request into ~/certificate_signed/{{inventory_hostname}}-{{service.keyname}}.csr, prefixed with the hostname so we can sign the same service keys on multiple machines.

Transfer the CSR to the CA server

The Certificate Authority will not usually be on the same machine as the service is running. Hence, we need to transfer the Certificate Signing Request to the Certificate authority, have that machine sign the certificate, then transfer the signed certificate back to the client. The Ansible fetch and copy modules can't do third-party transfers so we have to make the files travel via the Ansible runhost. We will create a local directory named ~/certificate_signed/ to hold the CSR and the singed certificate while in transit.

In the certificate_signed role, we will have an entire block delegated to the machine in the certificate_authority.server variable. We will create a temporary passphrase file, use that to sign the request, and transport the certificate back to the client where it is needed.

Signing the request

The actual signing of the key is done using the openssl ca command. This is the stanza that does this:

- name: Sign the request using the CA root key
  shell: >
    /usr/bin/openssl ca
    -batch
    -config {{certificate_authority.home_directory}}/openssl.cnf
    -passin file:{{certificate_authority.home_directory}}/private/passphrase.tmp
    -in {{certificate_authority.home_directory}}/csr/{{inventory_hostname}}-{{service.keyname}}.csr
    -out {{certificate_authority.home_directory}}/certs/{{inventory_hostname}}-{{service.keyname}}.crt

The lines are:

  • batch - Keeps OpenSSL from asking stupid questions.
  • config - Specifies the config file createrd when the certificate authority was built.
  • passin - The temporary passphrase file created earlier. More secure than specifying the actual password on the command line.
  • in - The request file from the client.
  • out - The signed certificate file.

Transfer the signed certificate back to the client

We have specified where the service needs the certificate to go, in the service.certificate_directory parameter to the role. Putting the certificate there is a simple matter of a fetch stanza to put the file where it is wanted.

With that done, we need to configure the service to actually use the certificate. But that is a matter for the service installation role.

Final remarks

These roles should now be inserted into the main Rebuild and Reconfigure playbooks. The certificate_signed role should be used whenever an OpenSSL certificate is needed.