User:Sgallagh/FirstTimeSetupDraft

This page is a DRAFT and is currently under development

First-time Service Setup

Many system services require some amount of initial setup before they can run properly for the first time. Common examples are the generation of private keys and certificates or a unique, system-specific identifier.

Traditionally, this was done by RPM scriptlets as part of the installation or upgrade of a package. This was sensible for a time when the majority of installations were performed by attended or unattended installers (such as anaconda and kickstart).

Today we see an increased reliance on generating virtual machine images for use in both traditional and cloud-computing environments. In those cases, having system-specific data created at package installation time is problematic. It means that the production of such images need to have significant care applied to remove any system-specific information about them and then additional tools written to apply the corrected information post-deployment. The goal of this guideline is to ensure that if a system clean-up service such as virt-sysprep is run on the system and then the machine is rebooted, any service that requires first-time configuration will re-run it. The mechanism by which we will accomplish this is to remove such first-time configuration from RPM scriptlets (e.g. %post) and instead execute this configuration as part of service startup with systemd.

This guideline describes a mechanism that can be used for both traditional and cloud-based deployment styles.

Note: this requirement can be waived if the equivalent functionality is incorporated as part of the service's own standard startup. These guidelines are meant to address services that require setup before the service can be started.

Defining System-Specific Setup

A particular setup task is defined thusly: "Any action that must be performed on the system where the service will be run that is not common to all systems running that service."

Some non-exhaustive examples of system-specific configuration:

• The SSH daemon generates a public/private host key
• The mod_ssl httpd module creates a self-signed certificate for the machine's hostname
• A remote logging service creates a UUID to represent this machine

A few examples that should not be considered system-specific configuration:

• Creating a service user and/or group. This is safe to copy to clones of the system.
• Anything automatically generated by the service when it is started and recreated if it gets deleted.

Common Guidelines

For all system-specific cases, we will take advantage of systemd's ExecStartPre functionality.

TODO: decide if this is the right location for scripts

Packagers will create a script in /usr/lib/systemd/system-init/ named after the package that it initializes. So for sshd, the script file would be /usr/lib/systemd/system-init/openssh-server (note that the language of the script is up to the packager, but the file must be executable). This script must implement all of the following steps:

1. Perform a test for whether the initialization has already completed. This may be a simple test of file existence or a more complicated examination of the configuration, as needed. If the initialization has already occurred, the script must immediately return zero (success).
2. Perform whatever steps are necessary to generate the configuration. If this cannot be accomplished, the script must return with a non-zero error code. This will prevent systemd from attempting to start the actual service. If this completes successfully, it must satisfy any and all requirements for the first step to to pass. On success, it must return zero.

The service's systemd unit file must be modified to include the following line within the [Service] section:

ExecStartPre=/usr/lib/systemd/system-init/<packagename>

Special Case: Self-signed Certificate Generation

OpenSSL and PEM Certificates

This tutorial will provide the steps to produce something akin to a self-signed certificate, except that the resulting service certificate cannot be used to sign further certificates (thus closing a potential security issue). In broad terms, what we are doing is creating a short-lived certificate authority, using that to sign a service certificate and then destroying the key material for the temporary authority.

(Note: for any instance of $package below, substitute the name of your package)

Create Short-Lived Certificate Authority

First, create an OpenSSL configuration file for the CA certificate. It should be similar to this:

[ req ]
distinguished_name     = req_distinguished_name
prompt                 = no
x509_extensions        = v3_ca
[ req_distinguished_name ]
C                      = --
ST                     = SomeState
L                      = SomeCity
O                      = Private CA for $package on <real_fqdn> at <date> UTC
OU                     = $package
CN                     = $package.<real_fqdn>
[ v3_ca ]
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = CA:TRUE

Most of the organizational values above can be changed, but CN must be $package.<real_fqdn>. Additionally, the Organization (O) field should remain as in the example so that it provides clues to administrators who may wish to import the public CA certificate. The date field should match the output of date -u +'%Y-%m-%d %H:%M:%S'

Create this file with the name $package-ssl-ca.cnf in a location owned by your package. You may retain or destroy this file later as you prefer.

Next, we will generate the private key for this temporary Certificate Authority. Set the umask so that only root/owner can read the files we create. Then create the key file with the openssl command. We will create the temporary key file in /dev/shm so that it is unlikely to ever be written to persistent media.

OLDUMASK=`umask`
umask 0077

TMPKEY=`mktemp --tmpdir=/dev/shm XXXXXXXXXXXX`
/usr/bin/openssl genrsa -out $TMPKEY 2048

Note: it is *highly* recommended to use a 2048-bit or greater key. Any package that is not compatible with 2048-bit keys should have a bug open to track this shortcoming.

Next, we will use this private key to create the temporary CA Certificate:

/usr/bin/openssl req -new -x509 -days 3650 \
                     -config /path/to/$package-ssl-ca.cnf \
                     -key $TMPKEY \
                     -out /path/to/$package-ssl-ca.crt

You can adjust the value of -days as you prefer. Store the $package-ssl-ca.crt file in the same location as the $package-ssl-ca.cnf. Now we have a certificate authority available to sign our service certificate.

Create and Sign a Service Certificate

Similar to the CA certificate generation, we need to create an OpenSSL configuration file for the service certificate as below:

[ req ]
distinguished_name     = req_distinguished_name
prompt                 = no
req_extensions         = v3_req
[ req_distinguished_name ]
C                      = --
ST                     = SomeState
L                      = SomeCity
O                      = The Fedora Project
OU                     = $package
CN                     = <real FQDN>
[ v3_req ]
basicConstraints       = CA:FALSE

Note the specific differences from the CA certificate configuration file: the v3_ca reference and section has been replaced by a v3_req section which asserts that this certificate is not a Certificate Authority (and cannot be used to sign other certificates). Also note that unlike the CA certificate, this configuration requires the CN to match the machine's real fully-qualified hostname. If you do not provide this, clients will be unable to validate this machine's certificate.

Save this file as $package-ssl-service.cnf in the same location as the CA configuration. As with that configuration, this file can be removed after processing is complete.

Next, we will generate a secure private key for the service certificate. This key must be retained and provided in the location that the service expects it for SSL to function properly. Substitute the correct location for your package below:

/usr/bin/openssl genrsa -out /path/to/service/key 2048

As above, this generates a 2048-bit key. This should be considered the recommended minimum key strength. If a package does not support 2048-bit or higher keys, a bug should be opened.

Next, we will create a signing request for this key and sign it with our temporary CA.

/usr/bin/openssl req -new \
                     -config /path/to/$package-ssl-service.cnf \
                     -key /path/to/service/key \
                     -out /path/to/$package-ssl.csr

/usr/bin/openssl x509 -req -days 3650 \
                      -in /path/to/$package-ssl.csr \
                      -CA /path/to/$package-ssl-ca.crt \
                      -CAkey $TMPKEY \
                      -CAcreateserial \
                      -set_serial 0x`/usr/bin/openssl rand -hex 8` \
                      -out /path/to/service/certificate \
                      -extfile /path/to/$package-ssl-ca.cnf

Carefully note the "-ca" suffixes there; it can be tricky to see where to use the CA certificate information vs. the service certificate information.

The last step is to clean up the resulting files, as they are all owned by root with restrictive permissions:

# Delete the temporary CA key so nothing else can be signed with it
# We'll also overwrite it with zeros before deleting it, just in case.
dd if=/dev/zero of=$TMPKEY bs=1k count=16
rm -f $TMPKEY

# Optionally delete unneeded CSR and the OpenSSL Configuration Files
rm -f /path/to/$package-ssl.csr \
      /path/to/$package-ssl-ca.cnf \
      /path/to/$package-ssl-service.cnf

# Restore the original umask
umask $OLDUMASK

chown pkguser:pkguser /path/to/service/certificate \
                      /path/to/service/key
# Appropriate chmod() calls here

As a final, optional step, the private temporary CA certificate can be added to the local root CA list so that requests from the local machine will be able to connect as trusted clients without having to skip validation (as would be normal for self-signed certificates). Unlike traditional self-signed certificates, since this temporary CA cannot be used to sign anything else, this does not open your system to a risk of the service certificate being used to falsely sign something.

cp /path/to/$package-ssl-ca.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust extract

Mozilla NSS Certificate Database

Pre-requisites

First, you must decide whether your package will rely on the system database or a private package database for storing certificates.

The system database is located at /etc/pki/nssdb/. This database should be used if you want to share certificate information with other services on the system (such as having a common CA certificate store).

A private database should be used when you want to isolate the private keys used by your package's certificate from other services and users on the system.

If you decide to create a private database, you will need to create a directory to store that database. We will call that directory NSSPATH in this document.

NSSPATH=/path/to/service/nss-database/directory

It is highly recommended to use the modern NSS file format, which means that whenever NSS utilities require the database directory (dbdir) as a parameter, you should prefix the NSSPATH with sql: to request the modern file format. (The resulting files in the NSSPATH directory will be key4.db, cert9.db, pkcs11.txt.) This file format is based on sqlite, and it's possible to allow multiple applications concurrent access the same NSSPATH (if desired). This means you may use the NSS command line tools to inspect NSSPATH while your service is running.

Initialization of the Database

If you are using a private database, you will need to initialize the NSSPATH database. If you are using the system database or another previously-existing database, you may skip to the next step.

You need to decide if you'd like a master password on the database that will protect the private keys stored in it. If your service is supposed to start up automatically, you probably don't want a password (equivalent to empty password).

Run:

certutil -d sql:$NSSPATH -N --empty-password

Create Short-Lived Certificate Authority

Next, we will create a temporary Certificate Authority that will be used to sign the service certificate. First, we will need to gather the identification text used to identify the CA certificate. This tutorial will use BASH syntax as an example, but it can be adapted as needed:

HOSTNAME=<system FQDN>

PACKAGE_NAME=<insert appropriate value>

HOSTNAME should be set to the fully qualified hostname, to ensure that client software that attempts to validate the service certificate won't complain about hostname mismatches.

We'll also include the service setup time in the CN, to minimize the risk that we'll ever create two certificates that have the same value of {name,serial-number} as that would cause issues for software verifying the certificates.

CURRENT_TIME=`date -u +'%Y-%m-%d %H:%M:%S UTC'`

CA_CN="O=Private CA for $PACKAGE_NAME on $HOSTNAME created at $CURRENT_TIME"

CERT_CN="O=Administrator for $PACKAGE_NAME on $HOSTNAME, CN=$HOSTNAME"

In an NSS database, a certificate is usually identified by a user-defined nickname. Set it to something that makes sense for your package.

CA_NICKNAME=my-ca-cert-nickname
CERT_NICKNAME=my-service-cert-nickname

The authors of this tutorial strongly recommend that any real certificate that is manually generated should have a much shorter valid lifetime. However, for automatically-generated certificates, we want to reduce the risk that they will expire too quickly, so we will set its lifetime to ten years.

VALIDITY_IN_MONTHS=120

In addition, we will generate a random serial number for the CA, and a value increased by one for the serial of the service cert.

CA_SERIAL=$RANDOM
CERT_SERIAL=$(($CA_SERIAL + 1))

We should use a certificate with a modern hash algorithm, let's use SHA256

HASH_ALG=SHA256

We should use at least 2048 bits for the RSA key size:

KEY_SIZE=2048

We need a random seed, to actually generate a random private key for our certificate. Let's read random data and save it to a file.

RANDOM_SEED_FILE_CA=$NSSPATH/seed_ca
RANDOM_SEED_FILE_CERT=$NSSPATH/seed_cert

SEED_BYTES=$(($KEY_SIZE / 8))

dd if=/dev/random bs=1 count=$SEED_BYTES of=$RANDOM_SEED_FILE_CA
dd if=/dev/random bs=1 count=$SEED_BYTES of=$RANDOM_SEED_FILE_CERT

The certutil tool will attempt to interactively read the choices for several parameters. In order to make it fully unattended, we must provide a file that can be passed to certutil to automatically read the choices from.

CA_CU_IN=$NSSPATH/ca-input
CERT_CU_IN=$NSSPATH/cert-input

rm -f $CA_CU_IN
echo "y" > $CA_CU_IN
echo "1" >> $CA_CU_IN
echo "n" >> $CA_CU_IN
echo "3" >> $CA_CU_IN
echo "localhost" >> $CA_CU_IN
echo "1" >> $CA_CU_IN
echo "n" >> $CA_CU_IN
echo "n" >> $CA_CU_IN

rm -f $CERT_CU_IN
echo "n" > $CERT_CU_IN
echo "" >> $CERT_CU_IN
echo "n" >> $CERT_CU_IN

Now to actually create the CA we execute:

certutil -d sql:$NSSPATH \
         -S -n "$CA_NICKNAME" \
         -s "$CA_CN" \
         -t C,C,C \
         -x -m $CA_SERIAL \
         -v $VALIDITY_IN_MONTHS \
         -Z $HASH_ALG \
         -g $KEY_SIZE \
         --extNC $HOSTNAME \
         -z $RANDOM_SEED_FILE_CA \
         -2 < $CA_CU_IN

Once the new certificate is created, you can optionally view it with

certutil -d sql:$NSSPATH -L -n "$CA_NICKNAME"

Create and Sign a Service Certificate

Now we need to generate (and at the same time, sign) a certificate for the package service to use. To create the certificate, run:

certutil -d sql:$NSSPATH \
         -S -n "$CERT_NICKNAME" \
         -s "$CERT_CN" \
         -t ,, \
         -c "$CA_NICKNAME" \
         -m $CERT_SERIAL \
         -v $VALIDITY_IN_MONTHS \
         -Z $HASH_ALG \
         -g $KEY_SIZE \
         -z $RANDOM_SEED_FILE_CERT \
         -2 < $CERT_CU_IN

Finally, we want to delete the private key of the CA certificate, to make it less likely that anyone can use this CA for any other purposes.

certutil doesn't offer a way to delete a private key directly (-F will also remove the associated certificate), so we must export the CA certificate without the key to a file, then delete it from the database, then import it back.

CA_FILE=$NSSPATH/ca.pem

certutil -d sql:$NSSPATH -L -n "$CA_NICKNAME" -a > $CA_FILE

certutil -d sql:$NSSPATH -F -n "$CA_NICKNAME"

certutil -d sql:$NSSPATH -A -n "$CA_NICKNAME" -t C,C,C -a -i $CA_FILE

You can verify that the service cert is indeed recognized as a valid TLS server cert by running:

 certutil -d sql:$NSSPATH -V -n "$CERT_NICKNAME" -u V

As a final, optional step, the private temporary CA certificate can be added to the local root CA list so that requests from the local machine will be able to connect as trusted clients without having to skip validation (as would be normal for self-signed certificates). Unlike traditional self-signed certificates, since this temporary CA cannot be used to sign anything else, this does not open your system to a risk of the service certificate being used to falsely sign something.

cp $CA_FILE /etc/pki/ca-trust/source/anchors/
update-ca-trust extract

Open Questions

• Should we change the Organization in the service certificate configuration examples? Kai Engert raises the following point: "I think the owner of the private key, and thus the one who is responsible for the actions that will be taken with that private key, isn't the fedora project... we're not making certificates for the package, but for a service instance"
• Should we modify the CA certificate capabilities so that it is only permitted to sign certificates for the machine hostname (or sub-names thereof)? Kai Engert raised this question, Miloslav Trmac countered that since the signing key is short-lived, this is probably not urgent.