First-time Service Setup
Many system services require some amount of initial setup before they can run properly for the first time. Common examples are the generation of private keys and certificates or a unique, system-specific identifier.
Traditionally, this was done by RPM scriptlets as part of the installation or upgrade of a package. This was sensible for a time when the majority of installations were performed by attended or unattended installers (such as anaconda and kickstart).
Today we see an increased reliance on generating virtual machine images for use in both traditional and cloud-computing environments. In those cases, having system-specific data created at package installation time is problematic. It means that the production of such images need to have significant care applied to remove any system-specific information about them and then additional tools written to apply the corrected information post-deployment. The goal of this guideline is to ensure that if a system clean-up service such as virt-sysprep is run on the system and then the machine is rebooted, any service that requires first-time configuration will re-run it. The mechanism by which we will accomplish this is to remove such first-time configuration from RPM scriptlets (e.g. %post) and instead execute this configuration as part of service startup with systemd.
This guideline describes a mechanism that can be used for both traditional and cloud-based deployment styles.
Note: this requirement can be waived if the equivalent functionality is incorporated as part of the service's own standard startup. These guidelines are meant to address services that require setup before the service can be started.
Defining System-Specific Setup
A particular setup task is defined thusly: "Any action that must be performed on the system where the service will be run that is not common to all systems running that service."
Some non-exhaustive examples of system-specific configuration:
- The SSH daemon generates a public/private host key
- The mod_ssl httpd module creates a self-signed certificate for the machine's hostname
- A remote logging service creates a UUID to represent this machine
A few examples that should not be considered system-specific configuration:
- Creating a service user and/or group. This is safe to copy to clones of the system.
- Anything automatically generated by the service when it is started and recreated if it gets deleted.
Common Guidelines
For all system-specific cases, we will take advantage of systemd's ExecStartPre functionality.
Packagers will create a script in /usr/lib/systemd/system-init/ named after the package that it initializes. So for sshd, the script file would be /usr/lib/systemd/system-init/openssh-server (note that the language of the script is up to the packager, but the file must be executable). This script must implement all of the following steps:
- Perform a test for whether the initialization has already completed. This may be a simple test of file existence or a more complicated examination of the configuration, as needed. If the initialization has already occurred, the script must immediately return zero (success).
- Perform whatever steps are necessary to generate the configuration. If this cannot be accomplished, the script must return with a non-zero error code. This will prevent systemd from attempting to start the actual service. If this completes successfully, it must satisfy any and all requirements for the first step to to pass. On success, it must return zero.
The service's systemd unit file must be modified to include the following line within the [Service] section:
ExecStartPre=/usr/lib/systemd/system-init/<packagename>
Special Case: Self-signed Certificate Generation
OpenSSL and PEM Certificates
This tutorial will provide the steps to produce something akin to a self-signed certificate, except that the resulting service certificate cannot be used to sign further certificates (thus closing a potential security issue). In broad terms, what we are doing is creating a short-lived certificate authority, using that to sign a service certificate and then destroying the key material for the temporary authority.
(Note: for any instance of $package below, substitute the name of your package)
Create Short-Lived Certificate Authority
First, create an OpenSSL configuration file for the CA certificate. It should be similar to this:
[ req ] distinguished_name = req_distinguished_name prompt = no x509_extensions = v3_ca [ req_distinguished_name ] C = -- ST = SomeState L = SomeCity O = Private CA for $package on <real_fqdn> at <date> UTC OU = $package CN = $package.<real_fqdn> [ v3_ca ] subjectKeyIdentifier=hash authorityKeyIdentifier=keyid:always,issuer basicConstraints = CA:TRUE
Most of the organizational values above can be changed, but CN must be $package.<real_fqdn>. Additionally, the Organization (O) field should remain as in the example so that it provides clues to administrators who may wish to import the public CA certificate. The date field should match the output of date -u +'%Y-%m-%d %H:%M:%S'
Create this file with the name $package-ssl-ca.cnf in a location owned by your package. You may retain or destroy this file later as you prefer.
Next, we will generate the private key for this temporary Certificate Authority. Set the umask so that only root/owner can read the files we create. Then create the key file with the openssl command. We will create the temporary key file in /dev/shm so that it is unlikely to ever be written to persistent media.
OLDUMASK=`umask` umask 0077 TMPKEY=`mktemp --tmpdir=/dev/shm XXXXXXXXXXXX` /usr/bin/openssl genrsa -out $TMPKEY 2048
Note: it is *highly* recommended to use a 2048-bit or greater key. Any package that is not compatible with 2048-bit keys should have a bug open to track this shortcoming.
Next, we will use this private key to create the temporary CA Certificate:
/usr/bin/openssl req -new -x509 -days 3650 \
-config /path/to/$package-ssl-ca.cnf \
-key $TMPKEY \
-out /path/to/$package-ssl-ca.crt
You can adjust the value of -days as you prefer. Store the $package-ssl-ca.crt file in the same location as the $package-ssl-ca.cnf. Now we have a certificate authority available to sign our service certificate.
Create and Sign a Service Certificate
Similar to the CA certificate generation, we need to create an OpenSSL configuration file for the service certificate as below:
[ req ] distinguished_name = req_distinguished_name prompt = no req_extensions = v3_req [ req_distinguished_name ] C = -- ST = SomeState L = SomeCity O = The Fedora Project OU = $package CN = <real FQDN> [ v3_req ] basicConstraints = CA:FALSE
Note the specific differences from the CA certificate configuration file: the v3_ca reference and section has been replaced by a v3_req section which asserts that this certificate is not a Certificate Authority (and cannot be used to sign other certificates). Also note that unlike the CA certificate, this configuration requires the CN to match the machine's real fully-qualified hostname. If you do not provide this, clients will be unable to validate this machine's certificate.
Save this file as $package-ssl-service.cnf in the same location as the CA configuration. As with that configuration, this file can be removed after processing is complete.
Next, we will generate a secure private key for the service certificate. This key must be retained and provided in the location that the service expects it for SSL to function properly. Substitute the correct location for your package below:
/usr/bin/openssl genrsa -out /path/to/service/key 2048
As above, this generates a 2048-bit key. This should be considered the recommended minimum key strength. If a package does not support 2048-bit or higher keys, a bug should be opened.
Next, we will create a signing request for this key and sign it with our temporary CA.
/usr/bin/openssl req -new \
-config /path/to/$package-ssl-service.cnf \
-key /path/to/service/key \
-out /path/to/$package-ssl.csr
/usr/bin/openssl x509 -req -days 3650 \
-in /path/to/$package-ssl.csr \
-CA /path/to/$package-ssl-ca.crt \
-CAkey $TMPKEY \
-CAcreateserial \
-set_serial 0x`/usr/bin/openssl rand -hex 8` \
-out /path/to/service/certificate \
-extfile /path/to/$package-ssl-ca.cnf
Carefully note the "-ca" suffixes there; it can be tricky to see where to use the CA certificate information vs. the service certificate information.
The last step is to clean up the resulting files, as they are all owned by root with restrictive permissions:
# Delete the temporary CA key so nothing else can be signed with it
# We'll also overwrite it with zeros before deleting it, just in case.
dd if=/dev/zero of=$TMPKEY bs=1k count=16
rm -f $TMPKEY
# Optionally delete unneeded CSR and the OpenSSL Configuration Files
rm -f /path/to/$package-ssl.csr \
/path/to/$package-ssl-ca.cnf \
/path/to/$package-ssl-service.cnf
# Restore the original umask
umask $OLDUMASK
chown pkguser:pkguser /path/to/service/certificate \
/path/to/service/key
# Appropriate chmod() calls here
As a final, optional step, the private temporary CA certificate can be added to the local root CA list so that requests from the local machine will be able to connect as trusted clients without having to skip validation (as would be normal for self-signed certificates). Unlike traditional self-signed certificates, since this temporary CA cannot be used to sign anything else, this does not open your system to a risk of the service certificate being used to falsely sign something.
cp /path/to/$package-ssl-ca.crt /etc/pki/ca-trust/source/anchors/ update-ca-trust extract
Mozilla NSS Certificate Database
Pre-requisites
First, you must decide whether your package will rely on the system database or a private package database for storing certificates.
The system database is located at /etc/pki/nssdb/. This database should be used if you want to share certificate information with other services on the system (such as having a common CA certificate store).
A private database should be used when you want to isolate the private keys used by your package's certificate from other services and users on the system.
If you decide to create a private database, you will need to create a directory to store that database. We will call that directory NSSPATH in this document.
NSSPATH=/path/to/service/nss-database/directory
It is highly recommended to use the modern NSS file format, which means that whenever NSS utilities require the database directory (dbdir) as a parameter, you should prefix the NSSPATH with sql: to request the modern file format. (The resulting files in the NSSPATH directory will be key4.db, cert9.db, pkcs11.txt.) This file format is based on sqlite, and it's possible to allow multiple applications concurrent access the same NSSPATH (if desired). This means you may use the NSS command line tools to inspect NSSPATH while your service is running.
Initialization of the Database
If you are using a private database, you will need to initialize the NSSPATH database. If you are using the system database or another previously-existing database, you may skip to the next step.
You need to decide if you'd like a master password on the database that will protect the private keys stored in it. If your service is supposed to start up automatically, you probably don't want a password (equivalent to empty password).
Run:
certutil -d sql:$NSSPATH -N --empty-password
Create Short-Lived Certificate Authority
Next, we will create a temporary Certificate Authority that will be used to sign the service certificate. First, we will need to gather the identification text used to identify the CA certificate. This tutorial will use BASH syntax as an example, but it can be adapted as needed:
HOSTNAME=<system FQDN> PACKAGE_NAME=<insert appropriate value>
HOSTNAME should be set to the fully qualified hostname, to ensure that client software that attempts to validate the service certificate won't complain about hostname mismatches.
We'll also include the service setup time in the CN, to minimize the risk that we'll ever create two certificates that have the same value of {name,serial-number} as that would cause issues for software verifying the certificates.
CURRENT_TIME=`date -u +'%Y-%m-%d %H:%M:%S UTC'` CA_CN="O=Private CA for $PACKAGE_NAME on $HOSTNAME created at $CURRENT_TIME" CERT_CN="O=Administrator for $PACKAGE_NAME on $HOSTNAME, CN=$HOSTNAME"
In an NSS database, a certificate is usually identified by a user-defined nickname. Set it to something that makes sense for your package.
CA_NICKNAME=my-ca-cert-nickname CERT_NICKNAME=my-service-cert-nickname
The authors of this tutorial strongly recommend that any real certificate that is manually generated should have a much shorter valid lifetime. However, for automatically-generated certificates, we want to reduce the risk that they will expire too quickly, so we will set its lifetime to ten years.
VALIDITY_IN_MONTHS=120
In addition, we will generate a random serial number for the CA, and a value increased by one for the serial of the service cert.
CA_SERIAL=$RANDOM CERT_SERIAL=$(($CA_SERIAL + 1))
We should use a certificate with a modern hash algorithm, let's use SHA256
HASH_ALG=SHA256
We should use at least 2048 bits for the RSA key size:
KEY_SIZE=2048
We need a random seed, to actually generate a random private key for our certificate. Let's read random data and save it to a file.
RANDOM_SEED_FILE_CA=$NSSPATH/seed_ca RANDOM_SEED_FILE_CERT=$NSSPATH/seed_cert SEED_BYTES=$(($KEY_SIZE / 8)) dd if=/dev/random bs=1 count=$SEED_BYTES of=$RANDOM_SEED_FILE_CA dd if=/dev/random bs=1 count=$SEED_BYTES of=$RANDOM_SEED_FILE_CERT
The certutil tool will attempt to interactively read the choices for several parameters. In order to make it fully unattended, we must provide a file that can be passed to certutil to automatically read the choices from.
CA_CU_IN=$NSSPATH/ca-input CERT_CU_IN=$NSSPATH/cert-input rm -f $CA_CU_IN echo "y" > $CA_CU_IN echo "1" >> $CA_CU_IN echo "n" >> $CA_CU_IN echo "3" >> $CA_CU_IN echo "localhost" >> $CA_CU_IN echo "1" >> $CA_CU_IN echo "n" >> $CA_CU_IN echo "n" >> $CA_CU_IN rm -f $CERT_CU_IN echo "n" > $CERT_CU_IN echo "" >> $CERT_CU_IN echo "n" >> $CERT_CU_IN
Now to actually create the CA we execute:
certutil -d sql:$NSSPATH \
-S -n "$CA_NICKNAME" \
-s "$CA_CN" \
-t C,C,C \
-x -m $CA_SERIAL \
-v $VALIDITY_IN_MONTHS \
-Z $HASH_ALG \
-g $KEY_SIZE \
--extNC $HOSTNAME \
-z $RANDOM_SEED_FILE_CA \
-2 < $CA_CU_IN
Once the new certificate is created, you can optionally view it with
certutil -d sql:$NSSPATH -L -n "$CA_NICKNAME"
Create and Sign a Service Certificate
Now we need to generate (and at the same time, sign) a certificate for the package service to use. To create the certificate, run:
certutil -d sql:$NSSPATH \
-S -n "$CERT_NICKNAME" \
-s "$CERT_CN" \
-t ,, \
-c "$CA_NICKNAME" \
-m $CERT_SERIAL \
-v $VALIDITY_IN_MONTHS \
-Z $HASH_ALG \
-g $KEY_SIZE \
-z $RANDOM_SEED_FILE_CERT \
-2 < $CERT_CU_IN
Finally, we want to delete the private key of the CA certificate, to make it less likely that anyone can use this CA for any other purposes.
certutil doesn't offer a way to delete a private key directly (-F will also remove the associated certificate), so we must export the CA certificate without the key to a file, then delete it from the database, then import it back.
CA_FILE=$NSSPATH/ca.pem certutil -d sql:$NSSPATH -L -n "$CA_NICKNAME" -a > $CA_FILE certutil -d sql:$NSSPATH -F -n "$CA_NICKNAME" certutil -d sql:$NSSPATH -A -n "$CA_NICKNAME" -t C,C,C -a -i $CA_FILE
You can verify that the service cert is indeed recognized as a valid TLS server cert by running:
certutil -d sql:$NSSPATH -V -n "$CERT_NICKNAME" -u V
As a final, optional step, the private temporary CA certificate can be added to the local root CA list so that requests from the local machine will be able to connect as trusted clients without having to skip validation (as would be normal for self-signed certificates). Unlike traditional self-signed certificates, since this temporary CA cannot be used to sign anything else, this does not open your system to a risk of the service certificate being used to falsely sign something.
cp $CA_FILE /etc/pki/ca-trust/source/anchors/ update-ca-trust extract
Open Questions
- Should we change the Organization in the service certificate configuration examples? Kai Engert raises the following point: "I think the owner of the private key, and thus the one who is responsible for the actions that will be taken with that private key, isn't the fedora project... we're not making certificates for the package, but for a service instance"
- Should we modify the CA certificate capabilities so that it is only permitted to sign certificates for the machine hostname (or sub-names thereof)? Kai Engert raised this question, Miloslav Trmac countered that since the signing key is short-lived, this is probably not urgent.
- Should we consider creation of a framework for runtime data (such as an empty database or log directory) to be "system-specific configuration"?. They may *become* populated with additional data that should be cleaned out before cloning, but the framework *itself* is not really system-specific.