| IPA HSM Test DAy | |
|---|---|
| |
| Date | 2024-07-09 to 2024-07-11 |
| Time | all day |
| Website | QA/Test Days |
| Matrix | #test-day:fedoraproject.org(other clients|?) |
| Mailing list | test |
If you come to this page after the test day is completed, your testing is still valuable, and you can use the information on this page to test, file any bugs you find, and add your results to the results section. If this page is more than a month old when you arrive here, please check the current schedule and see if a similar but more recent Test Day is planned or has already happened.
What to test?
This Test Day will focus on FIXME
Who's available
The following cast of characters will be available testing, workarounds, bug fixes, and general discussion:
You can chat with me on Matrix. See the infobox on top of the page to learn where to join.
Prerequisite for Test Day
- A virtual machine or a bare metal machine
- An installation of Fedora 40 (ideally Server). Make sure to fully update your system. If installing a fresh system, it's recommended to use the latest nightly image.
What to test
This will focus on testing IPA support for generating and storing CA private keys on a Hardware Security Module (HSM).
There are two supported HSMs: the nCipher nShield Connect XC (High) and the Thales TCT Luna Network HSM Luna-T7. Firmware versions can vary so only specific ones are supported.
Using softhsm2 as an HSM is usable for testing. It is not recommended for production because it is not a truly networked HSM and the private keys live on a file system (protected yes but not at a hardware level). Because it is not networked, users will need to carefully synchronize the token files whenever any private key generation is done to ensure the contents are identical.
How to test?
Install freeIPA packages
- dnf -y install freeipa-server-dns
Pre-configure the HSM
If you are using softhsm2, grant read access to the tokens:
# usermod pkiuser -a -G ods
Set up environment variables on each machine/VM
# export TOKEN_PASSWORD=password # export ADMIN_PASSWORD=password # export DM_PASSWORD=password
If using a supported hardware HSM ensure that it is working properly and have the token name and PKCS#11 library path handy.
In between tests
To re-use test machines in between installations:
On replica (if there is one)
# ipa server-del $HOSTNAME # ipa-server-install –uninstall -U
On the initial IPA server
# ipa-server-install –uninstall -U
If using softhsm2 you will also need to delete and re-create the token. To delete the token:
# softhsm2-util --delete-token --token ipa_token
This should return the machine(s) to the pre-installed state.
Test Cases
Test Case 1
Install a basic IPA server with HSM
How to test:
Install the freeipa packages
# dnf install freeipa-server freeipa-server-dns softhsm -y
Rename the hostname with the domain to be used with ipa
# hostnamectl hostname ipa.example.test # echo “<ip-address> ipa.example.test ” >> /etc/hosts
Create softhsm token
# runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token
Install the IPA server
# ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD
Ensure that certificate stored with the hsm token
# certutil -L -d /etc/pki/pki-tomcat/alias -h ipa_token
ipa_token:ocspSigningCert cert-pki-ca u,u,u ipa_token:subsystemCert cert-pki-ca u,u,u ipa_token:auditSigningCert cert-pki-ca u,u,Pu ipa_token:caSigningCert cert-pki-ca CTu,Cu,Cu
Basic IPA Sanity
Test that basic things within IPA work.
# kinit admin # ipa user-add --first tim --last user --password tuser # id tuser # kinit tuser (and reset password) # ipa user-show tuser
Test Case 2
Install an IPA server and replica with HSM
How to test:
Install the freeipa packages on both machines
# dnf install freeipa-server freeipa-server-dns softhsm -y
Rename the hostname with the domain to be used with ipa
server:
# hostnamectl hostname ipa.example.test # echo “<ip-address> ipa.example.test ” >> /etc/hosts
Create softhsm token on ipa.example.test only
# runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token
Install the IPA server
# ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD
Identify the token directory on ipa.example.test
# ls -1tr /var/lib/softhsm/tokens/ | tail -1
This will return a UUID like e373ded4-8763-29e9-dff9-e41f6930297e
Copy token data to replica
# export token=”<UUID>”
# rsync -avp $IPA_SERVER_IP:/var/lib/softhsm/tokens/${token} /var/lib/softhsm/tokens/
Add a DNS server to ipa.example.test (it will make things easier)
# ipa-dns-install --no-forwarders --auto-reverse
Add the replica IP information to DNS (on the IPA server)
# kinit admin # ipa dnsrecord-add example.test. replica --a-rec=$REPLICA_IP
Set replica hostname
# hostnamectl hostname replica.example.test
Configure the replica to use the IPA server DNS
# resolvectl dns eth0 $IPA_SERVER_IP:53
Install ipa-replica
# ipa-replica-install --domain example.test --realm EXAMPLE.TEST --admin-password $ADMIN_PASSWORD -U -N --setup-ca --token-password $TOKEN_PASSWORD
Verify that the certificate serial numbers are the same. Run this on both machines.
# certutil -L -d /etc/pki/pki-tomcat/alias -h ipa_token -n 'ipa_token:subsystemCert cert-pki-ca' |grep -A1 'Serial Number:'
Serial Number: 4 (0x4)
Test Case 3
Install an IPA server with a KRA
How to test:
Install the freeipa packages
# dnf install freeipa-server freeipa-server-dns softhsm -y
Rename the hostname with the domain to be used with ipa
# hostnamectl hostname ipa.example.test # echo “<ip-address> ipa.example.test” >> /etc/hosts
Create softhsm token
# runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token
Install the IPA server
# ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD --setup-kra
Ensure that certificate stored with the hsm token (note the kra certs)
# certutil -L -d /etc/pki/pki-tomcat/alias -h ipa_token
Verify that the KRA is functional
# kinit admin # ipa vault-add test # ipa vault-archive test --data Zm9vCg== # ipa vault-retrieve test
Test Case 4
Install an IPA server and replica with KRA
How to test:
Install the freeipa packages
# dnf install freeipa-server freeipa-server-dns softhsm -y
Rename the hostname with the domain to be used with ipa
# hostnamectl hostname ipa.example.test # echo “<ip-address> ipa.example.test” >> /etc/hosts
Create softhsm token
# runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token
Install IPA server with a KRA
# ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD --setup-kra
Identify the token directory on ipa.example.test
# ls -1tr /var/lib/softhsm/tokens/ | tail -1
This will return a UUID like e373ded4-8763-29e9-dff9-e41f6930297e
Copy token data to replica
# export token=”<UUID>”
# rsync -avp $IPA_SERVER_IP:/var/lib/softhsm/tokens/${token} /var/lib/softhsm/tokens/
Add a DNS server to ipa.example.test (it will make things easier)
# ipa-dns-install --no-forwarders --auto-reverse
Add the replica IP information to DNS (on the IPA server)
# kinit admin # ipa dnsrecord-add example.test. replica --a-rec=$REPLICA_IP
Set replica hostname
# hostnamectl hostname replica.example.test
Configure the replica to use the IPA server DNS
# resolvectl dns eth0 $IPA_SERVER_IP:53
Install an IPA replica with a KRA
# ipa-replica-install --domain example.test --realm EXAMPLE.TEST --admin-password $ADMIN_PASSWORD -U -N --setup-ca --token-password $TOKEN_PASSWORD --setup-kra
Verify that the KRA is functional
This vault can be created on either machine. Please verify that the vault is accessible on both.
# kinit admin # ipa vault-add test # ipa vault-archive test --data Zm9vCg== # ipa vault-retrieve test
Test Case 5
Install an IPA server with HSM and renew a certificate outside the grace period
How to test:
Install the freeipa packages
# dnf install freeipa-server freeipa-server-dns softhsm -y
Rename the hostname with the domain to be used with ipa
# hostnamectl hostname ipa.example.test # echo “<ip-address> ipa.example.test” >> /etc/hosts
Create softhsm token
# runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token
Install the IPA server
# ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD
Move date to within the expiration grace period
# date -s +2years+11months+20days # ipactl restart # sleep 90
Force issuance of new certs
# ipa-cert-fix (answer yes)
It will take a bit for new certs to be issued and for certmonger to notice. To monitor it: watch -n 5 'getcert list | grep status'
Expected results
The expired certificates (all but the CA cert) will be re-issued. As you monitor using getcert list you may see the certificates go through different states including: SUBMITTING, GENERATING_CSR, POST_SAVED_CERT, NEED_TO_SUBMIT and/or NEED_TO_SAVE_CERT
Return date to current time
Uninstall the IPA server prior to moving time backwards.
# date +s +2years+11months+20days
Test Case 6
Install an IPA server with HSM and renew a certificate inside the grace period
How to test:
Install the freeipa packages
# dnf install freeipa-server freeipa-server-dns softhsm -y
Rename the hostname with the domain to be used with ipa
# hostnamectl hostname ipa.example.test # echo “<ip-address> ipa.example.test” >> /etc/hosts
Create softhsm token
# runuser -u pkiuser -- /usr/bin/softhsm2-util --init-token --free --pin $TOKEN_PASSWORD --so-pin $TOKEN_PASSWORD --label ipa_token
Install the IPA server
# ipa-server-install -a $ADMIN_PASSWORD -p $DM_PASSWORD -r EXAMPLE.TEST -U --random-serial-numbers --token-name=ipa_token --token-library-path=/usr/lib64/pkcs11/libsofthsm2.so --token-password=$TOKEN_PASSWORD
Move date to near the end of the grace period
# date -s +1years+11months+20days
Monitor the renewal
It will take a bit for new certs to be issued and for certmonger to notice. To monitor it:
# watch -n 5 'getcert list | grep status'
The expired certificates (all but the CA cert) will be re-issued. As you monitor using getcert list you may see the certificates go through different states including: SUBMITTING, GENERATING_CSR, POST_SAVED_CERT, NEED_TO_SUBMIT and/or NEED_TO_SAVE_CERT
If one certificate fails to renew with CA_UNREACHABLE wait until all of the certs are either in this state or MONITORING. Then restart certmonger and run the watch again. Certificate renewal can be bumpy as lots of service restarts happen and the renewals can step on one another.
Reporting bugs
Perhaps you've found an already-reported bug. Please look at:
All new bugs should be reported into the upstream bug tracker. A less-preferred alternative is to file them into Red Hat JIRA, in most cases against the ipa component.
Please note that just mentioning your problem into the comments section on the results page is not very helpful. Very often those problems only happen in specific circumstances, or with specific steps taken. We need the logs and screenshots, and we need to be able to ask you followup questions. Please file bug reports, it's much more useful than a short comment. Thank you!
When filing the bug, it's very helpful to include:
- exact steps you've performed (and whether you can reproduce it again)
- screenshots or videos, if applicable
- system journal (log), which you can retrieve by
journalctl -b > journal.txt - all output in a terminal, if started from a terminal
- your system description
If you are unsure about exactly how to file the report or what other information to include, just ask us.
Please make sure to link to the bug when submitting your test result, thanks!
Test Results
Test results will be exported here once the test day is over. See How to test? section for information how to submit results and see the live results.