Changes/Automatic Cloud Reboot On Updates

Automatic Cloud Reboot On Updates

Summary

Cloud users can provide cloud-init metadata when creating a Fedora cloud instance and that metadata can contain instructions to update all packages on the system and reboot the system if any of those updated packages need a reboot to go into effect. Fedora cloud instances should write the /var/run/reboot-required file if a reboot is needed after a dnf update so that cloud-init can reboot the instance.

This issue originally surfaced in RHBZ 1275409.

Owner

• Name: Major Hayden
• Email: major@redhat.com

Current status

• Targeted release: Fedora Linux 39
• Last updated: 2023-09-28
• devel thread
• FESCo issue: #3010
• Tracker bug: #2233245
• Release notes tracker: #1021

Detailed Description

Fedora cloud instances use cloud-init to do the initial configuration of the instance. This includes setting up networking, assigning a hostname, adding users/groups, and arbitrary scripts. There are also two options that you can pass to cloud-init that are important for this change:

• package_update: If set to true, all installed packages are immediately updated on first boot
• package_reboot_if_required: If set to true, and the package_update step wrote to /var/run/reboot-required, reboot the system immediately after updating packages

📚 For more details, see cloud-init's module reference for package_update.

🚨 WAIT A MOMENT. ARE WE TALKING ABOUT REBOOTING EVERY CLOUD INSTANCE ON BOOT? 🚨 No! This change would require all three of these things to happen before a reboot occurs:

• User provides package_update: true on instance creation
• AND user provides package_reboot_if_required: true on instance creation
• AND tracer notices that at least one of the packages need a reboot to go into effect

🤔 Where does this /var/run/reboot-required file come from? On Debian and Ubuntu systems, apt automatically writes to /var/run/reboot-required if a reboot is needed after a package update. From there, cloud-init looks for the file (relevant cloud-init code) and if present, reboots the system immediately.

✏️ How do we write this file on Fedora? Fedora systems have a package called tracer and a corresponding dnf plugin, python3-dnf-plugin-tracer, that analyzes dnf updates and provides recommendations on reboots or user logouts to bring updates into effect on the system. A recent pull request added support for writing the /var/run/reboot-required file when a system reboot is recommended. The cloud-init tool can read this file after a package update and reboot if needed.

🔎 What does tracer's output look like?

   [root@tracer-testing ~]# tracer 
   You should restart:
   * Some applications using:
       sudo systemctl restart NetworkManager
       sudo systemctl restart auditd
       sudo systemctl restart chronyd
       sudo systemctl restart dbus-broker
       sudo systemctl restart qemu-guest-agent
       sudo systemctl restart sshd
       sudo systemctl restart systemd-journald
       sudo systemctl restart systemd-logind
       sudo systemctl restart systemd-oomd
       sudo systemctl restart systemd-resolved
       sudo systemctl restart systemd-udevd
       sudo systemctl restart systemd-userdbd
   
   * These applications manually:
       (sd-pam)
   
   Additionally, there are:
   - 3 processes requiring restart of your session (i.e. Logging out & Logging in again)
   - 1 processes requiring reboot
   [root@tracer-testing ~]# cat /var/run/reboot-required 
   Tracer says reboot is required

📋 What do we need to do? Add the python3-dnf-plugin-tracer plugin to Fedora cloud images. No additional configuration is necessary. This action pulls in five packages that are about 2.1MB after installation:

   =======================================================================================
   Package                               Arch       Version             Repository  Size
   =======================================================================================
   Installing:
   python3-dnf-plugin-tracer             noarch     4.1.0-1.fc38        fedora      14 k
   Installing dependencies:
   python3-dnf-plugins-extras-common     noarch     4.1.0-1.fc38        fedora      69 k
   python3-psutil                        x86_64     5.9.2-2.fc38        fedora     271 k
   python3-tracer                        noarch     0.7.8-5.fc38        fedora     172 k
   tracer-common                         noarch     0.7.8-5.fc38        fedora      22 k
   
   Transaction Summary
   =======================================================================================
   Install  5 Packages
   
   Total download size: 547 k
   Installed size: 2.1 M

Feedback

One of the other ideas was to patch cloud-init to run tracer directly and avoid the /var/run/reboot-required file altogether. That would require a lot of work upstream in cloud-init to enable the functionality and we would still need the same set of packages installed in Fedora anyway. 🥵

Benefit to Fedora

This change allows Fedora cloud instances to behave in the same way that Debian-based instances already behave. When users request package updates with a reboot now, cloud-init performs the update but never reboots the system. This is an unexpected and confusing result for users who come to Fedora from other distributions.

Rebooting automatically could also reduce the attack surface of an instance that just came online since it would immediately reboot to put all package updates into effect on the system. This reduces the time that an unpatched instance is online prior to being fully patched.

Scope

• Proposal owners: This change is fairly isolated and only affects Fedora cloud users who request package updates followed by a reboot in their cloud-init metadata.

• Other developers: N/A

• Release engineering: N/A

• Policies and guidelines: N/A

• Trademark approval: N/A

• Alignment with Community Initiatives: N/A

Upgrade/compatibility impact

Since this change only applies to cloud-init on the very first boot of the instance, this wouldn't affect a user upgrading from one version of Fedora to the next.

How To Test

1. Ensure you have a cloud image that has an update that needs a reboot (kernel, openssl, etc)
2. Boot an instance with the following cloud-init user data:

   #cloud-config
   package_update: true
   package_upgrade: true
   package_reboot_if_required: true

1. Wait for the package updates to finish on the instance and verify that it rebooted after updating

User Experience

First, if a user never uses the package_upgrade and package_reboot_if_required options in their cloud-init user data, they won't be affected by this change. These options are not enabled in cloud-init by default.

If a user does enable both of these options, they will see their cloud instance come online, apply updates, and reboot if required. Most cloud providers have very fast reboots, so the delay should not be a problem.

Dependencies

Nothing depends on this change.

Contingency Plan

• Contingency mechanism: Push to Fedora 40 if the work cannot be done in time
• Contingency deadline: N/A
• Blocks release? N/A

Documentation

Guidance for users in a blog post (Fedora Magazine) could be helpful for this change. Many users might not be aware that they had the option to ask for package updates and reboots via cloud-init for their Fedora cloud instances.

Release Notes

Fedora cloud instances now automatically reboot when a user requests package updates followed by a reboot on the first boot of the instance. The reboot only occurs if an updated package requires a reboot to go into effect (such as a kernel or critical system library).