Changes/Shorter Shutdown Timer

Shorter Shutdown Timer

Summary

A downstream configuration change to reduce the systemd unit timeout from 2 minutes to 45 seconds and send SIGABRT to generate a core dump before SIGKILL.

Owner

• Name: Michael Catanzaro, Allan Day, Zbigniew Jędrzejewski-Szmek
• Email: mcatanzaro at redhat dot com, aday at redhat dot com, zbyszek at in dot waw dot pl

Current status

• Targeted release: Fedora Linux 38
• Last updated: 2023-02-22
• devel thread
• FESCo issue: #2928
• Tracker bug: #2161753
• Release notes tracker: #954

Detailed Description

Currently, a service that fails to stop at shutdown time can block shutdown for up to 2 minutes. This is extremely frustrating for our users - someone goes to shutdown or reboot their system, and then unexpectedly has to wait for a long time before they can do anything else.

The most common service to cause this issue is PackageKit, but there are others.

When a service fails to shutdown when it is instructed to do so, it is not behaving properly, and it is preventing the system from behaving in an orderly and predictable manner. Desktop APIs exist for cases when services or apps legitimately need to prevent shutdown, and these allow the shutdown inhibit to be communicated to admins and users, so they understand what is happening. When the user decides to shut down anyway, services must terminate in a timely manner. The Workstation Working Group feels that 15 seconds is the maximum appropriate time for both system and user services, and that Fedora should be robust to buggy and misbehaving services that do not shut down in an appropriate manner. However, FESCo has requested that we start with a 45 second timeout instead of dropping immediately to 15 seconds.

To facilitate debugging when a service fails to stop cleanly, we will use TimeoutStopFailureMode=abort to crash services that fail to stop in the time allotted. This will cause the service to crash with SIGABRT so that a core dump will be generated.

History

The Workstation Working Group has been working on this issue for several years. Investigations have revealed that it's not possible to fix every misbehaving service: in some cases the misbehaviour comes from design flaws that are difficult to resolve.

An attempt has also been made to have the unit timeout changed in upstream systemd. That attempt did not go anywhere, despite various efforts to move it along. We are no longer comfortable waiting for upstream changes to land.

To our knowledge, there are no issues that will result from forcing services to stop after 45 seconds on typical systems. However, system administrators may need to configure a higher timeout if waiting longer for a particular service, which may be true for database services or virtual machine managers, for example. Sensitive services may disable the timeout altogether; Postgres and virt-manager already do this.

Feedback

• Fedora Server wishes to be cautious and use a longer shutdown timer, but this change proposal is implemented in a way that would affect all Fedora editions. We should find a way to allow different Fedora editions to have different defaults, perhaps by altering config files.
• The short shutdown timer might not be long enough for libvirt to shut down VMs. Databases and virtual machines really must not be killed forcibly. Service files may already request longer timeouts, but would need to be modified to do so.
• The short shutdown timer could brick Pinephone modems. This seems like a hardware bug rather than something that should affect Fedora's default behavior.
• This change proposal now incorporates use of TimeoutStopFailureMode=abort due to discussion feedback, to facilitate debugging of services that do not stop properly.

Benefit to Fedora

The primary benefit of the change will be to mitigate a very annoying and - frankly - embarrassing bug. Our users shouldn't have to randomly sit waiting for their machine to shutdown. It will also encourage the correct use of shutdown inhibit APIs.

Although this change will "paper over" bugs in services without fixing them, we emphasize that reducing the timeout is not merely a workaround for buggy services, but also the desired permanent design. Of course it is desirable to fix the underlying bugs as well, but it doesn't make sense to require this before fixing the service timeout to match our needs.

Scope

• Proposal owners:
  • Merge pull request to shorten timeout to 45 s to systemd.
  • Merge pull request to set TimeoutStopFailureMode=abort to systemd.
• Other developers:
  • Test their packages with the new behavior and report issues as necessary.
• Release engineering: #11193

• Policies and guidelines: No policy or guideline changes required

• Trademark approval: N/A (not needed for this Change)

• Alignment with Objectives: N/A (not needed for this Change)

Upgrade/compatibility impact

System and user services will be killed with SIGABRT 45 seconds after receiving SIGTERM. Previously they would receive SIGKILL 1 minute 30 seconds for most system and user services, or 2 minutes for user manager system services (the system service that runs all user services for a user). Services will have less time to shut down gracefully by default, and a core dump will be generated to facilitate bug reporting. These defaults are configurable and system administrators who require longer timeouts would need to adjust them before or after upgrade. You may edit the DefaultTimeoutStopSec= setting in /etc/systemd/user.conf and /etc/systemd/system.conf. You may also create a drop-in to change the TimeoutStopSec= setting for user@service.

How To Test

Given the intermittent and unpredictable nature of the bug that is being targeted, the best way to test is by using the upcoming Fedora release. Are shutdown delays eliminated as intended? Do system services experience issues as a result of the change?

User Experience

This change will make the Fedora user experience less annoying. It will also encourage the use of the existing inhibit APIs, which provide better feedback for users when system shutdown does need to be delayed.

Dependencies

No specific changes are required in other packages. However, service developers may want to take this opportunity to examine the shutdown behavior of their components.

Contingency Plan

• Contingency mechanism: the change owners will revert the change in systemd.
• Contingency deadline: if we back out the change it would be best to do it before beta freeze, but this can happen at any point.
• Blocks release? No.

Documentation

Documentation isn't required for this minor configuration change. Services that legitimately need to prevent system shutdown should use systemd inhibit. Desktop applications can use the XDG inhibit portal.