Updates Lessons

Introduction

There have been various issues with Fedora updates over the years. This page attempts to note such issues, not as a way of placing blame, but as a way of learning from these issues and preventing them from happening again. When a new issue comes up, it should be added to this page. If you know of one not noted here, please do add it.

Currently, this page just lists issues that caused problems for a large number of Fedora users. It's sorted in 'most recent first', so new issues should be added to the top. The date used is when the issue started. Ie, when the update landed in the stable updates repo.

2014-03 - updates pushed despite failing AutoQA tests

In February 2014, after a complaint about an update from Harald Reindl User:Adamwill noticed several cases where [updates had been submitted to stable despite a valid AutoQA test failure https://lists.fedoraproject.org/pipermail/devel/2014-February/196072.html], usually not via a manual push but via karma automatism. In response to this, Adam proposed and [FESCo agreed https://fedorahosted.org/fesco/ticket/1242] that AutoQA failures should case karma automatism to be disabled for updates, to prevent this kind of inadvertent stable promotion.

2012-10-06 - revelation security delay

On 2012-07-31 a security update for revelation got +3 karma for Fedora 17. On 2012-08-09 the maintainer went on a two month vacation without pushing the update first. He properly announced his leave to the fedora devel mailing list and on the wiki. Since nobody feeled responsible to take care of his packages during the leave the update was not pushed to stable until 2012-10-06. Even the regular reports to the fedora testing that include the age of the update did not help to get someones attention to push the security update in a timely manner.

• There needs to be better care for packages when a maintainer is on vacation.

2010-11-11 - proftpd remote code execution security update delay

On 2010-10-29 a new version of proftpd with a fix for a critical security vulnerability (remote code execution) was released. Updates for Fedora 12 to 14 were created between 2010-11-02 and 2010-11-03. The update received no karma points in Bodhi and therefore had to stay 7 days in testing, before the maintainer was allowed to push it to stable. On 2010-11-10 and 2010-11-11 bodhi noted in the updates feedback that the update has been pushed to stable. In conclusion, a highly important security update was only available in stable about two weeks after its upstream release. During this period the vulnerability was already being exploited.

• Critical security updates need to be distributed faster. If there is no manpower to test them shortly, they need to be published untested.

2010-09-09 - firefox/xulrunner/nspr broken dependency

A firefox/xulrunner security update was pushed on 2010-09-09 to stable updates. Unfortunately, there was a buildroot override in place for a new version of the nspr package, which was unpushed due to issues. This left stable updates with a broken dep on a newer nspr. See: https://admin.fedoraproject.org/updates/xulrunner-1.9.1.12-1.fc12,firefox-3.5.12-1.fc12,mozvoikko-1.0-12.fc12,perl-Gtk2-MozEmbed-0.08-6.fc12.15,gnome-web-photo-0.9-9.fc12,gnome-python2-extras-2.25.3-20.fc12,galeon-2.0.7-25.fc12 and https://admin.fedoraproject.org/updates/nss-util-3.12.7-2.fc12,nss-softokn-3.12.7-3.fc12,nss-3.12.7-4.fc12,nspr-4.8.6-1.fc12

This issue has yet to be fixed.

• Buildroot overrides can be a problem since they affect all packages

• The nss packages have a inordinate amount of issues, perhaps we could add manpower?

2010-07-07 - PackageKit not showing updates anymore

On 2010-07-07 https://admin.fedoraproject.org/updates/PackageKit-0.6.6-1.fc13 was pushed to stable updates after getting needed positive testing karma. Unfortunately, it moved an important binary from one location to another, which caused selinux to disallow it from checking for new updates correctly. This basically means all people who updated to this version and don't later do a yum update will be stuck with no updates. A fixed selinux policy was pushed out, but of course needed people to 'yum update' to get it. Additionally, PackageKit updates itself first after install, so this affects any new installs done at the same time. A updated PackageKit was pushed out in https://admin.fedoraproject.org/updates/PackageKit-0.6.6-2.fc13 that requires the fixed selinux policy. See also: https://bugzilla.redhat.com/show_bug.cgi?id=615099 and https://bugzilla.redhat.com/show_bug.cgi?id=620943

• Testing of updates should be done by testers with selinux enabled.

• Testing of updates system should take place over several days to confirm that updates notifications occur correctly, or a test repo with updates should be used.

• Packages shouldn't move binaries in stable releases without checking that selinux policy is affected.

• Updates to the update system should get a lot of scrutiny/testing.

2010-07-02 - celt updates-testing broken dependency

This update [which no longer has a bodhi link] was pushed to updates-testing on 2010-07-02. It was unpushed by the maintainer during/at the push time, but still pushed out. It was unpushed again by bodhi admins, but the f12 package was still tagged in updates-testing. The update was deleted from bodhi. On 2010-07-19 the f12 updates-testing was untagged. This update never reached stable updates.

• Removing updates entirely from bodhi makes it hard to figure out whats going on.

• Detecting updates that are not properly tagged is difficult. Perhaps an additional program that detects tags would be helpfull?

• The f12 one was only untagged after the proper people were notified. Perhaps a better means to notify admins on these problems.

2010-06-24 - Evolution abi breaking update

This update [ https://admin.fedoraproject.org/updates/evolution-mapi-0.30.2-1.fc13,evolution-exchange-2.30.2-1.fc13,evolution-2.30.2-1.fc13,evolution-data-server-2.30.2-2.fc13,gtkhtml3-3.30.2-1.fc13 ] was pushed out on 2010-06-24 (a thursday). It changed the so version in evolution, so all packages that depend on it or evolution-data server needed to be recompiled and pushed out as well. The issue was not fully solved until 2010-06-29 (5 days after it started).

Notes:

• Breakages that occur on thursday or friday are difficult to fix quickly due to no pushes sat/sun.

• ABI breaks in stable releases should be coordinated.

• Negative karma appeared very quickly, but the update was already pushed.

• Requesting stable updates after only 2 hours eariler requesting updates-testing is not a good idea. It didn't even have a chance to be pushed to updates-testing.

• AutoQA should be able to note these issues and block the update once it's implemented.

2010-05-27 - nss-softokn update problems

Due to a bug in mash, this nss-softokn update was pushed to the x86_64 update repos as x86_64-only, where previously nss-softokn had been a multilib package.

The update itself did not cause any problems, since technically this is a valid update - all requested dependencies were satisfied and there are some rare but legitimate circumstances where a package might change from being multi-arch to single-arch.

However, a subsequent update to glibc (which is multilib) was built against - and therefore required - the newer version of nss-softokn. This worked for glibc.x86_64, since the newer nss-softokn.x86_64 was present, but the glibc.i686 update had an unsatisfied dependency since there was no new version of nss-softokn.i686. And so we had broken dependencies in an update set.

Notes:

• This failure cannot be caught by dependency checking alone, since the "broken" package was technically valid until something else required the missing i686 package.

• Under normal circumstances, multilib packages should not drop one of their arches. AutoQA should test for this.

• Under the circumstances where a multilib package actually does intentionally stop being multilib, some special handling is involved. This should be documented somewhere.

2010-02-09 - dnssec-conf

This update [ https://admin.fedoraproject.org/updates/dnssec-conf-1.21-7.fc12 ] caused breakage in bind nameservers. It was solved 2010-02-13 with [ https://admin.fedoraproject.org/updates/dnssec-conf-1.21-8.fc12 ] (4 days after it started). Fedora-announce postings on the issue: http://lists.fedoraproject.org/pipermail/announce/2010-February/002765.html and http://lists.fedoraproject.org/pipermail/announce/2010-February/002768.html

• Update was pushed directly to stable with no testing.

• Modifies config files that users could modify, making updates very fragile.

fall of 2009 - PackageKit permissions too lax

We released F12 with default permissions that were too open for many people's taste, and had to quickly put out an update that fixed things up. Not sure it this entirely falls into 'Update' lessons, but there's a lesson there anyway.

2009-03-09 - NetworkManager unsigned issue

A NetworkManager update with an incorrect key was pushed out in updates. See http://lists.fedoraproject.org/pipermail/announce/2009-March/002620.html for the issue description. It was corrected 2009-03-10 (1 day after it started)

• Check to make sure signatures match the release on pushes.

2009-01-07 - Nautilus unsigned issue

A nautilus update was pushed out that was not signed. See http://lists.fedoraproject.org/pipermail/announce/2009-January/002590.html for more information. The issue was fixed 2009-01-08 (1 day after it started).

• Check to make sure packages are signed at all on push.

2008-02-28 - dbus security update issue

A dbus update was pushed on 2008-12-05 to fix CVE-2008-4311. This update was pushed directly to stable. It caused all dbus based services to be unable to run. See: http://lwn.net/Articles/311146/ and http://lists.fedoraproject.org/pipermail/announce/2008-December/002572.html for more information.

• Security updates pushed direct to stable got no testing.

• No way to back out changes that break PackageKit or other update methods.