Updates Lessons

Introduction

There have been various issues with Fedora updates over the years. This page attempts to note such issues, not as a way of placing blame, but as a way of learning from these issues and preventing them from happening again. When a new issue comes up, it should be added to this page. If you know of one not noted here, please do add it.

Currently, this page just lists issues that caused problems for a large number of Fedora users. It's sorted in 'most recent first', so new issues should be added to the top. The date used is when the issue started. Ie, when the update landed in the stable updates repo.

2010-09-09 - firefox/xulrunner/nspr broken dependency

A firefox/xulrunner security update was pushed on 2010-09-09 to stable updates. Unfortunately, there was a buildroot override in place for a new version of the nspr package, which was unpushed due to issues. This left stable updates with a broken dep on a newer nspr. See: https://admin.fedoraproject.org/updates/xulrunner-1.9.1.12-1.fc12,firefox-3.5.12-1.fc12,mozvoikko-1.0-12.fc12,perl-Gtk2-MozEmbed-0.08-6.fc12.15,gnome-web-photo-0.9-9.fc12,gnome-python2-extras-2.25.3-20.fc12,galeon-2.0.7-25.fc12 and https://admin.fedoraproject.org/updates/nss-util-3.12.7-2.fc12,nss-softokn-3.12.7-3.fc12,nss-3.12.7-4.fc12,nspr-4.8.6-1.fc12

This issue has yet to be fixed.

• Buildroot overrides can be a problem since they affect all packages

• The nss packages have a inordinate amount of issues, perhaps we could add manpower?

2010-07-02 - celt updates-testing broken dependency

This update [which no longer has a bodhi link] was pushed to updates-testing on 2010-07-02. It was unpushed by the maintainer during/at the push time, but still pushed out. It was unpushed again by bodhi admins, but the f12 package was still tagged in updates-testing. The update was deleted from bodhi. On 2010-07-19 the f12 updates-testing was untagged. This update never reached stable updates.

• Removing updates entirely from bodhi makes it hard to figure out whats going on.

• Detecting updates that are not properly tagged is difficult. Perhaps an additional program that detects tags would be helpfull?

• The f12 one was only untagged after the proper people were notified. Perhaps a better means to notify admins on these problems.

2010-06-24 - Evolution abi breaking update

This update [ https://admin.fedoraproject.org/updates/evolution-mapi-0.30.2-1.fc13,evolution-exchange-2.30.2-1.fc13,evolution-2.30.2-1.fc13,evolution-data-server-2.30.2-2.fc13,gtkhtml3-3.30.2-1.fc13 ] was pushed out on 2010-06-24 (a thursday). It changed the so version in evolution, so all packages that depend on it or evolution-data server needed to be recompiled and pushed out as well. The issue was not fully solved until 2010-06-29 (5 days after it started).

Notes:

• Breakages that occur on thursday or friday are difficult to fix quickly due to no pushes sat/sun.

• ABI breaks in stable releases should be coordinated.

• Negative karma appeared very quickly, but the update was already pushed.

• Requesting stable updates after only 2 hours eariler requesting updates-testing is not a good idea. It didn't even have a chance to be pushed to updates-testing.

• AutoQA should be able to note these issues and block the update once it's implemented.

2010-02-09 - dnssec-conf

This update [ https://admin.fedoraproject.org/updates/dnssec-conf-1.21-7.fc12 ] caused breakage in bind nameservers. It was solved 2010-02-13 with [ https://admin.fedoraproject.org/updates/dnssec-conf-1.21-8.fc12 ] (4 days after it started). Fedora-announce postings on the issue: http://lists.fedoraproject.org/pipermail/announce/2010-February/002765.html and http://lists.fedoraproject.org/pipermail/announce/2010-February/002768.html

• Update was pushed directly to stable with no testing.

• Modifies config files that users could modify, making updates very fragile.

2009-03-09 - NetworkManager unsigned issue

A NetworkManager update with an incorrect key was pushed out in updates. See http://lists.fedoraproject.org/pipermail/announce/2009-March/002620.html for the issue description. It was corrected 2009-03-10 (1 day after it started)

• Check to make sure signatures match the release on pushes.

2009-01-07 - Nautilus unsigned issue

A nautilus update was pushed out that was not signed. See http://lists.fedoraproject.org/pipermail/announce/2009-January/002590.html for more information. The issue was fixed 2009-01-08 (1 day after it started).

• Check to make sure packages are signed at all on push.

2008-02-28 - dbus security update issue

A dbus update was pushed on 2008-12-05 to fix CVE-2008-4311. This update was pushed directly to stable. It caused all dbus based services to be unable to run. See: http://lwn.net/Articles/311146/ and http://lists.fedoraproject.org/pipermail/announce/2008-December/002572.html for more information.

• Security updates pushed direct to stable got no testing.

• No way to back out changes that break PackageKit or other update methods.

fall of 2009 - PackageKit permissions too lax

We released F12 with default permissions that were too open for many people's taste, and had to quickly put out an update that fixed things up. Not sure it this entirely falls into 'Update' lessions, but there's a lession there anyway.