KernelBugTriage

Quick links to bug lists

If you are looking for bugs to triage, start here:

• Untriaged Fedora 30 Kernel Bugs

• Untriaged Fedora 31 Kernel Bugs

• Untriaged Fedora 32 Kernel Bugs

• Untriaged Rawhide Kernel Bugs

Full Bug Lists:

• Fedora 30 Kernel Bugs

• Fedora 31 Kernel Bugs

• Fedora 32 Kernel Bugs

• Fedora rawhide Kernel Bugs

• Fedora Kernel Security Bugs (all versions)

• Upstream Kernel Bugs

Triage Quickstart

So you've decided to do kernel bug triage. AWESOME! We could use the help. If you're already familiar with bugzilla and the kernel, here is a quickstart set of steps to help us triage the existing bugs. If you aren't familiar with kernel bugs, or need a refresher, the rest of the page has some good background on various aspects of kernel triage and we encourage you to read it before diving in.

Steps (Start at the top, work to the bottom):

• Read over the bug and come up with a basic classification (kernel panic, boot issue, RFE, missing/broken functionality, etc). Add a comment to the bug that provides this classification.
• Determine which kernel subsystem or driver is involved. Prefix the bug subject with the driver/subsytem (e.g. ALSA, USB, ACPI, etc).
  • If it is a bug in the video subsystem, reassign it to xorg-x11-drv-{ati,intel,nouveau}. Although the modesetting drivers live in the kernel, they are maintained by the same people who maintain our userspace graphics drivers. See KernelBugTriage#Video_Subsystem_bugs
  • If it is an i686 specific bug, please add a blocker for 1489998 to bring it to the i686 SIG attention.
  • If it is in one of these categories, please add the matching email address as a "CC" on the bug:

Subsystem/Driver	CC
ACPI	fedora-kernel-acpi@fedoraproject.org
Audit	fedora-kernel-audit@fedoraproject.org
block (Issues with block/* OR drivers/block/loop.c)	fedora-kernel-block@fedoraproject.org
DMAR (Intel IOMMU)	fedora-kernel-dmar@fedoraproject.org
Ethernet (all issues)	fedora-kernel-ethernet@fedoraproject.org
Ethernet (atl1*, alx)	fedora-kernel-ethernet-ath@fedoraproject.org
Ethernet (bnx2, bnx2x, b44, tg3)	fedora-kernel-ethernet-broadcom@fedoraproject.org
Ethernet (r8169)	fedora-kernel-ethernet-realtek@fedoraproject.org
Filesystems (AIO, fs/aio.c)	fedora-kernel-aio@fedoraproject.org
Filesystems (BTRFS)	fedora-kernel-btrfs@fedoraproject.org
Filesystems (CIFS)	nfs-maint@redhat.com
Filesystems (DirectIO, fs/direct-io.c)	fedora-kernel-directio@fedoraproject.org
Filesystems (EXT2/EXT3/EXT4/JBD)	fedora-kernel-extfs@fedoraproject.org
Filesystems (fs/buffer.c)	fedora-kernel-fsbuffer@fedoraproject.org
Filesystems (NFS)	nfs-maint@redhat.com
Filesystems (XFS)	fedora-kernel-xfs@fedoraproject.org
Firewire	fedora-kernel-firewire@fedoraproject.org
Graphics (DRM core, but not Radeon/Intel/Nouveau drivers)	fedora-kernel-drm@fedoraproject.org
HID/Input/Multitouch (but not WACOM)	fedora-kernel-input@fedoraproject.org
intel-pstate (cpufreq driver)	fedora-kernel-intelpstate@fedoraproject.org
kexec/kdump	kdump-team-bugs@redhat.com
KVM virtualization	fedora-kernel-kvm@fedoraproject.org
LVM and device-mapper	lvm-team@redhat.com
Networking (NFC)	fedora-kernel-nfc@fedoraproject.org
Networking (TCP, UDP, SCTP, IP)	fedora-kernel-networking@fedoraproject.org
OpenVSwitch	fedora-kernel-openvswitch@fedoraproject.org
ptrace and utrace	fedora-kernel-ptrace@fedoraproject.org
PCI / PnP resource allocation	fedora-kernel-pci@fedoraproject.org
RAID (drivers/md)	fedora-kernel-raid@fedoraproject.org
SATA and libata	fedora-kernel-ata@fedoraproject.org
SCSI and libsas	fedora-kernel-scsi@fedoraproject.org
SELinux	fedora-kernel-selinux@fedoraproject.org
UEFI	fedora-kernel-uefi@fedoraproject.org
USB Video Cameras	fedora-kernel-usb-cameras@fedoraproject.org
Video4Linux	fedora-kernel-v4l@fedoraproject.org
Xen virtualization	fedora-kernel-xen@fedoraproject.org
Wireless (generic issues, or issues that do not match one of the below drivers)	fedora-kernel-wireless@fedoraproject.org
Wireless (ath*k)	fedora-kernel-wireless-ath@fedoraproject.org
Wireless (Broadcom (b43))	fedora-kernel-wireless-b43@fedoraproject.org
Wireless (brcm80211)	fedora-kernel-wireless-brcm80211@fedoraproject.org
Wireless (Intel (iwlwifi and iwlegacy (iwl3945 and iwl4965)))	fedora-kernel-wireless-iwl@fedoraproject.org
Wireless (Ralink (rt2x00))	fedora-kernel-wireless-ralink@fedoraproject.org
Wireless (Realtek (rtlwifi, r8712u))	fedora-kernel-wireless-realtek@fedoraproject.org

• If multiple categories apply, add them each to the CC list.
• For oopses, add the oopsing function to the end of the bug subject. E.g. IP:hci_send_sco. For things like __list_del_entry, which are common kernel functions, try to find the latest function that was called before that and use that as the suffix.
• Look for duplicate bugs across all Fedora releases. If you find a duplicate, use whichever bug has the most information as the "active" bug and duplicate the other in bugzilla. If you duplicate a bug from a newer release against one in an older release, then change the version on the active bug to the newest version the issue has been seen on.
• If the bug is new, try to determine which kernel version it was first seen with. Note this in the bugzilla whiteboard with "first=X.Y.Z" (e.g. first=3.1.0). Also try to determine which kernel version is the latest tested and still having the issue. Note this in the bugzilla whiteboard with "latest=X.Y.Z" (e.g. latest=3.8.7).
• If the bug is missing information, ask for it and put the bug in NEEDINFO. This can include things like dmesg, modules in use, full panic/oops backtraces, or questions as to what was occurring on the machine at the time. Information should be attached as text/plain attachments, or simple comments in the bug if the information is fairly short. In order to avoid information overload, please ask for only relevant missing information (e.g. we don't need /proc/cpuinfo for wifi bugs, etc).
• If the bug looks fairly straightforward, try and see if upstream knows about it.
  • Look on the various mailing lists and kernel.org bugzilla to see if something similar has been reported upstream (or google search). If so, add a reference to the Fedora bug in the thread or in the kernel.org bug.
  • If it doesn't appear to be reported upstream, use the MAINTAINERS and scripts/get_maintainers.pl files in the kernel tree to narrow down who to report the issue to. Send them an email with all the relevant details from the bug, with kernel-team@fedoraproject.org on CC. Add a link to the mailing list thread archive of the email you sent to the Fedora bug, or use the external references field to pair it with a kernel.org bug.
• Mark the bug as Triaged with the bugzilla Triaged keyword.

Kernel Bug Classification

KernelBugClassification

Kernel Bugzilla Basics

• If there's an upstream (http://bugzilla.kernel.org) bug that matches
  • In the Fedora bug, set 'external bugzilla references' to point to the upstream bug number
  • In the upstream kernel.org bugzilla, add a comment along the lines of "also seen in Fedora. http://bugzilla.redhat.com/show_bug.cgi?id=253096"
• If the bug is against an already released Fedora (Ie, Fedora 7) and there isn't much information to go on "my machine locked up" for eg, request that the user try to reproduce the problem using the kernel-debug kernel. The additional debugging checks may trigger some clues.
• If the bug you are triaging contains a patch, please add [PATCH] to the beginning of the summary line of the bugzilla.
• Sometimes a user will attach a kernel panic as a jpeg, or in worst case, as a tarball of their /var/log/messages.
  • It saves the kernel team time if the kernel oops parts of these are transcribed/cut out and pasted into the bug as text.
• Additionally, making sure that text attachments of bugs have their MIME type set to text/plain can save some time.
• You can ask reporters to try and duplicate with the latest nightly live composes: http://alt.fedoraproject.org/pub/alt/nightly-composes/
• You can ask reporters to try the newest kernel available in koji: http://koji.fedoraproject.org/koji/packageinfo?packageID=8
• If the bug appears to be video/kernel modesetting related, reassign it to the xorg-x11-drv-{intel,ati,nouveau} component. See KernelBugTriage#Video_Subsystem_bugs
• If the reporter is using a custom compiled kernel, close the bug as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=126342

Bug state transitions

• A bug marked as MODIFIED has patches in testing and should not have their status changed. An exception to this is when a bug has been in MODIFIED state for some time (this usually indicates that the issue was fixed in an update, and no-one ever closed the bug. If in doubt, ping the reporter to retest with the latest build).
• If a bug has been in NEEDINFO for several weeks with no follow-up from the reporter, there's not a great deal we can do. In this situation, closing the bug is the only recourse available (which does actually tend to 'wake up' some reporters who then reopen the bug). See KernelBugTriage#Stale_Bugs below.
• When marking bugs as duplicates, we want the one with the most information to be the one that remains open, even if this means a higher numbered bug gets closed. Also, the Fedora release branches are kept on essentially the same kernel version. Therefore, if the same bug is reported in both releases then we close one of them as a duplicate of the other following the aforementioned rule. Duplicating bugs across releases is one way to control the bug count.
• Don't close bugs marked beginning with [mmconf] or [msi] (bugs beginning with [$something] are usually specifically marked so developers can quickly see the main issue within the bug)

NEW state bugs

If the bug is new and (this type of bug must be key-worded in either WHITEBOARD or BLOCKER): Hello,

Thank you for taking the time to report this bug. I have added the proper keyword to this bug to bring it to the attention of the <subsystem-name> subsystem maintainer.

Can you attach the following information to this bug:

Then pick the info that is useful in this case:

• dmesg output from boot before issue.
• dmesg output after issue.
• dmesg output from boot with 'quiet' option replaced with 'debug'
• lspci output
• lsusb output

For other necessary information check:

• X bugs see: http://fedoraproject.org/wiki/How_to_debug_Xorg_problems.
• Other troubleshooting ideas: http://fedoraproject.org/wiki/Common_kernel_problems.

Bug assignment

Assigning/cc'ing people responsible for the relevant subsystem is a good idea.

When reassigning bugs to yourself, add kernel-maint@redhat.com to Cc:

If you are an upstream developer, and would like to be added to the Cc: of relevant bugs, send an email to kernel-team at fedoraproject.org, (We used to maintain this here on the wiki, but for a number of reasons, we now maintain this list privately)

Possibly fixed in CURRENTRELEASE (Stick for prodding sleeping bugs with)

If the bug is against a previous point release kernel, and the reporter hasn't re-tested (this type of bugs must be set in NEEDINFO_REPORTER state):

Thank you for taking the time to report this bug. Updates to the kernel have been released since it was first reported. If you have time to update the package and re-test, please do so and report the results here. You can obtain the updated package by typing in Fedora 22 and later versions 'dnf upgrade kernel' and in Fedora 21 and earlier versions 'yum update kernel'or using the graphical updater, Software Update. If the problem no longer exists then please close this bug or it will be closed in a few weeks if there is no additional information lodged.

Installer bugs

If the bug refers to problems when installing Fedora in any of the possible ways CD, DVD, USB, network, etc (this type of bugs must be set in NEEDINFO_REPORTER state):

Thank you for taking the time to report this bug. Bugs involving the installation process cannot usually be resolved after release date. We therefore ask reporters to use one (or both) of the following options:

Test with an Alpha, Beta or Release Candidate for the next version. These can be found at http://fedoraproject.org/get-prerelease and by testing these we can work to resolve any installation issues so that the final release is free of such bugs.

However if the problem no longer exists then please close this bug or it will be closed in a few weeks if there is no additional information lodged.

get-prerelease link
The get-prerelease link sometimes is not available or have not updated development releases.

Video Subsystem bugs

The various video subsystems that have kernel modesetting support would just prefer their bugs to be moved over to the xorg-x11-drv-{intel|nouveau|ati} component. There, their triagers can gather needed info and debug the issue. If the problem is in the kernel, those maintainers can fix it, no need to leave the bug assigned to the kernel.

This bug is in a video subsystem that has a kernel part. We track and work on these bugs via the driver package name instead of leaving them assigned to the kernel

Rebase Regressions

The kernel is often rebased to the latest stable kernel release and pushed as an update. This fixes a large amount of bugs, but it inevitably introduces a few regressions. If a bug is reported and it is determined that it worked with the previous kernel release, then a rebase regression entry should be added to the whiteboard field in the bug. This entry takes the form of:

rebase-regression-<kernel version>

E.g. if there is a regression when moving to the 3.1 kernel, it should be:

rebase-regression-3.1

The latest working Fedora kernel version and the earliest broken version should be noted in the comment section if possible.

This will allow the kernel developers to track regressions introduced by a rebased kernel. That will also help facilitate our communication with the upstream kernel developers, as we can report these bugs to them to be included in the regression lists that are published upstream.

Bugs with TAINTED modules

Certain bug reports refer to a kernel module that is not open source software (mark this bugs with the keyword TAINTED):

Thank you for taking the time to report this bug. Sadly, your report contains information about a not open-sourced module. This means you have loaded a module into your kernel that we can't debug, making it difficult to isolate the issue. Can you reproduce the issue without loading the tainted <module-name> module?.

If you cannot duplicate the bug without the tainted module loaded it will be closed.

However, you can report it to the vendor of your non-free module.

Stale Bugs

If a bug has been in NEEDINFO with no response from the reporter for 2 weeks, close the bug out with INSUFFICIENT_DATA and the following text:

This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 2 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.

Common Problems

Point reporters to: KernelCommonProblems