SELinux/Debugging

How to debug SELinux issues

This page was created to set up the system to gather additional information to help with debugging issues related to SELinux.

Install packages useful for debugging

$ sudo dnf -y install setools-console selinux-policy-devel policycoreutils-newrole strace initscripts-service bzip2

Enable full auditing

For performance reasons, full auditing is not enabled by default. Instructions how to enable it:

1. Open the /etc/audit/rules.d/audit.rules file in an editor.

2. Remove the following line if it exists:

-a task,never

3. Add the following line to the end of the file:

-w /etc/shadow -p w

4. Restart the audit daemon using the legacy service command, or reboot the system:

$ sudo service auditd restart

5. Run the scenario which effects in SELinux denials.

6. Collect AVC denials:

$ sudo ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts today

This command output needs to be further troubleshooted.

Switch the system to SELinux permissive mode

For testing purposes and to gather as many denials as possible, the permissive mode is useful so that the debugging person is not blocked.

Open the /etc/selinux/config file in an editor, change the SELINUX=enforcing line to

SELINUX=permissive

and reboot the system. After tests finish, switch the system back to enforcing.

For a one-time change to permissive, execute

$ sudo setenforce 0

This setting will be valid till the next reboot, or till setenforce 1 was executed.

Alternatively, one particular SELinux domain can be made permissive (httpd_t in this example), leaving the rest of the system in enforcing mode:

$ sudo semanage permissive -a httpd_t

Use ausearch to gather audited information

Numerous switches can be used with the ausearch command to gather audited data, interpret them, limit the results, etc. These switches have particular importance:

Interpret numeric values:

-i

Display only entries with a particular key:

-k key-value

Look only for data starting at some point - 10 minutes ago, today, reboot:

-ts recent
-ts today
-ts boot

Refer to ausearch(1) for additional information.

Advanced debugging

In many cases it is helpful to track the failing syscalls or the kernel code path through which the denial occurs. For that, use the strace and perf commands, or use the kernel's tracing support to get the kernel (or even userspace) backtraces for SELinux denials. Strace helps with a failing syscall, ltrace with userspace library call, perf in intricate problems, tracefs where kernel is directly involved.

Install additional tools and debugging information for affected packages (systemd in this example).

$ sudo dnf -y install dnf-utils strace ltrace perf
$ debuginfo-install "systemd*"

Use strace to trace process syscalls

The strace tool can be used to trace system calls to isolate a particular syscall which fails.

$ sudo strace -vxy -s 256 -o /tmp/command.strace /path/command -options

To track also SELinux context changes, use the --secontext switch:

$ sudo strace -vxy -s 256 --secontext -o /tmp/command.strace /path/command -options

Refer to strace(1) for additional information.

Use ltrace to trace library calls

The ltrace tool can be used in case it is a userspace library call which fails.

$ sudo ltrace -C -s 256 -o /tmp/command.ltrace /path/command -options

Refer to ltrace(1) for additional information.

Use auditctl to audit a particular syscall

The audit service can be set to audit particular system calls.

1. Read the syscall's man page in the 2nd man section to identify all related syscalls. For example, to create a directory, 2 distinct syscalls can be used: mkdir and mkdirat.

2. Open the /etc/audit/rules.d/syscalls.rules file in an editor.

3. Add the following line to the end of the file (example for mkdir):

-a always,exit -F arch=b64 -S mkdir,mkdirat -F key=mkdir

4. Restart the audit daemon using the legacy service command, or reboot the system:

$ sudo service auditd restart

5. Run the scenario which uses the syscalls.

6. Collect the audited events for further troubleshooting:

$ sudo ausearch -i -k mkdir -ts recent

Refer to auditctl(8) for explanation and additional information.

Using perf to trace denials triggered by a command

Install perf and debugging information for the involved package and its libraries. Execute a command through perf:

$ sudo perf record -o perf.data -a -g --call-graph dwarf -e avc:selinux_audited -- /path/command -options

The perf.data file with information about audited denials caused by the command is stored.

Using perf to trace all system denials

Install perf and debugging information for the involved package and its libraries. Execute perf and terminate it when the scenario is finished:

$ sudo perf record -o perf.data -a -g --call-graph dwarf -e avc:selinux_audited
^C

The perf.data file with information about all denials audited during the perf command run is stored.

Interpret data gathered by perf

Run perf script to read stored data and display stack trace:

$ sudo perf script -i perf.data

Run perf report to display call chains:

$ sudo perf report -g

Refer to perf-record(1) for additional information.

Using tracefs

1. Run the following commands as root:

# echo stacktrace >/sys/kernel/tracing/trace_options
# echo 1 >/sys/kernel/tracing/events/avc/selinux_audited/enable

2. Run the scenario which triggers the SELinux denials.

3. Dump the backtraces of captured AVC events to report them:

# cat /sys/kernel/tracing/trace
[...]

4. (Optional) Reset the tracing settings by running the following commands (or just rebooting the machine):

# echo nostacktrace >/sys/kernel/tracing/trace_options
# echo 0 >/sys/kernel/tracing/events/avc/selinux_audited/enable