User:Dmalcolm/StaticAnalysisErgonomics

Proposal: Static Analysis Ergonomics

I want:

• to make it easier to run static analysis tools during the development of code and of patches
• to provide *mechanisms* to allow distributions such as Fedora to set *policy* about what static analysis tools are run during a build, and which results are expected to gate a build. For example, I want a distribution to be able to create e.g. a "network facing daemon" policy file that describes what analysis should be run, and be able to have a one-liner in an rpm .spec file/.deb file that enforces that policy, -Werror, style (with the ability to override it)

I'm looking at adding new functionality to gcc 8 upstream, and this proposal is intended as an RFC to check that I'm on the right track, and to gather ideas.

The problem and the proposed solution

• IMHO static analysis is done too late, if at all: static analysis tools are run as an optional extra, "on the side", rather than in developers' normal workflow. Analysis results are reviewed (if at all) in some kind of review tool, rather than when the code is being edited, or patches being prepared. It's a major PITA to be told via a logfile that tool $FOO reports a problem in bar.c line 1150, 3 hours after I edited that code; better to tell the developer immediately during the initial compile when they're right there in the editor.

• better to have an easy way to run analyzer(s) as development happens, as part of the edit-compile-test cycle - problems are reported immediately, and can be acted on immediately

• hence the proposal is to give gcc a way to integrate invoking 3rd-party static analysis tools, emitting the results through gcc's diagnostic subsystem as if they were plugins

• also the ability to run analyzer(s) when packages are built. Have the ability for distributions to create a variety of pre-canned policies for analyzers (e.g. "network-facing daemons", "suid binaries", etc)

• metadata tagging: all binaries to be tagged with information on what analyzers were run, what options etc (and also, which compiler was run, and what options). Potentially have "dump_file" information from optimization passes stored in the metadata also. Have a tool to query all of this, so that we can do a distro-wide query: "tell me all suid binaries that weren't compiled with such-and-such warnings enabled and at -Werror".

Questions

• Can/should we break the build if there are issues?
  • Yes: but have a way to opt-in easily: if the tool is well-integrated with the compiler: e.g.

       -frun-analyzers=/usr/share/analyzers/userspace/network-facing-service

then upstream developers and packagers can turn on the setting, and see what breaks, and fix it naturally within an compile-edit-test cycle

• Does this slow the build down?
  • Yes: but you can choose which analyzers run, and can choose to turn them off. It ought to parallelize well. I believe users will prefer to turn them on, and have Koji builders burn up the extra CPU cycles.

Users

• upstream developer
• distribution packager (Fedora, Debian)
• Fedora security team (setting policy etc)
  • setting policy
  • after a vulnerability, looking to assess the scope; could we have caught this earlier?

Which analyzers?

 * clang-analyzer
 * cppcheck
 * findbugs
 * cpychecker???
 * anything else?

UI ideas

A possible new option in GCC 8:

 -frun-analyzers=PATH_TO_POLICY_FILE

e.g.:

 -frun-analyzers=/usr/share/analyzers/userspace/network-facing-service
 -frun-analyzers=/usr/share/analyzers/userspace/suid-binary
 -frun-analyzers=/usr/share/analyzers/userspace/application
 -frun-analyzers=/usr/share/analyzers/userspace/default
 -frun-analyzers=/usr/share/analyzers/kernel

or whatnot.

Alternatively, these would simply be gcc "command" files via @.

Idea is to provide mechanism, and for the distribution to decide on some standard policies.

 $ rpm -qf /usr/share/analyzers/userspace/network-facing-service
 analysis-policy-userspace-network-facing-service-0.1-1.noarch.fc27

hence would have:

 BuildRequires: analysis-policy-userspace-network-facing-service

which in turn would Require the analyzers themselves.

Notes

Look at Firehose:

 https://github.com/fedora-static-analysis/firehose

and look at mock-with-analysis:

 https://github.com/fedora-static-analysis/mock-with-analysis

See also:

 https://fedoraproject.org/wiki/Toolchain/Watermark

for watermarking binaries with metadata.

Do we run the risk of breaking "configure" tests?

Code

Proof-of-concept patch kit for gcc: RFC: integrated 3rd-party static analysis support