OLPC/ABIVerifier

ABI Verifier

There are two requirements for ABI verification tools

1. A tool to perform a 'semantic diff' between two builds of the OS image - to verify that we don't break APIs between OS updates 1. A tool to validate that an applicaiton only uses the officially published API & not any internal APIs

Working notes

Some notes from discussions with Mike Hearn from #olpc on freenode.net

(11:51:14) TD: do the abi verifier tools exist yet?
(11:52:43) danpb_ltop: nothing significant has been written yet
(11:53:09) TD: ok
(11:55:25) TD: we need it for autopackage anyway. my time is pretty limited right now but come the summer hopefully i'll have some hours to hack on such a tool
(11:55:31) TD: don't let me stop you writing it first though :)
(12:04:37) danpb_ltop: TD: well there are two bits to it really
(12:05:08) danpb_ltop: first we need a tool to perform a 'semantic diff'  between two builds of the OS image - to verify that we don't break APIs between OS updates
(12:05:27) danpb_ltop: second we need a tool to validate that an applicaiton only  uses the officially published API & not any internal stuff
(12:05:48) danpb_ltop: we've only even considered the first of those tools thus far
(12:07:41) TD: i'm interested in the second
(12:07:59) TD: in autopackage we have big problems with the linux "platform" (i use that term loosely) doing unexpected things behind developers backs
(12:08:11) TD: eg, people think they are targetted GTK 2.2, but actually end up with binaries that depend on GTK 2.8
(12:08:28) TD: right now we special case that stuff. but we've long wanted to define some kind of platform, and have a tool that automatically checks for compliance
(12:09:35) TD: the lsb solves this problem by using magic build wrappers and stub libs, but i think a better way to solve it is just to have a binary linting tool of some form (maybe working in conjunction with the compiler)
(12:10:18) danpb_ltop: yeah, its pretty much impossible to target particularly versions of any library unless you have tools to verify your linkage
(12:12:34) TD: it's fixable but last time i talked to owen about this, he wasn't interested
(12:12:44) danpb_ltop: TD: at a simplest level i think an API checker would consist of two bits
(12:12:54) TD: he felt developers should just build on the oldest system they wanted to target using VMs or something.
(12:13:07) TD: so i made a start on scrubbing the gtk headers myself. but didn't finish
(12:13:08) danpb_ltop: first, we'd run something against the supported libraries in  the OS image to generate a list of exported symbols
(12:13:30) danpb_ltop: and then we'd run a similar tool against the binary to extract the required symbols
(12:13:38) danpb_ltop: and finally just match 'em up
(12:13:46) TD: it's a little more complicated than that :(
(12:13:50) TD: but yes it'd be a good start
(12:13:52) danpb_ltop: (at least for ELF binaries)
(12:14:06) danpb_ltop: scripting languages get much tricker / fuzzier
(12:14:22) TD: yes. scripts are one issue
(12:14:34) TD: dependency scanning through binary analysis isn't possible in the general case
(12:14:42) TD: the best you can do is warn on some obvious mistakes
(12:14:48) danpb_ltop: also we need to verify that an app isn't, for example, directly linking to X11, rather using GTK for all its windowing stuff
(12:14:51) TD: eg, symbol analysis won't catch the case of something like
(12:15:08) TD: if (gtk_version_minor >= 4) gtk_foo(GTK_NEW_FLAG); else gtk_foo(GTK_OLD_FLAG);
(12:15:31) TD: is making xlib a part of the platform really that big a deal?
(12:15:40) TD: lots of interesting programs make lightweight use of X directly
(12:15:40) danpb_ltop: TD: no, i just used that as an example
(12:15:46) TD: ok
(12:16:02) TD: there are some low level changes that can be made to simplify these things (for developers)
(12:16:16) TD: 1) making --as-needed the default for ld in binutils
(12:16:34) TD: 2) merging michael meeks direct linkage patches into glibc/binutils, then activating them by default
(12:16:48) TD: this will simplify things for 3rd party developers _considerably_
(12:16:55) TD: and probably for you guys too
(12:17:05) danpb_ltop: yeah i've been checking up on the state of --as-needed in fedora builds
(12:17:33) danpb_ltop: seems we might need to modify pkg-config and/or libtool  to let us take full advantage of it
(12:18:01) TD: in autopackage we have been using it by default for some time now, and it's not caused any significant problems
(12:18:01) danpb_ltop: TD: have you got a URL / info for  2) ?
(12:18:20) TD: the biggest issue is that libpng upstream is buggy, and somehow doesn't list its dependencies in its own ELF headers
(12:18:27) TD: which is a problem for us, but not for you
(12:18:30) danpb_ltop: TD: yeah the only problems it will cause are where there are actually bugs in the app's makefile
(12:18:36) ***TD nods
(12:18:38) TD: but they are trivially fixed
(12:19:08) danpb_ltop: exactly, so its a one time, fairly mechanical straightforward process to sort it out
(12:21:04) TD: most of the -Bdirect info is buried on michaels blog. it's not well documented unfortunately
(12:21:45) danpb_ltop: wheres his blog ?
(12:22:05) danpb_ltop: ah, no worry, i got it
(12:23:26) TD: http://www.sourceware.org/ml/libc-alpha/2006-01/msg00128.html
(12:24:18) TD: he is merging it into suse 10.1, though i do not know if it's for all binaries
(12:24:22) TD: i suspect just for OO at this point
(12:25:12) TD: yeah
(12:25:16) TD: OO is compiled with this feature
(12:25:37) TD: however gedit isn't
(12:27:56) TD: danpb_ltop: so there are a couple of reasons to use this set of patches
(12:28:23) TD: danpb_ltop: first is that it can improve performance. specifically it gives the same/similar benefits to prelink but without the need to run huge cache/flash-trashing cron jobs all the time
(12:28:44) TD: danpb_ltop: secondly it can massively improve performance of C++ apps in certain situations. OO is a pathological example of that, but others may benefit too
(12:29:09) TD: danpb_ltop: thirdly it makes elf work as you'd expect, which will reduce the amount of unexpected random weird crap going on, and generally make it easier to debug/support and port software
(12:30:25) TD: danpb_ltop: OTOH i don't know of any distros except suse currently using it, as the glibc guys basically ignored it (as they tend to do), and even in suse it's only being used for OO
(12:30:34) TD: so i'd definitely drop michael meeks a line before using it globally
(12:34:49) TD: the good news is solaris has used a variant of this technique globally for years and no problems

---