From Fedora Project Wiki
(Redirected from Classroom/Package Taxonomy and Techniques)
Fedora Classroom - Package Taxonomy and Techniques - Ignacio Vazquez-Abrams - Saturday, November 8, 2008
IRC Log of the Class
| -!- nirik changed the topic of #fedora-classroom to: Fedora Classroom - Package Taxonomy and Techniques with your teacher: ivazquez - See Communicate/IRC/Classroom for more info | 23:00 | |
| Ineluctable | thanks stickster | 23:00 |
|---|---|---|
| stickster | neverho0d: It is, but you could work with another person to edit your summaries | 23:00 |
| * kdn *bell rings* | 23:00 | |
| brunowolff | If it isn't, then translating it could be another area to help out. | 23:00 |
| neverho0d | stickster: thank you | 23:01 |
| stickster | brunowolff: +1, exactly | 23:01 |
| * stickster really scoots now | 23:01 | |
| ivazquez | Hello everyone, and thank you for coming. | 23:01 |
| mattia | hi ivazquez | 23:01 |
| Abd4llA | Helloo Mr ivazquez :P | 23:01 |
| neverho0d | brunowolff: I'll be glad to contribute as much as I can | 23:01 |
| thomasj | helloiva | 23:01 |
| thomasj | ops | 23:01 |
| ivazquez | You see them go by all the time in anaconda and PackageKit. | 23:01 |
| kiakli | hi | 23:01 |
| cga | ciao ivazquez, i'm here too ;) | 23:02 |
| thomasj | hello ivazquez | 23:02 |
| ivazquez | libfoo-3.2-5.fc9... barprogs-5.2.6-0.cvs20081016... | 23:02 |
| ivazquez | But just what is it that makes a package... | 23:02 |
| ivazquez | ... a package? | 23:02 |
| domg472 | rpmbuil | 23:02 |
| domg472 | nvm | 23:02 |
| brunowolff | A spec file. | 23:03 |
| ivazquez | At its simplest, a package is a set of files, and metadata about each file as well as about the entire bundle. | 23:03 |
| kdn | clean installs, clean uninstalls, clean upgrades? | 23:03 |
| ivazquez | Things like permissions, ownership, scripts executed during installation, etc. | 23:03 |
| ivazquez | So let's spend some time looking at these things. | 23:04 |
| ivazquez | The most important thing you need to know about a package is its NEVRA. | 23:04 |
| ivazquez | This stands for Name, Epoch, Version, Release, and Architecture. | 23:04 |
| ivazquez | It's what determines when a package can be installed or upgraded. | 23:04 |
| ivazquez | The name is the part we're most familiar with. | 23:05 |
| ivazquez | glibc, firefox, etc. | 23:05 |
| ivazquez | It is the smallest part of a package we can use and still talk about the package itself. | 23:05 |
| ivazquez | The epoch, version, and release determine the "order" of packages. | 23:06 |
| ivazquez | That is to say, which package is considered "newer" than others. | 23:06 |
| ivazquez | Normally we don't need to worry about the epoch, so we'll hold out on that one for just a moment. | 23:07 |
| ivazquez | Version is mostly self-explanatory. | 23:07 |
| ivazquez | 2.8, 3.0.2, etc. | 23:07 |
| ivazquez | When upstream releases software, they give it a version to keep it separate from all the other versions they release. | 23:07 |
| ivazquez | The first part of a package that we really get to handle as a distro is the release. | 23:08 |
| EvilBob | how should a packager deal with packaging a beta or prerelease version from upstream? | 23:08 |
| ivazquez | That is actually covered in Fedora's packaging guidelines. | 23:09 |
| ivazquez | I'll be happy to discuss packaging issues after. | 23:09 |
| @nirik | http://fedoraproject.org/wiki/Packaging/NamingGuidelines#Pre-Release_packages | 23:09 |
| ivazquez | The release is used to distinguish one package of a specific version from another. | 23:09 |
| ivazquez | So you package libfoo 3.2, and the first package gets a release of "1". | 23:10 |
| ivazquez | Then let's say someone notices a problem in your package, and you fix it. | 23:10 |
| ivazquez | When you go to fix it, you keep the version the same (since it's still the same code from upstream), but you bump the release up to "2". | 23:11 |
| ivazquez | That way people and tools realize that libfoo-3.2-2 is an update for libfoo-3.2-1. | 23:11 |
| ivazquez | And when upstream releases a new version, say, libfoo 3.3, you can drop the release back down to "1". | 23:12 |
| ivazquez | libfoo-3.3-1 is seen as an update to libfoo-3.2-2 since the version is higher. | 23:12 |
| brunowolff | The kernel seems to keep an increase release number even after minor releases? | 23:13 |
| ivazquez | It used to. They fixed that. | 23:13 |
| ivazquez | The exact details involve all sorts of ugliness regarding the "base" commit. | 23:13 |
| ivazquez | We can come back to that later. | 23:14 |
| brunowolff | When 2.27.5 was released it had -88 for the release number? | 23:14 |
| ivazquez | Now, epoch. | 23:14 |
| brunowolff | OK | 23:14 |
| ivazquez | The epoch is the rarest-seen and the most troublesome part of a package. | 23:14 |
| ivazquez | By default it's not shown when you use rpm to query a package. | 23:15 |
| ivazquez | (and we'll come back to querying after) | 23:15 |
| zcat | if you have rpmdevtools installed you can play with the version/epoch comparisons. rpmdev-vercmp libfoo-3.2-2 libfoo-3.2-1 | 23:15 |
| ivazquez | Additionally, when writing a spec file, you must include this invisible epoch in listing packages that you are dependent upon. | 23:16 |
| ivazquez | Many a packager has been bitten by omitting the epoch in this scenario. | 23:16 |
| ivazquez | Now, why would anyone use the epoch, given all this trouble? | 23:16 |
| ivazquez | The simple answer is that the version has had issues. | 23:17 |
| ivazquez | For instance, going from, say, "3.2preview" to "3.2final" will be a problem, since 3.2preview actually looks newer than 3.2final. | 23:18 |
| cga | how comes? | 23:19 |
| ivazquez | So what must be done is the epoch must be bumped to 1 (it is 0 by default), and everyone involved lets out a collective groan because of the issues mentioned earlier. | 23:19 |
| thomasj | cga, p - f | 23:19 |
| Sid | because 2.3preview comes later in an alphabetical listing than 2.3final | 23:19 |
| ivazquez | Strings in a VR are compared lexicographically. | 23:19 |
| Sid | 3.2 even | 23:19 |
| cga | i see thanks | 23:19 |
| EvilBob | cga: alphanumeric ordering | 23:19 |
| cga | fair enough | 23:19 |
| ivazquez | This brings us to the fifth part, the architecture. | 23:20 |
| ivazquez | You would think that this is a very small topic. | 23:20 |
| ivazquez | After all, i386 goes on i386 machines, x86_64 on x86_64, and so on. | 23:20 |
| ivazquez | Unfortunately this is not true. | 23:21 |
| ivazquez | And this is for 2 reasons. | 23:21 |
| ivazquez | The first is that there are architectures that are "equivalent" to each other. | 23:21 |
| ivazquez | E.g., i386, i486, i586, i686, etc. | 23:22 |
| ivazquez | You can only have a single package (with exceptions we'll cover later) with the same NA installed at a time. | 23:22 |
| ivazquez | So you can't have libfoo.i386 installed at the same time as libfoo.i686. | 23:23 |
| ivazquez | The second is a little feature called "multilib". | 23:23 |
| ivazquez | This allows packages from different equivalent architectures to be installed at the same time. | 23:24 |
| ivazquez | It's what lets you install glibc.x86_64 and glibc.i686 on the same system. | 23:24 |
| ivazquez | However, it also has its issues. | 23:25 |
| Discordian | Doesn't that increase the memory residency? | 23:25 |
| ivazquez | One of the issues is when the x86_64 package and the ix86 (i386, i686, etc.) package contain the same files. | 23:26 |
| ivazquez | Sorry, memory residency? | 23:26 |
| Discordian | Don't shared libs take up memory? | 23:26 |
| Discordian | Sorry ignore that please don't want to distract you | 23:27 |
| ivazquez | Only when they're loaded. Packages on the disk take up no memory until they're requested. | 23:27 |
| ivazquez | Alright, same files. | 23:27 |
| cga | Discordian: see differences between application and process (ie. installed versus running) | 23:28 |
| ivazquez | rpm does not have a problem with the x86_64 package and the ix86 package containing the same files, provided that the files are *exactly* the same. | 23:28 |
| ivazquez | Timestamp, contents, permissions, etc. | 23:28 |
| Discordian | cga: thank you | 23:28 |
| ivazquez | But even this is not true. | 23:28 |
| ivazquez | And this is where we get into trouble. | 23:28 |
| ivazquez | If the x86_64 package and the ix86 package are installed in the same "transaction", rpm is perfectly willing to overlook differences in the files and will install both packages, with the x86_64 files overwriting the ix86 files. | 23:29 |
| ivazquez | HOWEVER. | 23:29 |
| ivazquez | If either one of the packages is already installed, and we try to install the other package, rpm will complain about file conflicts since they don't match. | 23:30 |
| ivazquez | This sort of issue has spawned many more bug reports than are deserved. | 23:31 |
| ivazquez | So, something to keep in mind when you install or build packages. | 23:31 |
| ivazquez | So, now that we understand (I hope) these 5 elements, let's look at what else a package's metadata provides. | 23:32 |
| @nirik | I have a question: where does rpm store the ix86 binaries? If the x86_64 package is removed it would have to place the ix86 ones back right? | 23:33 |
| ivazquez | It does not. It simply doesn't erase the files provided by the x86_64 package. | 23:33 |
| ivazquez | Yay multilib. | 23:33 |
| @nirik | yikes. ;( | 23:34 |
| Discordian | Does that mean you could rebuild both i386 and x64 rpms using rpmrebuild and they'd both be valid? | 23:34 |
| ivazquez | I don't know. I've never looked at how rpmrebuild works. | 23:35 |
| Discordian | My impression is it just uses the rpm db to reconstruct the rpm file | 23:35 |
| ivazquez | I'm glad you bring up the rpmdb. | 23:36 |
| ivazquez | A package in a file has a set of metadata. | 23:36 |
| ivazquez | When said package is installed, the metadata needs to be stored in order to retain information about the package. | 23:36 |
| ivazquez | The metadata is stored in what is called the rpmdb. | 23:37 |
| ivazquez | We can query the rpmdb via "rpm -q". | 23:37 |
| ivazquez | We pass it a name, and rpm will give us the metadata in a pre-programmed format. | 23:38 |
| ivazquez | Normally it looks like <name>-<version>-<release>. | 23:38 |
| ivazquez | I'm not sure if they've added .<arch> to that recently. I believe so though. | 23:38 |
| ivazquez | You can see an example by opening up a terminal and typing "rpm -q filesystem". | 23:39 |
| Discordian | That is the case on F9 yes | 23:39 |
| ivazquez | If someone could provide the output from theirs please? | 23:39 |
| Sid | filesystem-2.4.13-1.fc9.i386 | 23:40 |
| ivazquez | Right, there we are. | 23:40 |
| ivazquez | The name is, of course, "filesystem". | 23:40 |
| ivazquez | The version is "2.4.13", the release is "1.fc9", and the arch is "i386". | 23:41 |
| EvilBob | [root@mediapc1 ~]# rpm -q filesystem | 23:41 |
| EvilBob | filesystem-2.4.13-1.fc9.i386 | 23:41 |
| EvilBob | grrr | 23:41 |
| ivazquez | Sometimes we need more or less than is provided by the default format. | 23:41 |
| zcat | ah. so showing the arch is default now? i've been adding "%_query_all_fmt %%{name}-%%{version}-%%{release}.%%{arch}" to ~/.rpmmacros so i didn't notice | 23:41 |
| EvilBob | filesystem-2.4.11-1.fc8 | 23:41 |
| Discordian | filesystem-2.4.13-1.fc9.i386 is what I have | 23:41 |
| ivazquez | Right, and we're get to query formats right now. | 23:42 |
| ivazquez | *getting | 23:42 |
| ivazquez | Let's say that you have a script that verifies the version of certain packages on your system. | 23:42 |
| ivazquez | You want to narrow down your focus to only the version. | 23:42 |
| ivazquez | Normally you'd do post-processing of the output of "rpm -q". | 23:43 |
| ivazquez | But the default output can be changed, as demonstrated by zcat just above. | 23:43 |
| Discordian | you mean using awk or perl? | 23:43 |
| ivazquez | Correct. | 23:44 |
| ivazquez | What you can do is you can pass rpm a parameter that tells it the format you want. | 23:44 |
| Discordian | cool I can see that would be useful | 23:44 |
| ivazquez | This argument is "--qf", and it takes the format in a special, err, format. | 23:44 |
| ivazquez | Let's deal with just the version for now. | 23:45 |
| ivazquez | To get the version of filesystem we run "rpm -q --qf '%{version}'". | 23:45 |
| ivazquez | This will return just the value, with no fancy formatting. | 23:45 |
| ivazquez | Naturally if the output will be consumed by a person we'll at least want to put it on its own line. | 23:46 |
| ivazquez | The query format string takes several C-style escapes, including \n for a newline. | 23:46 |
| nuonguy | you mean rpm -q filesystem --qf '%{version}' right? | 23:47 |
| ivazquez | "rpm -q --qf '%{version}\n'" will give us something that we can deal with easier. | 23:47 |
| ivazquez | Yes. | 23:47 |
| neverho0d | rpm -q --qf '%{version}\n' filesystem | 23:47 |
| ivazquez | Right. | 23:47 |
| ivazquez | There are a large number of these "query tags" built into rpm. | 23:48 |
| ivazquez | You can see a full list by running "rpm --querytags | less". | 23:48 |
| ivazquez | The query tags in the query format are not case-sensitive. | 23:48 |
| neverho0d | wow 163! | 23:48 |
| ivazquez | Most of them you'll never need though. | 23:49 |
| daMaestro | and shortcuts for those that don't want to type ;-) | 23:49 |
| ivazquez | Among the important ones are the 5 parts that we started with here: | 23:49 |
| ivazquez | name, epoch, version, release, arch. | 23:50 |
| ivazquez | Also important is sourcerpm, which gives you the name of the component in Bugzilla to file bugs against. | 23:50 |
| ivazquez | installtime is also useful. | 23:51 |
| ivazquez | But there are 2 things to know about it. | 23:51 |
| ivazquez | 1) It only makes sense in the rpmdb. Querying it from a package in a file makes no sense. | 23:51 |
| ivazquez | And we'll get to querying a package in a file in a moment. | 23:51 |
| ivazquez | 2) It gives its result as a *nix timestamp. Not exactly human-friendly. | 23:52 |
| ivazquez | So "rpm -q --qf '%{name}: %{installtime}\n' filesystem", while technically correct, isn't all that useful. | 23:53 |
| ivazquez | Fortunately query tags support a suffix that you can add to modify them. | 23:53 |
| ivazquez | The suffix ":date" will convert a timestamp into a human-readable date and time. | 23:54 |
| ivazquez | So with a small change we get "rpm -q --qf '%{name}: %{installtime:date}\n' filesystem", and we can see more clearly when our system was installed or upgraded. | 23:54 |
| ivazquez | Another query tag of interest is dsaheader. | 23:56 |
| ivazquez | This tag tells us how, when, and who signed a package. | 23:56 |
| ivazquez | We'll get into signing packages a bit later. | 23:56 |
| Discordian | Cool, this is good stuff | 23:57 |
| cga | night all, this is interesting but i'm falling asleep. i'll read the logs tomorrow. i find the classroom idea very interesting. kudos. | 23:57 |
| ivazquez | Take care. | 23:57 |
| ivazquez | Again, it spews a long hex string that will make our brains asplode if we try to consume it. | 23:57 |
| * nirik didn't know the :date stuff. very cool. | 23:57 | |
| ivazquez | Suffixing it with ":pgpsig" processes it and gives us a result we can use. | 23:58 |
| daMaestro | yes, i just wasted a few moments trying to pipe the unix timestamp to data -d via xargs | 23:58 |
| daMaestro | thanks. | 23:58 |
| kdn | Thanks to all. This was a Good Thing(tm). | 23:58 |
| * delhage agrees | 23:58 | |
| ivazquez | rpm -q --qf '%{name}: %{dsaheader:pgpsig}\n' filesystem | 23:58 |
| daMaestro | it's not over | 23:58 |
| Discordian | I'm not going | 23:58 |
| ivazquez | I'll leave it as an exercise for the reader as to whose key that is ;) | 23:58 |
| Discordian | heh | 23:59 |
| brunowolff | How well do you have to know the code to package something from upstream? | 23:59 |
| * nirik notes that this is the last class scheduled today... more classess tomorrow. | 23:59 | |
| ivazquez | Almost not at all. | 23:59 |
| domg472 | why --qf and not -qf, would that conflict? | 23:59 |
| brunowolff | So a good working relationship with upstream would be good enough? | 00:00 |
| ivazquez | -qf is seen as a combination of 2 flags, -q and -f. | 00:00 |
| domg472 | i see | 00:00 |
| Discordian | -qf is which package does this file come from | 00:00 |
| ivazquez | brunowolff: Not even. There are a number of instances where upstream is not even aware of it. | 00:00 |
| ivazquez | Okay, packages on disk. | 00:00 |
| Sid | is it possible to query packages that aren't installed? | 00:00 |
| ivazquez | Yes it is. | 00:01 |
| ivazquez | There isn't much to say about packages on disk, other than you need to add -p to rpm. | 00:01 |
| ivazquez | And of course, some tags such as installtime don't make sense. | 00:01 |
| brunowolff | Is there a good place to get info on how to package java apps? I have one I build from ant locally, but I don't know where fedora wants | 00:01 |
| ivazquez | After all, a package on disk isn't considered to be installed. | 00:02 |
| brunowolff | java apps placed or if there are any particular build practices to use. | 00:02 |
| neverho0d | ivazquez: it would be helpful to explain a package breaking on docs, devel and other things.... | 00:02 |
| Sid | ok, what if the package isn't on disk but simply in the yum repo, can I then query it? | 00:02 |
| ivazquez | brunowolff: Fedora has an extensive set of packaging guidelines in the wiki. | 00:02 |
| Sid | or do I at least need to download the .rpm? | 00:02 |
| ivazquez | Another excellent question. | 00:02 |
| ivazquez | neverho0d: We'll handle that in a bit. | 00:02 |
| ivazquez | Packages in a repo. | 00:03 |
| neverho0d | ivazquez: ok. thnks | 00:03 |
| ivazquez | yum-utils has a script, repoquery, which can be used to query packages in a repo. | 00:03 |
| @nirik | brunowolff: Packaging/Java | 00:03 |
| ivazquez | It takes a subset of the query tags that rpm does, due to the fact that the repo metadata has only a small fraction of the package metadata. | 00:04 |
| ivazquez | Other than that it takes -q and --qf just like rpm. | 00:04 |
| ivazquez | Alright, I think I've chewed up enough time on my bit. Which topics did I say I'd defer? | 00:05 |
| brunowolff | nirik: Thanks that looks pretty readable. I'll see if I can make it work locally. There are other issues with packaging the app (Colossus) that I am interested in. | 00:05 |
| * nirik is sad that it doesn't provide sourcerpm. ;( | 00:05 | |
| domg472 | thanks | 00:05 |
| domg472 | very informative day | 00:05 |
| ivazquez | Package breakup. | 00:06 |
| Discordian | Thank you very much | 00:06 |
| erinlea80 | Thanks ivazquez :) Its been informative. | 00:06 |
| Discordian | I learned a lot | 00:06 |
| ivazquez | Not quite done yet, but I understand if you have other things to do. | 00:06 |
| mattia | thanks ivazquez | 00:06 |
| Discordian | No no I'll stay | 00:06 |
| Bugz | thanks ivazquez | 00:06 |
| ivazquez | Part of the metadata in a package is what "type" a file is. | 00:06 |
| ivazquez | That is to say, whether it's a normal file, a configuration file, or documentation. | 00:07 |
| Discordian | or a script? | 00:08 |
| ivazquez | rpm allows you to query only these specific files by passing -c or -d, but most of the query tags are at the package level and so don't apply. | 00:08 |
| ivazquez | Package scripts are actually stored separately. | 00:08 |
| ivazquez | They're in the metadata, not the files. | 00:08 |
| Discordian | Okay I've wondered about that | 00:08 |
| ivazquez | They're viewable as either any of the *prog query tags, or by passing --scripts when querying a package. | 00:09 |
| domg472 | rpm -q --scripts | 00:09 |
| Discordian | Thank you | 00:09 |
| ivazquez | No, I lie. Not the *prog tags. | 00:09 |
| ivazquez | Those are what is used to actually run the scripts. | 00:10 |
| ivazquez | The scripts themselves are in the prein, postin, preun, postun, and other related tags. | 00:10 |
| ivazquez | Okay, I can't seem to find what other audience topics I decided to defer. | 00:11 |
| ivazquez | Does anyone have any questions?] | 00:11 |
| domg472 | it was all clear except the x86 x86_64 bit but that must be my fault | 00:12 |
| nuonguy | is there a virtual file system like /selinux or /proc to browse the rpm database? | 00:12 |
| domg472 | ill digginto that myself | 00:12 |
| Discordian | since this is Fedora : what impact will the new version of rpm have? | 00:12 |
| nuonguy | domg472: +1 | 00:12 |
| Discordian | or is that unfair? | 00:12 |
| ivazquez | No. The rpmdb is stored as a set of bdb or sqlite databases in /var/lib/rpm. | 00:12 |
| erinlea80 | My questions were answered up above. | 00:13 |
| ivazquez | Which one, rpm5? | 00:13 |
| * nirik muses that someone could make a fuse-rpmdb someday... | 00:13 | |
| Discordian | Yeah | 00:13 |
| domg472 | by the way nuonguy id like to hear what is your reason to contribute to fedora or oss in general? | 00:13 |
| ivazquez | rpm5 is an independent project. I cannot speak for Red Hat, but it has little or no impact in Fedora as far as I can see. | 00:14 |
| ivazquez | the rpm maintainers in RH could probably give you a more complete answer on that one. | 00:14 |
| Discordian | Okay I noted that F10 had flagged up a new version of rpm | 00:15 |
| Discordian | I think perhaps it's 4.x | 00:15 |
| ivazquez | That would be further development in the 4.x branch. | 00:15 |
| Discordian | Yes | 00:15 |
| @nirik | yes, 4.6.0 | 00:15 |
| sdodson | F10 will use 4.6. | 00:15 |
| sdodson | Though, I believe it's been requested that newer features of 4.6 not actually be used until a later date. | 00:15 |
| Discordian | Ahhh cool | 00:16 |
| ivazquez | Anyone else? | 00:16 |
| @nirik | thanks ivazquez. Great session! | 00:16 |
| ivazquez | In that case, thank you for coming, and thank you for listening. | 00:16 |
| Discordian | yes thank you very much ivazquez | 00:17 |
| nuonguy | domg472: when I interview candidates, I ask them if they contribute to any open source projects | 00:17 |
| * erinlea80 applauds | 00:17 | |
| neverho0d | ivazquez: may be one another class om spec-file building? ;) | 00:17 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!