Introduction to Mock (20090308 Classroom)

IRC Log

-!- nirik changed the topic of #fedora-classroom to: Fedora IRC Classroom - An introduction to mock, the chroot package builder with your teacher ( nirik ) - See https://fedoraproject.org/wiki/Classroom for todays class schedule and more info.		14:45
@nirik	welcome back everyone.	14:45
@nirik	Our next talk is an Introduction to mock.	14:45
DiscordianUK	Hi	14:45
@nirik	with your teacher... me. ;)	14:45
@nirik	So, I am gonna go over what mock is, why it's needed and how to use it, then some tricks and tips.	14:46
sdodson	Sounds good.	14:46
@nirik	feel free to ask questions at anytime.	14:46
@nirik	ok, so some intro.	14:47
@nirik	mock is a package that makes a chroot and builds other packages in it.	14:47
@nirik	it's available in any of the fedora repos. You can install it with a simple 'yum install mock'	14:47
@nirik	You can also build packages other releases with it. For example, if you have F10 and want to build a rawhide package you can, or if you have f10 and want to build a f9 package.	14:48
@nirik	mock downloads and installs all the needed packages in it's chroot to build your package.	14:48
@nirik	It has a bunch of caching ability, so you only usually suffer a large download the first time you build something. I will talk about the cache later.	14:49
@nirik	So, why do we want to go to all this trouble? why not just build the package locally and be done with it?	14:49
@nirik	The reason is rerproducability. Some packages look for things at compile time. We have to know that when we build something it's reproducable and we can build it the same way again.	14:50
@nirik	so, we have a known set of base packages that everything gets and then after that, packages must specify exactly what they need to build.	14:50
@nirik	The Fedora build system uses mock to build each package that maintainers build.	14:50
@nirik	any questions on what mock is or why we have it?	14:51
@nirik	ok, moving on...	14:51
Ritmo2k	Yeah, wo its the std base set of packages that make it advantagous over a non root build?	14:51
@nirik	ah, excellect question.	14:51
@nirik	for fedora, look at http://fedoraproject.org/wiki/Packaging/Guidelines and the buildrequires section...	14:52
zcat	what was used before mock? custom rpmbuild scripts?	14:53
@nirik	basically it's a group called 'buildsys-build'	14:53
@nirik	so if you look at that group in yum you can see exactly whats in it.	14:53
@nirik	zcat: yeah, I think it was a simpler version in the buildsys....	14:54
DiscordianUK	so what if yu're	14:54
DiscordianUK	so what if what you're building exceeds the buildsys-build specs?	14:54
@nirik	despite mock being made for building packages you can also use it for testing or making populated chroots. I will talk about that a bit later as well.	14:54
@nirik	DiscordianUK: then you need to add to your package spec file a "BuildRequires: whatever you need"	14:55
DiscordianUK	ok	14:55
@nirik	it will then install those things as well and their dependencies.	14:55
DiscordianUK	cool	14:55
@nirik	ok, so moving on to some simple usage. If anyone would like to follow along, you can 'yum install mock' first.	14:55
@nirik	the first thing you want to do is add your user (or the user you want to use mock) to the 'mock' group so it can run the command.	14:56
@nirik	usermod -a -G mock myusername	14:56
@nirik	for example	14:56
@nirik	you may need to logout and back on to get the new group or run 'newgrp mock' after that.	14:56
@nirik	After that you can run mock. The easiest command that most people will use is 'rebuild'	14:57
@nirik	so, you do 'mock rebuild my.src.rpm'	14:57
@nirik	when you issue that command, mock creates a chroot of your current fedora version, installs the buildsys-build group into it, then installs the src.rpm's BuildRequirers packages and all their dependencies into it,	14:58
Ritmo2k	how do specify where the chroot takes place in the event the deps are huge and need lotsa space?	14:58
@nirik	it then calls the rpmbuild in the chroot and rebuilds the package. You should ideally end up with your binary and src.rpm packages at the end.	14:58
@nirik	Ritmo2k: good question. By default mock uses /var/lib/mock/ and cache in /var/cache/mock. You can specify other dirs on the command line.	14:59
@nirik	or you can edit the mock config file in /etc/mock/	14:59
@nirik	/etc/mock/site-defaults.cfg is the config you want to change there.	15:00
@nirik	depending on your package mock could take up a fair bit of space.	15:00
@nirik	mock has some other base command line arguments as well...	15:01
@nirik	init - init a chroot, but don't do anything with it yet.	15:01
@nirik	chroot <cmd> - chroot to the mock chroot you made and run command <cmd>	15:01
@nirik	installdeps / install <pkg> - will install a package or a package and all it's deps into the chroot.	15:02
Ritmo2k	Can you elaborate?	15:02
@nirik	on which item? :)	15:02
Ritmo2k	The first,	15:03
@nirik	using those commands you can use mock to make a chroot that has whatever packages you like in it.	15:03
AndyCap	Hmm, my mock on F10 didn't have an /etc/mock/default.cfg	15:03
@nirik	on init? Basically it does the create the chroot step and installs buildsys-build into it. It doesn't build anything after that or do anything until you run other commands on that chroot	15:04
Ritmo2k	ok	15:04
@nirik	AndyCap: should be site-default.cfg	15:04
@nirik	AndyCap: but I am looking at rawhide, so it might be a difference with the f10 version. ;(	15:04
@nirik	so, when mock is running on say a rebuild of a src.rpm, it makes some log files with all the things it's doing.	15:05
AndyCap	nirik: what I did was symlink a distro-arch.cfg to default.cfg, at least that seemed to be what the manpage meant	15:05
@nirik	yeah, I would have to check a f10 machine. Not sure whats going on there. ;(	15:05
@nirik	any further questions on the base commands? AndyCap: I can get a f10 machine here after the talk and see whats going on for you...	15:06
AndyCap	I'll do fine. don't worry	15:07
@nirik	ok, so lets talk a bit about log files. ;) mock makes logs of it's actions, which are often quite helpfull.	15:07
@nirik	Since the fedora buildsys uses mock, we can take a look at an example there.	15:07
@nirik	lets take a look at: http://kojipkgs.fedoraproject.org/packages/rkhunter/1.3.4/4.fc11/data/logs/noarch/	15:07
@nirik	these are the 3 log files mock makes on a build run. First state.log shows high level commands it's running.	15:08
@nirik	then root.log shows the packages it's installing in the chroot... both the buildsys-build and the deps.	15:08
@nirik	then build.log is the output of rpmbuild on the src.rpm in the chroot.	15:09
* nirik will pause for a sec and let everyone look over logs. Any questions on those?		15:09
Ritmo2k	Wow, so do those build servers grab deps from a cached mirror :) That would amount to alot of dl'ing for every package!	15:10
@nirik	ok, lets move on to talking about mock's caching ability.	15:10
Ritmo2k	cached 'local' mirror	15:11
@nirik	Ritmo2k: they do. They have a caching proxy over a local mirror. ;)	15:11
@nirik	but mock also does some nice caching. ;)	15:11
@nirik	so mock can cache at 3 points (and does by default).	15:11
@nirik	You can see the buildsys-build download there was 113MB. It would be anoying to download that everytime you rebuilt your package.	15:12
@nirik	So, mock keeps a cache of the chroot it last used for your package. It can just unpack that again and apply any updates since it was last used, and use it for the build.	15:12
sdodson	That's per package?	15:13
AndyCap	mock uses a "clean" chroot for each build?	15:13
@nirik	sdodson: I think it's per os-arch. So, if you built a package with f9/i386 it would re-use that cache.	15:13
@nirik	you can of course tell it not to use the cache.	15:14
@nirik	AndyCap: unless it's using a cached one from a previous build.	15:14
@nirik	mock also uses yum to install things, so it can use your yum cache if you have that enabled to pull packages from.	15:14
Ritmo2k	So the chroot is a minimal installation of your target OS, then it adds what ever deps it needs?	15:15
@nirik	finally mock can use ccache for compiled binaries	15:15
@nirik	Ritmo2k: yes.	15:15
AndyCap	nirik: I guess what I'm asking is it unpacking a pristine cache like rpm does with sources or is it using the same directory if I were to build more than one package without asking mock to clean up?	15:15
@nirik	AndyCap: it does unpack a new one. What it does is the first time you compile a given os/arch it will make a root cache file... basically a tar.gz of the root	15:16
@nirik	then it you compile another thing using that same os/arch, it will unpackage that 'clean' root into the new build.	15:16
AndyCap	good, thank you.	15:17
@nirik	so there is always unpacking time/disk io involved.	15:17
@nirik	but if you are testing rebuilds of the same package over and over, the caching should really help you not use too much network BW on downloads.	15:18
@nirik	Here you can see an example of a mockbuild I did the other day...	15:18
@nirik	http://scrye.com/pastebin/196	15:18
@nirik	this shows the typical output from a rebuild...	15:19
@nirik	you can see there that it used a existing cache for the root	15:19
@nirik	it also tells you how long the build took.	15:20
@nirik	any further questions on cache or the output?	15:20
@nirik	ok, moving along to some talk of the config files...	15:21
@nirik	mock has it's config under /etc/mock as I mentioned earier. There are config files for each os/arch...	15:22
@nirik	if you need to modify what repository you are using, or add a local repo for some packages not in fedora, thats where you do that.	15:22
@nirik	or if you need to get more debugging from yum or mock when running it to track down a problem.	15:22
@nirik	in those files you find the yum.conf that mock uses to install with.	15:23
@nirik	also, this is a good time to mention smock... it's a wrapper around mock that rwmjones made to allow you to use local repos easily in your builds.	15:24
@nirik	http://hg.et.redhat.com/cgi-bin/hg-misc.cgi/fedora-mingw--devel/file/tip/smock	15:24
@nirik	any questions on the config files?	15:25
@nirik	AndyCap: yeah, it looks like you should ln default.cfg to whatever os/arch config you want to use by default.	15:26
@nirik	ok, moving along then...	15:26
AndyCap	nirik: ah. read it again and I see you can choose release-arch with -r	15:27
@nirik	AndyCap: yes, you can choose which specific config you want to use with -r	15:27
@nirik	note that you can also use centos repos and build against centos4/centos5 with those configs.	15:27
@nirik	ok, so you can also use mock to generate chroots for testing packages or just whatever reason you might have to use a fedora chroot.	15:28
@nirik	you can use the init and installdeps and chroot for this.	15:29
@nirik	along with the 'shell' command to run a shell in the chroot.	15:30
@nirik	one thing worth noting, if you are running multiple mock commands you will want to use the --uniqueext option, so it makes different chroots for each instance.	15:31
@nirik	if you do that it should work fine to run multiple mock instances at a time.	15:31
* nirik checks his outline to see what else he wanted to cover...		15:32
@nirik	any other general questions.	15:33
@nirik	Another thing to note is that if you have a x86_64 machine, you can use mock to build both i386 and x86_64 packages, simply choose the right config with -r when you run it.	15:33
@nirik	There are some more tips and suggestions at the mock tips page on the fedora wiki: https://fedoraproject.org/wiki/Extras/MockTricks	15:34
@nirik	sorry, thats an old link. You want: https://fedoraproject.org/wiki/Using_Mock_to_test_package_builds	15:35
@nirik	'man mock' has a good deal of info as well.	15:35
@nirik	ok, I think thats about all I was hoping to cover...	15:36
AndyCap	nirik: thank you. it was interesting	15:37
@nirik	no problem... mock is handy for building and testing things for sure. :)	15:38
AndyCap	and killing your laptop. :P	15:38
@nirik	depending on what you are building, yeah. ;)	15:39
baconfork	nirik: Thank you	15:39
sdodson	If you were working in an environment where the buildroot changes a lot, like a koji builder, is squid probably the best caching tool?	15:39
@nirik	sdodson: yeah, squid works fine. I think thats what the fedora buildsys uses to cache packages like that.	15:40
@nirik	all the ones in buildsys-build then are always hot in the cache.	15:40
sdodson	nirik: *nod* that doc you referenced has squid settings in it	15:40
sdodson	We have a koji hub that has horrible connectivity between it and the builders.	15:41
@nirik	yeah, really you can use whatever you like, anything that caches http and works with yum	15:41
@nirik	but squid is pretty quick and easy to setup to do this.	15:41
@nirik	ok, if there are no further questions, I guess we can close up this session. ;)	15:42
@nirik	next up is the wrap up session... which we can start anytime.	15:42
ndim	nirik: Would it be difficult to make mock not just build a chroot, but a KVM/qemu image for different arches?	15:43
@nirik	ndim: humm... it would need to make a image of the chroot and also install a boot loader, etc... but it should be possible.	15:44
-!- nirik changed the topic of #fedora-classroom to: Fedora IRC Classroom - Wrap up session - See https://fedoraproject.org/wiki/Classroom for todays class schedule and more info.		15:44
@nirik	so, I guess we can take a break here, or folks can feel free to chime in now with any feedback. Suggestions for how to better run classes? ideas or comments? all feedback welcome.	15:45
@nirik	I'd like to thank everyone for coming. Please suggest new topics on the wiki, or better sign up to teach something you know!	15:46
* nirik notes its officially the start of the wrap up session time. Feel free to bring up any feedback... or catch herlo or I with any suggestions.		16:01
-!- nirik changed the topic of #fedora-classroom to: Fedora IRC Classroom - Next Classes (tenative) 10:00 UTC 2009-04-04/2009-04-05 - See https://fedoraproject.org/wiki/Classroom more information. End user help /join #fedora		16:12

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!