From Fedora Project Wiki
Fedora Classroom - Configuration Management using Puppet - Jeroen van Meeuwen - Sunday, November 9, 2008
IRC Log of the Class
Nov 09 18:04:12 <kanarip> so this session is about configuration management, which does *not* include the soft side of configuration management for a change Nov 09 18:04:12 * Bugz_ is Nov 09 18:04:12 <kanarip> we're *not* going to talk about procedural issues, but keep it on the technical side Nov 09 18:04:12 <kanarip> i'd like to invite you all to take a look at http://www.kanarip.com/courses/puppet/puppet.pdf Nov 09 18:04:12 <kanarip> which is a nice reader i wrote for a one-day workshop i'm giving on this topic, and that is commercially available Nov 09 18:04:12 <kanarip> community people like us however *do not have to pay* Nov 09 18:04:12 <kanarip> sounds great, huh? ;-) Nov 09 18:04:12 <domg472_> yes Nov 09 18:04:12 <erinlea80> :) Nov 09 18:04:12 <delhage> from Delft eh? Nov 09 18:04:12 <kanarip> so, a little something on configuration management first, then a sneak peak at the installation of puppet, and a quick howto on writing your "manifest" as they call the puppet configuration syntax Nov 09 18:04:12 <kanarip> delhage, Utrecht Nov 09 18:04:12 <delhage> ok Nov 09 18:04:12 <kanarip> so, what is it we would want to do with configuration management in the technical sense of the words Nov 09 18:04:12 <kanarip> we all know that the more systems we need to maintain, the harder it becomes to keep some extend of consistency between different systems Nov 09 18:05:14 <kanarip> some organizations use scripts from an NFS share, or use an SVN tree with scripts that is periodically updated and execute scripts Nov 09 18:05:14 <kanarip> does that sound familiar to anyone? Nov 09 18:05:14 * erinlea80 nods Nov 09 18:05:14 <kanarip> so what happens in these cases is: Nov 09 18:05:14 <kanarip> 1) your scripts tend to get very large Nov 09 18:05:50 <kanarip> describing every unique situation, every exemption, every update, for every operating system, distribution, distribution version, makes the script less maintainable Nov 09 18:06:10 <kanarip> 2) your scripts are executed at arbitrary intervals Nov 09 18:06:34 <kanarip> either a cronjob, on boot, may or may not fail, and you cannot make the changes you want pronto Nov 09 18:06:58 <kanarip> 3) your scripts replace configuration and reconfigure stuff without actual changes being applied Nov 09 18:07:18 <kanarip> either of these 3 things cries out for a configuration management like... Nov 09 18:07:26 <kanarip> Puppet (or CFEngine) Nov 09 18:07:32 <kanarip> anyone heard of CFEngine before? Nov 09 18:07:37 <delhage> yup Nov 09 18:07:39 <jds2001> however cfengine is fail Nov 09 18:07:43 <jds2001> we use it :) Nov 09 18:07:51 <kanarip> jds2001, you're in the right spot ;-) Nov 09 18:08:09 <kanarip> what CFEngine does is very accurately describe the configuration state a system has to be in Nov 09 18:08:32 <kanarip> it does it so accurately, in fact, that it ends up being a very low-level tool Nov 09 18:08:36 <kanarip> to give you an example; Nov 09 18:09:22 <kanarip> in CFEngine, to install a package called foo, you will have to manually assign a package provider to each operating system, distribution or distribution version, and then use that provider to install the package Nov 09 18:09:43 <kanarip> in addition, you would need to change the name for each OS, distro and distribution version Nov 09 18:10:01 <kanarip> lots to type, lots to care about, while in fact all you care about is that the package is installed Nov 09 18:10:18 * kanarip introduces... the high-level abstraction of Puppet Nov 09 18:10:36 <kanarip> no matter what the operating system, distribution or distribution may be, Puppet speaks the local language Nov 09 18:11:13 <kanarip> it'll figure out on it's own whether to use yum, up2date, dpkg, apt, PackageKit, alien, smart, rpm or what-package-manager-have-you Nov 09 18:11:30 <kanarip> this way, ensuring that package foo is installed on a system becomes: Nov 09 18:11:35 <kanarip> package { "foo": Nov 09 18:11:40 <kanarip> ensure => installed Nov 09 18:11:41 <kanarip> } Nov 09 18:11:44 <kanarip> and you're done Nov 09 18:11:58 <kanarip> jds2001, can you confirm that is easier then CFEngine please? ;-) Nov 09 18:12:06 <jds2001> yes :) Nov 09 18:12:11 <kanarip> thanks ;-) Nov 09 18:12:19 <kanarip> so, how does puppet work? Nov 09 18:13:15 <kanarip> given these abstracted "system resources" such as packages, services, cronjobs, users, ..., this and that and more, puppet uses so-called "types" you can use in a configuration "manifest" to describe what state a managed system should be in Nov 09 18:13:29 <kanarip> here's a list of types: http://reductivelabs.com/trac/puppet/wiki/TypeReference Nov 09 18:14:56 <kanarip> now let me describe to you what a working setup looks like; Nov 09 18:15:15 <kanarip> there is a single server, called the puppetmaster Nov 09 18:15:34 <kanarip> personally i think the puppeteer would have been a more appropriate name, but puppetmaster it is Nov 09 18:15:56 <kanarip> the managed system is called a "puppet" - whos strings you can pull using the puppetmaster Nov 09 18:16:20 <kanarip> the puppet, or managed system, polls the puppetmaster at a default interval of every 30 minutes Nov 09 18:17:08 <kanarip> the puppetmaster, having *all* the configuration for *all* nodes in an environment, will parse *all* it's manifests, and send to the puppet a set of resources to be managed on that specific node Nov 09 18:17:09 <nirik> Q: do they all pull at once? or is it staggered? Nov 09 18:17:34 <kanarip> nirik, it depends on the time at which the puppet starts counting Nov 09 18:17:56 <kanarip> it has a default 1 minute randomization i believe Nov 09 18:18:34 <kanarip> but it'll poll at least once every 30.9 minutes, if that makes sense ;-) Nov 09 18:18:52 <domg472_> why does it do that? Nov 09 18:19:21 <kanarip> to check if there's any changes to the configuration for the node Nov 09 18:19:21 <domg472_> if yo have changes on the master can you just tell it to push them? Nov 09 18:19:40 <domg472_> ok thanks Nov 09 18:20:38 <kanarip> if you change anything, you let the master pick it up and at most 59 minutes later the puppet will have picked up on the changes Nov 09 18:20:51 <kanarip> that is, if you are using the default interval of 30 minutes Nov 09 18:21:10 <kanarip> sorry, should have said: at most 30.9 minutes after you apply changes, the puppet will have picked up the changes Nov 09 18:21:33 <kanarip> so here's how the puppetmaster distinguishes configuration to be applied to one node: http://git.puppetmanaged.org/?p=domain-kanarip.com;a=blob;f=puppet/manifests/nodes/anakin.kanarip.com.pp Nov 09 18:21:46 <daMaestro> is there any way to push changes? or is it all pull operations? Nov 09 18:21:56 <daMaestro> (in the case of needing an isolated DMZ/VLAN) Nov 09 18:22:09 <kanarip> the puppetmaster will load that file and know that anakin.kanarip.com which is an actual system is to be configured accordingly Nov 09 18:22:32 <neverho0d> is there way to stop propagating changes? Nov 09 18:22:40 <domg472_> evry 30 minutes sounds like overhead Nov 09 18:22:41 <kanarip> daMaestro, puppet provides a utility called puppetrun which tells the client or clients to do a puppet run Nov 09 18:22:59 <daMaestro> kanarip, yet this is still a pull? Nov 09 18:23:02 <kanarip> neverho0d, yes; Nov 09 18:23:13 <kanarip> 1) stage your changes in environments Nov 09 18:23:14 <neverho0d> kanarip: ok Nov 09 18:23:17 <kanarip> 2) stop the puppetmaster Nov 09 18:23:23 <kanarip> 3) stop the puppet client daemon Nov 09 18:23:30 * daMaestro attempts to not interrupt anymore Nov 09 18:23:37 <kanarip> 4) let the puppetmaster not pull from a SCM to load new configuration Nov 09 18:23:52 <kanarip> daMaestro, the puppet client will perform a pull from the puppetmaster, yes Nov 09 18:24:03 <daMaestro> k Nov 09 18:24:28 <kanarip> any other questions so far? Nov 09 18:24:38 <kanarip> i'd like to dive into modules, if that's ok? Nov 09 18:24:45 <erinlea80> any utilities to automate data/config collections on pre-existing servers? Nov 09 18:24:53 <erinlea80> or is it all by hand? Nov 09 18:25:38 <kanarip> erinlea80, it's very complex to harvest data and then roll it out but i guess a certain scale justifies putting in such effort Nov 09 18:25:56 <kanarip> the harvesting would only happen once is what i'm concerned with Nov 09 18:26:00 <erinlea80> agreed -- was wondering what the options are. Nov 09 18:26:13 <kanarip> so here's a sample module for puppet: http://git.puppetmanaged.org/?p=sudo;a=tree Nov 09 18:26:52 <kanarip> a module is a collection of the manifest: manifests/init.pp, the files used by that module (in files/), any templates used by the module (in templates/), and more Nov 09 18:27:06 <kanarip> http://reductivelabs.com/trac/puppet/wiki/ModuleOrganisation for a reference Nov 09 18:27:25 <kanarip> and of course there's something mentioned in http://www.kanarip.com/courses/puppet/puppet.pdf as well Nov 09 18:27:55 <kanarip> what makes modules so great is a couple of things; Nov 09 18:28:13 <kanarip> 1) i can share it with you and you can just start using it Nov 09 18:28:45 <kanarip> 2) you can customize it and still pull changes from an upstream SCM like the GIT repository I just pointed you to and send me/upstream patches Nov 09 18:28:55 <kanarip> 3) modules allow "staging" Nov 09 18:29:03 <kanarip> i guess the next topic is "staging" ;-) Nov 09 18:29:25 <kanarip> 4) modules keep your files, manifests, templates and plugins organized Nov 09 18:29:52 <kanarip> believe me when I say puppet configuration trees can become a mess if you start managing more and more with puppet, and not use modules Nov 09 18:29:58 <kanarip> == Staging == Nov 09 18:30:38 <kanarip> What I mean by staging is simply this; stage your changes from a development environment, possibly via a testing environment onto your production environment Nov 09 18:31:26 <kanarip> you can develop all you want in your development environment, make syntax errors, rm -rf / sorta speak, and never nuke that business critical application server you have running Nov 09 18:32:08 <kanarip> then once you're satisfied, you may move the changes into the testing or production environment Nov 09 18:32:12 <kanarip> does that make sense? Nov 09 18:32:21 <nuonguy> oh yeah Nov 09 18:32:24 <neverho0d> sure Nov 09 18:32:24 <jds2001> yes, but how does one do this in practice? Nov 09 18:32:26 <kanarip> awesome Nov 09 18:32:26 <erinlea80> mmmm yes Nov 09 18:32:42 <erinlea80> and does it require a unique environment for each corresponding production environment? Nov 09 18:32:43 <kanarip> i can recommend anyone getting his feet wet in using environments Nov 09 18:32:48 * nirik has some questions, but can save them for the end as well. Nov 09 18:32:49 <erinlea80> unique test environ. Nov 09 18:32:52 <kanarip> jds2001, here's a practical example; Nov 09 18:33:27 <kanarip> all the repositories you see on http://git.puppetmanaged.org/ have a development, a testing, and a production branch Nov 09 18:34:10 <kanarip> what i do when i'm satisfied with my developments, is I change the working branch to testing, and git pull development Nov 09 18:34:40 <kanarip> then when i push, the puppetmaster runs a puppet that is configured so that it will pull the changes from the GIT repo Nov 09 18:34:57 <jds2001> then you have post-receive hooks to put them someplace? Nov 09 18:35:05 <kanarip> http://git.puppetmanaged.org/?p=domain-puppetmanaged.org;a=blob;f=puppet/manifests/nodes/master.puppetmanaged.org.pp as a reference Nov 09 18:35:28 <kanarip> jds2001, no, the puppetmaster in my case runs a puppet that has ^^ configuration applied to itself Nov 09 18:36:10 <kanarip> a post-hook making the puppetmaster pull in the changes is a viable solution too; the Fedora Project does such Nov 09 18:36:25 <kanarip> however, in my situation, the puppetmaster and the SCM are not on the same machine Nov 09 18:37:08 <jds2001> ok, makes sense Nov 09 18:37:12 <erinlea80> kk Nov 09 18:37:14 <kanarip> http://www.kanarip.com/courses/puppet/puppet.odp is a presentation slide-deck BTW, that goes along with the .pdf i've already linked Nov 09 18:37:38 <kanarip> note that what you see on puppetmanaged.org is heavily dependent on the other modules available from puppetmanaged.org Nov 09 18:38:43 <kanarip> so... what do you think about setting it up? Nov 09 18:39:15 <erinlea80> lets do it? Nov 09 18:39:24 <kanarip> there you go ;-) Nov 09 18:39:37 <kanarip> let me refer to the documentation i shared earlier because it's a lot to type Nov 09 18:39:49 <kanarip> http://www.kanarip.com/courses/PuppetWorkshop-SettingUpPuppet.html Nov 09 18:40:21 <kanarip> it's a yum install puppet-server to install it on any Fedora system Nov 09 18:40:36 <kanarip> for Red Hat and CentOS, you'll need EPEL configured Nov 09 18:40:43 <kanarip> http://fedoraproject.org/wiki/EPEL Nov 09 18:40:56 * erinlea80 nods Nov 09 18:41:26 <kanarip> for anyone who's wondering about what EPEL is... I maintain or co-maintain the entire Ruby stack to puppet in that repository ;-) Nov 09 18:41:49 <kanarip> and that brings me to one more nifty feature of puppet; Nov 09 18:42:08 <kanarip> but first... facts! Nov 09 18:42:17 <kanarip> facts are... well... facts... Nov 09 18:42:33 <kanarip> little information about the current state of a client Nov 09 18:43:02 <kanarip> it includes the kernel version, number of network interfaces, the architecture, operating system name, operating system version, and lots more Nov 09 18:43:16 <kanarip> yum install facter and run "facter" on your machine to check it out Nov 09 18:43:38 <kanarip> these are all "variables" you can use in your puppet manifests Nov 09 18:44:09 <kanarip> given a small amount of RAM for example, you may want to apply a snippet similar to: package { "firefox": ensure => absent } Nov 09 18:44:35 <kanarip> anyway, what i really wanted to say rather then to give a rant on firefox's memory consumption is this; Nov 09 18:44:52 <kanarip> facts, types and providers, these things that make puppet do smart things... Nov 09 18:45:01 <kanarip> facts for information about the client... Nov 09 18:45:09 <erinlea80> do we have a sample facter output? Nov 09 18:45:21 <kanarip> i could fpaste you some Nov 09 18:45:35 <erinlea80> that would be great :) Nov 09 18:45:53 <zcat> kanarip, is facter supposed to spit out "virtual => vmware_server" if detected? Nov 09 18:45:56 <domg472_> creating new ones doesnt look that easy Nov 09 18:46:10 <kanarip> http://fpaste.org/paste/15 Nov 09 18:46:41 <domg472_> but theres good documentation on create types fact etc Nov 09 18:46:45 <kanarip> types to define resources to be managed on the puppet... Nov 09 18:47:00 <erinlea80> thanks Nov 09 18:47:03 <kanarip> and providers to make puppet speak the local language... Nov 09 18:47:14 <kanarip> the great thing is... you can write your own for all of these Nov 09 18:47:53 <kanarip> it does require ruby knowledge, but then again... there's sample recipes on reductivelabs.com for...for example zcat, a vmware custom fact ;-) Nov 09 18:47:59 <kanarip> see, i was getting there ;-) Nov 09 18:48:22 <kanarip> actually i think the custom fact is called "this is a vmware machine" and "whether it has vm-tools installed" Nov 09 18:48:59 <kanarip> one more thing i really, really need to stress out to you Nov 09 18:49:17 <kanarip> all communication between the puppet and it's puppetmaster is *secure* Nov 09 18:49:45 * nirik has one of his questions answered. ;) Nov 09 18:50:02 <kanarip> that being said, the security is based on the puppetmaster running the Certificate Authority, the client generating a certificate and putting in a request with the puppetmaster to get a signed copy of that certificate Nov 09 18:50:05 <erinlea80> can communication be tunneled over ssh if necessary? (firewall restrictions, etc.) Nov 09 18:50:11 <domg472_> but the puppets have root access obviously? Nov 09 18:50:18 <domg472_> and the master Nov 09 18:50:22 <kanarip> erinlea80, you can make the puppetmaster run on any port you want Nov 09 18:50:43 <kanarip> erinlea80, may i point out "autossh" with which you can create tunnels to communicate over if you wish Nov 09 18:50:49 <erinlea80> ty :) Nov 09 18:51:01 <kanarip> domg472_, the puppet client daemon runs with root privileges, yes Nov 09 18:51:16 <domg472_> has anyone tested this with selinux? Nov 09 18:51:16 <kanarip> domg472_, the master runs with under the puppet user Nov 09 18:51:51 <domg472_> ok thanks Nov 09 18:52:20 <kanarip> domg472_, i have, and i can tell you it requires a custom selinux policy in which you basically describe the puppet daemon can do this and that and more Nov 09 18:52:31 <kanarip> in my case, the puppet daemon can do anything it likes Nov 09 18:52:45 <domg472_> youd need custom modules i bet Nov 09 18:52:58 <domg472_> interesting Nov 09 18:53:21 <sdodson> There are also a few minor bugs which I think have been fixed recently where puppet creates files without the proper contexts. Nov 09 18:53:34 <kanarip> selinux is being integrated better into puppet as we speak Nov 09 18:53:39 <domg472_> great Nov 09 18:53:49 <kanarip> it's a big issue, i concur Nov 09 18:53:54 <nuonguy> Q: if I enable, for example httpd in a manifest, and 2 hours later comment it out, does the puppet uninstall/disable httpd or ignore it from then on? Nov 09 18:54:17 <kanarip> nuonguy, puppet only applies changes Nov 09 18:54:34 <kanarip> nuonguy, so it only enforces what you describe needs to happen Nov 09 18:54:46 <kanarip> if you say package httpd should be installed at one point Nov 09 18:54:46 <nuonguy> so I'd have to explicity turn it off, if that's what I meant? Nov 09 18:55:20 <kanarip> and then comment that out... no other machine is going to care whether the package should be installed, but it's not going to be removed on puppet's behalf Nov 09 18:55:26 <kanarip> nuonguy, yes Nov 09 18:55:37 <nuonguy> I dig, thanks Nov 09 18:56:00 <kanarip> nuonguy, same goes for the service type; ensure => running, enable => true would need to become ensure => stopped, enable => false Nov 09 18:56:10 <nuonguy> precisely Nov 09 18:56:22 <nuonguy> I can just see myself not remembering that... Nov 09 18:56:33 <kanarip> so, another thing on the topic; what do you think happens if the puppet daemon cannot contact the puppetmaster? Nov 09 18:56:34 <nuonguy> and commenting it out instead Nov 09 18:56:56 <nirik> does it do them in order in the manifest? ie, make sure to stop before ensure absent? Nov 09 18:57:08 <kanarip> nuonguy, there's some discussion going on about supporting an "exclude" statement like we do "include webserver" now Nov 09 18:57:18 <nuonguy> nice Nov 09 18:57:24 <kanarip> nuonguy, and then flipping all the false's into true's and so forth Nov 09 18:57:37 <kanarip> nirik, good point! Nov 09 18:57:41 <sdodson> Could create a baseline that ensures none of that is installed/running then include webserver for only your webservers which would override those baselines? Nov 09 18:57:46 <kanarip> puppet allows ordering the resources; Nov 09 18:57:52 <kanarip> let me look you up an example Nov 09 18:58:55 <domg472_> why was ruby chosen as the language for writing types etc? Nov 09 18:59:05 <domg472_> why not bash? Nov 09 18:59:42 <kanarip> here's a yp manifest: http://fpaste.org/paste/16 Nov 09 19:00:21 <kanarip> domg472_, i'm not sure i'm not one of the core developers, but i can try and answer it if you promise not to quote me on it ;-) Nov 09 19:00:32 <domg472_> hah ok Nov 09 19:00:40 <domg472_> i think most sysadmins know bash Nov 09 19:00:49 <kanarip> nirik, you can see how "require => (...)" in that yp manifest causes one resource to be dependent on another Nov 09 19:00:59 <domg472_> and they will need to write custom types probably Nov 09 19:01:51 <kanarip> domg472_, i think bash is what you use to do a quick thing, clean and simple. I think ruby allows a framework to be built, reusing snippets of code you've already written in a different place. I think ruby is more modular then is bash Nov 09 19:02:06 <domg472_> ok thanks Nov 09 19:02:16 <kanarip> you could write a website in bash but you won't, right? Nov 09 19:02:21 <domg472_> i guess ill be learning ruby then Nov 09 19:02:23 <domg472_> true Nov 09 19:02:33 <domg472_> but these are simplate admin tasks Nov 09 19:02:40 <domg472_> simple Nov 09 19:02:42 <kanarip> not saying you should choose php to do it BTW ;-) Nov 09 19:02:56 * nirik notes we are over time, but this is also the last class today. Nov 09 19:03:18 <kanarip> in puppet terms; distribute the bash scripts amongst the clients that need it using the file type, and exec that script Nov 09 19:03:24 <nirik> I had one last question: how do you decide what should be in a module? whats the logical boundry there? Nov 09 19:03:36 <kanarip> anything you; Nov 09 19:03:42 <domg472_> ah right, thanks kanarip Nov 09 19:03:47 <kanarip> 1) pull from an upstream source Nov 09 19:04:00 <kanarip> 2) need to share (with the general public) Nov 09 19:04:11 <kanarip> 3) want to stage Nov 09 19:04:33 <nirik> so it doesn't need to be a specific service or anything? just whatever you want to put in one? Nov 09 19:04:38 <kanarip> 4) that is a concise set of files, manifests, etc. Nov 09 19:04:57 <sdodson> They're advising people to get away from writing classes and move towards modules. Nov 09 19:05:04 <kanarip> nirik, you're almost right, a module called everything-else doesn't make sense ;-) Nov 09 19:05:12 <kanarip> sdodson, yes, very much so Nov 09 19:05:51 <kanarip> nirik, a module webserver makes sense; keeping it close and concise within a module is way easier then dispersing it all over the place Nov 09 19:06:12 <nirik> ok, makes sense. Nov 09 19:06:27 <tmz> domg472_: regarding why ruby was used to write puppet, here's a short interview with Luke Kanies (primary puppet author), where he touches on that choice: Nov 09 19:06:31 <tmz> http://on-ruby.blogspot.com/2008/02/puppet-interview-with-luke-kanies.html Nov 09 19:06:40 <domg472_> thanks Nov 09 19:06:46 <kanarip> what i tend to recommend to customers is also, to keep the systems they manage as consistent as possible Nov 09 19:07:20 <kanarip> being able to determine the exceptions to the rule you've layed down saves you a lot of time in determining what the cause might be for a specific issue Nov 09 19:08:11 <nirik> shall we wrap up and let kanarip get some sleep? Nov 09 19:08:17 <kanarip> ;-) Nov 09 19:08:23 <daMaestro> no, no sleep for kanarip ! Nov 09 19:08:34 <kanarip> thank you all for your interest, and thank you for attending Nov 09 19:08:37 * daMaestro /dcc kanarip latte Nov 09 19:08:38 <jds2001> he must stay up all night! Nov 09 19:08:41 <nirik> thanks so much for the session kanarip ! Nov 09 19:08:42 <erinlea80> thanks kanarip! Nov 09 19:08:46 <domg472_> thanks again Nov 09 19:08:47 <erinlea80> :) Nov 09 19:08:49 <iondrip> thanks Nov 09 19:08:53 <kanarip> i hope you find a lot of use in the pdf/html/odp i've given you ;-) Nov 09 19:08:57 <neverho0d> kanarip: thanks Nov 09 19:09:04 <nirik> thanks everyone for coming... Nov 09 19:09:08 <Bugz_> kanarip: Thanks Nov 09 19:09:15 <erinlea80> :) Nov 09 19:09:31 <domg472_> thanks for hosting Nov 09 19:09:57 <SSlater> Thanks