From Fedora Project Wiki
Fedora Classroom - Configuration Management using Puppet - Jeroen van Meeuwen - Sunday, November 9, 2008
IRC Log of the Class
| kanarip | so this session is about configuration management, which does *not* include the soft side of configuration management for a change | 23:04 |
|---|---|---|
| *Bugz_ is | 23:04 | |
| kanarip | we're *not* going to talk about procedural issues, but keep it on the technical side | 23:04 |
| kanarip | i'd like to invite you all to take a look at http://www.kanarip.com/courses/puppet/puppet.pdf | 23:04 |
| kanarip | which is a nice reader i wrote for a one-day workshop i'm giving on this topic, and that is commercially available | 23:04 |
| kanarip | community people like us however *do not have to pay* | 23:04 |
| kanarip | sounds great, huh? ;-) | 23:04 |
| domg472_ | yes | 23:04 |
| erinlea80 | :) | 23:04 |
| delhage | from Delft eh? | 23:04 |
| kanarip | so, a little something on configuration management first, then a sneak peak at the installation of puppet, and a quick howto on writing your "manifest" as they call the puppet configuration syntax | 23:04 |
| kanarip | delhage, Utrecht | 23:04 |
| delhage | ok | 23:04 |
| kanarip | so, what is it we would want to do with configuration management in the technical sense of the words | 23:04 |
| kanarip | we all know that the more systems we need to maintain, the harder it becomes to keep some extend of consistency between different systems | 23:04 |
| kanarip | some organizations use scripts from an NFS share, or use an SVN tree with scripts periodically updated and execute scripts | 23:05 |
| kanarip | does that sound familiar to anyone? | 23:05 |
| *erinlea80 nods | 23:05 | |
| kanarip | so what happens in these cases is: | 23:05 |
| kanarip | 1) your scripts tend to get very large | 23:05 |
| kanarip | describing every unique situation, every exemption, every update, for every operating system, distribution, distribution version, makes the script less maintainable | 23:05 |
| kanarip | 2) your scripts are executed at arbitrary intervals | 23:06 |
| kanarip | either a cronjob, on boot, may or may not fail, and you cannot make the changes you want pronto | 23:06 |
| kanarip | 3) your scripts replace configuration and reconfigure stuff without actual changes being applied | 23:06 |
| kanarip | either of these 3 things cries out for a configuration management like... | 23:07 |
| kanarip | Puppet (or CFEngine) | 23:07 |
| kanarip | anyone heard of CFEngine before? | 23:07 |
| delhage | yup | 23:07 |
| jds2001 | however cfengine is fail | 23:07 |
| jds2001 | we use it :) | 23:07 |
| kanarip | jds2001, you're in the right spot ;-) | 23:07 |
| kanarip | what CFEngine does is very accurately describe the configuration state a system has to be in | 23:08 |
| kanarip | it does it so accurately, in fact, that it ends up being a very low-level tool | 23:08 |
| kanarip | to give you an example; | 23:08 |
| kanarip | in CFEngine, to install a package called foo, you will have to manually assign a package provider to each operating system, distribution or distribution version, and then use that provider to install the package | 23:09 |
| kanarip | in addition, you would need to change the name for each OS, distro and distribution version | 23:09 |
| kanarip | lots to type, lots to care about, while in fact all you care about is that the package is installed | 23:10 |
| *kanarip introduces... the high-level abstraction of Puppet | 23:10 | |
| kanarip | no matter what the operating system, distribution or distribution may be, Puppet speaks the local language | 23:10 |
| kanarip | it'll figure out on it's own whether to use yum, up2date, dpkg, apt, PackageKit, alien, smart, rpm or what-package-manager-have-you | 23:11 |
| kanarip | this way, ensuring that package foo is installed on a system becomes: | 23:11 |
| kanarip | package { "foo": | 23:11 |
| kanarip | ensure => installed | 23:11 |
| kanarip | } | 23:11 |
| kanarip | and you're done | 23:11 |
| kanarip | jds2001, can you confirm that is easier then CFEngine please? ;-) | 23:11 |
| jds2001 | yes :) | 23:12 |
| kanarip | thanks ;-) | 23:12 |
| kanarip | so, how does puppet work? | 23:12 |
| kanarip | given these abstracted "system resources" such as packages, services, cronjobs, users, ..., this and that and more, puppet uses so-called "types" you can use in a configuration "manifest" to describe what state a managed system should be in | 23:13 |
| kanarip | here's a list of types: http://reductivelabs.com/trac/puppet/wiki/TypeReference | 23:13 |
| kanarip | now let me describe to you what a working setup looks like; | 23:14 |
| kanarip | there is a single server, called the puppetmaster | 23:15 |
| kanarip | personally i think the puppeteer would have been a more appropriate name, but puppetmaster it is | 23:15 |
| kanarip | the managed system is called a "puppet" - whos strings you can pull using the puppetmaster | 23:15 |
| kanarip | the puppet, or managed system, polls the puppetmaster at a default interval of every 30 minutes | 23:16 |
| kanarip | the puppetmaster, having *all* the configuration for *all* nodes in an environment, will parse *all* it's manifests, and send to the puppet a set of resources to be managed on that specific node | 23:17 |
| nirik | Q: do they all pull at once? or is it staggered? | 23:17 |
| kanarip | nirik, it depends on the time at which the puppet starts counting | 23:17 |
| kanarip | it has a default 1 minute randomization i believe | 23:17 |
| kanarip | but it'll poll at least once every 30.9 minutes, if that makes sense ;-) | 23:18 |
| domg472_ | why does it do that? | 23:18 |
| kanarip | to check if there's any changes to the configuration for the node | 23:19 |
| domg472_ | if yo have changes on the master can you just tell it to push them? | 23:19 |
| domg472_ | ok thanks | 23:19 |
| kanarip | if you change anything, you let the master pick it up and at most 59 later the puppet will have picked up on the changes | 23:20 |
| kanarip | that is, if you are using the default interval of 30 minutes | 23:20 |
| kanarip | sorry, should have said: at most 30.9 minutes after you apply changes, puppet will have picked up the changes | 23:21 |
| kanarip | so here's how the puppetmaster distinguishes configuration to be applied to one node: http://git.puppetmanaged.org/?p=domain-kanarip.com;a=blob;f=puppet/manifests/nodes/anakin.kanarip.com.pp | 23:21 |
| daMaestro | is there any way to push changes? or is it all pull operations? | 23:21 |
| daMaestro | (in the case of needing an isolated DMZ/VLAN) | 23:21 |
| kanarip | the puppetmaster will load that file and know that anakin.kanarip.com which is an actual system is to be configured accordingly | 23:22 |
| neverho0d | is there way to stop propagating changes? | 23:22 |
| domg472_ | evry 30 minutes sounds like overhead | 23:22 |
| kanarip | daMaestro, puppet provides a utility called puppetrun which tells the client or clients to do a puppet run | 23:22 |
| daMaestro | kanarip, yet this is still a pull? | 23:22 |
| kanarip | neverho0d, yes; | 23:23 |
| kanarip | 1) stage your changes in environments | 23:23 |
| neverho0d | kanarip: ok | 23:23 |
| kanarip | 2) stop the puppetmaster | 23:23 |
| kanarip | 3) stop the puppet client daemon | 23:23 |
| *daMaestro attempts to not interrupt anymore | 23:23 | |
| kanarip | 4) let the puppetmaster not pull from a SCM to load new configuration | 23:23 |
| kanarip | daMaestro, the puppet client will perform a pull from the puppetmaster, yes | 23:23 |
| daMaestro | k | 23:24 |
| kanarip | any other questions so far? | 23:24 |
| kanarip | i'd like to dive into modules, if that's ok? | 23:24 |
| erinlea80 | any utilities to automate data/config collections on pre-existing servers? | 23:24 |
| erinlea80 | or is it all by hand? | 23:24 |
| kanarip | erinlea80, it's very complex to harvest data and then roll it out but i guess a certain scale justifies putting in such effort | 23:25 |
| kanarip | the harvesting would only happen once is what i'm concerned with | 23:25 |
| erinlea80 | agreed -- was wondering what the options are. | 23:26 |
| kanarip | so here's a sample module for puppet: http://git.puppetmanaged.org/?p=sudo;a=tree | 23:26 |
| kanarip | a module is a collection of the manifest: manifests/init.pp, the files used by that module (in files/), any templates used by the module (in templates/), and more | 23:26 |
| kanarip | http://reductivelabs.com/trac/puppet/wiki/ModuleOrganisation for a reference | 23:27 |
| kanarip | and of course there's something mentioned in http://www.kanarip.com/courses/puppet/puppet.pdf as well | 23:27 |
| kanarip | what makes modules so great is a couple of things; | 23:27 |
| kanarip | 1) i can share it with you and you can just start using it | 23:28 |
| kanarip | 2) you can customize it and still pull changes from an upstream SCM like the GIT repository I just pointed you to and send me/upstream patches | 23:28 |
| kanarip | 3) modules allow "staging" | 23:28 |
| kanarip | i guess the next topic is "staging" ;-) | 23:29 |
| kanarip | 4) modules keep your files, manifests, templates and plugins organized | 23:29 |
| kanarip | believe me when I say puppet configuration trees can become a mess if you start managing more and more with puppet, and not use modules | 23:29 |
| kanarip | == Staging == | 23:29 |
| kanarip | What I mean by staging is simply this; stage your changes from a development environment, possibly via a testing environment onto your production environment | 23:30 |
| kanarip | you can develop all you want in your development environment, make syntax errors, rm -rf / sorta speak, and never nuke that business critical application server you have running | 23:31 |
| kanarip | then once you're satisfied, you may move the changes into the testing or production environment | 23:32 |
| kanarip | does that make sense? | 23:32 |
| nuonguy | oh yeah | 23:32 |
| neverho0d | sure | 23:32 |
| jds2001 | yes, but how does one do this in practice? | 23:32 |
| kanarip | awesome | 23:32 |
| erinlea80 | mmmm yes | 23:32 |
| erinlea80 | and does it require a unique environment for each corresponding production environment? | 23:32 |
| kanarip | i can recommend anyone getting his feet wet in using environments | 23:32 |
| *nirik has some questions, but can save them for the end as well. | 23:32 | |
| erinlea80 | unique test environ. | 23:32 |
| kanarip | jds2001, here's a practical example; | 23:32 |
| kanarip | all the repositories you see on http://git.puppetmanaged.org/ have a development, a testing, and a production branch | 23:33 |
| kanarip | what i do when i'm satisfied with my developments, is I change the working branch to testing, and git pull development | 23:34 |
| kanarip | then when i push, the puppetmaster runs a puppet that is configured so that it will pull the changes from the GIT repo | 23:34 |
| jds2001 | then you have post-receive hooks to put them someplace? | 23:34 |
| kanarip | http://git.puppetmanaged.org/?p=domain-puppetmanaged.org;a=blob;f=puppet/manifests/nodes/master.puppetmanaged.org.pp as a reference | 23:35 |
| kanarip | jds2001, no, the puppetmaster in my case runs a puppet that has ^^ configuration applied to itself | 23:35 |
| kanarip | a post-hook making the puppetmaster pull in the changes is a viable solution too; the Fedora Project does such | 23:36 |
| kanarip | however, in my situation, the puppetmaster and the SCM are not on the same machine | 23:36 |
| jds2001 | ok, makes sense | 23:37 |
| erinlea80 | kk | 23:37 |
| kanarip | http://www.kanarip.com/courses/puppet/puppet.odp is a presentation slide-deck BTW, that goes along with the .pdf i've already linked | 23:37 |
| kanarip | note that what you see on puppetmanaged.org is heavily dependent on the other modules available from puppetmanaged.org | 23:37 |
| kanarip | so... what do you think about setting it up? | 23:38 |
| erinlea80 | lets do it? | 23:39 |
| kanarip | there you go ;-) | 23:39 |
| kanarip | let me refer to the documentation i shared earlier because it's a lot to type | 23:39 |
| kanarip | http://www.kanarip.com/courses/PuppetWorkshop-SettingUpPuppet.html | 23:39 |
| kanarip | it's a yum install puppet-server to install it on any Fedora system | 23:40 |
| kanarip | for Red Hat and CentOS, you'll need EPEL configured | 23:40 |
| kanarip | http://fedoraproject.org/wiki/EPEL | 23:40 |
| *erinlea80 nods | 23:40 | |
| kanarip | for anyone who's wondering about what EPEL is... I maintain or co-maintain the entire Ruby stack to puppet in that repository ;-) | 23:41 |
| kanarip | and that brings me to one more nifty feature of puppet; | 23:41 |
| kanarip | but first... facts! | 23:42 |
| kanarip | facts are... well... facts... | 23:42 |
| kanarip | little information about the current state of a client | 23:42 |
| kanarip | it includes the kernel version, number of network interfaces, the architecture, operating system name, operating system version, and lots more | 23:43 |
| kanarip | yum install facter and run "facter" on your machine to check it out | 23:43 |
| kanarip | these are all "variables" you can use in your puppet manifests | 23:43 |
| kanarip | given a small amount of RAM for example, you may want to apply a snippet similar to: package { "firefox": ensure => absent } | 23:44 |
| kanarip | anyway, what i really wanted to say rather then to give a rant on firefox's memory consumption is this; | 23:44 |
| kanarip | facts, types and providers, these things that make puppet do smart things... | 23:44 |
| kanarip | facts for information about the client... | 23:45 |
| erinlea80 | do we have a sample facter output? | 23:45 |
| kanarip | i could fpaste you some | 23:45 |
| erinlea80 | that would be great :) | 23:45 |
| zcat | kanarip, is facter supposed to spit out "virtual => vmware_server" if detected? | 23:45 |
| domg472_ | creating new ones doesnt look that easy | 23:45 |
| kanarip | http://fpaste.org/paste/15 | 23:46 |
| domg472_ | but theres good documentation on create types fact etc | 23:46 |
| kanarip | types to define resources to be managed on the puppet... | 23:46 |
| erinlea80 | thanks | 23:47 |
| kanarip | and providers to make puppet speak the local language... | 23:47 |
| kanarip | the great thing is... you can write your own for all of these | 23:47 |
| kanarip | it does require ruby knowledge, but then again... there's sample recipes on reductivelabs.com for...for example zcat, a vmware custom fact ;-) | 23:47 |
| kanarip | see, i was getting there ;-) | 23:47 |
| kanarip | actually i think the custom fact is called "this is a vmware machine" and "whether it has vm-tools installed" | 23:48 |
| kanarip | one more thing i really, really need to stress out to you | 23:48 |
| kanarip | all communication between the puppet and it's puppetmaster is *secure* | 23:49 |
| *nirik has one of his questions answered. ;) | 23:49 | |
| kanarip | that being said, the security is based on the puppetmaster running the Certificate Authority, the client generating a certificate and putting in a request with the puppetmaster to get a signed copy of that certificate | 23:50 |
| erinlea80 | can communication be tunneled over ssh if necessary? (firewall restrictions, etc.) | 23:50 |
| domg472_ | but the puppets have root access obviously? | 23:50 |
| domg472_ | and the master | 23:50 |
| kanarip | erinlea80, you can make the puppetmaster run on any port you want | 23:50 |
| kanarip | erinlea80, may i point out "autossh" with which you can create tunnels to communicate over if you wish | 23:50 |
| erinlea80 | ty :) | 23:50 |
| kanarip | domg472_, the puppet client daemon runs with root privileges, yes | 23:51 |
| domg472_ | has anyone tested this with selinux? | 23:51 |
| kanarip | domg472_, the master runs with under the puppet user | 23:51 |
| domg472_ | ok thanks | 23:51 |
| kanarip | domg472_, i have, and i can tell you it requires a custom selinux policy in which you basically describe the puppet daemon can do this and that and more | 23:52 |
| kanarip | in my case, the puppet daemon can do anything it likes | 23:52 |
| domg472_ | youd need custom modules i bet | 23:52 |
| domg472_ | interesting | 23:52 |
| sdodson | There are also a few minor bugs which I think have been fixed recently where puppet creates files without the proper contexts. | 23:53 |
| kanarip | selinux is being integrated better into puppet as we speak | 23:53 |
| domg472_ | great | 23:53 |
| kanarip | it's a big issue, i concur | 23:53 |
| nuonguy | Q: if I enable, for example httpd in a manifest, and 2 hours later comment it out, does the puppet uninstall/disable httpd or ignore it from then on? | 23:53 |
| kanarip | nuonguy, puppet only applies changes | 23:54 |
| kanarip | nuonguy, so it only enforces what you describe needs to happen | 23:54 |
| kanarip | if you say package httpd should be installed at one point | 23:54 |
| nuonguy | so I'd have to explicity turn it off, if that's what I meant? | 23:54 |
| kanarip | and then comment that out... no other machine is going to care whether the package should be installed, but it's not going to be removed on puppet's behalf | 23:55 |
| kanarip | nuonguy, yes | 23:55 |
| nuonguy | I dig, thanks | 23:55 |
| kanarip | nuonguy, same goes for the service type; ensure => running, enable => true would need to become ensure => stopped, enable => false | 23:56 |
| nuonguy | precisely | 23:56 |
| nuonguy | I can just see myself not remembering that... | 23:56 |
| kanarip | so, another thing on the topic; what do you think happens if the puppet daemon cannot contact the puppetmaster? | 23:56 |
| nuonguy | and commenting it out instead | 23:56 |
| nirik | does it do them in order in the manifest? ie, make sure to stop before ensure | 23:56 |
| kanarip | nuonguy, there's some discussion going on about supporting an "exclude" statement like we do "include webserver" now | 23:57 |
| nuonguy | nice | 23:57 |
| kanarip | nuonguy, and then flipping all the false's into true's and so forth | 23:57 |
| kanarip | nirik, good point! | 23:57 |
| sdodson | Could create a baseline that ensures none of that is installed/running then include webserver for only your webservers which would override those baselines? | 23:57 |
| kanarip | puppet allows ordering the resources; | 23:57 |
| kanarip | let me look you up an example | 23:57 |
| domg472_ | why was ruby chosen as the language for writing types etc? | 23:58 |
| domg472_ | why not bash? | 23:59 |
| kanarip | here's a yp manifest: http://fpaste.org/paste/16 | 23:59 |
| kanarip | domg472_, i'm not sure i'm not one of the core developers, but i can try and answer it if you promise not to quote me on it ;-) | 00:00 |
| domg472_ | hah ok | 00:00 |
| domg472_ | i think most sysadmins know bash | 00:00 |
| kanarip | nirik, you can see how "require => (...)" in that yp manifest causes one resource to be dependent on another | 00:00 |
| domg472_ | and they will need to write custom types probably | 00:00 |
| kanarip | domg472_, i think bash is what you use to do a quick thing, clean and simple. I think ruby allows a framework to be built, reusing snippets of code you've already written in a different place. I think ruby is more modular then is bash | 00:01 |
| domg472_ | ok thanks | 00:02 |
| kanarip | you could write a website in bash but you won't, right? | 00:02 |
| domg472_ | i guess ill be learning ruby then | 00:02 |
| domg472_ | true | 00:02 |
| domg472_ | but these are simplate admin tasks | 00:02 |
| domg472_ | simple | 00:02 |
| kanarip | not saying you should choose php to do it BTW ;-) | 00:02 |
| *nirik notes we are over time, but this is also the last class today. | 00:02 | |
| kanarip | in puppet terms; distribute the bash scripts amongst the clients that need it using the file type, and exec that script | 00:03 |
| nirik | I had one last question: how do you decide what should be in a module? whats the logical boundry there? | 00:03 |
| kanarip | anything you; | 00:03 |
| domg472_ | ah right, thanks kanarip | 00:03 |
| kanarip | 1) pull from an upstream source | 00:03 |
| kanarip | 2) need to share (with the general public) | 00:04 |
| kanarip | 3) want to stage | 00:04 |
| nirik | so it doesn't need to be a specific service or anything? just whatever you want to put in one? | 00:04 |
| kanarip | 4) that is a concise set of files, manifests, etc. | 00:04 |
| sdodson | They're advising people to get away from writing classes and move towards modules. | 00:04 |
| kanarip | nirik, you're almost right, a module called everything-else doesn't make sense ;-) | 00:05 |
| kanarip | sdodson, yes, very much so | 00:05 |
| kanarip | nirik, a module webserver makes sense; keeping it close and concise within a module is way easier then dispersing it all over the place | 00:05 |
| nirik | ok, makes sense. | 00:06 |
| tmz | domg472_: regarding why ruby was used to write puppet, here's a short interview with Luke Kanies (primary puppet author), where he touches on that choice: | 00:06 |
| tmz | http://on-ruby.blogspot.com/2008/02/puppet-interview-with-luke-kanies.html | 00:06 |
| domg472_ | thanks | 00:06 |
| kanarip | what i tend to recommend to customers is also, to keep the systems they manage as consistent as possible | 00:06 |
| kanarip | being able to determine the exceptions to the rule you've layed down saves you a lot of time in determining what the cause might be for a specific issue | 00:07 |
| nirik | shall we wrap up and let kanarip get some sleep? | 00:08 |
| kanarip | ;-) | 00:08 |
| daMaestro | no, no sleep for kanarip ! | 00:08 |
| kanarip | thank you all for your interest, and thank you for attending | 00:08 |
| *daMaestro /dcc kanarip latte | 00:08 | |
| jds2001 | he must stay up all night! | 00:08 |
| nirik | thanks so much for the session kanarip ! | 00:08 |
| erinlea80 | thanks kanarip! | 00:08 |
| domg472_ | thanks again | 00:08 |
| erinlea80 | :) | 00:08 |
| iondrip | thanks | 00:08 |
| kanarip | i hope you find a lot of use in the pdf/html/odp i've given you ;-) | 00:08 |
| neverho0d | kanarip: thanks | 00:08 |
| nirik | thanks everyone for coming... | 00:09 |
| Bugz_ | kanarip: Thanks | 00:09 |
| erinlea80 | :) | 00:09 |
| domg472_ | thanks for hosting | 00:09 |
| SSlater | Thanks | 00:09 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!