Infrastructure/Meetings/2006-10-26

Meeting of 2006-10-26

*** Time shown in EDT

16:00 < mmcgrath> Lets get started.
16:00 < mmcgrath> Who's here?
16:00  * abadger1999 1/2 here
16:00 < lyz> yo
16:01  * dgilmore is here
16:01 < mmcgrath> So FC6 is out (hurray)
16:02 < lyz> running it at home already
16:02 < mmcgrath> same here.
16:02 < warren> oops!
16:02 -!- nman64 [n=n-man@fedora/nman64]  has joined #fedora-admin
16:02 < warren> We never did use smtp.fedora did we
16:02 < mmcgrath> warren: not yet, there's a few final touches we need.
16:02 < smooge> ok
16:02 < mmcgrath> But as far as I know, its clos.
16:02 < dgilmore> warren: mostly  it needs some testing
16:02 < mmcgrath> nman64: you here?
16:03 < dgilmore> warren: did you want to put postgrey on it?
16:03 < warren> Why would test guests require ssh access?
16:03 < warren> dgilmore, sqlgrey preferably
16:03 < dgilmore> warren: ok  is that in extras now?
16:03 < warren> let me check..
16:03 < lmacken> warren: pong
16:03 < warren> lmacken, sent you mail
16:03 < lmacken> k
16:03  * dgilmore will upgrade smtp to fc6
16:04 < nman64> mmcgrath: Yeah.
16:04 < dgilmore> we should upgrade all the xen guests  to it also
16:04 < warren> as well as the hosts
16:04 < mmcgrath> uhh, is anyone else having issues bringing up the schedule page?
16:04 < warren> I updated the InfrastructurePrivate xen page install example too
16:04 < lmacken> dgilmore: i can upgrade test6
16:04 < warren> anybody with root access on xen1/2 should be able to easily install a guest.
16:05 < dgilmore> lmacken: :)  all yours
16:05 < dgilmore> mmcgrath: nope
16:06 < mmcgrath> Ok, well first off lets talk about the VCS.
16:06 < f13> dgilmore: I don't know what box you touched, but I had to fix ssh when I got back to my desk (:
16:06 < mmcgrath> abadger1999: you and f13 were talking yesterday.
16:06 < mmcgrath> how are we going to cooridnate this with the developers?
16:06 < warren> f13, where?
16:06 < warren> mmcgrath, which aspect of what?
16:06 < abadger1999> So far we only have one developer on each VCS.
16:06 < dgilmore> f13: its acting goofy.  the console keeps locking up
16:07 < abadger1999> But we do need to coordinate what they're doing so they can learn from each other.
16:07 < mmcgrath> warren: the new VCS, I think f13 has shown a paticular interest in mercurial.
16:07 < warren> dgilmore, what is acting goofy?
16:07 < f13> mmcgrath: so I'm setting up a trial box with mercurial
16:07 < dgilmore> warren: console on test1
16:07 < abadger1999> (Although there's several new people expressing interest in svn to me.)
16:07 < mmcgrath> abadger1999: :)
16:07 < f13> mmcgrath: I want to figure out some tasks I need, and then post to various lists asking for folks to help me with what I need
16:08 < f13> things like make file adjustments, or plague edits.
16:08 < f13> brb
16:08 < warren> I don't think jeremy will ever accept svn.  he doesn't see it as an improvement over CVS.
16:08 < warren> (at least not enough of an improvement to switch to it)
16:08 < mmcgrath> warren: thats the other question.
16:08 < mmcgrath> who's going to decide the new vcs?
16:08 -!- abompard [n=gauret@bne75-8-88-161-125-228.fbx.proxad.net]  has joined #fedora-admin
16:08 < mmcgrath> Not us.
16:09 < mmcgrath> should it be completely community driven?  Or will we leave this to the core dev's and maybe the FESCo?
16:09 < abadger1999> Fedora Extras?
16:09 < abadger1999> FESCo?
16:09 < warren> It probably goes down to whoever implements a proof of concept that satisfies all goals first.
16:09 < mmcgrath> but who decides the goals?  I mean, everyone will want everything.
16:09 < abadger1999> warren: No testing?
16:09 < warren> The VCS goals have been on that web page for a while now.
16:10 < abadger1999> For scalabilty and user interface feel?
16:10 < warren> Essentially what package CVS currently does, except with the ability to have fine grained ACL's and distributed operation.
16:10 < dgilmore> i think  the Extras/Core developers should decide
16:10 < f13> testing would be manditory
16:10 < warren> user interface actually isn't important, abstraction layers can make it look anyway we want later
16:10 < warren> the core capabilities are what is important
16:10 < f13> but thats what the proof of ocncepts are for
16:11 < f13> import a big chunk of packages and have people use it and abuse it for a bit.
16:11 < warren> Anyway, Jeremy suggested that we (RH) finally sit down and focus on this when Max visits the Westford office on the week of November 6th.
16:11 < mmcgrath> I've never quite gotten whats required with 'distributed operation'
16:11 < mmcgrath> someone educate me :D
16:11 < dgilmore> f13: test1 is back
16:11 < f13> dgilmore: I know, I fixed it myself.
16:11  * f13 wonders where the disconnect is.
16:11 < dgilmore> f13: i just rebooted it
16:12 < f13> ?!
16:12 < f13> why?
16:12 < dgilmore> a feww minutes ago
16:12 < warren> VCS is a big first step, part of many pieces necessary for the merge.  Our RH meeting November 6th is really the first time RH has the room to breathe after FC5 and FC6 and focus on this problem.
16:12 < warren> Meanwhile, we can provide proof of concept examples of how mercurial or (something else) is awesome for package VCS needs.
16:13  * f13 discovers hg grep and wets himself.
16:13 < mmcgrath> Ok, should we continue this on the list or do people have more to discuss right now?
16:13 < warren> Jeremy suggested that we will fail if we attempt to implement everything at once.  (Everything being
VCS, package database, next gen account system, etc.)
16:13 < dgilmore> lets discuss on the list
16:13 < warren> This is just my status update.  Conclusion is infrastructure team needs to provide proof of concept implementations
16:14 < warren> November 6th is the target date
16:14 < abadger1999> warren: Jeremy's right on that.  I've been way too stretched just dealing with two of those areas.
16:14 < warren> nod
16:14 < mmcgrath> K,  so the VCS is priority 1 then.
16:14 < warren> I think VCS will just happen with all packages (Core + Extras), and we'll figure out the ACL policy at that point.
16:14 < warren> ACL will be course at first, and later we can make it well controlled.
16:15 < dgilmore> ACL stuff will depend  on what VCS we use
16:15 < warren> nod
16:15 < mmcgrath> Ok, regarding the package database.  Whats the word?
16:16 < abadger1999> I didn't work on it this weekend: c4chris, did you?
16:16 < mmcgrath> lmacken: anything?
16:16 < lmacken> on the package database? no
16:17 < mmcgrath> heh, sorry
16:17 -!- rmm [i=keefejoh@gateway/web/cgi-irc/ircatwork.com/session]  has joined #fedora-admin
16:17 -!- kimo [n=ahmed@196.202.18.251]  has quit ["rebooting for FC6"] 
16:17 < abadger1999> lmacken: Did you get any sense of how hard multiple projects from one TG server would be?
16:17 -!- rmm [i=keefejoh@gateway/web/cgi-irc/ircatwork.com/x-cbf51c6b60953a39]  has quit [Client Quit] 
16:18 < mmcgrath> Ok, how about the new accounts system
16:18 < mmcgrath> lyz: ?
16:18 < mmcgrath> abompard: ?
16:18 < lmacken> abadger1999: i didn't get a chance to read that thread yet, but i'm sure we'll figure something out
16:18 -!- rmm [i=keefejoh@gateway/web/cgi-irc/ircatwork.com/session]  has joined #fedora-admin
16:18 < lyz> FDS is up with the schema
16:18 < abompard> no news from my front..... busy time
16:18 < abompard> cool
16:18 < lyz> I sent a screenshot to the list
16:18 < mmcgrath> Saw the screenshot.
16:18 < mmcgrath> where do you want to proceed from here?
16:18 < lyz> next task is export from the db and import to LDAP
16:19 < lyz> I may need a dump of the current db
16:19 < lyz> to test with
16:19 < c4chris> abadger1999, got pretty swamped, and my network ADSL broke down :(
16:19 < mmcgrath> Hmmm, can you create your own to test with for now?
16:20 < lyz> yeah,, but it's not the best way to do it
16:20 < lyz> brb
16:20 < mmcgrath> Yeah.  I'll come up with something for you.
16:20 -!- rmm [i=keefejoh@gateway/web/cgi-irc/ircatwork.com/x-278c16f77914ca43]  has quit [Client Quit] 
16:21 < lyz> thanks
16:21 < c4chris> I started to grab old review data from the ml and put some data on test3
16:21 < lyz> After I get an importer going, we can start testing some of our systems on it
16:22 < lyz> tickets perhaps?
16:22 < abadger1999> lmacken: We could accept that all projects would live in the same DB -- I just like paritioning
it because it seems more organized, secure, and easier for new users to get a handle on.
16:22 < abadger1999> lyz: What about porting the Accounts API?
16:23 < lyz> abadger1999 what's the accounts API,  is that the web site?
16:23 < mmcgrath> lyz: nope, thats all you ;-)
16:23 < mmcgrath> step 2.
16:24 < mmcgrath> We'll need to provide everyone with an easy way to send requests to the system.  Perhaps code snipits in some of the primary languages (python, perl, bash?)
16:24 < mmcgrath> Just curious, of the new guys I've only had a couple actually come to me asking for work.  Of the other officers have you been contacted?
16:24 < dgilmore> mmcgrath: no one has contacted me
16:24 < lyz> isn't LDAP the API?
16:24 < abadger1999> lyz: Sorta.  The website uses it, but at least the voting app uses functions from website.py to
authenticate the users.
16:25 < abadger1999> mmcgrath: Two people contacted me.
16:25 < mmcgrath> lyz: yeah, but we should set it up so that people can use LDAP in our situation for authentication
and authorization.
16:25 < lyz> abadger1999 I see.  I'll check that out
16:25 < lmacken> mmcgrath: i've been contacted a couple of times
16:25 < abadger1999> Both interested in the VCS.
16:25 < lmacken> not recently though
16:25 < mmcgrath> thats something at least.
16:26 < nman64> Websites always has people interested in helping.
16:26 < lyz> I suggested tickets as a sample cause it just uses apache pgsql authentication.  Should be easy to move
to LDAP
16:26 < abadger1999> nman64: Any of them coders?
16:26 < mmcgrath> nman64: I sent an email to DaMaestro but so far haven't heard back from him about sponsorship :(
16:26 < nman64> abadger1999: Haven't seen any Python coders that I can recall.
16:27 < mmcgrath> Ok, I only have a couple of more things.
16:27 < mmcgrath> 1) network stuff on the xen guests.
16:27 < nman64> abadger1999: Plenty with basic knowledge of PHP, though that's not useful for our purposes.
16:27 < mmcgrath> what would we like stacy to do?
16:27 < nman64> mmcgrath: What does he need to do?
16:29 < dgilmore> mmcgrath: probably port 80 and 443.  and maybe on  two of them 8888 and 8889
16:30 < dgilmore> then  we could have a test on each xen host for building
16:30 < mmcgrath> he keeps pushing the plone instance so we're giving it to him, the doc guys have been wanting it too.
16:30 < abadger1999> nman64: We might be able to have them work on front end stuff (accounts interface, packageDB interface) since I'd like to use kid templates within TurboGears.  The backend isn't formalized enough to be able to get them started yet, though.
16:30 < mmcgrath> My only concern is that he doesn't seem to be coordinating with anyone
16:30 < dgilmore> unless we wanted ports  for different VCS's
16:30 < abadger1999> mmcgrath, f13: Do we need more ips for people to connect to ssh on the xenguests?
16:30 < abadger1999> Similar to cvs-int and cvs-ext?
16:30 -!- rannis [n=lycos@did75-5-82-224-183-241.fbx.proxad.net]  has quit [Remote closed the connection] 
16:31 < f13> abadger1999: I have no idea.
16:31 -!- warren [i=warren@redhat/wombat/warren]  has quit [Read error: 104 (Connection reset by peer)] 
16:32 < mmcgrath> abadger1999: we can always give some boxes multiple test ip's.
16:32 -!- warren [i=warren@nat/redhat/x-9dade552838d22ff]  has joined #fedora-admin
16:32 < mmcgrath> we'll just have to see how it goes.
16:32 < dgilmore> do we want to talk about hammer3?
16:32 < mmcgrath> warren: how's that sound
16:32 < nman64> abadger1999: That sounds like a good idea.  Whenever you see an opportunity for an ambitious volunteer to help out with simple tasks, let me know.  We might start a wiki page full of tasks within Websites to point them to.
16:33 < warren> sorry, lost my connection =(
16:33 < abadger1999> mmcgrath: That should be fine as long as we're testing internally.
16:33 < warren> Stacy said in private e-mail he'll take a look at hammer3
16:33 < warren> at this point I don't think it is important
16:33 < warren> next topic
16:33 < dgilmore> warren: :D  cool   its not really
16:33 < abadger1999> mmcgrath: We're down to two ip addresses in the list on the wiki.
16:33 < warren> BTW, I need to leave for another meeting at 5pm sharp
16:33 < mmcgrath> wow, that went QUICK.
16:34 < warren> I want to talk about network addresses
16:34 < dgilmore> warren: your floor
16:34 < warren> Should we request more test addresses?
16:34 < mmcgrath> k, we'll go quick.
16:34 < mmcgrath> warren: we're talking about it now, what do you have in mind :D
16:34 < dgilmore> warren: probably yes
16:34 < warren> we need to put a limit on the number
16:34 < warren> OTOH, we will have capacity to run even more guests when the two new dell servers come in
16:34 < abadger1999> yes
16:34 < mmcgrath> +1
16:34 < mmcgrath> or should I say +10
16:34 < abadger1999> I'm pretty sure VCS prototypes need two IPs apiece.
16:34 < warren> why two?
16:35 < dgilmore> warren: what do you think the hosts will support as guests ?  4 on xen1  and 8 on xen2?
16:35 < abadger1999> internal and external ssh
16:35 < abadger1999> Like the cvs server has now
16:35 < abadger1999> Or do you have another plan?
16:35 < warren> I strongly believe that the future is best to run ALL services within xen guests.
16:36 < f13> at some point the mercurial box will need external people to log in to it
16:36 < f13> and be able to get http or ssh repo checkouts
16:36 < mmcgrath> warren: in our infrastructure I'm fine with that.
16:36 < warren> Before we add +10 test addresses, let us think about how test* addresses are to be used.
16:36 < dgilmore> warren: so do I,  we should test the migration of guests between the hosts
16:36 < warren> dgilmore, I think we would need SAN for that?
16:36 < warren> one option...
16:36 < abadger1999> f13: So make the second IP external right from the start?
16:37 < mmcgrath> warren: actually it would be easier in our xen environment if we used files instead of partitions.
16:37 < mmcgrath> Small performance hit, but not much.
16:37 < mmcgrath> then we're an scp away.l
16:37 < warren> test1-20 have both an internal and external IP address, with port 80, 443, 8888 and 8889 forwarded by default.
16:37 < dgilmore> warren:  test ip's  should be throwaway systems
16:37 < warren> They are to be used for TEST purposes until we are ready to launch a production service.  At that point we request a new internal and external IP assignment and new names.
16:37 < mmcgrath> dgilmore: +1
16:38 < warren> Then change the IP address of the test guest to the new address.
16:38 < f13> abadger1999: I"d thin kso yes.
16:38 < dgilmore> warren: if  we know  we are going to have a production system  why not request it from the start
16:39 < dgilmore> like we did with smtp
16:39 < warren> *but*... is 80 and 443 sufficient to allow users to test mercurial?
16:39 < dgilmore> we know we want db2  request it
16:40 < warren> dgilmore, does db2 require external IP and port forwards?  (i'm guessing no)
16:40 -!- tibbs [n=tibbs@fedora/tibbs]  has quit [Remote closed the connection] 
16:40 < dgilmore> warren: no it shouldn't
16:40 < mmcgrath> uhh, what happened to hammer1?
16:40 < warren> ??
16:40 < dgilmore> hammer1  was ok last i checked
16:40 < abadger1999> warren: I think port 80 and 443 are good defaults.  mercurial and bazaar would need 22 as well.
16:40 < warren> ssh wtogami@hammer1.fedora.phx.redhat.com
16:40 < warren> ssh_exchange_identification: Connection closed by remote host
16:40 < mmcgrath> try it now.
16:41 < warren> hmm... what happened indeed
16:41 < dgilmore> thats the first sign i got on hammer3
16:41 < warren> abadger1999, would that port 22 be a different sshd than the one used to login to the system itself?
16:41 < abadger1999> Since commit access needs to work over ssh.
16:42 < warren> dgilmore, maybe the hammer's are committing suicide at the same time...
16:42 < abadger1999> warren: Yes.
16:42 < mmcgrath> should we run ssh on a nonstandard port?
16:42 < dgilmore> warren: i hope not
16:42 < warren> OK, I suppose this is fine for now.
16:43 < dgilmore> maybe the warranty expired on them last week
16:43 < warren> hehe
16:43 < mmcgrath> well at least we have the xen boxes.
16:43 < f13> oh man.
16:43 < warren> hammer2 is OK so far?
16:43 < mmcgrath> Ok, so honestly I don't have anything else required for this meeting.  its been a long, long week.
16:43 < mmcgrath> Anyone else have it.
16:43 < mmcgrath> err anyone else have anything?
16:43 < f13> looking at how dist-cvs is setup, since EVERYTHING is the same repo, conversion is going to be a BITCH
16:43 < lmacken> yeah
16:44 < lmacken> just a quick status update on the updates system
16:44 < f13> that is conversion with keeping any kind of history
16:44 < lmacken> I did a little bit of hacking on the updates system this week (not as much as I would have liked, due to exams), and I hope to have a bunch of code checked into CVS in the very near future.
16:44 < lmacken> I also updated the Infrastructure/UpdatesSystem page with screenshots of the current system (RH internal), an ideal workflow design (comments/suggestions are encouraged), and where to get the new code.  Once I get a solid codebase and some unit tests committed, we should have plenty of tasks to go around.
16:44 < mmcgrath> f13: yep.
16:44 < lmacken> and I'll hopefully have our test6 xen guest up and running as our TG application server soon as well.
16:44 < nman64> nagios is reporting the problem on hammer1, but not on hammer3.
16:44 -!- BobJensen is now known as BobJensen-Away
16:44 < abadger1999> f13: I was thinking about that a bit.
16:44 < mmcgrath> nman64: we acknowledged hammer 3 last week sometime.
16:44 < abadger1999> Does it help if we just save the tagged history?
16:44 < mmcgrath> lmacken: thats cool.
16:45 < abadger1999> Plus whatever's in the head revision
16:45 < warren> I should request port 22 to be opened to test1 now?
16:45 < f13> abadger1999: possibly.
16:46 < dgilmore> hammer1  looks to be ok
16:47 < f13> hrm, let me try something...
16:47 < mmcgrath> dgilmore: echo | nc hammer1 22
16:47 < mmcgrath> says different :D
16:47 < abadger1999> warren: and test2.  If you want all prototypes in a week and a half.
16:47 < dgilmore> mmcgrath: well hardware wise
16:48 < dgilmore> [root@hammer1 ~] # /etc/init.d/sshd restart
16:48 < dgilmore> Stopping sshd: [  OK  ] 
16:48 < dgilmore> Starting sshd: Privilege separation user sshd does not exist
16:48 < dgilmore> [FAILED] 
16:48 < warren> Hmm... inconsistent firewall policy between test1/2 and the other test boxes.
16:48  * iWolf wanders back in
16:48 < warren> dgilmore, that's special!
16:48 < mmcgrath> snmp is responding.
16:48 < dgilmore> warren: yeah  i just saw selinux logs of it
16:48 < warren> some FC5 update broke it?
16:48 < mmcgrath> dgilmore: did a package build do that by chance?
16:48 < mmcgrath> err, check /etc/passwd /etc/shadow
16:49 < warren> i'm not able to ssh into that box at all
16:50 < mmcgrath> dgilmore: you on the console?
16:50 < dgilmore> mmcgrath: yeah i am
16:50 < mmcgrath> grep ssh /etc/passwd
16:50 < mmcgrath> ?
16:51 < warren> no ssh in /etc/passwd
16:51 < warren> i'm on KVM now
16:51  * lmacken has to head out for a bit
16:51  * lmacken &
16:51 < dgilmore> [root@hammer1 ~] # grep ssh /etc/passwd
16:51 < dgilmore> [root@hammer1 ~] #
16:51 < abadger1999> warren: Are firewall configs for xen guests checked into fedora-config?
16:52 < mmcgrath> dgilmore: less /etc/passwd find out why its not in there.
16:52 < lmacken> abadger1999: there are no pyroman configs for xen guest yet.  they should be pretty trivial to make
though
16:52 < warren> abadger1999, AFAIK no configs of xen guests are checked into fedora-config
16:52 < warren> fedora-config itself does not match our servers too well
16:52 < warren> one of the things we need to do is re-explore how we will handle config management
16:53 < warren> and talk to skvidal about his awesome system
16:53 < lmacken> yeah, dist-conf is pretty painful
16:53 < mmcgrath> yeah, skvidal is trying to get his system gpl'd.
16:53 < mmcgrath> warren: its not 'awesome' its simple :D
16:53 < abadger1999> warren: I made a start on using fedora-config for the xenguests
16:53 < warren> ah
16:54 < warren> hmm.... according to yum.log the last time something changed on the buildhost of hammer1 was september 24th
16:54 < warren> it is a little worrisome that something somehow broke sshd in this way
16:54 < warren> *HOW* did this happen?
16:54 < abadger1999> So I could send changes to all the relevant guests instead of individually configuring.
16:54 < mmcgrath> dgilmore: ls -l /etc/passwd
16:55 < dgilmore> -rw-r--r-- 1 root root 637 Jul  6 01:58 /etc/passwd
16:55 < warren> that might be the default passwd as shipped in the setup package?
16:55 < warren> Perhaps somehow mock screwed up and modified stuff in the / ?
16:55 < mmcgrath> wow.
16:56 < mmcgrath> dgilmore: there should be a sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
16:57 < warren> openssh-server is not even installed here!
16:57 < f13> abadger1999: ok, I'm able to convert a single module (rpms/yum) to an HG repo
16:57 < f13> abadger1999: now, how this all sorts out is goign to be... interesting.
16:57 < dgilmore> ok  ive reinstalled sshd  its up now
16:57 < warren> gotta go to my next meeting, bbl
16:58 < mmcgrath> later warren:
16:58 < mmcgrath> does anyone else have anything for the meeting or should we just get to hammer?
16:58 < warren> hammer the hammer
16:58 < dgilmore> mmcgrath: just the hammer
16:58 < dgilmore> mmcgrath: meeting done
16:58 < iWolf> mmcgrath: I have nothing else.  I can grab the logs later and email them to the lsit.
16:59 < abadger1999> f13: Great!  Do you have that scripted?
17:00 < mmcgrath> ------------------- MEETING END -----------------------