From Fedora Project Wiki
Fedora Release Engineering Meeting :: Monday 2008-08-25
Schedule
- Propose three week slip to schedule to FESCo
- http://poelstra.fedorapeople.org/schedules/f-10/f-10-three-week-change.html
- Key dates:
- Beta Freeze/Feature Freeze 2008-09-09
- GA 2008-11-18
Package Signing & Go Forward Proposal
- See IRC log below
- Proposal for FESCo: http://lists.fedoraproject.org/pipermail/rel-eng/2008-August/001622.html
IRC Transcript
f13 | ping: notting jeremy lmacken warren jwb rdieter wwoods spot | 10:18 |
---|---|---|
* lmacken | 10:18 | |
* rdieter | 10:18 | |
* rdieter . | 10:18 | |
* spot .. | 10:19 | |
* warren . | 10:19 | |
notting | sorry, back | 10:21 |
* poelcat here | 10:21 | |
f13 | ok. | 10:21 |
f13 | poelcat: oh sorry, that's who I forgot to ping (: | 10:21 |
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - rawhide | 10:21 | |
f13 | so lets start with the 800lb gorilla. | 10:22 |
f13 | We obviously haven't had a rawhide in a while, and I'm working to fix that. | 10:22 |
f13 | Provided all of Fedora leadership is on board with putting new bits out, we should at least have trees of packages for rawhide tonight | 10:22 |
f13 | with any luck, I'll get the "make it installable" part done too. | 10:22 |
f13 | that's primarly what I'm working on today | 10:23 |
warren | why wouldn't we be onboard? | 10:24 |
notting | onboard, waterboard, close enough | 10:24 |
warren | check | 10:24 |
f13 | warren: I can't imagine why not, but since I've been afk for the past few days I'm not sure where we stand with starting to put out new content. | 10:25 |
f13 | Unless told otherwise, I plan to let mash do as it has always done, and take the highest signed version of the rpms to put out. | 10:26 |
notting | i'm not sure it's been formally discussed | 10:26 |
f13 | we have no reason to expect our existing signed content to be "bad" | 10:26 |
f13 | we're going to create new keys before signing anything else new | 10:26 |
warren | f13: what is the plan for a signing host? | 10:26 |
notting | are we? | 10:26 |
notting | f13: you get a chicken/egg with updates if you do that | 10:27 |
f13 | notting: this is know AFAIK. The fedora project lead sent me an email saying that we would be retiring the existing keys | 10:27 |
f13 | warren: complicated, and not fully drafted. The proposals thus far involve a xen guest with no listening daemons, accessable only via xm console on the xen server, that has read access to the koji nfs for the purpose of signing, and nothing else. | 10:28 |
f13 | warren: that's the short term. Longer term I have openpgp smartcards finally in route to me to experiment with as a storage source for our future keys | 10:28 |
f13 | and sigul as signing server software | 10:29 |
warren | perhaps I shouldn't ask where signing happens in the short-term then | 10:29 |
warren | is sigul released as OSS? | 10:29 |
notting | f13: last i heard was issuing a new signing key, didn't here where it fit in the timeline | 10:29 |
f13 | it'll be much more secure (even in the short term) than it was before. | 10:29 |
warren | f13: and the procedure of issuing that new signing key | 10:29 |
f13 | warren: sigul is not exactly written yet, and being written from the get go as OSS | 10:29 |
f13 | notting: k, I was under the assumption that new key was a blocking item to pushing any future Fedora updates to 8/9 | 10:30 |
lmacken | so what's stopping us from regenerating new keys, and resuming where we left off ? | 10:30 |
warren | f13: the procedure including the user experience. I think there is no avoiding users needing to manually import it right? | 10:30 |
notting | lmacken: need to get a way to get it to the users sanely. whether that's a one-off fedora-release update signed with old key (with manual fingerprints we can post) that refers to and includes the new key, i'm not sure | 10:31 |
f13 | lmacken: getting those new keys onto people's systems | 10:31 |
f13 | warren: that has been 'vague' in the messaging. | 10:32 |
f13 | warren: but the possibility of manual action was made known in Paul's public messages | 10:32 |
warren | f13: sure, but I haven't seen discussion of executing on that | 10:32 |
f13 | neither have I, efforts have been focused elsewhere | 10:33 |
f13 | from releng side, all we need to do is get sign1 into the desired state, generate a new set of keys (test/gold), and adjust existing signing software and mash to expect the new keys | 10:33 |
f13 | as to how those keys will be delivered to users, that's another discussion, perhaps at the FESCo level | 10:34 |
warren | anything I can do to prepare the current signing environment or the user experience for key transition? | 10:34 |
warren | do we really want to wait until later this week to have a new key? | 10:34 |
f13 | warren: if by "later this week" you mean tomorrow.. yes | 10:34 |
warren | what is so special about tomorrow? | 10:35 |
warren | I would really think the key is rel-eng's responsibiltiy? | 10:35 |
f13 | warren: I'm not entirely sure of what all work is necessary, until we get the key generated and start trying ot use it. | 10:35 |
notting | f13: opinions on my meta-proposal above? | 10:35 |
lmacken | then it sounds like we need to generate it, and start trying to use it. | 10:35 |
f13 | warren: tomorrow being the day after today, where today is the day I'm spending getting rawhide out the door and installable, since it doesn't block on a new signing key. | 10:35 |
f13 | notting: it's not a terrible idea. | 10:35 |
warren | notting: let's just do it then. | 10:36 |
warren | who will generate the key? | 10:36 |
lmacken | other than a fedora-release one-off, do we have any other options for getting the new keys out ? | 10:36 |
f13 | notting: put the details of the package on a https site with an actually verifiable ssl cert | 10:36 |
f13 | lmacken: warren: so work on getting a xen guest (el5) up without any external access, and somewhere around 100gigs of local storage, and readonly access to /mnt/koji, and git installed. | 10:36 |
warren | I have a fresh installed RHEL-5.2 box here to generate a key in known safety. | 10:36 |
lmacken | f13: i rebuilt sign1 last week | 10:37 |
f13 | lmacken: is it in the above state? | 10:37 |
notting | heck, i was debating generating the key on a non-FAS box and then uploading it | 10:37 |
lmacken | f13: "no external access" meaning, xen console only... I don't know about the /mnt/koji stuff | 10:37 |
f13 | notting: I was just going to generate it on a throwaway guest and then upload it. | 10:37 |
f13 | and burn it to physical media | 10:37 |
notting | heh | 10:37 |
lmacken | f13: I just rebuilt it, ran yum update, and turned it off... so getting it to your mentioned state should be trivial | 10:38 |
warren | I don't care where or who generates it, just that it happens, and why not today? | 10:38 |
warren | notting's meta-proposal is fine, and we could do that today. | 10:38 |
notting | well, do we even know that bodhi works with the rebuilt infrastructure push-wise, just using the old keys? :) | 10:38 |
f13 | right | 10:38 |
lmacken | notting: nada | 10:39 |
f13 | having a new key doesn't help unless we know that bodhi is operational | 10:39 |
lmacken | i'm going to trying mashing a bunch today | 10:39 |
lmacken | but it should Just Work... even better than before | 10:39 |
lmacken | in theory | 10:39 |
warren | does a new xen guest have enough entropy to generate a key? | 10:39 |
f13 | lmacken: which will be fun as I try to do rawhide composes :/ | 10:39 |
f13 | warren: why wouldn't it? | 10:39 |
notting | gpg will yell if it doesn't, and we can do something else. no biggie | 10:40 |
f13 | right | 10:40 |
warren | f13: because there is no keyboard or mouse to give it entropy, and network interrupts are not used anymore | 10:40 |
f13 | I think this seriously a non-issue. | 10:41 |
warren | I have serious issues generating enough entropy for 16 bytes on a fresh booted kvm machine, so I wonder how you can get several kilobytes of entropy on a xen guest without any devices that generate entropy. | 10:41 |
* poelcat wonders if we are going to have time to talk about the schedule? | 10:42 | |
f13 | warren: if we can't, we'll do something on the box to generate more entropy | 10:43 |
f13 | warren: this is the least of my concerns, can we move on? | 10:43 |
warren | nod | 10:43 |
f13 | ok. | 10:43 |
f13 | Provided we get rawhide rolling today (and I have many hours left in my version of "today"), and provided we get some form of signing working in the next day or so, we should think about Beta | 10:44 |
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - Beta | 10:44 | |
f13 | we're obviously not frozen, other than by infrastructure issues | 10:44 |
f13 | but we need to pick a new beta freeze target, and adjust out the rest of the schedule. | 10:44 |
poelcat | f13 did you see my mail? | 10:44 |
f13 | I think it's pretty darned impossible to work the delays of the last week or so into our existing end schedule. | 10:45 |
* poelcat created a couple of scenarios | 10:45 | |
f13 | poelcat: I think so | 10:45 |
* f13 checks | 10:45 | |
poelcat | i couldn't mail the list because it bounced | 10:45 |
f13 | ok | 10:45 |
poelcat | i proposed a freeze of 9/2 http://poelstra.fedorapeople.org/schedules/f-10/f-10-two-week-change.html | 10:46 |
f13 | I was just about to post that. | 10:46 |
poelcat | other scenarios was freeze on 9/9... found in same dir | 10:46 |
f13 | so one week of recovery time is pretty short IMHO | 10:47 |
notting | hm, i don't suppose there's a 'diff' version? | 10:47 |
poelcat | notting: yes, but you won't like the presentation :) | 10:47 |
poelcat | http://poelstra.fedorapeople.org/schedules/f-10/f-10-stacked.html | 10:47 |
warren | poelcat: assuming we get installable rawhide out for a few days before 9/2 it might be OK. | 10:47 |
poelcat | for each task it shows.. original, 2 week change, 3 week change | 10:47 |
f13 | warren: I woudln't bet on that. | 10:47 |
warren | f13: are we redoing the keys for testing as well? | 10:47 |
f13 | yes | 10:48 |
warren | k | 10:48 |
f13 | I think http://poelstra.fedorapeople.org/schedules/f-10/f-10-three-week-change.html is probably our most reasonable option | 10:48 |
f13 | although there is an error here showing GA dates on a Monday | 10:48 |
f13 | (instead of a Tuesday) | 10:48 |
poelcat | f13: that is the maint phase | 10:48 |
warren | 9/9 beta freeze? | 10:48 |
poelcat | f13: go up a little | 10:48 |
poelcat | i'm including maint phase now so we realize how long a release really is alive | 10:49 |
f13 | Beta Public Availability Mon 2008-09-22 Mon 2008-09-22 | 10:49 |
poelcat | what :( | 10:49 |
f13 | warren: yes, freeze on 9/9 | 10:50 |
poelcat | sorry, i missed that... i can fix it | 10:50 |
warren | I'm +1 on 9/9 beta freeze | 10:50 |
notting | it's a date. having a date is good. +1 | 10:50 |
warren | actually I would prefer something after 9/2 and before 9/9, but 9/9 is OK | 10:50 |
f13 | poelcat: your concern with the 9/9 scenario is that if we have to slip again, we slip into thanksgiving? | 10:51 |
poelcat | yes | 10:51 |
warren | is 9/5 impossible for beta freeze? | 10:51 |
f13 | ok, while that's a valid concern, I think we can deal with it then. | 10:51 |
f13 | freezing on a Friday is rather pointless | 10:51 |
f13 | and I'd rather stick to our standard days of the week for freezing. | 10:52 |
f13 | we had rawhide issues leading up to the shutdown as it was, we really need to get a week or so of working rawhide for pre-freeze testing | 10:52 |
f13 | so that we can fix up stuff before we freeze | 10:52 |
f13 | otherwise we're just going ot have a lengthy freeze period doing nobody any good | 10:52 |
warren | ok | 10:53 |
f13 | alright just for the record: Proposal, slip by 3 weeks and have a new beta freeze date of 9/9, adjust all later schedule dates to match (http://poelstra.fedorapeople.org/schedules/f-10/f-10-three-week-change.html) | 10:54 |
f13 | +1 | 10:54 |
f13 | warren: and notting already +1d | 10:54 |
f13 | spot: rdieter: lmacken: jwb: jeremy: ping | 10:54 |
rdieter | totally +1 | 10:55 |
lmacken | +1 | 10:55 |
* stickster interjects that the proposed slip means the week *after* the proposed date is Thanksgiving week, right? | 10:56 | |
stickster | sorry, s/date/final release date/ | 10:56 |
stickster | i.e. GA == 2008-11-18 | 10:57 |
f13 | stickster: that is correct. | 10:57 |
stickster | Nothing like a little pressure to induce our best work then :-) | 10:57 |
wwoods | sounds reasonable | 10:58 |
f13 | oh crap, yeah, wwoods you get a vote too | 10:58 |
rdieter | more reason to be thankful | 10:58 |
f13 | anyway, I don't see any real objections to this, so I'm going to stamp it as APPROVED. | 10:58 |
stickster | rdieter: heh | 10:58 |
f13 | I suppose one of us needs to drive this to FESCo | 10:59 |
f13 | and jwb doesn't seem to be on right now | 10:59 |
notting | i can bring it up | 10:59 |
poelcat | f13: can you double check the latest? i adjusted to make sure Tues is release day | 10:59 |
f13 | poelcat: that looks better | 10:59 |
wwoods | I'm sure none of us are exactly thrilled about slipping 3 full weeks but I don't really see any other sane plan | 11:00 |
f13 | poelcat: although I just noticed that the snapshot dates look funny | 11:00 |
f13 | the snapshot periods are for thu->fri, but the compose/stage/sync items just list fri->fri | 11:00 |
f13 | nothing is happening on a thursday under the snapshot items. | 11:00 |
f13 | wwoods: it also helps us catch up to the alpha delays | 11:01 |
poelcat | f13: i'll fix that too .. sorry about that | 11:01 |
f13 | no worries, there is a lot of data to cover | 11:02 |
f13 | well, that brings us to the hour mark | 11:03 |
f13 | given that we (hopefully, given FESCo approval) have a bit more breathing room before beta, we can concentrate more on repairing infrastructure. | 11:03 |
warren | OK, so is the new key and fedora-release signed by old key happening today? | 11:03 |
warren | Pushing that sooner before new updates will make it work automatically for more people. | 11:04 |
f13 | wwoods: The next qa meeting should focus on what can be done before beta freeze to prepare ourselves for a better freeze | 11:04 |
f13 | warren: need to get fesco involved before doing anything. | 11:04 |
f13 | warren: this isn't something releng can just decide and run with. | 11:05 |
warren | why? | 11:05 |
notting | in any case, i doubt we'll be in a position today | 11:05 |
f13 | warren: because fesco is the engineering steering, above releng. | 11:05 |
f13 | warren: we don't have carte blanc to just do whatever the hell we want to do | 11:05 |
warren | f13: I'll ask them to vote on the list ASAP | 11:05 |
f13 | particularly in things like this that greatly impact the user base. | 11:05 |
f13 | this is not something I want to just rush out in an afternoon and hope it's OK | 11:05 |
warren | I can't see why they would say no, but I'll ask anyway. | 11:05 |
f13 | warren: they probably want a more formulated plan | 11:06 |
warren | Specifically: 1) Generate new key. 2) Add it to fedora-release 3) push it as the only update signed by the old key | 11:06 |
warren | f13: who writes the new "more formulated plan"? | 11:06 |
f13 | warren: sounds like you just volunteered | 11:06 |
warren | well sure, but what beyond these 3 steps? | 11:06 |
nirik | so what happens to people who don't update that, and update down the road? fails right? | 11:07 |
warren | nirik: yes | 11:07 |
warren | nirik: no avoiding that | 11:07 |
warren | announcements need to go out | 11:07 |
f13 | nirik: yes, all future updates would fail to apply until they either A) install the new keys, or B) disable gpg checking | 11:07 |
nirik | and packagekit can import the new keys? | 11:07 |
warren | nope | 11:07 |
warren | and this isn't something you can safely do in %post | 11:07 |
warren | I think | 11:07 |
* nirik would like to see examples of these failures on a Q&A page we can point people at....as there are going to be tons of people hitting those issues. | 11:08 | |
warren | yes | 11:08 |
wwoods | wait. PK *can* import keys now | 11:08 |
warren | the current PK in F9 updates? | 11:08 |
f13 | warren: well, if they're trusting the package itself, there isn't ar eason to not trust the keys within | 11:08 |
wwoods | the problem is.. that requires an update to PK from F9 gold | 11:08 |
wwoods | and *that* update is signed with the old key | 11:08 |
f13 | warren: why don't you run with this issue today, from the end user aspect. | 11:08 |
warren | f13: we are forbidden to use rpm within %post | 11:09 |
warren | f13: I'm going to ask FESCO to vote on the initial 3 steps ASAP, then immediately thereafter work on the announcements and FAQ. | 11:09 |
f13 | warren: right, safely from a trust aspect, vs from a rpm within rpm aspect. | 11:09 |
nirik | so we are also revoking the old key, right? | 11:10 |
f13 | nirik: that's my understanding from stickster_food | 11:10 |
warren | nirik: if we do, then we have to resign and push all packages on the mirrors | 11:10 |
f13 | warren: I'm not sure how far down that rabbit hole we're going | 11:10 |
notting | ... which i'd like to avoid | 11:10 |
nirik | and how do we revoke it? remove it from their rpmdb? (horror) | 11:10 |
nirik | yeah, then do we resign gold too? | 11:11 |
f13 | nirik: we don't have a way to "revoke" it, we can just not use it anymore. | 11:11 |
nirik | f8 as well? ;( | 11:11 |
f13 | honestly I think we need to just use the new key for all future updates | 11:11 |
f13 | since all the printed media has the old signed content still on it, and hasn't been hacked. | 11:11 |
warren | f13: the entire point of a new key would be to revoke the old key, as the "caution" | 11:12 |
wwoods | so we're going to continue to accept packages signed with the old key? | 11:12 |
warren | f13: if the old key continues to allow installs, no point in doing a new key. | 11:12 |
f13 | warren: do you have any way to "revoke" the old key? | 11:12 |
warren | f13: Conflicts: gpg-whatever? | 11:12 |
notting | that's just wrong | 11:13 |
warren | seriously, if the old key allows package installs, what is the point of a new key? | 11:13 |
f13 | warren: new installs | 11:13 |
warren | users wont notice the difference between old signed packages and new signed packages with old key. | 11:13 |
f13 | warren: one of the suggestions made, that I like, is using a new key per release. | 11:13 |
warren | that isn't a bad idea. | 11:14 |
f13 | right, so new keys will accomplish that. | 11:14 |
f13 | the question is how far back to go with the new keys | 11:14 |
warren | So the "caution" of this new key is only to make F10 known safe | 11:14 |
f13 | and I don't know if this is the body to make that decision | 11:14 |
warren | while F8 and F9 are only "high confidence"? | 11:14 |
skvidal | so we're not resigning the whole shooting match? | 11:14 |
warren | skvidal: that's the question being asked now | 11:15 |
f13 | skvidal: to what end? | 11:15 |
skvidal | f8, f8+updates*, f9, f9-updates* | 11:15 |
f13 | if we resign all GOLD content out there, respin new isos, we would make new installs of F8/9 safer. | 11:15 |
skvidal | to the end of knowing that the content is what we intended | 11:15 |
skvidal | yes, and it is an effective revocation of the former key | 11:15 |
warren | We have "high confidence" that the key passphrases were not stolen | 11:15 |
f13 | we do not much for existing installs, other than breaking their update stream. | 11:15 |
skvidal | f13: coupled with a new fedora-release pkg | 11:16 |
warren | but if we don't revoke the old key, the new key isn't raising "high confidence" to "fully known confidence" | 11:16 |
f13 | it's still breaking their stream for not much added confidence, unless they manually remove the old key from their rpmdb | 11:16 |
skvidal | as notting mentioned to me earlier - do a one time old-key-sign of fedora-release | 11:16 |
notting | respin ISOs? owwwww. | 11:16 |
skvidal | f13: we don't have to manually do so | 11:16 |
f13 | yeah, iso respinning doesn't sound very fun to me. | 11:16 |
warren | is conflicts on the old key to force its removal not an option? | 11:16 |
notting | isos are already hashed for integrity | 11:16 |
skvidal | f13: we could yank the gpg key via other mechanisms | 11:17 |
f13 | skvidal: how are you going to automatically remove the old key? | 11:17 |
skvidal | warren: conflicts don't remove | 11:17 |
f13 | "other mechanisms" | 11:17 |
skvidal | f13: %post ? | 11:17 |
f13 | fail | 11:17 |
skvidal | why? | 11:17 |
f13 | skvidal: don't call rpm from within rpm | 11:17 |
wwoods | can't run rpm in %post | 11:17 |
skvidal | you can for gpg keys | 11:17 |
warren | skvidal: can we safely use rpm within rpm %post? we have rules against that. | 11:17 |
skvidal | try it | 11:17 |
warren | oh | 11:17 |
drago01 | yum-plugin | 11:17 |
rdieter | don't cross the streams | 11:17 |
skvidal | aiui gpg keys are in a separate db | 11:17 |
warren | wwoods: does the current F9 updates PK allow graphical GPG importing? | 11:17 |
notting | do we really need to worry about *revoking* the old key (remove from db, etc.) | 11:17 |
skvidal | we can nuke them w/o hosing the rpmdb for pkgs | 11:17 |
drago01 | warren: afaik yes | 11:18 |
skvidal | but ask panu to be sure | 11:18 |
warren | drago01: that's good then. | 11:18 |
skvidal | notting: yes | 11:18 |
f13 | notting: if we don't, what good does a new key do? | 11:18 |
skvidal | notting: if the old key is compromised | 11:18 |
skvidal | notting: then they can slip trojanned data onto a rogue mirror | 11:18 |
f13 | notting: if the old key is still in their rpmdb, yum/rpm will happily install new content signed with old key | 11:18 |
drago01 | adding a second key and leaving the old is imho a waste of time and effort | 11:18 |
warren | BTW, will rsync efficiently redo all packages on mirrors without downloading the entire thing again? | 11:18 |
f13 | the other option is to use the existing signed content we know to be OK and go to gpg signed repodata | 11:18 |
wwoods | warren: IIRC yes, and without the horrid rpm-transaction-restart baloney that we had for F9 Gold | 11:18 |
f13 | warren: probably not | 11:18 |
skvidal | warren: as long as it is either checksum or timestamp based, yes | 11:18 |
skvidal | f13: which requires an update to yum oob | 11:19 |
warren | where is the signature payload in the rpm file, beginning or end of file? | 11:19 |
nphilipp | skvidal, warren, notting: how about obsoletes? | 11:19 |
f13 | skvidal: we're already doing an update to fedora-release | 11:19 |
skvidal | f13: signed with the old key? | 11:19 |
notting | nphilipp: Obsoletes: gpg-pubkey? hmmm... | 11:19 |
skvidal | nphilipp: I don't know if that works on pubkeys | 11:19 |
skvidal | nphilipp: give it a test? | 11:19 |
warren | perhaps signing with the old key isn't a good idea, if we aim on revoking the old key | 11:20 |
nphilipp | skvidal, notting: forget it -- removes all keys | 11:20 |
warren | and people need to install it manually anyway | 11:20 |
notting | nphilipp: hahaha | 11:20 |
skvidal | nphilipp: version it | 11:20 |
skvidal | nphilipp: obsoletes gpg-pubkey-stamp-stamp | 11:20 |
* nphilipp didn't realize that the name is "gpg-pubkey", not "gpg-pubkey-<...>" | 11:20 | |
skvidal | rpm -qa gpg-pubkey | 11:20 |
skvidal | gpg-pubkey-4f2a6fd2-3f9d9d3b | 11:20 |
skvidal | versioned obsoletes | 11:20 |
* skvidal tests | 11:20 | |
warren | f13: signing with the old key actually wont work | 11:21 |
nphilipp | well versioned should work | 11:21 |
skvidal | nphilipp: let's fine out | 11:21 |
notting | does not work | 11:21 |
f13 | warren: pardon? | 11:21 |
skvidal | okay | 11:21 |
skvidal | so | 11:21 |
nphilipp | I'm really missing an easy way to find out what gpg-pubkey "package"§ corresponds to what key | 11:21 |
skvidal | obsoletes gpg-pubkey doesn't do anything | 11:21 |
warren | f13: it might help for the short period of time where that fedora-release is the ONLY update, but after that point it doesn't help us. | 11:21 |
skvidal | nphilipp: rpm -qi gpg-pubkey | 11:21 |
notting | unless i did it wrong | 11:22 |
warren | f13: we might as well sign it with the new key and make everyone install it manually. it is a lot more consistent that way. | 11:22 |
f13 | warren: this update would be put into a special location as well as the same location as the other updates. | 11:22 |
nphilipp | skvidal: argh didn't notice that | 11:22 |
skvidal | notting: argh | 11:22 |
skvidal | notting: hahah | 11:22 |
nphilipp | *tomatoes on the eyes* | 11:22 |
f13 | warren: but I agree, I don't see too much value to signing it with the old key vs the new key. | 11:22 |
skvidal | notting: so I did a global obsoletes gpg-pubkey | 11:22 |
skvidal | notting: and it makes rpm segfault :( | 11:22 |
f13 | wewt. | 11:22 |
nphilipp | ooh | 11:23 |
notting | skvidal: versioned does not work | 11:23 |
skvidal | nod | 11:23 |
skvidal | for me neither | 11:23 |
skvidal | so, as I was saying | 11:23 |
f13 | so we're back to rpm -e in %post | 11:23 |
warren | OK, so the 3 step initial plan for FESCO vote: 1) Generate new key. 2) New fedora-release containing that key, signed with the new key 3) Push that in updates, and send out announcements and FAQ. | 11:23 |
skvidal | I bet we can rpm -e in %post :) | 11:23 |
f13 | warren: that's decidedly lacking on details for FESCo to vote on | 11:23 |
nphilipp | why is it always me to come up with ways to break RPM? | 11:23 |
drago01 | warren: no that won't reach ALL users | 11:23 |
notting | warren: 2) was signed with the old key, but then new stuff is signed with new key, i thought | 11:23 |
f13 | notting: what good does that do? | 11:24 |
warren | notting: it doesn't help us to sign with the old key | 11:24 |
skvidal | yes it does | 11:24 |
f13 | "don't trust this key, but trust this update signed with the key we just told you not to trust" | 11:24 |
notting | f13: we can verify the package | 11:24 |
skvidal | if we issue the warning with checksums of the pkg | 11:24 |
warren | drago01: there is no avoiding manual importing | 11:24 |
skvidal | essentially telling people to run | 11:24 |
skvidal | yumdownloader fedora-release-ver-rel | 11:24 |
skvidal | then | 11:24 |
skvidal | sha1sum fedora-release-ver-rel | 11:24 |
skvidal | then yum update /path/to/that/file | 11:25 |
f13 | if they're going to do that, why not have it signed with the new key? | 11:25 |
warren | skvidal: not everyone has yumdownloader by default | 11:25 |
skvidal | and that pkg nukes the old key, if it can | 11:25 |
skvidal | warren: yes they do | 11:25 |
skvidal | warren: yum-utils is in base | 11:25 |
warren | oh, didn't realize that | 11:25 |
f13 | does yumdownloader do gpg checking? | 11:25 |
skvidal | f13: b/c they cannot install it without yum --nogpgcheck | 11:25 |
nirik | is that also the case for f8? | 11:25 |
drago01 | warren: a new fedora-release will ask the users to import after update .. so everyone can see it but a news "please update your keys" wont | 11:25 |
f13 | skvidal: <obvious wanking> not everybody has (all of) base </> | 11:25 |
notting | meh. too bad we can't tie keys -> repos -> packages directly | 11:25 |
skvidal | f13: don't give a shit | 11:26 |
skvidal | f13: those people are smart enough to figure out their own crap, then :) | 11:26 |
skvidal | f13: hell, we can tell them to run urlgrabber someurl, if you're really worried | 11:26 |
f13 | skvidal: why make people use yum install instead of just rpm -Uvh ? | 11:26 |
warren | drago01: no no... updates will fail to install entirely (if you're talking about automatic) because there will be updates in addition to the fedora-release package. | 11:26 |
skvidal | f13: b/c I like yum? :) | 11:26 |
nirik | if they are doing manual stuff, wouldn't it be easier to: rpm -e oldgpgkey ; rpm -ivh https://newgpgkey; yum update? | 11:26 |
f13 | skvidal: that way it can be signed with the new key and we don't have to dick with the old key. | 11:26 |
f13 | skvidal: so no other reason? | 11:26 |
drago01 | warren: ouch ... FAIL | 11:27 |
skvidal | b/c I think the more things we do to keep people away from the rpm cli the better off we are | 11:27 |
warren | drago01: yes, thus there is no avoiding manually installing the new fedora-release. | 11:27 |
f13 | .... | 11:27 |
f13 | this still doesn't answer how far back we go. | 11:28 |
drago01 | do we really need a new key? | 11:28 |
f13 | drago01: we need a new key for F10 at least. | 11:28 |
nirik | f10 would be much easier... | 11:28 |
warren | drago01: if we want to go from "high confidence" to "full confidence" then yes. | 11:28 |
f13 | (and should be doing successive new keys for each new release) | 11:28 |
drago01 | f13: that does not worry me ... but for f9 it will be pain | 11:28 |
warren | How high is our confidence? | 11:28 |
warren | Is this rekeying being done for mostly paranoia reasons? | 11:29 |
drago01 | and for the future we should have some infrastructure to allow us to change keys | 11:29 |
drago01 | without asking the users "run this commands" | 11:29 |
nirik | warren: if the private key was copied off by intruder, they might be able to brute force the passphrase... | 11:29 |
rdieter | another option, do a new key, but do *not* revoke the old one, until there's stronger reason to do so? (or just plain don't do a new one for f9, like warren...) | 11:30 |
warren | I'm saying, if we do a new key, there is no point if we don't revoke the old one. | 11:30 |
f13 | so this isn't the group of people to decide this. | 11:30 |
rdieter | bleh, that's icky, ignore me. | 11:30 |
f13 | warren: again you're wrong. | 11:30 |
f13 | warren: if we do a new key it's extremely valueable for F10 | 11:31 |
warren | Is FESCO really more versed than us in the workings of the rel-eng systems? | 11:31 |
warren | f13: sure, no doubt about F10 | 11:31 |
notting | nirik: brute force is not likely | 11:31 |
warren | f13: but we're talking about F8 and F9. | 11:31 |
f13 | warren: this is more than a technical decision | 11:31 |
rdieter | rel-eng can't provide a recommendation, but f13's right, it's ultimately fesco's call | 11:31 |
warren | can't? | 11:31 |
f13 | s/can't/can/ | 11:31 |
rdieter | can , duh. :) | 11:31 |
warren | sure, but what is our recommendation? | 11:31 |
f13 | go for a bike ride. | 11:32 |
warren | If we are sufficiently not confident in the F8 and F9 key, then we need to revoke it. | 11:32 |
drago01 | new key for F10 (and each future release) and leave f8/9 until we really need it | 11:32 |
warren | We shouldn't leave this meeting without a recommendation and a plan for FESCO to vote on. | 11:32 |
f13 | warren: that's a tough question to answer without having all the details, and not many people here do. | 11:32 |
f13 | drago01: "until we really need it" sounds like after we discover somebody putting out bad software with our key, which means we just gambled on our users' saftey and lost. | 11:33 |
f13 | drago01: not a very good scenario. | 11:33 |
warren | f13: +1 | 11:33 |
rdieter | ask stickster ? :) | 11:33 |
skvidal | warren: I can assure you of this | 11:34 |
skvidal | warren: we | 11:34 |
skvidal | 're replacing the key | 11:34 |
drago01 | f13: well that would be indeed to late ..."need it" = "until we have any kind of evidence that somebody has the passphrase or could have got it" | 11:34 |
f13 | In my opinion, we should stop using the old key. period. | 11:34 |
warren | OK, if we're clear on replacing the key, then we need to resign the entire repo and revoke it. | 11:34 |
warren | And there is no avoiding users installing a new key manually. | 11:34 |
drago01 | f13: ok, but there is another question too not only if we want to do it but HOW? | 11:34 |
skvidal | this is the official word on what we said we would do | 11:35 |
f13 | drago01: that's the ongoing debate. | 11:35 |
skvidal | "While there is no definitive evidence that the Fedora key has been | 11:35 |
skvidal | compromised, because Fedora packages are distributed via multiple | 11:35 |
skvidal | third-party mirrors and repositories, we have decided to convert to new | 11:35 |
skvidal | Fedora signing keys. This may require affirmative steps from every | 11:35 |
skvidal | Fedora system owner or administrator. We will widely and clearly | 11:35 |
skvidal | communicate any such steps to help users when available. | 11:35 |
skvidal | " | 11:35 |
skvidal | we need to stick to that | 11:35 |
rdieter | there ya go. | 11:35 |
* stickster reads back in buffer | 11:35 | |
f13 | and we need a higher body to determine how far back we go | 11:35 |
skvidal | stickster: I pasted directly from the public announcement | 11:35 |
warren | skvidal: for the future might we consider a separate master key locked away from any server that might allow automatic key revocation and migration? | 11:35 |
f13 | existing updates, GOLD packages, old distros, isos, etc... | 11:35 |
stickster | skvidal: *nod | 11:36 |
skvidal | warren: rekeying for EVERY release buys us more, imo | 11:36 |
skvidal | then we can discard the ancient | 11:36 |
warren | skvidal: nod | 11:36 |
stickster | skvidal: compartmentalization++ | 11:36 |
warren | no disagreement | 11:36 |
smooge | we would discard after each release or when one went EOL? | 11:37 |
smooge | sorry for jumping in | 11:37 |
skvidal | smooge: EOL | 11:37 |
f13 | smooge: a key would be valid for the entire life of a release, which ends at EOL | 11:37 |
f13 | we just simply don't use it anymore | 11:37 |
f13 | (man, updates are going to be fun when we're signing with 3 different keys) | 11:38 |
smooge | hmmm I wonder if we could use a trail of signing keys. F10 signs F9 signs F8 signs F7.... | 11:38 |
f13 | smooge: rpm doesn't grok that at all | 11:38 |
f13 | smooge: otherwise we'd just use new subkeys from a master key each release | 11:39 |
f13 | (master key would exist only to issue new subkeys) | 11:39 |
smooge | well we should use new RPM :). | 11:39 |
drago01 | skvidal: what does yum do if one repo has badly/unsigned packages and the other one has signed ones? will the entire transaction fail? | 11:39 |
f13 | but we're off on a tangent now and not getting any where close to a decision. | 11:39 |
smooge | anyway.. back to conversation thansk for the clarification | 11:39 |
warren | we can discuss the master key idea later. | 11:39 |
warren | OK, we have agreement that we are redoing the key, which means resigning everything in the repos and pushing to mirrors. | 11:40 |
skvidal | drago01: any package failing a signature check fails the whole process | 11:40 |
warren | That is the recommendation we give to FESCO? | 11:40 |
f13 | so the F10 stuff is easy, we just make the rawhide fedora-release reference the new key. | 11:40 |
drago01 | skvidal: hmm ok | 11:40 |
notting | s/new/f10/ | 11:40 |
f13 | yes, the f10 key | 11:40 |
f13 | it's really teh F9 and F8 stuff that's the problem. | 11:40 |
f13 | if we revoke the old key as part of a fedora-release update, we have to resign everything that is sitting in the releases/8/ and releases/9/ directoies | 11:41 |
f13 | directories. | 11:41 |
f13 | which makes the content not match what's in the isos | 11:41 |
warren | right | 11:41 |
drago01 | and how do we get the users to import the new key? | 11:41 |
warren | So let's make that our recommendation? | 11:41 |
warren | drago01: there is NO AVOIDING users doing it manually | 11:42 |
warren | unless somebody has a brilliant idea | 11:42 |
notting | f13: and the isos are in the wild and burned.... | 11:42 |
f13 | are we all OK with leaving the isos alone, since they're validated in different ways (sha1sums), and they don't import the key anyway? | 11:42 |
f13 | (let alone check the key) | 11:43 |
drago01 | well some users might not get to read this mail/faq and end up with no updates / no way to install packages ... which isn't fun.... | 11:43 |
notting | f13: livecds do. ;) | 11:43 |
warren | f13: only drawback there is users of those iso's have to manually import the new key. | 11:43 |
smooge | how long until F8 EOL? | 11:43 |
warren | f13: but I might be OK with that. | 11:43 |
skvidal | notting: we don't actually need to if we have signed sha1sums | 11:43 |
warren | f13: currently scheduled for late December | 11:43 |
skvidal | hmm | 11:43 |
f13 | warren: if you install from iso but enable updates, you'll get the new fedora-release | 11:43 |
skvidal | depending on what they're signed with :) | 11:43 |
warren | f13: which you need to install manualy | 11:43 |
f13 | warren: no, we'd also put the fedora-release in the updates/ tree | 11:43 |
f13 | just like all other updates we do to the distro | 11:44 |
nirik | but it's signed with the new key they don't have? | 11:44 |
f13 | anaconda "wins" by not doing gpg checking | 11:44 |
warren | f13: if there are multiple updates, none of them will install automatically | 11:44 |
warren | f13: haha | 11:44 |
f13 | so anaconda will happily install the latest fedora-release which has the new key content and sets up the user for any further updates. | 11:44 |
warren | huh, anaconda will install updates by default? | 11:45 |
f13 | warren: not by default. | 11:45 |
drago01 | no... | 11:45 |
f13 | warren: you simply add the updates repo at install time. | 11:45 |
f13 | F9 anaconda had a checkbox for this IIRC | 11:45 |
warren | ok, F9 install with network is good | 11:45 |
f13 | only works well if you're doing some kind of network install, non-iso install | 11:45 |
drago01 | and btw this is bug | 11:46 |
f13 | drago01: define "this" | 11:46 |
drago01 | installing packages without checking the key | 11:46 |
warren | So our recommendation to FESCO: 1) Generate new key 2) Resign all F8, F9 and update packages in repos, leave ISO's alone. 3) Announce new key and instructions for users because most people will need to upgrade manually. | 11:46 |
f13 | drago01: that's bug 998 to be exact. | 11:46 |
warren | ? | 11:46 |
buggbot | Bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=998 medium, high, ---, clumens@redhat.com, ASSIGNED, FTP/NFS install/upgrade is unsafe, should check GPG signatures. | 11:46 |
drago01 | f13: ;) | 11:46 |
f13 | well, if we're doing keys per release, wouldn't 1 be generate F8 keys, F9 keys, and F10 keys ? | 11:47 |
skvidal | f13: we don't have to do a key per release for the past | 11:47 |
warren | do we really need a separate key from F8 and F9? | 11:47 |
notting | f13: oh feh. koji. | 11:47 |
skvidal | f13: only going forward | 11:47 |
f13 | notting: what about koji? | 11:47 |
notting | f13: well, we need to generate all the signed stuff in koji | 11:47 |
notting | too | 11:48 |
f13 | how else were we going to do it? | 11:48 |
f13 | rpmsign --resign directly on the fedora netapp? no. | 11:48 |
notting | yeah, just realizing the level of the pain | 11:49 |
warren | Recommendation to FESCO: 1) Generate new key for F8 and F9, and a separate key for testing. 2) Resign all F8, F9, update and testing packages in repos, leave ISO's alone. 3) Announce new key and instructions for users because most people will need to upgrade manually. | 11:49 |
f13 | regarding koji, for dist-f8+ we can garbage collect all the invalid signed copies. | 11:49 |
f13 | notting: just a looooooong screen session and a lot of pasting. | 11:49 |
f13 | and probably 3 weeks to get it all done. | 11:50 |
f13 | if not longer | 11:50 |
warren | 3 weeks? | 11:51 |
warren | to resign? | 11:51 |
drago01 | well 3+ weeks with no updates ..... | 11:51 |
f13 | although in theory you could just resign directly on the netapp, then import from the netapp into koji | 11:51 |
f13 | warren: it takes a long time to copy every package out of koji, sign them, import them back into koji, and ask koji to write out a version of the package with the new key header. | 11:51 |
warren | f13: "resign directly on the netapp, then import from the netapp into koji" that sounds good | 11:52 |
f13 | warren: especially given command argument lengths and the need to paste in the key passphrase every few minutes. | 11:52 |
f13 | warren: where good == extremely risky | 11:52 |
f13 | and no good way to stage it unless we turn off rsync for a day or so | 11:52 |
warren | hm | 11:52 |
warren | ok, so what do we actually want to do? | 11:52 |
f13 | or else you're going to get rpms that don't match the metadata. | 11:52 |
f13 | warren: I'd prefer to do it via koji, and stage the change on the master mirror. it's just going to take a rather long time. | 11:53 |
f13 | we can do each release in stages though, instead of everything at once. | 11:53 |
f13 | (if that "helps" at all) | 11:53 |
warren | so we begin using the new key, and revoke the old key only after a few weeks? | 11:54 |
warren | (this still wont avoid the need for folks to manually install the new key) | 11:54 |
f13 | that's one possibility | 11:54 |
f13 | start using the new key(s) immediately, stage the replacement of old signed data as new content becomes available. | 11:55 |
f13 | (this feels like a FESCo discussion / decision) | 11:55 |
f13 | lets start with your proposal warren, as a high level, and drill into details at FESCo | 11:55 |
warren | Recommendation to FESCO: 1) Generate new key for F8 and F9, and a separate key for testing. 2) Announce new key and instructions for users because most people will need to upgrade manually. 3) Begin using the new key to sign packages immediately to push updates. 4) Begin resigning all F8, F9, update and testing packages in repos, leave ISO's alone. Replace content on mirrors over time. 5) In a few weeks revoke the old key when all packa | 11:56 |
warren | ges are resigned and old key packages are eliminated from mirrors. 6) Generate a new key for F10. | 11:56 |
warren | ? | 11:57 |
f13 | erm. I'm not comfortable with recommending 4 without a fesco discussion. | 11:57 |
warren | Uh... we're giving this recommendation | 11:57 |
warren | we need to recommend something | 11:57 |
f13 | we need to recommend what actions need to be taken | 11:57 |
warren | They have the option of modifying the recommendations. | 11:58 |
f13 | whatever. | 11:58 |
warren | I'll draft up a longer written version of this including reasons | 11:58 |
warren | (reason: signing everything will take weeks) | 11:58 |
mbonnet | How do we revoke a key that already exists in people's rpmdb? | 11:58 |
warren | mbonnet: skvidal is testing if rpm -e gpgkey is safe during %post | 11:59 |
mbonnet | warren: doing that removes people's ability to verify packages that they already have installed | 11:59 |
skvidal | mbonnet: you can't verify an installed pkg | 11:59 |
skvidal | mbonnet: rpm -Va doesn't use the gpg sig | 11:59 |
mbonnet | ok | 11:59 |
drago01 | warren: can we do this: "while signing the new packages give instructions to users how to update to the new key and let livna/freshrpm etc ship the key in theire release package to reach more users" ? | 11:59 |
mbonnet | does resigning all existing F8/F9 content include spinning new isos? | 12:00 |
drago01 | I know part 2 sounds stupid but it might help | 12:00 |
skvidal | mbonnet: read up | 12:00 |
warren | drago01: livna/freshrpms wont help because the entire transaction will fail | 12:00 |
wwoods | the PK on the F9 media has trouble importing keys | 12:00 |
f13 | mbonnet: short answer, no. | 12:01 |
drago01 | warren: how so ? if we wait until the new key is used (signing done) so the users have 3 weeks time to get the new key | 12:01 |
warren | drago01: we are not going to wait between issuing new key and using it to sign packages. | 12:01 |
f13 | drago01: 3 weeks is only if we wait to push anything with the new key until /all/ content is replaced. | 12:01 |
* skvidal has a bad feeling | 12:02 | |
warren | about? | 12:02 |
f13 | warren: that's a rather decisive statement to make without having a FESCo vote. | 12:02 |
skvidal | rpm -e gpg-pubkey | 12:02 |
warren | f13: no, you misunderstand | 12:02 |
drago01 | f13: I thought that was the plan | 12:03 |
f13 | skvidal: that takes 'em all doesn't it? | 12:03 |
warren | f13: he's suggesting that everyone will update fedora-release signed by existing keys before packages signed by that key are pushed. | 12:03 |
skvidal | f13: not about takes them all | 12:03 |
skvidal | f13: atm it is locks the transaction | 12:03 |
f13 | warren: he's suggesting that we use livna repos which use livna's keys to get our new key into place | 12:03 |
f13 | skvidal: haha, fail. | 12:03 |
warren | f13: which wont work if the update transaction has packages signed by the new key | 12:03 |
f13 | *sigh* | 12:04 |
warren | am I wrong? | 12:04 |
f13 | warren: IF WE DON"T PUSH UPDATES UNTIL ALL CONTENT IS SWITCHED | 12:04 |
warren | f13: but that isn't our recommendation right? | 12:04 |
f13 | that gives a period of time that users may get updates from livna, before any updates from us | 12:04 |
warren | you're also assuming everyone is using livna | 12:04 |
drago01 | f13: exactly | 12:04 |
f13 | warren: as I stated before, I'm not comfortable making a recommendation on the timeline of events. | 12:04 |
f13 | warren: no I'm not. | 12:04 |
f13 | warren: neither is drago01 | 12:04 |
warren | f13: if we are going with that plan, then we might as well push fedora-release signed with old key for the 3 weeks of resigning. | 12:05 |
f13 | it'll get it onto some people's systems. | 12:05 |
drago01 | warren: no bu it will reach a big part of our users .. better than "go read a faq" | 12:05 |
f13 | Can we please get out of this rabbit hole? | 12:05 |
warren | If we do not discuss this, how are we going to climb out? | 12:06 |
warren | We have an uncomfortable situation sure | 12:06 |
warren | but we need to give SOME recommendation | 12:06 |
f13 | drago01: I appreciate the suggestion (I understand it), but its not going to work for everybody, and would only hwork for a short period of time. I think it would just confuse users more to have a longer FAQ about it | 12:06 |
skvidal | okay | 12:06 |
f13 | "If you use livna and updated betwen X and X, you may not have to do any of this..." | 12:06 |
skvidal | well rpm -e gpg-pubkey-anything will not work | 12:06 |
skvidal | in %post | 12:06 |
f13 | awesome. | 12:06 |
skvidal | yah | 12:06 |
f13 | well, our "recommendation" doesn't list /how/ we would do things. | 12:07 |
mbonnet | skvidal: ok, push a new fedora-release that installs a cron job that'll remove the old fedora keys and install the new one? | 12:07 |
dgilmore | skvidal: what about a shell script that does it? | 12:07 |
f13 | cron, the answer to everything. | 12:07 |
skvidal | dgilmore: ? | 12:07 |
mbonnet | f13: :-P | 12:07 |
f13 | ok, guys, NOT IMPORTANT at this moment. | 12:07 |
skvidal | mbonnet: I don't think we can count on cron | 12:07 |
skvidal | ... | 12:07 |
warren | what is not important? | 12:07 |
f13 | we can drill into details once we get a FESCo mandate to make it happen. | 12:07 |
dgilmore | skvidal: for the rpm -e in %post | 12:08 |
warren | FESCO needs a recommendation | 12:08 |
drago01 | f13: reimporing the key wont hurt (so if the key is already imported and they read the faq nothing will break right?) ... and no it won't work for everyone but it would make it easier for xx% of our users | 12:08 |
warren | FESCO needs all the facts | 12:08 |
dgilmore | skvidal: just an idea. no clue if it could workl | 12:08 |
f13 | warren: which we obviously can't make right now because WE DONT HAVE A FUCKING WAY THAT WORKS | 12:08 |
warren | "We could do this, we can't do this because of foo." | 12:08 |
* skvidal suggests that the process be manual | 12:08 | |
warren | skvidal: +1 | 12:08 |
skvidal | I don't think we have much choice | 12:09 |
warren | oh wait, the revoking part? | 12:09 |
skvidal | all of it | 12:09 |
dgilmore | skvidal: i think it should be manual | 12:09 |
skvidal | we cannot revoke the key | 12:09 |
warren | f13: stating that might be true, but we have to work on it anyway. | 12:09 |
skvidal | we've got no mechanism to do that | 12:09 |
* nirik suggests someone ping Panu and see if there is some clever way we don't know, otherwise manual and make sure it's very clear what to do and why. | 12:09 | |
f13 | warren: then why don't you go and find a way that works, and then come back here later to talk about working options. | 12:09 |
f13 | warren: because right now we're just arguing over theories, without any actual facts. not helpful. | 12:09 |
warren | It seems that we have some facts from the discussion | 12:10 |
f13 | warren: we also have only you and I left from releng even participating in the discussion | 12:10 |
f13 | warren: not much of a quorum for a releng recommendation. | 12:11 |
warren | which means we've failed at our job? | 12:11 |
sharkcz | what about adding the key removal into rc.local? it will be delayed until next reboot ... | 12:11 |
f13 | warren: no it means we're beyond our meeting time. Lets work on this outside of the meeting time. | 12:11 |
notting | rc.local can't be modifed reliably | 12:11 |
f13 | warren: once we have some things that work, we can bring the people back and get a vote. There is just no reason to continue doing this as part of our meeting time. | 12:12 |
* f13 goes away to a real life meeting. | 12:12 | |
drago01_ | f13, warren :so what did rel-eng agree on? (sorry wireless died) | 12:13 |
warren | drago01_: nothing | 12:14 |
drago01_ | ok | 12:14 |
warren | I'll write up everything we have, but I have to make some phone calls first. | 12:14 |
f13 | drago01_: one assumption we had, rpm -e the gpg key would work, proved to not work. warren wants to have a very detailed recommendation to fesco, which would mean actually testing what we want to do, so I asked him to actually do that and bring releng folks back to vote on it once we know our recommendation would actually work. | 12:15 |
drago01_ | f13: ok, thx for the summary | 12:16 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!