From Fedora Project Wiki

Fedora Release Engineering Meeting :: Monday 2008-08-25

Schedule

Package Signing & Go Forward Proposal

IRC Transcript

f13 ping: notting jeremy lmacken warren jwb rdieter wwoods spot 10:18
* lmacken 10:18
* rdieter 10:18
* rdieter . 10:18
* spot .. 10:19
* warren . 10:19
notting sorry, back 10:21
* poelcat here 10:21
f13 ok. 10:21
f13 poelcat: oh sorry, that's who I forgot to ping (: 10:21
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - rawhide 10:21
f13 so lets start with the 800lb gorilla. 10:22
f13 We obviously haven't had a rawhide in a while, and I'm working to fix that. 10:22
f13 Provided all of Fedora leadership is on board with putting new bits out, we should at least have trees of packages for rawhide tonight 10:22
f13 with any luck, I'll get the "make it installable" part done too. 10:22
f13 that's primarly what I'm working on today 10:23
warren why wouldn't we be onboard? 10:24
notting onboard, waterboard, close enough 10:24
warren check 10:24
f13 warren: I can't imagine why not, but since I've been afk for the past few days I'm not sure where we stand with starting to put out new content. 10:25
f13 Unless told otherwise, I plan to let mash do as it has always done, and take the highest signed version of the rpms to put out. 10:26
notting i'm not sure it's been formally discussed 10:26
f13 we have no reason to expect our existing signed content to be "bad" 10:26
f13 we're going to create new keys before signing anything else new 10:26
warren f13: what is the plan for a signing host? 10:26
notting are we? 10:26
notting f13: you get a chicken/egg with updates if you do that 10:27
f13 notting: this is know AFAIK. The fedora project lead sent me an email saying that we would be retiring the existing keys 10:27
f13 warren: complicated, and not fully drafted. The proposals thus far involve a xen guest with no listening daemons, accessable only via xm console on the xen server, that has read access to the koji nfs for the purpose of signing, and nothing else. 10:28
f13 warren: that's the short term. Longer term I have openpgp smartcards finally in route to me to experiment with as a storage source for our future keys 10:28
f13 and sigul as signing server software 10:29
warren perhaps I shouldn't ask where signing happens in the short-term then 10:29
warren is sigul released as OSS? 10:29
notting f13: last i heard was issuing a new signing key, didn't here where it fit in the timeline 10:29
f13 it'll be much more secure (even in the short term) than it was before. 10:29
warren f13: and the procedure of issuing that new signing key 10:29
f13 warren: sigul is not exactly written yet, and being written from the get go as OSS 10:29
f13 notting: k, I was under the assumption that new key was a blocking item to pushing any future Fedora updates to 8/9 10:30
lmacken so what's stopping us from regenerating new keys, and resuming where we left off ? 10:30
warren f13: the procedure including the user experience. I think there is no avoiding users needing to manually import it right? 10:30
notting lmacken: need to get a way to get it to the users sanely. whether that's a one-off fedora-release update signed with old key (with manual fingerprints we can post) that refers to and includes the new key, i'm not sure 10:31
f13 lmacken: getting those new keys onto people's systems 10:31
f13 warren: that has been 'vague' in the messaging. 10:32
f13 warren: but the possibility of manual action was made known in Paul's public messages 10:32
warren f13: sure, but I haven't seen discussion of executing on that 10:32
f13 neither have I, efforts have been focused elsewhere 10:33
f13 from releng side, all we need to do is get sign1 into the desired state, generate a new set of keys (test/gold), and adjust existing signing software and mash to expect the new keys 10:33
f13 as to how those keys will be delivered to users, that's another discussion, perhaps at the FESCo level 10:34
warren anything I can do to prepare the current signing environment or the user experience for key transition? 10:34
warren do we really want to wait until later this week to have a new key? 10:34
f13 warren: if by "later this week" you mean tomorrow.. yes 10:34
warren what is so special about tomorrow? 10:35
warren I would really think the key is rel-eng's responsibiltiy? 10:35
f13 warren: I'm not entirely sure of what all work is necessary, until we get the key generated and start trying ot use it. 10:35
notting f13: opinions on my meta-proposal above? 10:35
lmacken then it sounds like we need to generate it, and start trying to use it. 10:35
f13 warren: tomorrow being the day after today, where today is the day I'm spending getting rawhide out the door and installable, since it doesn't block on a new signing key. 10:35
f13 notting: it's not a terrible idea. 10:35
warren notting: let's just do it then. 10:36
warren who will generate the key? 10:36
lmacken other than a fedora-release one-off, do we have any other options for getting the new keys out ? 10:36
f13 notting: put the details of the package on a https site with an actually verifiable ssl cert 10:36
f13 lmacken: warren: so work on getting a xen guest (el5) up without any external access, and somewhere around 100gigs of local storage, and readonly access to /mnt/koji, and git installed. 10:36
warren I have a fresh installed RHEL-5.2 box here to generate a key in known safety. 10:36
lmacken f13: i rebuilt sign1 last week 10:37
f13 lmacken: is it in the above state? 10:37
notting heck, i was debating generating the key on a non-FAS box and then uploading it 10:37
lmacken f13: "no external access" meaning, xen console only... I don't know about the /mnt/koji stuff 10:37
f13 notting: I was just going to generate it on a throwaway guest and then upload it. 10:37
f13 and burn it to physical media 10:37
notting heh 10:37
lmacken f13: I just rebuilt it, ran yum update, and turned it off... so getting it to your mentioned state should be trivial 10:38
warren I don't care where or who generates it, just that it happens, and why not today? 10:38
warren notting's meta-proposal is fine, and we could do that today. 10:38
notting well, do we even know that bodhi works with the rebuilt infrastructure push-wise, just using the old keys? :) 10:38
f13 right 10:38
lmacken notting: nada 10:39
f13 having a new key doesn't help unless we know that bodhi is operational 10:39
lmacken i'm going to trying mashing a bunch today 10:39
lmacken but it should Just Work... even better than before 10:39
lmacken in theory 10:39
warren does a new xen guest have enough entropy to generate a key? 10:39
f13 lmacken: which will be fun as I try to do rawhide composes :/ 10:39
f13 warren: why wouldn't it? 10:39
notting gpg will yell if it doesn't, and we can do something else. no biggie 10:40
f13 right 10:40
warren f13: because there is no keyboard or mouse to give it entropy, and network interrupts are not used anymore 10:40
f13 I think this seriously a non-issue. 10:41
warren I have serious issues generating enough entropy for 16 bytes on a fresh booted kvm machine, so I wonder how you can get several kilobytes of entropy on a xen guest without any devices that generate entropy. 10:41
* poelcat wonders if we are going to have time to talk about the schedule? 10:42
f13 warren: if we can't, we'll do something on the box to generate more entropy 10:43
f13 warren: this is the least of my concerns, can we move on? 10:43
warren nod 10:43
f13 ok. 10:43
f13 Provided we get rawhide rolling today (and I have many hours left in my version of "today"), and provided we get some form of signing working in the next day or so, we should think about Beta 10:44
-!- f13 changed the topic of #fedora-meeting to: Fedora releng - Beta 10:44
f13 we're obviously not frozen, other than by infrastructure issues 10:44
f13 but we need to pick a new beta freeze target, and adjust out the rest of the schedule. 10:44
poelcat f13 did you see my mail? 10:44
f13 I think it's pretty darned impossible to work the delays of the last week or so into our existing end schedule. 10:45
* poelcat created a couple of scenarios 10:45
f13 poelcat: I think so 10:45
* f13 checks 10:45
poelcat i couldn't mail the list because it bounced 10:45
f13 ok 10:45
poelcat i proposed a freeze of 9/2 http://poelstra.fedorapeople.org/schedules/f-10/f-10-two-week-change.html 10:46
f13 I was just about to post that. 10:46
poelcat other scenarios was freeze on 9/9... found in same dir 10:46
f13 so one week of recovery time is pretty short IMHO 10:47
notting hm, i don't suppose there's a 'diff' version? 10:47
poelcat notting: yes, but you won't like the presentation :) 10:47
poelcat http://poelstra.fedorapeople.org/schedules/f-10/f-10-stacked.html 10:47
warren poelcat: assuming we get installable rawhide out for a few days before 9/2 it might be OK. 10:47
poelcat for each task it shows.. original, 2 week change, 3 week change 10:47
f13 warren: I woudln't bet on that. 10:47
warren f13: are we redoing the keys for testing as well? 10:47
f13 yes 10:48
warren k 10:48
f13 I think http://poelstra.fedorapeople.org/schedules/f-10/f-10-three-week-change.html is probably our most reasonable option 10:48
f13 although there is an error here showing GA dates on a Monday 10:48
f13 (instead of a Tuesday) 10:48
poelcat f13: that is the maint phase 10:48
warren 9/9 beta freeze? 10:48
poelcat f13: go up a little 10:48
poelcat i'm including maint phase now so we realize how long a release really is alive 10:49
f13 Beta Public Availability Mon 2008-09-22 Mon 2008-09-22 10:49
poelcat what :( 10:49
f13 warren: yes, freeze on 9/9 10:50
poelcat sorry, i missed that... i can fix it 10:50
warren I'm +1 on 9/9 beta freeze 10:50
notting it's a date. having a date is good. +1 10:50
warren actually I would prefer something after 9/2 and before 9/9, but 9/9 is OK 10:50
f13 poelcat: your concern with the 9/9 scenario is that if we have to slip again, we slip into thanksgiving? 10:51
poelcat yes 10:51
warren is 9/5 impossible for beta freeze? 10:51
f13 ok, while that's a valid concern, I think we can deal with it then. 10:51
f13 freezing on a Friday is rather pointless 10:51
f13 and I'd rather stick to our standard days of the week for freezing. 10:52
f13 we had rawhide issues leading up to the shutdown as it was, we really need to get a week or so of working rawhide for pre-freeze testing 10:52
f13 so that we can fix up stuff before we freeze 10:52
f13 otherwise we're just going ot have a lengthy freeze period doing nobody any good 10:52
warren ok 10:53
f13 alright just for the record: Proposal, slip by 3 weeks and have a new beta freeze date of 9/9, adjust all later schedule dates to match (http://poelstra.fedorapeople.org/schedules/f-10/f-10-three-week-change.html) 10:54
f13 +1 10:54
f13 warren: and notting already +1d 10:54
f13 spot: rdieter: lmacken: jwb: jeremy: ping 10:54
rdieter totally +1 10:55
lmacken +1 10:55
* stickster interjects that the proposed slip means the week *after* the proposed date is Thanksgiving week, right? 10:56
stickster sorry, s/date/final release date/ 10:56
stickster i.e. GA == 2008-11-18 10:57
f13 stickster: that is correct. 10:57
stickster Nothing like a little pressure to induce our best work then :-) 10:57
wwoods sounds reasonable 10:58
f13 oh crap, yeah, wwoods you get a vote too 10:58
rdieter more reason to be thankful 10:58
f13 anyway, I don't see any real objections to this, so I'm going to stamp it as APPROVED. 10:58
stickster rdieter: heh 10:58
f13 I suppose one of us needs to drive this to FESCo 10:59
f13 and jwb doesn't seem to be on right now 10:59
notting i can bring it up 10:59
poelcat f13: can you double check the latest? i adjusted to make sure Tues is release day 10:59
f13 poelcat: that looks better 10:59
wwoods I'm sure none of us are exactly thrilled about slipping 3 full weeks but I don't really see any other sane plan 11:00
f13 poelcat: although I just noticed that the snapshot dates look funny 11:00
f13 the snapshot periods are for thu->fri, but the compose/stage/sync items just list fri->fri 11:00
f13 nothing is happening on a thursday under the snapshot items. 11:00
f13 wwoods: it also helps us catch up to the alpha delays 11:01
poelcat f13: i'll fix that too .. sorry about that 11:01
f13 no worries, there is a lot of data to cover 11:02
f13 well, that brings us to the hour mark 11:03
f13 given that we (hopefully, given FESCo approval) have a bit more breathing room before beta, we can concentrate more on repairing infrastructure. 11:03
warren OK, so is the new key and fedora-release signed by old key happening today? 11:03
warren Pushing that sooner before new updates will make it work automatically for more people. 11:04
f13 wwoods: The next qa meeting should focus on what can be done before beta freeze to prepare ourselves for a better freeze 11:04
f13 warren: need to get fesco involved before doing anything. 11:04
f13 warren: this isn't something releng can just decide and run with. 11:05
warren why? 11:05
notting in any case, i doubt we'll be in a position today 11:05
f13 warren: because fesco is the engineering steering, above releng. 11:05
f13 warren: we don't have carte blanc to just do whatever the hell we want to do 11:05
warren f13: I'll ask them to vote on the list ASAP 11:05
f13 particularly in things like this that greatly impact the user base. 11:05
f13 this is not something I want to just rush out in an afternoon and hope it's OK 11:05
warren I can't see why they would say no, but I'll ask anyway. 11:05
f13 warren: they probably want a more formulated plan 11:06
warren Specifically: 1) Generate new key. 2) Add it to fedora-release 3) push it as the only update signed by the old key 11:06
warren f13: who writes the new "more formulated plan"? 11:06
f13 warren: sounds like you just volunteered 11:06
warren well sure, but what beyond these 3 steps? 11:06
nirik so what happens to people who don't update that, and update down the road? fails right? 11:07
warren nirik: yes 11:07
warren nirik: no avoiding that 11:07
warren announcements need to go out 11:07
f13 nirik: yes, all future updates would fail to apply until they either A) install the new keys, or B) disable gpg checking 11:07
nirik and packagekit can import the new keys? 11:07
warren nope 11:07
warren and this isn't something you can safely do in %post 11:07
warren I think 11:07
* nirik would like to see examples of these failures on a Q&A page we can point people at....as there are going to be tons of people hitting those issues. 11:08
warren yes 11:08
wwoods wait. PK *can* import keys now 11:08
warren the current PK in F9 updates? 11:08
f13 warren: well, if they're trusting the package itself, there isn't ar eason to not trust the keys within 11:08
wwoods the problem is.. that requires an update to PK from F9 gold 11:08
wwoods and *that* update is signed with the old key 11:08
f13 warren: why don't you run with this issue today, from the end user aspect. 11:08
warren f13: we are forbidden to use rpm within %post 11:09
warren f13: I'm going to ask FESCO to vote on the initial 3 steps ASAP, then immediately thereafter work on the announcements and FAQ. 11:09
f13 warren: right, safely from a trust aspect, vs from a rpm within rpm aspect. 11:09
nirik so we are also revoking the old key, right? 11:10
f13 nirik: that's my understanding from stickster_food 11:10
warren nirik: if we do, then we have to resign and push all packages on the mirrors 11:10
f13 warren: I'm not sure how far down that rabbit hole we're going 11:10
notting ... which i'd like to avoid 11:10
nirik and how do we revoke it? remove it from their rpmdb? (horror) 11:10
nirik yeah, then do we resign gold too? 11:11
f13 nirik: we don't have a way to "revoke" it, we can just not use it anymore. 11:11
nirik f8 as well? ;( 11:11
f13 honestly I think we need to just use the new key for all future updates 11:11
f13 since all the printed media has the old signed content still on it, and hasn't been hacked. 11:11
warren f13: the entire point of a new key would be to revoke the old key, as the "caution" 11:12
wwoods so we're going to continue to accept packages signed with the old key? 11:12
warren f13: if the old key continues to allow installs, no point in doing a new key. 11:12
f13 warren: do you have any way to "revoke" the old key? 11:12
warren f13: Conflicts: gpg-whatever? 11:12
notting that's just wrong 11:13
warren seriously, if the old key allows package installs, what is the point of a new key? 11:13
f13 warren: new installs 11:13
warren users wont notice the difference between old signed packages and new signed packages with old key. 11:13
f13 warren: one of the suggestions made, that I like, is using a new key per release. 11:13
warren that isn't a bad idea. 11:14
f13 right, so new keys will accomplish that. 11:14
f13 the question is how far back to go with the new keys 11:14
warren So the "caution" of this new key is only to make F10 known safe 11:14
f13 and I don't know if this is the body to make that decision 11:14
warren while F8 and F9 are only "high confidence"? 11:14
skvidal so we're not resigning the whole shooting match? 11:14
warren skvidal: that's the question being asked now 11:15
f13 skvidal: to what end? 11:15
skvidal f8, f8+updates*, f9, f9-updates* 11:15
f13 if we resign all GOLD content out there, respin new isos, we would make new installs of F8/9 safer. 11:15
skvidal to the end of knowing that the content is what we intended 11:15
skvidal yes, and it is an effective revocation of the former key 11:15
warren We have "high confidence" that the key passphrases were not stolen 11:15
f13 we do not much for existing installs, other than breaking their update stream. 11:15
skvidal f13: coupled with a new fedora-release pkg 11:16
warren but if we don't revoke the old key, the new key isn't raising "high confidence" to "fully known confidence" 11:16
f13 it's still breaking their stream for not much added confidence, unless they manually remove the old key from their rpmdb 11:16
skvidal as notting mentioned to me earlier - do a one time old-key-sign of fedora-release 11:16
notting respin ISOs? owwwww. 11:16
skvidal f13: we don't have to manually do so 11:16
f13 yeah, iso respinning doesn't sound very fun to me. 11:16
warren is conflicts on the old key to force its removal not an option? 11:16
notting isos are already hashed for integrity 11:16
skvidal f13: we could yank the gpg key via other mechanisms 11:17
f13 skvidal: how are you going to automatically remove the old key? 11:17
skvidal warren: conflicts don't remove 11:17
f13 "other mechanisms" 11:17
skvidal f13: %post ? 11:17
f13 fail 11:17
skvidal why? 11:17
f13 skvidal: don't call rpm from within rpm 11:17
wwoods can't run rpm in %post 11:17
skvidal you can for gpg keys 11:17
warren skvidal: can we safely use rpm within rpm %post? we have rules against that. 11:17
skvidal try it 11:17
warren oh 11:17
drago01 yum-plugin 11:17
rdieter don't cross the streams 11:17
skvidal aiui gpg keys are in a separate db 11:17
warren wwoods: does the current F9 updates PK allow graphical GPG importing? 11:17
notting do we really need to worry about *revoking* the old key (remove from db, etc.) 11:17
skvidal we can nuke them w/o hosing the rpmdb for pkgs 11:17
drago01 warren: afaik yes 11:18
skvidal but ask panu to be sure 11:18
warren drago01: that's good then. 11:18
skvidal notting: yes 11:18
f13 notting: if we don't, what good does a new key do? 11:18
skvidal notting: if the old key is compromised 11:18
skvidal notting: then they can slip trojanned data onto a rogue mirror 11:18
f13 notting: if the old key is still in their rpmdb, yum/rpm will happily install new content signed with old key 11:18
drago01 adding a second key and leaving the old is imho a waste of time and effort 11:18
warren BTW, will rsync efficiently redo all packages on mirrors without downloading the entire thing again? 11:18
f13 the other option is to use the existing signed content we know to be OK and go to gpg signed repodata 11:18
wwoods warren: IIRC yes, and without the horrid rpm-transaction-restart baloney that we had for F9 Gold 11:18
f13 warren: probably not 11:18
skvidal warren: as long as it is either checksum or timestamp based, yes 11:18
skvidal f13: which requires an update to yum oob 11:19
warren where is the signature payload in the rpm file, beginning or end of file? 11:19
nphilipp skvidal, warren, notting: how about obsoletes? 11:19
f13 skvidal: we're already doing an update to fedora-release 11:19
skvidal f13: signed with the old key? 11:19
notting nphilipp: Obsoletes: gpg-pubkey? hmmm... 11:19
skvidal nphilipp: I don't know if that works on pubkeys 11:19
skvidal nphilipp: give it a test? 11:19
warren perhaps signing with the old key isn't a good idea, if we aim on revoking the old key 11:20
nphilipp skvidal, notting: forget it -- removes all keys 11:20
warren and people need to install it manually anyway 11:20
notting nphilipp: hahaha 11:20
skvidal nphilipp: version it 11:20
skvidal nphilipp: obsoletes gpg-pubkey-stamp-stamp 11:20
* nphilipp didn't realize that the name is "gpg-pubkey", not "gpg-pubkey-<...>" 11:20
skvidal rpm -qa gpg-pubkey 11:20
skvidal gpg-pubkey-4f2a6fd2-3f9d9d3b 11:20
skvidal versioned obsoletes 11:20
* skvidal tests 11:20
warren f13: signing with the old key actually wont work 11:21
nphilipp well versioned should work 11:21
skvidal nphilipp: let's fine out 11:21
notting does not work 11:21
f13 warren: pardon? 11:21
skvidal okay 11:21
skvidal so 11:21
nphilipp I'm really missing an easy way to find out what gpg-pubkey "package"§ corresponds to what key 11:21
skvidal obsoletes gpg-pubkey doesn't do anything 11:21
warren f13: it might help for the short period of time where that fedora-release is the ONLY update, but after that point it doesn't help us. 11:21
skvidal nphilipp: rpm -qi gpg-pubkey 11:21
notting unless i did it wrong 11:22
warren f13: we might as well sign it with the new key and make everyone install it manually. it is a lot more consistent that way. 11:22
f13 warren: this update would be put into a special location as well as the same location as the other updates. 11:22
nphilipp skvidal: argh didn't notice that 11:22
skvidal notting: argh 11:22
skvidal notting: hahah 11:22
nphilipp *tomatoes on the eyes* 11:22
f13 warren: but I agree, I don't see too much value to signing it with the old key vs the new key. 11:22
skvidal notting: so I did a global obsoletes gpg-pubkey 11:22
skvidal notting: and it makes rpm segfault :( 11:22
f13 wewt. 11:22
nphilipp ooh 11:23
notting skvidal: versioned does not work 11:23
skvidal nod 11:23
skvidal for me neither 11:23
skvidal so, as I was saying 11:23
f13 so we're back to rpm -e in %post 11:23
warren OK, so the 3 step initial plan for FESCO vote: 1) Generate new key. 2) New fedora-release containing that key, signed with the new key 3) Push that in updates, and send out announcements and FAQ. 11:23
skvidal I bet we can rpm -e in %post :) 11:23
f13 warren: that's decidedly lacking on details for FESCo to vote on 11:23
nphilipp why is it always me to come up with ways to break RPM? 11:23
drago01 warren: no that won't reach ALL users 11:23
notting warren: 2) was signed with the old key, but then new stuff is signed with new key, i thought 11:23
f13 notting: what good does that do? 11:24
warren notting: it doesn't help us to sign with the old key 11:24
skvidal yes it does 11:24
f13 "don't trust this key, but trust this update signed with the key we just told you not to trust" 11:24
notting f13: we can verify the package 11:24
skvidal if we issue the warning with checksums of the pkg 11:24
warren drago01: there is no avoiding manual importing 11:24
skvidal essentially telling people to run 11:24
skvidal yumdownloader fedora-release-ver-rel 11:24
skvidal then 11:24
skvidal sha1sum fedora-release-ver-rel 11:24
skvidal then yum update /path/to/that/file 11:25
f13 if they're going to do that, why not have it signed with the new key? 11:25
warren skvidal: not everyone has yumdownloader by default 11:25
skvidal and that pkg nukes the old key, if it can 11:25
skvidal warren: yes they do 11:25
skvidal warren: yum-utils is in base 11:25
warren oh, didn't realize that 11:25
f13 does yumdownloader do gpg checking? 11:25
skvidal f13: b/c they cannot install it without yum --nogpgcheck 11:25
nirik is that also the case for f8? 11:25
drago01 warren: a new fedora-release will ask the users to import after update .. so everyone can see it but a news "please update your keys" wont 11:25
f13 skvidal: <obvious wanking> not everybody has (all of) base </> 11:25
notting meh. too bad we can't tie keys -> repos -> packages directly 11:25
skvidal f13: don't give a shit 11:26
skvidal f13: those people are smart enough to figure out their own crap, then :) 11:26
skvidal f13: hell, we can tell them to run urlgrabber someurl, if you're really worried 11:26
f13 skvidal: why make people use yum install instead of just rpm -Uvh ? 11:26
warren drago01: no no... updates will fail to install entirely (if you're talking about automatic) because there will be updates in addition to the fedora-release package. 11:26
skvidal f13: b/c I like yum? :) 11:26
nirik if they are doing manual stuff, wouldn't it be easier to: rpm -e oldgpgkey ; rpm -ivh https://newgpgkey; yum update? 11:26
f13 skvidal: that way it can be signed with the new key and we don't have to dick with the old key. 11:26
f13 skvidal: so no other reason? 11:26
drago01 warren: ouch ... FAIL 11:27
skvidal b/c I think the more things we do to keep people away from the rpm cli the better off we are 11:27
warren drago01: yes, thus there is no avoiding manually installing the new fedora-release. 11:27
f13 .... 11:27
f13 this still doesn't answer how far back we go. 11:28
drago01 do we really need a new key? 11:28
f13 drago01: we need a new key for F10 at least. 11:28
nirik f10 would be much easier... 11:28
warren drago01: if we want to go from "high confidence" to "full confidence" then yes. 11:28
f13 (and should be doing successive new keys for each new release) 11:28
drago01 f13: that does not worry me ... but for f9 it will be pain 11:28
warren How high is our confidence? 11:28
warren Is this rekeying being done for mostly paranoia reasons? 11:29
drago01 and for the future we should have some infrastructure to allow us to change keys 11:29
drago01 without asking the users "run this commands" 11:29
nirik warren: if the private key was copied off by intruder, they might be able to brute force the passphrase... 11:29
rdieter another option, do a new key, but do *not* revoke the old one, until there's stronger reason to do so? (or just plain don't do a new one for f9, like warren...) 11:30
warren I'm saying, if we do a new key, there is no point if we don't revoke the old one. 11:30
f13 so this isn't the group of people to decide this. 11:30
rdieter bleh, that's icky, ignore me. 11:30
f13 warren: again you're wrong. 11:30
f13 warren: if we do a new key it's extremely valueable for F10 11:31
warren Is FESCO really more versed than us in the workings of the rel-eng systems? 11:31
warren f13: sure, no doubt about F10 11:31
notting nirik: brute force is not likely 11:31
warren f13: but we're talking about F8 and F9. 11:31
f13 warren: this is more than a technical decision 11:31
rdieter rel-eng can't provide a recommendation, but f13's right, it's ultimately fesco's call 11:31
warren can't? 11:31
f13 s/can't/can/ 11:31
rdieter can , duh. :) 11:31
warren sure, but what is our recommendation? 11:31
f13 go for a bike ride. 11:32
warren If we are sufficiently not confident in the F8 and F9 key, then we need to revoke it. 11:32
drago01 new key for F10 (and each future release) and leave f8/9 until we really need it 11:32
warren We shouldn't leave this meeting without a recommendation and a plan for FESCO to vote on. 11:32
f13 warren: that's a tough question to answer without having all the details, and not many people here do. 11:32
f13 drago01: "until we really need it" sounds like after we discover somebody putting out bad software with our key, which means we just gambled on our users' saftey and lost. 11:33
f13 drago01: not a very good scenario. 11:33
warren f13: +1 11:33
rdieter ask stickster ?  :) 11:33
skvidal warren: I can assure you of this 11:34
skvidal warren: we 11:34
skvidal 're replacing the key 11:34
drago01 f13: well that would be indeed to late ..."need it" = "until we have any kind of evidence that somebody has the passphrase or could have got it" 11:34
f13 In my opinion, we should stop using the old key. period. 11:34
warren OK, if we're clear on replacing the key, then we need to resign the entire repo and revoke it. 11:34
warren And there is no avoiding users installing a new key manually. 11:34
drago01 f13: ok, but there is another question too not only if we want to do it but HOW? 11:34
skvidal this is the official word on what we said we would do 11:35
f13 drago01: that's the ongoing debate. 11:35
skvidal "While there is no definitive evidence that the Fedora key has been 11:35
skvidal compromised, because Fedora packages are distributed via multiple 11:35
skvidal third-party mirrors and repositories, we have decided to convert to new 11:35
skvidal Fedora signing keys. This may require affirmative steps from every 11:35
skvidal Fedora system owner or administrator. We will widely and clearly 11:35
skvidal communicate any such steps to help users when available. 11:35
skvidal " 11:35
skvidal we need to stick to that 11:35
rdieter there ya go. 11:35
* stickster reads back in buffer 11:35
f13 and we need a higher body to determine how far back we go 11:35
skvidal stickster: I pasted directly from the public announcement 11:35
warren skvidal: for the future might we consider a separate master key locked away from any server that might allow automatic key revocation and migration? 11:35
f13 existing updates, GOLD packages, old distros, isos, etc... 11:35
stickster skvidal: *nod 11:36
skvidal warren: rekeying for EVERY release buys us more, imo 11:36
skvidal then we can discard the ancient 11:36
warren skvidal: nod 11:36
stickster skvidal: compartmentalization++ 11:36
warren no disagreement 11:36
smooge we would discard after each release or when one went EOL? 11:37
smooge sorry for jumping in 11:37
skvidal smooge: EOL 11:37
f13 smooge: a key would be valid for the entire life of a release, which ends at EOL 11:37
f13 we just simply don't use it anymore 11:37
f13 (man, updates are going to be fun when we're signing with 3 different keys) 11:38
smooge hmmm I wonder if we could use a trail of signing keys. F10 signs F9 signs F8 signs F7.... 11:38
f13 smooge: rpm doesn't grok that at all 11:38
f13 smooge: otherwise we'd just use new subkeys from a master key each release 11:39
f13 (master key would exist only to issue new subkeys) 11:39
smooge well we should use new RPM :). 11:39
drago01 skvidal: what does yum do if one repo has badly/unsigned packages and the other one has signed ones? will the entire transaction fail? 11:39
f13 but we're off on a tangent now and not getting any where close to a decision. 11:39
smooge anyway.. back to conversation thansk for the clarification 11:39
warren we can discuss the master key idea later. 11:39
warren OK, we have agreement that we are redoing the key, which means resigning everything in the repos and pushing to mirrors. 11:40
skvidal drago01: any package failing a signature check fails the whole process 11:40
warren That is the recommendation we give to FESCO? 11:40
f13 so the F10 stuff is easy, we just make the rawhide fedora-release reference the new key. 11:40
drago01 skvidal: hmm ok 11:40
notting s/new/f10/ 11:40
f13 yes, the f10 key 11:40
f13 it's really teh F9 and F8 stuff that's the problem. 11:40
f13 if we revoke the old key as part of a fedora-release update, we have to resign everything that is sitting in the releases/8/ and releases/9/ directoies 11:41
f13 directories. 11:41
f13 which makes the content not match what's in the isos 11:41
warren right 11:41
drago01 and how do we get the users to import the new key? 11:41
warren So let's make that our recommendation? 11:41
warren drago01: there is NO AVOIDING users doing it manually 11:42
warren unless somebody has a brilliant idea 11:42
notting f13: and the isos are in the wild and burned.... 11:42
f13 are we all OK with leaving the isos alone, since they're validated in different ways (sha1sums), and they don't import the key anyway? 11:42
f13 (let alone check the key) 11:43
drago01 well some users might not get to read this mail/faq and end up with no updates / no way to install packages ... which isn't fun.... 11:43
notting f13: livecds do. ;) 11:43
warren f13: only drawback there is users of those iso's have to manually import the new key. 11:43
smooge how long until F8 EOL? 11:43
warren f13: but I might be OK with that. 11:43
skvidal notting: we don't actually need to if we have signed sha1sums 11:43
warren f13: currently scheduled for late December 11:43
skvidal hmm 11:43
f13 warren: if you install from iso but enable updates, you'll get the new fedora-release 11:43
skvidal depending on what they're signed with :) 11:43
warren f13: which you need to install manualy 11:43
f13 warren: no, we'd also put the fedora-release in the updates/ tree 11:43
f13 just like all other updates we do to the distro 11:44
nirik but it's signed with the new key they don't have? 11:44
f13 anaconda "wins" by not doing gpg checking 11:44
warren f13: if there are multiple updates, none of them will install automatically 11:44
warren f13: haha 11:44
f13 so anaconda will happily install the latest fedora-release which has the new key content and sets up the user for any further updates. 11:44
warren huh, anaconda will install updates by default? 11:45
f13 warren: not by default. 11:45
drago01 no... 11:45
f13 warren: you simply add the updates repo at install time. 11:45
f13 F9 anaconda had a checkbox for this IIRC 11:45
warren ok, F9 install with network is good 11:45
f13 only works well if you're doing some kind of network install, non-iso install 11:45
drago01 and btw this is bug 11:46
f13 drago01: define "this" 11:46
drago01 installing packages without checking the key 11:46
warren So our recommendation to FESCO: 1) Generate new key 2) Resign all F8, F9 and update packages in repos, leave ISO's alone. 3) Announce new key and instructions for users because most people will need to upgrade manually. 11:46
f13 drago01: that's bug 998 to be exact. 11:46
warren ? 11:46
buggbot Bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=998 medium, high, ---, clumens@redhat.com, ASSIGNED, FTP/NFS install/upgrade is unsafe, should check GPG signatures. 11:46
drago01 f13: ;) 11:46
f13 well, if we're doing keys per release, wouldn't 1 be generate F8 keys, F9 keys, and F10 keys ? 11:47
skvidal f13: we don't have to do a key per release for the past 11:47
warren do we really need a separate key from F8 and F9? 11:47
notting f13: oh feh. koji. 11:47
skvidal f13: only going forward 11:47
f13 notting: what about koji? 11:47
notting f13: well, we need to generate all the signed stuff in koji 11:47
notting too 11:48
f13 how else were we going to do it? 11:48
f13 rpmsign --resign directly on the fedora netapp? no. 11:48
notting yeah, just realizing the level of the pain 11:49
warren Recommendation to FESCO: 1) Generate new key for F8 and F9, and a separate key for testing. 2) Resign all F8, F9, update and testing packages in repos, leave ISO's alone. 3) Announce new key and instructions for users because most people will need to upgrade manually. 11:49
f13 regarding koji, for dist-f8+ we can garbage collect all the invalid signed copies. 11:49
f13 notting: just a looooooong screen session and a lot of pasting. 11:49
f13 and probably 3 weeks to get it all done. 11:50
f13 if not longer 11:50
warren 3 weeks? 11:51
warren to resign? 11:51
drago01 well 3+ weeks with no updates ..... 11:51
f13 although in theory you could just resign directly on the netapp, then import from the netapp into koji 11:51
f13 warren: it takes a long time to copy every package out of koji, sign them, import them back into koji, and ask koji to write out a version of the package with the new key header. 11:51
warren f13: "resign directly on the netapp, then import from the netapp into koji" that sounds good 11:52
f13 warren: especially given command argument lengths and the need to paste in the key passphrase every few minutes. 11:52
f13 warren: where good == extremely risky 11:52
f13 and no good way to stage it unless we turn off rsync for a day or so 11:52
warren hm 11:52
warren ok, so what do we actually want to do? 11:52
f13 or else you're going to get rpms that don't match the metadata. 11:52
f13 warren: I'd prefer to do it via koji, and stage the change on the master mirror. it's just going to take a rather long time. 11:53
f13 we can do each release in stages though, instead of everything at once. 11:53
f13 (if that "helps" at all) 11:53
warren so we begin using the new key, and revoke the old key only after a few weeks? 11:54
warren (this still wont avoid the need for folks to manually install the new key) 11:54
f13 that's one possibility 11:54
f13 start using the new key(s) immediately, stage the replacement of old signed data as new content becomes available. 11:55
f13 (this feels like a FESCo discussion / decision) 11:55
f13 lets start with your proposal warren, as a high level, and drill into details at FESCo 11:55
warren Recommendation to FESCO: 1) Generate new key for F8 and F9, and a separate key for testing. 2) Announce new key and instructions for users because most people will need to upgrade manually. 3) Begin using the new key to sign packages immediately to push updates. 4) Begin resigning all F8, F9, update and testing packages in repos, leave ISO's alone. Replace content on mirrors over time. 5) In a few weeks revoke the old key when all packa 11:56
warren ges are resigned and old key packages are eliminated from mirrors. 6) Generate a new key for F10. 11:56
warren ? 11:57
f13 erm. I'm not comfortable with recommending 4 without a fesco discussion. 11:57
warren Uh... we're giving this recommendation 11:57
warren we need to recommend something 11:57
f13 we need to recommend what actions need to be taken 11:57
warren They have the option of modifying the recommendations. 11:58
f13 whatever. 11:58
warren I'll draft up a longer written version of this including reasons 11:58
warren (reason: signing everything will take weeks) 11:58
mbonnet How do we revoke a key that already exists in people's rpmdb? 11:58
warren mbonnet: skvidal is testing if rpm -e gpgkey is safe during %post 11:59
mbonnet warren: doing that removes people's ability to verify packages that they already have installed 11:59
skvidal mbonnet: you can't verify an installed pkg 11:59
skvidal mbonnet: rpm -Va doesn't use the gpg sig 11:59
mbonnet ok 11:59
drago01 warren: can we do this: "while signing the new packages give instructions to users how to update to the new key and let livna/freshrpm etc ship the key in theire release package to reach more users" ? 11:59
mbonnet does resigning all existing F8/F9 content include spinning new isos? 12:00
drago01 I know part 2 sounds stupid but it might help 12:00
skvidal mbonnet: read up 12:00
warren drago01: livna/freshrpms wont help because the entire transaction will fail 12:00
wwoods the PK on the F9 media has trouble importing keys 12:00
f13 mbonnet: short answer, no. 12:01
drago01 warren: how so ? if we wait until the new key is used (signing done) so the users have 3 weeks time to get the new key 12:01
warren drago01: we are not going to wait between issuing new key and using it to sign packages. 12:01
f13 drago01: 3 weeks is only if we wait to push anything with the new key until /all/ content is replaced. 12:01
* skvidal has a bad feeling 12:02
warren about? 12:02
f13 warren: that's a rather decisive statement to make without having a FESCo vote. 12:02
skvidal rpm -e gpg-pubkey 12:02
warren f13: no, you misunderstand 12:02
drago01 f13: I thought that was the plan 12:03
f13 skvidal: that takes 'em all doesn't it? 12:03
warren f13: he's suggesting that everyone will update fedora-release signed by existing keys before packages signed by that key are pushed. 12:03
skvidal f13: not about takes them all 12:03
skvidal f13: atm it is locks the transaction 12:03
f13 warren: he's suggesting that we use livna repos which use livna's keys to get our new key into place 12:03
f13 skvidal: haha, fail. 12:03
warren f13: which wont work if the update transaction has packages signed by the new key 12:03
f13 *sigh* 12:04
warren am I wrong? 12:04
f13 warren: IF WE DON"T PUSH UPDATES UNTIL ALL CONTENT IS SWITCHED 12:04
warren f13: but that isn't our recommendation right? 12:04
f13 that gives a period of time that users may get updates from livna, before any updates from us 12:04
warren you're also assuming everyone is using livna 12:04
drago01 f13: exactly 12:04
f13 warren: as I stated before, I'm not comfortable making a recommendation on the timeline of events. 12:04
f13 warren: no I'm not. 12:04
f13 warren: neither is drago01 12:04
warren f13: if we are going with that plan, then we might as well push fedora-release signed with old key for the 3 weeks of resigning. 12:05
f13 it'll get it onto some people's systems. 12:05
drago01 warren: no bu it will reach a big part of our users .. better than "go read a faq" 12:05
f13 Can we please get out of this rabbit hole? 12:05
warren If we do not discuss this, how are we going to climb out? 12:06
warren We have an uncomfortable situation sure 12:06
warren but we need to give SOME recommendation 12:06
f13 drago01: I appreciate the suggestion (I understand it), but its not going to work for everybody, and would only hwork for a short period of time. I think it would just confuse users more to have a longer FAQ about it 12:06
skvidal okay 12:06
f13 "If you use livna and updated betwen X and X, you may not have to do any of this..." 12:06
skvidal well rpm -e gpg-pubkey-anything will not work 12:06
skvidal in %post 12:06
f13 awesome. 12:06
skvidal yah 12:06
f13 well, our "recommendation" doesn't list /how/ we would do things. 12:07
mbonnet skvidal: ok, push a new fedora-release that installs a cron job that'll remove the old fedora keys and install the new one? 12:07
dgilmore skvidal: what about a shell script that does it? 12:07
f13 cron, the answer to everything. 12:07
skvidal dgilmore: ? 12:07
mbonnet f13: :-P 12:07
f13 ok, guys, NOT IMPORTANT at this moment. 12:07
skvidal mbonnet: I don't think we can count on cron 12:07
skvidal ... 12:07
warren what is not important? 12:07
f13 we can drill into details once we get a FESCo mandate to make it happen. 12:07
dgilmore skvidal: for the rpm -e in %post 12:08
warren FESCO needs a recommendation 12:08
drago01 f13: reimporing the key wont hurt (so if the key is already imported and they read the faq nothing will break right?) ... and no it won't work for everyone but it would make it easier for xx% of our users 12:08
warren FESCO needs all the facts 12:08
dgilmore skvidal: just an idea. no clue if it could workl 12:08
f13 warren: which we obviously can't make right now because WE DONT HAVE A FUCKING WAY THAT WORKS 12:08
warren "We could do this, we can't do this because of foo." 12:08
* skvidal suggests that the process be manual 12:08
warren skvidal: +1 12:08
skvidal I don't think we have much choice 12:09
warren oh wait, the revoking part? 12:09
skvidal all of it 12:09
dgilmore skvidal: i think it should be manual 12:09
skvidal we cannot revoke the key 12:09
warren f13: stating that might be true, but we have to work on it anyway. 12:09
skvidal we've got no mechanism to do that 12:09
* nirik suggests someone ping Panu and see if there is some clever way we don't know, otherwise manual and make sure it's very clear what to do and why. 12:09
f13 warren: then why don't you go and find a way that works, and then come back here later to talk about working options. 12:09
f13 warren: because right now we're just arguing over theories, without any actual facts. not helpful. 12:09
warren It seems that we have some facts from the discussion 12:10
f13 warren: we also have only you and I left from releng even participating in the discussion 12:10
f13 warren: not much of a quorum for a releng recommendation. 12:11
warren which means we've failed at our job? 12:11
sharkcz what about adding the key removal into rc.local? it will be delayed until next reboot ... 12:11
f13 warren: no it means we're beyond our meeting time. Lets work on this outside of the meeting time. 12:11
notting rc.local can't be modifed reliably 12:11
f13 warren: once we have some things that work, we can bring the people back and get a vote. There is just no reason to continue doing this as part of our meeting time. 12:12
* f13 goes away to a real life meeting. 12:12
drago01_ f13, warren :so what did rel-eng agree on? (sorry wireless died) 12:13
warren drago01_: nothing 12:14
drago01_ ok 12:14
warren I'll write up everything we have, but I have to make some phone calls first. 12:14
f13 drago01_: one assumption we had, rpm -e the gpg key would work, proved to not work. warren wants to have a very detailed recommendation to fesco, which would mean actually testing what we want to do, so I asked him to actually do that and bring releng folks back to vote on it once we know our recommendation would actually work. 12:15
drago01_ f13: ok, thx for the summary 12:16

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!

Retrieved from "https://fedoraproject.org/w/index.php?title=ReleaseEngineering/Meetings/2008-aug-25&oldid=46524"