From Fedora Project Wiki
| wwoods | okay! let's talk about the Beta for a bit. | 08:06 |
|---|---|---|
| wwoods | It's out. Yay. | 08:06 |
| wwoods | Some good testing has been going on, and I've been trying to keep track of stuff reported against the Beta | 08:06 |
| * f13 | 08:06 | |
| wwoods | but a question occurred to me - do we care about keeping track of the major Beta issues as a group | 08:07 |
| * jds2001 hasn't had a ton of free time, but my last few rawhide attempts have gone up in flames. | 08:07 | |
| jlaska | wwoods: how do you mean? | 08:07 |
| wwoods | like, should we continue to maintain the F10Beta tracker to keep track of Common Bugs reported against the beta | 08:08 |
| jds2001 | interesting idea. | 08:08 |
| wwoods | (or some other way of keeping track of F10Beta common bugs, separate from the big F10Target/F10Blocker lists) | 08:08 |
| jds2001 | kind of 'go here for common issues we already know about' type of thing? | 08:08 |
| wwoods | right. For the final releases we typically have a CommonBugs page | 08:08 |
| f13 | don't we have a common bugs page for beta already, linked from the release notes? | 08:09 |
| jlaska | http://fedoraproject.org/wiki/Releases/10/Beta/ReleaseNotes#Known_bugs_and_issues | 08:09 |
| jlaska | only 2 issues :( | 08:09 |
| f13 | there ya go, a place to track the common bugs we know are in the shipped Beta | 08:10 |
| mcepl | jlaska: I would venture to say that we have more problems than that ;-) | 08:11 |
| wwoods | wonderful | 08:11 |
| f13 | the bugs themselves should be moved over to the appropriate tracker. | 08:11 |
| wwoods | yeah I've been putting them on F10Blocker/F10Target as appropriate | 08:11 |
| wwoods | but wanted a summary page somewhere. the release notes seems as good a place as any | 08:12 |
| f13 | worksforme | 08:12 |
| stickster | https://fedoraproject.org/wiki/Common_F10_bugs | 08:12 |
| wwoods | We could use that page too, so long as we maintain it carefully | 08:12 |
| jds2001 | stickster: is that the common bugs page for final? | 08:14 |
| stickster | jds2001: It will be, yes | 08:14 |
| stickster | So the Intel entry is deprecated, I suppose | 08:14 |
| wwoods | so should we use that to track F10 bugs from Beta on | 08:15 |
| wwoods | or just leave it for a roundup at the end? | 08:15 |
| poelcat | how about the Common Bugs page points to the blocker/trackers ? | 08:16 |
| jlaska | wwoods: it'd be great if we could maintain a wiki page with the bugs ... but I fear that might be too much mgmt of wiki ... | 08:16 |
| jlaska | yeah ... I was just going to suggest putting a link in that wiki page to the blocker dependency bugzilla page? | 08:16 |
| wwoods | but *which*? | 08:17 |
| wwoods | F10Blocker? F10Preview? F10Beta? | 08:17 |
| wwoods | how do we track the stuff that's open in F10Beta but closed in Rawhide since then? | 08:17 |
| f13 | in my opinion, we need different wiki pages, or sections for each milestone | 08:18 |
| poelcat | list them all... removing them as they are no longer relevant | 08:18 |
| f13 | we definitely need an ongoing record of known bugs in Beta (until the next milestone is out) so that people who install beta, see that their bug is known. | 08:18 |
| jlaska | yeah, I like the break it out for milestones approach | 08:18 |
| f13 | but for rawhide users, or the next milestone user, the bug may not be there, we don't want it listed | 08:18 |
| wwoods | poelcat: I don't think that scales | 08:18 |
| poelcat | no, list the links | 08:19 |
| f13 | but if we remove it, then people installing beta will hit it, not see it listed, etc... | 08:19 |
| wwoods | poelcat: that doesn't help | 08:19 |
| jlaska | we can just drop it from the blocks:F10Beta | 08:19 |
| jlaska | err nm | 08:19 |
| stickster | The purpose of the "Common F10 bugs" page needs to be a readable summary for non-experts. That may not fit what you guys need. | 08:19 |
| jds2001 | why not, link to the bug, and they can see the status from there. | 08:19 |
| wwoods | giving people a link to all the trackers does not tell them "yes, we know that f10beta will wait for 30sec if your CD drive is empty" | 08:19 |
| jds2001 | there's even a rhbug:123456 notation in the wiki | 08:20 |
| jlaska | if we have a manageable (smallish) list of bugs ...I suspect we could engage folks with doc skills to help word smith the Common_BUgs entries | 08:20 |
| stickster | jlaska: darn tootin' | 08:20 |
| jds2001 | stickster: you're sounding like you're from the midwest, not DC :) | 08:21 |
| stickster | oh yah | 08:21 |
| wwoods | maybe we should just set the fedora_requires_release_note flag on bugs that should go into the common_bugs page? | 08:22 |
| jlaska | I think that's a great idea | 08:22 |
| wwoods | I mean, that's kind of why we have it.. right? | 08:22 |
| jlaska | wwoods: yeah, you got it | 08:22 |
| jlaska | do we have folks in thne fedora space that monitor that queue of fedora_requires_release_note? | 08:23 |
| jds2001 | there's even some list that gets email notification of that flag being requested :) | 08:23 |
| poelcat | why not just call the whole thing "known rawhide bugs?" | 08:23 |
| wwoods | stickster might know more about that..? | 08:23 |
| jds2001 | i think that quaid does that, not sure. | 08:23 |
| poelcat | people keep telling me that 3 or 4 days after "the beta" is released it is no longer relevant... "use rawhide" | 08:23 |
| stickster | wwoods: Yes, relnotes@fp.o writes to the fedora-relnotes-content list when that happens. | 08:23 |
| stickster | http://www.redhat.com/mailman/listinfo/fedora-relnotes-content | 08:24 |
| wwoods | poelcat: it's relevant as a way to get rawhide on your system. we've had this conversation a dozen times. | 08:24 |
| poelcat | wwoods: i guess we keep having it because it has never been answered in a way most people can understand :) | 08:25 |
| f13 | poelcat: here is the work flow. | 08:25 |
| f13 | poelcat: people install beta, they run into bugs. Often times they file those bugs or bother people on forums/IRC about them. | 08:25 |
| f13 | poelcat: they get told to update, since it's fixed in rawhide. | 08:25 |
| wwoods | maybe we need to rename the post-Beta snapshots | 08:26 |
| wwoods | this week's snapshot should be Beta 2 | 08:26 |
| f13 | poelcat: if we had a place and led people to it that would list the known bugs they're going to encounter with a fresh beta install, that also shows them that it has been fixed since beta, they can update (when possible due to broken deps), see their bug fixed, and move on. | 08:26 |
| wwoods | increment the number 'til we hit Preview 1 | 08:26 |
| f13 | wwoods: erm... I didn't really want to use that naming scheme since it's a much much different distribution method for snapshots | 08:27 |
| f13 | torrent only, potentially smaller payload | 08:27 |
| f13 | and it's a trial (twice, thursday/friday) and if it still doesn't work, punt until next week | 08:27 |
| f13 | not guarenteed to actually show up | 08:27 |
| wwoods | right | 08:27 |
| wwoods | so here's the thing, poelcat | 08:27 |
| wwoods | you're not being told that Beta is irrelevant | 08:28 |
| viking_ice | why not just drop the whole alpha beta etc and have just rawhide-snapshot-$date ? | 08:28 |
| wwoods | it's actually the case that the bug is fixed and it'll be in the Preview | 08:28 |
| wwoods | but if you *want* you can use rawhide to get early access | 08:28 |
| wwoods | now. since rawhide is public and just as easy to install.. everyone uses that instead | 08:28 |
| f13 | viking_ice: people and press don't look at those. | 08:28 |
| f13 | the Beta iso set is not irrelevant. It provides a relatively known stable install point | 08:29 |
| f13 | just installing and immediately testing the Beta is not very relevant, particularly the older it gets | 08:29 |
| wwoods | that doesn't negate the relevance of the Beta, it just provides early access to fixes destined for Preview | 08:29 |
| f13 | installing, /updating/, and then testing is absolutely relevant. | 08:30 |
| wwoods | right. but reporting bugs about the Beta itself is not real useful. but apparently people aren't getting this message. | 08:30 |
| jlaska | wwoods: I think that's just it | 08:31 |
| wwoods | so how do we make that plainer? I think the obvious first step is: keep a list of known bugs | 08:31 |
| viking_ice | like always better documentation would fix that | 08:31 |
| wwoods | bugs *in* beta, and their current status | 08:31 |
| jlaska | I don't think this is the case ... but if we don't want people ti file bugs against the beta, they need ot rawhide update first ... which opens up the whole rawhide failure loop | 08:31 |
| f13 | right | 08:32 |
| wwoods | so. F10Beta should continue to be used for tracking Beta bugs post-release | 08:32 |
| f13 | for the love of god, we need a document on people's desktops that explains this. | 08:32 |
| poelcat | jlaska: +1 | 08:32 |
| jlaska | ;) | 08:32 |
| wwoods | f13: seriously | 08:32 |
| poelcat | f13: or we need a better/different process | 08:32 |
| jlaska | f13: wwoods: I understand the model | 08:32 |
| f13 | poelcat: hey I'm all ears. | 08:32 |
| wwoods | poelcat: write up a proposal for a better model and we'll talk about it | 08:32 |
| jlaska | I'm just saying that I can see how for a newbie the interaction between the rawhide and a milestone can be hazy | 08:33 |
| wwoods | right, so let's unhaze it | 08:33 |
| viking_ice | why not just have the default web pages in the browser common_f10_bugs ? instead of search ? | 08:33 |
| wwoods | desktop document / panel applet that says "here's the current list of known bugs, don't bother reporting these, just upgrade your system already" | 08:34 |
| viking_ice | what the first thing that people do after install go to the internet.. | 08:34 |
| viking_ice | easy-er to maintain.. | 08:34 |
| viking_ice | then just change it for final | 08:34 |
| viking_ice | back to search | 08:34 |
| wwoods | What controls the default page? | 08:34 |
| jlaska | it is an interesting model ... here install the beta which ideally we put a lot of effort into, then update your system to rawhide which has no guarruntee of success. | 08:35 |
| f13 | wwoods: start.fedoraproject.org | 08:35 |
| f13 | wwoods: we could possibly trigger on client headers and direct them to a different page | 08:35 |
| jlaska | we definitely want folks to be running the latest-greatest ... it just often comes with warts | 08:35 |
| wwoods | jlaska: post-beta that's not supposed to be the case | 08:36 |
| viking_ice | f13: was going to suggest that.. | 08:36 |
| wwoods | post-beta people aren't supposed to be putting destabilizing changes into rawhide | 08:36 |
| f13 | man, if only | 08:36 |
| poelcat | but everyone knows that isn't the case | 08:36 |
| wwoods | so "no guarantee of success" != "guaranteed unstable as all hell" | 08:36 |
| * f13 cries a little | 08:36 | |
| jlaska | f13: cry for baby jesus :) | 08:36 |
| wwoods | okay, fine, rawhide sucks. what do you want to do about it? | 08:36 |
| viking_ice | complain... | 08:37 |
| * viking_ice ducks...+ | 08:37 | |
| jlaska | just thinking out loud ... I'm not providing any alternatives or trying to break things down | 08:37 |
| wwoods | we're 30 minutes into the meeting and I'm hearing pushback on the very idea that people should *update their systems* after Beta | 08:38 |
| wwoods | so, okay, fine. What would be better? | 08:38 |
| viking_ice | to get rawhide more stable we need more testing on rawhide.. simple | 08:38 |
| wwoods | and how do we accomplish that? by *not* having people update to rawhide? | 08:38 |
| wwoods | automated testing? great. who's writing those tools? | 08:38 |
| viking_ice | more reports == more bugs == same amount of people behind the project == we need more developers.. | 08:38 |
| wwoods | let's take it as read that the biggest QA problem we have right now is that rawhide isn't stable enough | 08:39 |
| wwoods | because that's what I'm hearing | 08:39 |
| wwoods | put everything else aside and give me suggestions on making rawhide more stable | 08:39 |
| viking_ice | developers put their own baby's to the test on rawhide | 08:39 |
| wwoods | except developers are all savvy enough to deal with instability | 08:40 |
| viking_ice | should filter out simple mistakes typing errors etc.. | 08:40 |
| wwoods | a lot of our devs *do* run rawhide pretty much all the time | 08:40 |
| f13 | this is the age old argument. | 08:40 |
| jlaska | true true | 08:40 |
| f13 | We ask developers to produce better builds for rawhide | 08:40 |
| wwoods | obviously we can't just say "hey. make it more stable KTHX" | 08:41 |
| f13 | they ask us to develop pre-testing environments to prevent bad builds from going into rawhide | 08:41 |
| f13 | we stare at eachother from across a table. | 08:41 |
| wwoods | pre-testing environments. okay. so we want automated testing, then? | 08:41 |
| wwoods | rawhide builds that don't pass the test don't get synced out? | 08:41 |
| wwoods | cool. who's writing that? | 08:41 |
| f13 | right, like I said, it's the age old argument that isn't going to resolve itself here in this meeting. | 08:42 |
| wwoods | or are there other things we should be doing? | 08:42 |
| * viking_ice is taking early steps in python slowly getting there.. | 08:42 | |
| f13 | and I don't even think that's the right solution (preventing the rawhide push) | 08:42 |
| wwoods | that's not a rhetorical question - seriously, if that's the biggest problem we have, then that's what we're going to work on as soon as F10 is out | 08:42 |
| jlaska | based on our FUDCon talk ... I guess that depends on what aspect of rawhide fails (the install source vs yum repo issue) | 08:42 |
| wwoods | until F11a or F11b | 08:43 |
| wwoods | at which point we are invariably overwhelmed by actual testing and bug tracking and, y'know, this conversation happens again | 08:43 |
| wwoods | f13: so what's the solution? | 08:43 |
| f13 | wwoods: I don't necessarily have one. | 08:44 |
| wwoods | I want at least two automated systems: 1) rpmdiff for each build, and 2) a set of automated tests for nightly anaconda image builds | 08:44 |
| f13 | wwoods: however, are there not people within Red Hat working on automated testing systems? | 08:44 |
| wwoods | that's what I was told | 08:44 |
| wwoods | haven't heard a peep since FUDCon | 08:44 |
| pjones | rpmdiff is mostly hype, though. | 08:44 |
| f13 | "rpmdiff like" | 08:45 |
| f13 | perhaps not 'rpmdiff' as you know it. | 08:45 |
| wwoods | I'm not suggesting literal rpmdiff | 08:45 |
| f13 | to state it better, we want 1) post-package-build testing of package, and 2) automated tests for nightly rawhide trees. | 08:45 |
| wwoods | but I want to know when files are dropped/added to packages, or change perms, or sonames | 08:45 |
| pjones | f13: yeah, that does sound better. | 08:46 |
| wwoods | especially the latter, which should trigger a mass of emails and/or automated rebuilds | 08:46 |
| stickster | wwoods: What about the stuff dmalcolm put out recently? | 08:46 |
| wwoods | stickster: what stuff? | 08:46 |
| f13 | once we have this magical world where developers can get automated feedback on the build they just did, we can go into a mode where builds don't automatically get tagged. | 08:47 |
| f13 | the build goes through, gets tested, and IF the maintainer is happy with it, the maintainer (or somebody else) can fiddle a bit and that build is now tagged. | 08:47 |
| stickster | I'm pretty sure he recently released some sort of initial code for doing a bunch of in-depth RPM analysis, at the symbol level | 08:47 |
| jlaska | https://fedorahosted.org/rpmfluff/ | 08:47 |
| f13 | wow what an unfortunate name. | 08:47 |
| jlaska | :) | 08:48 |
| wwoods | stickster: no, that's a tool for creating sets of RPMs with weird dependencies or conflicts or whatever | 08:49 |
| wwoods | it's for testing tools that handle RPMs - yum, rpm, packagekit, etc. | 08:49 |
| f13 | going to a mode where builds don't automatically get tagged, we'd be closer to a sourcecontrol world, where changes committed don't automatically get tagged for "stable" | 08:49 |
| wwoods | stickster: it doesn't actually analyze RPMs at all, which is what we're talking about | 08:49 |
| stickster | https://fedorahosted.org/rpmgrok/ | 08:50 |
| stickster | There you go ^^ | 08:50 |
| f13 | In my opinion, aside from doing the actual testing, the most important things the Fedora project needs is a tool to manage quality tests cases/results, and an automated system to run said tests and report said results. | 08:51 |
| stickster | "rpmgrok is a web application for browsing the payloads of a large collection of RPM software packages. It digests a set of input RPMs, analysing the metadata and payload, and storing the results in a database. There's a web UI for viewing the data, an XML-RPC interface for querying it, and a command-line tool for using the XML-RPC interface." | 08:51 |
| wwoods | interesting. that's more of a distro-wide info-viewer, although you could probably plug that into bodhi and hit it with a wrench until rpmdiff fell out | 08:51 |
| wwoods | f13: right. and that ain't gonna be Testopia, apparently :/ | 08:51 |
| f13 | secondary to that we need to get more frequent and better information to our developers as they complete builds | 08:52 |
| stickster | wwoods: I think rpmgrok will do what you asked for though -- filename changes, sonames, etc. | 08:52 |
| f13 | throwing it over the rawhide wall and waiting for the users to report bugs has "worked" for a number of years, but could be so much better. | 08:52 |
| pjones | f13: really the test infrastructure needs to be part of the build system, at least from the developer's point of view | 08:53 |
| f13 | pjones: correct. | 08:53 |
| f13 | pjones: my vision is koji reports a build complete, the test infrastructure gets this notice and runs the available/configured tests on the build and reports the results. | 08:54 |
| f13 | and we can play games with when/if a tag is applied to the build somewhere in that timeline. | 08:54 |
| wwoods | pjones: do you have a suggested structure for the dev view of the testing stuff? | 08:54 |
| wwoods | mostly I'm wondering about where devs put their tests | 08:55 |
| wwoods | you'd think we'd want a %test section in the spec, or a test dir in package CVS, or similar | 08:55 |
| pjones | well, there are three aspects: scratch builds, non-scratch builds, and trees | 08:55 |
| f13 | pjones: 4 really. | 08:55 |
| pjones | the middle is the easiest one -- best case, if a problem is detected, it results in a build failure. | 08:55 |
| f13 | pjones: scratch builds, tagless builds, tagged builds, freeze tags. | 08:55 |
| pjones | f13: man, we really have too much jargon. which of those is which? | 08:56 |
| f13 | pjones: I really don't want to have koji have to know specifics about a test infrastructure that would come between a build "complete" and a build "failure" | 08:56 |
| f13 | pjones: scratch-build is an anonymous build in koji, can come from srpm, no restrictions on what the package is, where it comes from, and what it's built against. | 08:57 |
| f13 | scratch-build isn't recorded in the database, the output isn't kept. | 08:57 |
| pjones | f13: without that, I fear we'll have a situation like we have with rpmdiff now, where it's not really part of the process, and developers get to it as an afterthought | 08:57 |
| f13 | tagless build is in all ways a "normal" build, comes from package scm, recorded in the database, must be a known package in a known tag. | 08:57 |
| pjones | oh, by tagged and untagged you meant WRT CVS | 08:57 |
| pjones | or not. | 08:57 |
| f13 | the kicker with a tagless build, is that the final step of the build process, applying a tag to the build (eg dist-f10), isn't done. | 08:58 |
| pjones | ok | 08:58 |
| pjones | so it's basically a second form of scratch build | 08:58 |
| f13 | the build completes, the rpms are stored, but the tag itself isn't applied | 08:58 |
| pjones | except with some more rigorous restrictions | 08:58 |
| f13 | pjones: with a scratch-build, you can never tag it | 08:58 |
| f13 | pjones: with a tagless build, you can go back and apply a tag | 08:58 |
| pjones | ok. | 08:58 |
| f13 | so you tagless build, you run said build through testing environment | 08:59 |
| f13 | if it passes, or if the developer forces, the tag gets applied | 08:59 |
| pjones | right. | 08:59 |
| f13 | so in essense, you have your "failure" if it doesn't pass qa | 08:59 |
| pjones | right. | 08:59 |
| f13 | but we don't have to teach koji about a testing environment | 08:59 |
| f13 | all of that logic can live outside the koji codebase. | 08:59 |
| wwoods | hey, that's pretty clever. | 09:00 |
| f13 | wwoods: it was a design goal when koji was written. | 09:01 |
| pjones | f13: I'm still worried about the rpmdiff issue, where you can basically ignore it if you choose to. | 09:01 |
| f13 | pjones: that comes down to policy | 09:01 |
| f13 | pjones: but if we go this route, if you ignore rpmdiff, your build doesn't get tagged, doesn't show up in rawhide, doesn't show up in the release | 09:01 |
| pjones | right. | 09:02 |
| wwoods | okay, so. some assumptions for the test system. will it be in the same lab as the buildsys? should we be able to assume filesystem (NFS, etc) access to the builds? | 09:02 |
| f13 | we can send email until we're blue in the fingers if you've got a pending build, but unless the maintainer does something about it.... | 09:02 |
| pjones | that can work. I just don't want a process like the RHEL errata process, where I have to go ask somebody to unlock something so that I can go fix something I accidentally skipped. | 09:02 |
| f13 | pjones: no, as long as I'm at the helm, the ability to do cvs and do builds of some kind will never be locked down. | 09:03 |
| f13 | pjones: I've chosen to draw my line in the sand at the tagging of the completed build level, no lower. | 09:03 |
| pjones | ok. | 09:03 |
| pjones | good :) | 09:03 |
| f13 | I've listened to enough people rail on the RHEL process to follow that disaster. | 09:03 |
| f13 | wwoods: good quesitons for mmcgrath | 09:04 |
| f13 | wwoods: ideally it would be in the PHX infrastructure, where the koji filestore is, and that you'd have either nfs, or http access to the builds. | 09:04 |
| f13 | nfs-readonly is probably doable. | 09:05 |
| wwoods | will that be access to the entire koji db? ideally we want to be able to query, you know, what tag this build wants, so we can find the previous package with that tag | 09:05 |
| wwoods | so we can do diffs | 09:05 |
| f13 | the koji db is public | 09:06 |
| f13 | you can do anonymous queries like that via the API | 09:06 |
| f13 | so you'd have both access to the API to get information, and access to the filesystem where the packages are stored to poke at the rpms themselves. | 09:06 |
| wwoods | okay, but you need to have the newly-built package, and some idea of what tag to look at | 09:08 |
| wwoods | anyway, implementation detail | 09:08 |
| wwoods | in the long run this should help make rawhide more stable | 09:08 |
| f13 | wwoods: you'd have that. | 09:09 |
| f13 | via the api | 09:09 |
| wwoods | f13: no, I mean, the notification to the test system needs those two things | 09:09 |
| wwoods | the way you described it, new builds would be tagless | 09:09 |
| wwoods | so something has to pass that buildID and, I guess, the target tag | 09:10 |
| f13 | wwoods: they would be tagless, however you'd have information in the builddata about what they were targetting | 09:10 |
| wwoods | ah - the builddata would still contain it, even if it's not applied? | 09:10 |
| f13 | you /target/ dist-f10, you just don't apply the final tag. | 09:10 |
| f13 | a build target has 3 peices of data. | 09:10 |
| f13 | the target name, the buildroot tag to use, and the tag to apply to builds that complete. | 09:11 |
| wwoods | "target name" is.. the package name? | 09:11 |
| f13 | $ koji list-targets --name dist-f10 | 09:11 |
| f13 | Name Buildroot Destination | 09:11 |
| f13 | --------------------------------------------------------------------------------------------- | 09:11 |
| f13 | dist-f10 dist-f10-build dist-f10 | 09:11 |
| wwoods | ahhh | 09:11 |
| wwoods | okay, I get it now. | 09:12 |
| f13 | when you do a build from the devel/ branch, it executes: koji build --target dist-f10 cvs://path | 09:12 |
| wwoods | so the other thing we talked about is automated testing of installer images | 09:12 |
| f13 | yep. | 09:13 |
| f13 | <future world> | 09:13 |
| f13 | compose process goes through the rawhide compose, builds images, and stages them on the kojifilestore. | 09:13 |
| f13 | compose process drops a message on our communication bus that says "New tree, path here" | 09:13 |
| f13 | testing system hears the notification on teh bus and starts it's pre-configured 'rawhide' tests on the location | 09:14 |
| f13 | results from testing are dropped on the bus | 09:14 |
| f13 | sync tool gets results from bus, makes decision about what parts of the tree gets synced to the master mirror | 09:14 |
| wwoods | I'd think we'd want a simpler test input, though - it'd be better if anaconda devs could kick off automated tests without needing to wait for the rawhide compose | 09:15 |
| f13 | information about the whole compose, including test results and whether or not images were synced dropped on the bus. | 09:15 |
| f13 | notification bot gets this info from the bus, sends various mails, rss notices, etc.. | 09:15 |
| f13 | </future world> | 09:15 |
| f13 | wwoods: there is no reason why we couldn't manually schedule a run of tests on a tree location | 09:16 |
| wwoods | but that requires composing a tree | 09:16 |
| f13 | or even upload a boot.iso via a webform to run a manual set of tests on | 09:16 |
| wwoods | especially with the way that stage2 works these days, you'd think you could take, as inputs, kernel/initrd + tree URL | 09:17 |
| wwoods | or boot.iso + tree URL, sure | 09:17 |
| f13 | the trick is coming up with the boot.iso | 09:17 |
| f13 | IE stage1/2 made from the changes they're attempting | 09:17 |
| f13 | given the way that buildinstall works, that's not an easy task, since the goddamn thing pulls a anaconda package from a repo, exploads it, and re-executes itself from the exploaded package. | 09:18 |
| wwoods | right, but we can work on that within anaconda | 09:19 |
| wwoods | there's supposedly a makefile target to build a boot.iso | 09:19 |
| wwoods | but it's unreliable | 09:19 |
| f13 | right | 09:19 |
| f13 | my above scenario was focused on the nightly compose | 09:19 |
| wwoods | right, which is also an important thing | 09:19 |
| f13 | since we could make a decision there to sync the new repos and packages, but hold back bad images. | 09:19 |
| wwoods | but ideally this test service should be able to handle either one | 09:20 |
| f13 | yep | 09:20 |
| f13 | I wouldn't think of it any other way | 09:20 |
| wwoods | so clumens and dcantrell and friends can throw bits at it until something sticks, and then *that* goes through the compose | 09:20 |
| f13 | realistically, we're likely going to need to provide them shell access on a machine close to the test system, because asking them to upload stage2 images from their house (dcantrell), or even the bos office (clumens) is going to introduce a lot of lag. | 09:21 |
| f13 | or we make the test system capable of having nodes in their house, and in the bos office | 09:22 |
| f13 | so that they can run the tests locally with local input | 09:22 |
| wwoods | right, I want this system to be something you can instantiate lolcally | 09:23 |
| wwoods | err, locally. lol-cat-ly | 09:23 |
| wwoods | running on bare metal will require things like a PXE/tftp server for recovery | 09:24 |
| wwoods | so the initial test run might be virt stuff | 09:24 |
| wwoods | anyway, we're waaay over our normal meeting time | 09:24 |
| wwoods | and I need food before my brain leaks out my ears | 09:24 |
| wwoods | so let's summarize | 09:25 |
| wwoods | 1) we need to track Beta common bugs and put them somewhere, like the Beta release notes page | 09:25 |
| wwoods | 2) people need to be informed about that page, and about the fact that many bugs will be fixed by updating | 09:26 |
| wwoods | 2a) we need to figure out a way to get that info onto people's desktops for pre-releases | 09:26 |
| wwoods | 3) if we're going to continue encouraging people to update their Beta/etc. systems to Rawhide, we need to make Rawhide more stable. | 09:27 |
| wwoods | 3a) we should have a per-build test system attached to koji that runs tests on each package built - before the package is tagged in koji - and can refuse to tag the package if the test fails. | 09:28 |
| wwoods | 3ai) Devs can still force-tag stuff that fails tests, but they do so at their own peril. | 09:29 |
| jwb | erm.. | 09:30 |
| wwoods | 3b) we should have an installer test system that takes the nightly compose and runs some "rawhide" tests on it - if the tests fail, we don't sync out the broken images/data | 09:30 |
| wwoods | Does that cover it? | 09:30 |
| jwb | shouldn't 3a be done by the developer before they submit the build to koji? | 09:31 |
| wwoods | jwb: we're talking about koji tags here, not CVS tags, and only the koji tags that normal devs can apply, like 'dist-f10' for rawhide builds | 09:31 |
| jwb | i think (maybe) my comment still applies | 09:32 |
| wwoods | jwb: so you run 'make build', koji builds your new package, but doesn't tag it 'dist-f10' until the tests pass | 09:32 |
| f13 | jwb: please read some of the backscroll. | 09:32 |
| jwb | yeah ok | 09:32 |
| jwb | ignore me for now | 09:32 |
| f13 | jwb: essentially it's a "scratch build" until the tests pass, or the developer forces the tag application. | 09:33 |
| f13 | with the bonus that the developer wouldn't have to re-submit the build "officially" if the tests passed. | 09:33 |
| jwb | this is for post-Beta, or all the time? | 09:33 |
| f13 | up for discussion | 09:33 |
| f13 | I'd push hard for at /least/ post-beta | 09:33 |
| f13 | but there could also be different "pass" criteria for different periods of the development cycle. | 09:34 |
| f13 | more lenient earlier, more strict later. | 09:34 |
| wwoods | it depends on us writing some infrastructure bits first | 09:35 |
| wwoods | but, basically, ASAP | 09:35 |
| wwoods | stuff like: hey your library changed soname | 09:36 |
| wwoods | should probably require some intervention / notifications to maintainers of dependent packages | 09:36 |
| f13 | yeah, this brings in a wrinkle. | 09:36 |
| wwoods | *immediately* on package build, not during the nightly compose | 09:36 |
| f13 | chain-builds | 09:36 |
| wwoods | I think it's obvious that we move too fast to wait for nightly builds for all our tests | 09:36 |
| f13 | chain-builds allow you to set up a chain of N+ packages, say 'build foo, once done and in buildroots, build bar, baz, and bangle' | 09:37 |
| jwb | where are these tests coming from? | 09:37 |
| f13 | since the building of foo will break bar, baz, bangle. We'd probably want to be able to allow those builds, and do the testing on the whole set. | 09:37 |
| f13 | jwb: combination of common tests for all packages, and then developer/QA generated tests that hopefully live near hte package SCM | 09:38 |
| f13 | or in the package SCM | 09:38 |
| f13 | like a tests/ subdir for each branch dir | 09:38 |
| jwb | ok... so for the kernel, you'd run LTP? | 09:38 |
| f13 | *shrug* up to the kernel people | 09:38 |
| wwoods | that's a little ambitious | 09:38 |
| jwb | so is this whole idea | 09:39 |
| wwoods | not really | 09:39 |
| wwoods | the current test suite consists of nothing, so everything passes | 09:39 |
| wwoods | so don't freak out just yet | 09:39 |
| f13 | jwb: sadly it's something that Red Hat has had in one form or fashion for a very long time, and Fedora lost it when we moved Fedora out of Red Hat infrastructure. | 09:39 |
| jwb | i'm not freaking out | 09:39 |
| jwb | i think the idea is great. but i also think it's ambitious | 09:40 |
| wwoods | the initial test suite would probably be something like: did we lose any files? did anything change perms? | 09:40 |
| f13 | and Fedora has suffered because of this | 09:40 |
| wwoods | it's really not that ambitious. we're just adding an "are you sure?" step for builds where weird things are happening | 09:40 |
| f13 | "did this build break deps?" | 09:40 |
| wwoods | everything beyond that is up to the package maintainers discretion | 09:40 |
| f13 | I think we're finally to the point in Fedora that focusing on this is our #1 priority | 09:42 |
| f13 | we've got the opensource buildsystem thing down, we've got good infrastructure now, we've solved some of our storage issues | 09:42 |
| wwoods | stickster specifically called out QA as a priority for improvement | 09:43 |
| f13 | we've got compose tools pretty well done | 09:43 |
| stickster | f13: agreed | 09:43 |
| wwoods | building good QA depends on everything else being fairly stable already | 09:43 |
| f13 | it's time to focus on getting the QA superstructure built up. | 09:43 |
| wwoods | and I think we're just now finally reaching that point | 09:43 |
| wwoods | luckily we've got a *lot* of really good ideas | 09:43 |
| wwoods | and a lot of really talented people | 09:43 |
| f13 | it's taken a while, but goddamn does 6 month release cycles drain a lot of energy :/ | 09:43 |
| wwoods | but, y'know, no matter how many women you hire, it takes 9 months to make a baby | 09:43 |
| wwoods | f13: for serious | 09:44 |
| f13 | wwoods: not according to Fringe! | 09:44 |
| wwoods | my wife advises people not to talk to me during October and April | 09:44 |
| wwoods | heh | 09:44 |
| wwoods | okay, so, aside from tracking bugs for PR and doing Rawhide testing | 09:45 |
| wwoods | which should be first priority - per-build tests or per-compose installer tests? | 09:46 |
| f13 | I feel like per-compose installer tests because it seems like those could more easily be done hodge-podge | 09:48 |
| wwoods | also I'd like to ensure - and have proof - that rawhide is installable more of the time, so people stop using anaconda as a scapegoat | 09:49 |
| wwoods | unfortunately I think this one is the tougher problem.. but hey, I've got two machines with virt stuff handy | 09:49 |
| wwoods | so I'll see what I can get cooking | 09:50 |
| wwoods | okay. anything else we need to discuss? | 09:50 |
| wwoods | oh! right! Features. | 09:50 |
| -!- wwoods changed the topic of #fedora-meeting to: Fedora QA | Feature testing | 09:50 | |
| wwoods | bpepple (on behalf of FESCo) has asked about testing accepted features so FESCo can review ones that need review | 09:51 |
| wwoods | https://fedoraproject.org/wiki/Category:FeatureAcceptedF10 has the feature list | 09:51 |
| wwoods | https://fedoraproject.org/wiki/QA/TestResults/Fedora10Features/Beta has the ongoing results (which needs updating) | 09:52 |
| wwoods | if you can provide a quick status on any of those things | 09:52 |
| wwoods | or want to nominate something for a Test Day | 09:52 |
| wwoods | *please* let me know.. or just go ahead and do it | 09:52 |
| -!- wwoods changed the topic of #fedora-meeting to: Fedora QA | misc | 09:55 | |
| wwoods | anything else? | 09:55 |
| * wwoods takes that as a no | 09:56 | |
| wwoods | okay - thanks for the time and discussion, folks | 09:56 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!