From Fedora Project Wiki
Attendees
People present (lines said)
- jlaska (93)
- kparal (49)
- adamw (33)
- bmwiedemann (19)
- wwoods (10)
- Cerlyn (2)
- mkrizek (1)
- Viking-Ice (1)
Unable to attend:
Agenda
Previous meeting follow-up
- Bodhi feedback patch from fcami (see
infrastructure ticket#701awaiting review
- adamw rescheduled xorg-x11-drv test events
Update on critpath test definition (ticket#154)
- Owner
- adamwill
- Summary
- See
fedora-qa ticket#154 - Still experimenting with some mock-ups/examples
- Next steps ...
- HELP - More feedback needed from testers and developers
- If no additional feedback, and we assume no news is good news, the next step is converting existing test cases, and getting people started writing new ones developed
- Reach out to f-e-k and bodhi teams to discuss tools integration
Latest and greatest on autoqa-0.4.4
- Owner
- kparal
- Summary
- Kparal pushed mkrizek's support for staging server into master
- Next steps ...
- What's the status of depcheck?
- Merge clumens branch into master
SUSE's openqa project
- Owner
- bmwiedemann
- Summary
- Project home - http://openqa.opensuse.org
- Sample automated results available at - http://openqa.opensuse.org/results/
- Source code hosted at http://www.os-autoinst.org/
Open discussion - <Your topic here>
Action items
IRC Transcript
| jlaska | #startmeeting Fedora QA Meeting | 16:00 |
|---|---|---|
| zodbot | Meeting started Mon Jan 10 16:00:21 2011 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| zodbot | Useful Commands: #action #agreed #halp #info #idea #link #topic. | 16:00 |
| jlaska | #meetingname fedora-qa | 16:00 |
| zodbot | The meeting name has been set to 'fedora-qa' | 16:00 |
| jlaska | #topic Gathering people | 16:00 |
| * kparal here | 16:01 | |
| jlaska | okay, show of nicks, who is here? | 16:01 |
| * mkrizek is here | 16:01 | |
| jlaska | hi kparal + mkrizek | 16:01 |
| jlaska | anyone else lurking ... adamw Viking-Ice robatino bmwiedemann? | 16:02 |
| adamw | yo | 16:02 |
| bmwiedemann | lurking :) | 16:02 |
| * adamw busy playing with wikis | 16:02 | |
| Cerlyn | here | 16:02 |
| * Viking-Ice here | 16:03 | |
| jlaska | Cerlyn: adamw: Viking-Ice: bmwiedemann: hello all | 16:03 |
| jlaska | okay, let's get started | 16:03 |
| jlaska | quick recap from last week | 16:03 |
| jlaska | #topic Previous meeting follow-up | 16:03 |
| jlaska | #info Bodhi feedback patch from fcami (see ticket#701 (infrastructure) awaiting review | 16:04 |
| jlaska | not much to add here ... looks like it's still pending review | 16:04 |
| jlaska | I don't see lmacken around to ping on this | 16:04 |
| jlaska | unless no other ideas ... | 16:05 |
| jlaska | #info Adamw and Hurry did some f15 test day rescheduling (adjusted Xorg dates and added preupgrade) -- https://fedoraproject.org/wiki/QA/Fedora_15_test_days | 16:06 |
| adamw | that was in response to viking's suggestion last week | 16:06 |
| jlaska | right on, thanks Viking-Ice and adamw | 16:06 |
| adamw | X test week now comes between the first and second GNOME test days | 16:06 |
| * jlaska adjusts a cell in the test day schedule | 16:07 | |
| adamw | brb | 16:07 |
| jlaska | adamw: you're next on the agenda, I'll shuffle that topic until you're bafck | 16:07 |
| jlaska | back | 16:07 |
| jlaska | kparal are you ready to talk about autoqa updates? | 16:08 |
| kparal | jlaska: yes | 16:08 |
| jlaska | #topic Latest and greatest on autoqa-0.4.4 | 16:08 |
| jlaska | kparal: okay, take it away | 16:08 |
| kparal | ok, welcome all to the regular news from autoqa world :) | 16:08 |
| adamw | back | 16:08 |
| kparal | the changes for the past week: | 16:08 |
| jlaska | adamw: okay, I'll queue you up right after kparal | 16:08 |
| kparal | 1. mkrizek's patch "Add support for a staging server" has been pushed to master. | 16:09 |
| kparal | https://fedorahosted.org/autoqa/ticket/241 | 16:09 |
| kparal | That means that AutoQA should be now fully configurable when it comes to interaction with various external services. Sending email with results (we have 3 different types of them) can be turned on or off, sending Bodhi comments can be turned on or off, URLs for Koji and Bodhi instances are configurable now. There are a few more options available. | 16:09 |
| jlaska | #info mkrizek's patch "Add support for a staging server" has been pushed to master (https://fedorahosted.org/autoqa/ticket/241) | 16:09 |
| kparal | 2. my patch "Load config files from current directory by default" has been pushed to master. | 16:09 |
| kparal | https://fedorahosted.org/autoqa/ticket/253 | 16:09 |
| kparal | All config files are now copied to the autotest client to the test's directory and used from there. That means we don't have to maintain /etc/autoqa conf files on each and every client anymore. | 16:09 |
| jlaska | very cool, nice work mkrizek and kparal | 16:09 |
| kparal | Unfortunately in one of the following patches we hit some issues and partly broke this functionality again. But it works for the most important config file - fas.conf. And I'll try to fix the rest soon :) | 16:09 |
| jlaska | #info my patch "Load config files from current directory by default" has been pushed to master (https://fedorahosted.org/autoqa/ticket/253) | 16:09 |
| jlaska | kparal: ah, so that's why you wanted site_tests writable by autotest? | 16:10 |
| kparal | jlaska: yes, that's the reason, all the config files must be first copied to site_tests to be tarred and transfered to the client | 16:10 |
| kparal | but as I say, maybe we will devise some other solution | 16:11 |
| jlaska | yeah, I added a comment in the ticket regarding packaging guidelines | 16:11 |
| kparal | because the current one causes some problems, we can't fully on CWD being set properly on the client | 16:11 |
| kparal | next on | 16:11 |
| kparal | 3. clumens asked for merging his clumens branch onto master. His branch expanded again, now it contains not just anaconda_storage test, but also compose_tree and anaconda_checkbot tests and git-post-receive hook. I started to review it, albeit slowly. | 16:11 |
| jlaska | #info clumens asked for merging his clumens branch onto master (tests: anaconda_storage, compose_tree, anaconda_checkbot, watcher: git-post-receive) | 16:12 |
| kparal | clumens and jlaska implemented lots of stuff in that patchset, lot of work done in that. so thanks. | 16:13 |
| * jlaska notes, it was fun writing test wrappers for existing tests | 16:13 | |
| kparal | 4. jlaska found out that autoqa package will probably need to add autotest as its dependency. (This stuff is still a little unclear, maybe we will need to do a few things differently because of ticket https://fedorahosted.org/autoqa/ticket/256, so this may change). | 16:13 |
| jlaska | #info jlaska found out that autoqa package will probably need to add autotest as its dependency (due to fas.conf changes) | 16:14 |
| kparal | everything's interconnected, this is the problem we spoke about a few paragraphs above :) | 16:14 |
| kparal | 5. A few bugfixes found its way into master branch. Kudos to jskladan for finding them out. | 16:14 |
| jlaska | #info A few bugfixes found its way into master branch. Kudos to jskladan for finding them out. | 16:14 |
| kparal | 6. jskladan posted his "New Koji Watcher" patch to autoqa-devel: | 16:15 |
| kparal | https://fedorahosted.org/pipermail/autoqa-devel/2011-January/001485.html | 16:15 |
| kparal | That should allow us to get rid of the outdated post-bodhi-update watcher and make use of the new -pending tags in koji. Still waiting for review. | 16:15 |
| jlaska | #info jskladan posted his "New Koji Watcher" patch to autoqa-devel, ready for review (https://fedorahosted.org/pipermail/autoqa-devel/2011-January/001485.html) | 16:15 |
| jlaska | kparal: ah okay, I'll queue that up for some feedback today, thanks for reminder | 16:15 |
| kparal | it will also allow us to run depcheck-style tests effectively, just once for many received update notifications | 16:15 |
| kparal | 7. Documentation has been improved a little to accommodate the new changes. | 16:16 |
| jlaska | #info Documentation has been improved a little to accommodate the new changes. | 16:16 |
| kparal | 8. I just pushed a 'use self.__class__ in super() calls' patch that should simplify a little our test template (one place less requiring manual changes). | 16:16 |
| kparal | https://fedorahosted.org/pipermail/autoqa-devel/2011-January/001494.html | 16:16 |
| jlaska | kparal: feel free to prefix with #info if you have more :) | 16:16 |
| kparal | jlaska: next time, sorry :) | 16:16 |
| kparal | this is all for today :) | 16:17 |
| jlaska | #info I just pushed a 'use self.__class__ in super() calls' patch that should simplify a little our test template (https://fedorahosted.org/pipermail/autoqa-devel/2011-January/001494.html) | 16:17 |
| jlaska | kparal: awesome, thank you for the updates | 16:17 |
| jlaska | how's it all looking for the next release | 16:17 |
| jlaska | is the end in sight for the original est date of January? | 16:17 |
| kparal | we have to solve a few unexpected issues, but I don't see any major obstacle | 16:18 |
| kparal | I'm just a little worried about testing | 16:18 |
| kparal | we tested the individual patches manually | 16:18 |
| kparal | but I can't really say everything's gonna work perfectly when deployed | 16:18 |
| jlaska | kparal: yeah, it will be interesting to get a sense for how long integration testing takes | 16:18 |
| * kparal looking forward to a staging server in the future | 16:19 | |
| jlaska | kparal: note, our new staging hardware has been delivered ... I don't have any updates on ETA for setup | 16:19 |
| kparal | jlaska: that sounds great | 16:19 |
| jlaska | kparal: thanks again for the updates | 16:20 |
| jlaska | adamw: you ready, we can touch on your topic next | 16:20 |
| adamw | sure | 16:20 |
| jlaska | #topic Update on critpath test definition (ticket#154) | 16:20 |
| jlaska | adamw: alrighty, what's the latest n' greatest | 16:20 |
| adamw | lemme remember, where were we up to last week | 16:21 |
| adamw | just a sec | 16:21 |
| jlaska | adamw: https://fedoraproject.org/wiki/QA/Meetings/20110103#Critical_Path_test_case_development | 16:21 |
| adamw | yeah thanks | 16:21 |
| adamw | I think the current pages are pretty much good to go, so i'll put them into production soon | 16:21 |
| adamw | i think the last current easily-resolvable question (ignoring test dependencies for now) is whether we should standardize *naming* | 16:22 |
| adamw | the benefit of that would be to allow tools like bodhi/f-e-k to display 'nice' test names, but it may be a bit cumbersome | 16:22 |
| jlaska | adamw: this is for test case page names? | 16:23 |
| adamw | yes | 16:23 |
| adamw | so for instance if we standardised on the name QA:Testcase_package_(packagename)_testname | 16:23 |
| adamw | bodhi could easily weed out all the parameters except 'testname' | 16:23 |
| adamw | but it may not work well for some situations, i guess. it may be a step too far | 16:24 |
| jlaska | what's the "standard" now? | 16:24 |
| adamw | there isn't one | 16:24 |
| jlaska | the page name has to be unique? | 16:24 |
| adamw | it has to be QA:Testcase, but then you can do whatever you like | 16:24 |
| adamw | so far the policy only requires test cases to be in the right categories | 16:24 |
| adamw | we don't have a de facto standard either, all test cases tend to be named according to somewhat different schemes | 16:25 |
| jlaska | okay | 16:25 |
| adamw | (some of them aren't even in the correct QA:Testcase_blah format yet) | 16:25 |
| jlaska | adamw: yeah, we've got a lot of migration still with older tests | 16:26 |
| adamw | yeah | 16:26 |
| adamw | i'm inclined towards not requiring a specific name format as it may restrict some cases | 16:26 |
| adamw | like creating a test case that can apply as-is to several packages, and adding it to the categories for each package | 16:27 |
| jlaska | I agree, my take ... let the categories you've previously define handle the organization and metadata ... and let the test case names be more human readable | 16:27 |
| adamw | and the benefit of being able to display a 'nice' test name isn't that great | 16:27 |
| jlaska | adamw: that reminds me of one feature of the wiki that I'll need to list on Hurry's feature comparison: human readible test case links | 16:28 |
| adamw | yeah | 16:28 |
| jlaska | instead of http://some.server.com/test?id=12435 | 16:28 |
| jlaska | on right, test 12435! that's a good one | 16:28 |
| adamw | one of my all-time favourites | 16:28 |
| jlaska | alright ... so any next steps you wanted to highlight? | 16:28 |
| adamw | i'm migrating some more test cases to the new format at present, still need to move the pages into 'production', announce to the lists, solicit test creation, and talk to tools maintainers | 16:29 |
| jlaska | #info next steps ... migrating some more test cases to the new format at present, still need to move the pages into 'production', announce to the lists, solicit test creation, and talk to tools maintainers | 16:31 |
| jlaska | adamw: thanks for the updates. Nice job on the SOP's too | 16:31 |
| adamw | thanks | 16:31 |
| * jlaska adds a comment regarding human-readable test case URLs to https://fedoraproject.org/wiki/Talk:Rhe/tcms_Comparison | 16:31 | |
| jlaska | #topic SUSE openqa project | 16:31 |
| jlaska | kparal: suggested adding this topic | 16:32 |
| jlaska | bmwiedemann: joined #fedora-qa last week to let everyone know about a cool test automation effort he started | 16:32 |
| jlaska | #info Project home - http://openqa.opensuse.org | 16:32 |
| jlaska | #info Sample automated results available at - http://openqa.opensuse.org/results/ | 16:32 |
| jlaska | #info Source code hosted at http://www.os-autoinst.org/ | 16:32 |
| kparal | #info Example Fedora installation video - http://www3.zq1.de/fedora/Fedora-14-i386-netinst.iso.ogv | 16:33 |
| jlaska | It was nice that bmwiedemann and _lmr_ were both in channel, so they could discuss similarities between the two screenshoting GUI automation approaches (openqa and kvm-autotest) | 16:33 |
| adamw | ooh! shiny! what's it do? | 16:33 |
| kparal | adamw: click on the video link | 16:33 |
| jlaska | adamw: it does GUI test automation by using screen region matching | 16:33 |
| jlaska | so it's different from LDTP or dogtail, as those are at-spi based | 16:33 |
| jlaska | I've reached out to hongqing and hurry with some links and for their thoughts on this approach | 16:34 |
| jlaska | I'll me meeting with _lmr_ after this meeting to talk through his experiences with using GUI matching like this in kvm-autotest | 16:34 |
| bmwiedemann | The effort started in May 2010 and had produced videos that were also used by openSUSE-marketing on youtube | 16:34 |
| kparal | bmwiedemann: what is it used for by opensuse - just installation testing, or also desktop applications testing? | 16:35 |
| adamw | that looks like an awesome step towards project mojito | 16:35 |
| jlaska | the source is all perl-based ... so if anyone in fedora-qa is a perl guru, this might be fun to get involved in | 16:35 |
| bmwiedemann | kparal: it does test some important applications like KDE,GNOME, libreoffice, firefox - but the main reason is to prevent breakage in install, boot and updates | 16:36 |
| * adamw not a perl guru but he bets he knows where to hire one | 16:36 | |
| kparal | I believe this project could nicely supplement our current rats-install test | 16:36 |
| bmwiedemann | aka critical-path | 16:36 |
| kparal | maybe we can execute it from autoqa regularly for Branched composes | 16:37 |
| jlaska | kparal: it could be ... I've asked hongqing for his thoughts on the approach. He's still coming up to speed with the roadmap | 16:37 |
| jlaska | bmwiedemann: what do you call your GUI region matches again? | 16:38 |
| jlaska | kvm-autotest calls them stepfiles | 16:38 |
| bmwiedemann | I would say "test modules" would be the best equivalent. they contain a "run" method with the keypresses and waits, plus a checklist method with the MD5sums of the regions | 16:39 |
| jlaska | bmwiedemann: how do you get the md5sums of the regions? Is there a tool that watches a manual install, and records keypresses/clicks etc..? | 16:39 |
| kparal | bmwiedemann: are you able to ignore minor differences somehow? like changed icon in the menu would not break the test | 16:41 |
| bmwiedemann | jlaska: every automated install can also take manual interaction via VNC or qemu-monitor commands ("sendkey ctrl-alt-f2"), so I usually interact with the automation, which will write things to a log. | 16:41 |
| bmwiedemann | but still a lot of things done manually there. | 16:41 |
| bmwiedemann | kparal: one way to avoid breakage is to only use MD5sum of a certain interesting region. | 16:41 |
| kparal | bmwiedemann: I see, so I don't have to match the whole screen, just a part of it, right? | 16:42 |
| bmwiedemann | the alternative is to live with it and have several "known-good" MD5sums for a test | 16:42 |
| bmwiedemann | kparal: yes. | 16:42 |
| jlaska | bmwiedemann: I tried running it against a Fedora 14 ISO, but it wasn't happy, and I wasn't sure how/where to make adjustments since I haven't touched perl in ages | 16:42 |
| wwoods | augh, Perl | 16:43 |
| bmwiedemann | jlaska: you should know that "wasn't happy" is not a good bug report ;) we can have a look at that later. | 16:43 |
| jlaska | bmwiedemann: understood :) | 16:43 |
| kparal | there are some tools in the world that are able to intelligently recognize "almost same" images - they give percentage of similarity. it's done by image downscaling, re-coloring, etc. I am sure some of that approach could be used also here to improve image matching | 16:44 |
| jlaska | there are some potential licensing things I wasn't sure about with the tool (since it's using some packaging from rpmfusion) to generate the videos. Someone smarter than me would need to work that out | 16:44 |
| bmwiedemann | kparal: I have one simple approach: thresholding the image before checksumming | 16:44 |
| kparal | bmwiedemann: how does it work? | 16:45 |
| wwoods | I'm convinced that for certain apps we could be using a 'translation' that put unique unicode glyphs on each button | 16:45 |
| wwoods | and then just scan for those glyphs | 16:45 |
| bmwiedemann | kparal: substituting each byte<128 to 0 and every other to 255 | 16:45 |
| wwoods | heh - we could use tiny QR codes | 16:46 |
| jlaska | wwoods: would that be any better than just using at-spi instead? | 16:46 |
| Cerlyn | bmwiedemann: So does your tool support detecting slight shifts of the image region in question (new field, moved for alignment purposes, etc.)? | 16:47 |
| bmwiedemann | I know, there is "sikuli" out there, which uses opencv for computer-vision to compensate for small changes | 16:47 |
| wwoods | at-spi requires actual access to the accessibility layer - i.e. the dbus session in the client | 16:47 |
| kparal | I think both approach have their uses. os-autoinst is very usable for installation testing and boot process testing | 16:47 |
| bmwiedemann | Cerlyn: no | 16:47 |
| jlaska | kparal: yeah, I was thinking over the weekend that hte perfect solution would likely be a mix of the two approaches | 16:48 |
| kparal | Cerlyn: I am sure that can be worked out to some extent | 16:48 |
| kparal | jlaska: agreed | 16:48 |
| wwoods | but if we included a special 'translation' of the app, and provided a font to display its 'language', then you can find all the buttons by their labels using image recognition | 16:48 |
| jlaska | kparal: as we do now with lili's sample DVD auto install proof of concept (uses dogtail + kickstart) | 16:48 |
| wwoods | so then you could just set anaconda to lang=QR and the button for 'Next' would have the QR-ese word for 'Next', which would be some easily-recognized symbol | 16:49 |
| wwoods | e.g. a QR code | 16:49 |
| kparal | bmwiedemann: so what's the tool name - openqa or os-autoinst? :) | 16:49 |
| bmwiedemann | wwoods: that would only be important, if you wanted to click the buttons, instead of using the keyboard. | 16:49 |
| bmwiedemann | the test tool is "os-autoinst" | 16:49 |
| jlaska | wwoods: sounds good, I'd have no idea where+how to start there :) | 16:49 |
| bmwiedemann | openqa.opensuse.org is just the machine running it | 16:50 |
| bmwiedemann | one of the two | 16:50 |
| jlaska | bmwiedemann: thanks for stopping by for the meeting | 16:50 |
| wwoods | bmwiedemann: true, keyboad will suffice if there's keyboard accelerators for everything you want to test | 16:50 |
| wwoods | anyway, interesting stuff! | 16:50 |
| jlaska | all: let's move discussion of further details into #fedora-qa | 16:50 |
| jlaska | and in the interest of time ... we'll move on to open discussion | 16:50 |
| kparal | yes, certainly very nice tool | 16:50 |
| jlaska | agreed, nice work bmwiedemann | 16:51 |
| jlaska | #topic Open Discussion <your topic here> | 16:51 |
| jlaska | Alright ... anything folks want to raise that we haven't already discussed? | 16:51 |
| jlaska | if not ... we'll close out in 2 minutes | 16:52 |
| jlaska | Meeting end in 1 minute ... | 16:53 |
| jlaska | thanks everyone for your time! | 16:54 |
| jlaska | #endmeeting | 16:54 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!