ReleaseEngineering/StatusReport

Monthly Status for Priority Pipeline Projects

20MAR2017

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done
Public hardware for AutoCloud	Low	Done
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	Done
Taskotron	High	Done
Atomic release reorientation	High	Done
Fedora dockerfiles support for Beta	Medium	Done
Bodhi: Design for supporting for containers	Low	Done
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	Done
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Done		Kernel bug is causing image creation to fail but dev work complete
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	In progress	Will be blocked on a new Koji release that has been planned for April 5	Update Koji in Stage after April 5 & Prod after freeze lifts
Move releng, mash & pungi from yum to dnf	High	In progress	Rust support requirements from advanced functionality in DNF are not included in this work but are being requested; need to route those to the right owner for decision making and prioritization; need to get some sort of "shadow" compose created to begin comparing dnf output to yum	releng scripts not started; Mash update is in progress; python-mutlilib ported to support DNF; Pungi support is almost done
Ansible: Docker 2 week build/test/release workflow automated	High	In-Progress	Rewriting some pieces for better future tooling integration based on feedback from releng	Build, test, release automation exist individually; next step is wiring them all together
OSBS has redundancy and support for multi-master / multi-node	High	In progress	None	Multi node infra is live in prod. Pending scale testing on multi node infra.
OSBS: Fedora Docker Registry	High	In progress	awaiting stage testing & monitoring and a prod VM deploy	Now that the minimal registry is deployed, next step is scaling
Support for non x86_64 ostrees as part of standard build & release process	Medium	Not started	Resourcing	Awaiting confirmation from Atomic WG that this is desirable & a priority

Other

• F27 planning will be scheduled in April

06MAR2017

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done		Live in stage & prod
Public hardware for AutoCloud	Low	Done
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	Done
Taskotron	High	Done		Feedback from users
Atomic release reorientation	High	Done
Fedora dockerfiles support for Beta	Medium	Done
Bodhi: Design for supporting for containers	Low	Done
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	Blocked	UPDATE 15MAR16 - temporarily unblocked but will be blocked on a new Koji release that has been planned for April 5	Bug identifed in Koji#306 that requires a release & update; blocked on deploying update in stage for testing
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Delayed	UPDATE 15MAR16 - image composes are working in ImageFactory except for one arch that has a kernel bug
Move releng, mash & pungi from yum to dnf	High	In progress	Rust support requirements from advanced functionality in DNF are not included in this work but are being requested; need to route those to the right owner for decision making and prioritization	releng scripts not started; Mash update is in progress; python-mutlilib ported to support DNF; Pungi support is almost done
Ansible: Docker 2 week build/test/release workflow automated	High	In-Progress	None	Build, test, release automation exist individually; next step is wiring them all together
OSBS has redundancy and support for multi-master / multi-node	High	In progress	None	Multi node infra is live in prod. Pending scale testing on multi node infra.
OSBS: Fedora Docker Registry	High	In progress	awaiting stage testing & monitoring and a prod VM deploy	Now that the minimal registry is deployed, next step is scaling
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	In Progress	None	Requirements for aarch64 & Power are complete; s390 import is running and may make it in for F26 also
Support for non x86_64 ostrees as part of standard build & release process	Medium	Not started	Resourcing	Awaiting confirmation from Atomic WG that this is desirable & a priority

Other

08FEB2017

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done	None	Live in stage & prod
Public hardware for AutoCloud	Low	Done	None
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	Done	None
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Delayed	Decision to focus on moving from appliance creator to Image Factory temporarily
Taskotron	High	Delayed	Remaining work	Stage update live; prod update pending decision about storing tests in dist-git and
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	Delayed	Koji updates deployed; bug identifed in Koji#306 that requires a release & update
Ansible: Docker 2 week build/test/release workflow automated	High	In-Progress	None	Automation Notes and Action Items Etherpad
Atomic release reorientation	High	In progress	Deployment is in progress this week
OSBS has redundancy and support for multi-master / multi-node	High	In progress	None	Multi node infra is live in prod. Pending scale testing on multi node infra.
Move releng, mash & pungi from yum to dnf	High	In progress	None	releng scripts & pungi port in progress; pending work to port python multi-lib to dnf instead of updating mash itself
OSBS: Fedora Docker Registry	High	In progress		Decision made to run registry in OSD; planning completed Dec 14; implementation in progress
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	In Progress	Pending hardware configuration for aarch64/Power; s390 prep on target & may make it in early
Fedora dockerfiles support for Beta	Medium	Not started	Planned for February activity day
Bodhi: Design for supporting for containers	Low	Not started	Resourcing
Support for non x86_64 ostrees as part of standard build & release process	Medium	Not started	Resourcing

Other

• Moved deliverables that are not yet started and not going to make it in for F26 out to the F27 candidates list
  • Koji support for disk images
  • Deploy koji signed repo support - scope work for Pungi & Bodhi to use it
  • add index.asc files for cloud images for virt-builder
  • ostree mirroring
  • Ansible: Atomic 2 week build/test/release workflow automated

14DEC2016

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done	None	Live in stage & prod
Atomic release reorientation	High	Blocked	No available resources to work on this issue; open as a high priority for a year	???
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Blocked	Mohan's testing is failing - requires fixes from LMC team Live cd creator is very fragile and could break at any moment, moving off is crucial	Create problem report for LMC to review
Taskotron	High	Blocked	Remaining work	Dev update live; prod update pending decision about storing tests in dist-git and fixing fedmsg events
Ansible: Docker 2 week build/test/release workflow automated	High	In-Progress	None	Automation Notes and Action Items Etherpad
OSBS has redundancy and support for multi-master / multi-node	High	In progress	None	Multi node infra is live in prod. Scale testing on multi node infra.
Move releng, mash & pungi from yum to dnf	High	In progress	DGilmore is the only active contributor to Mash	Identify backup for Mash work; begin to update releng scripts
OSBS: Fedora Docker Registry	High	In progress		Decision made to run registry in OSD; planning completed Dec 14; implementation to begin before shut down
Koji support for disk images	High	Not started	Resourcing & prioritization from Brew	Pending functional disk image builds in LMC
Deploy koji signed repo support - scope work for Pungi & Bodhi to use it	Medium	Blocked	Need PR revisions due to the amount of elapsed time Also need to support DNF in PR ongoing delays on the merge will impact our ability to scope work for Pungi & Bodhi
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	Blocked	Need koji updates deployed to stage for testing and dev
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	In Progress	Need status update from Mohan
add index.asc files for cloud images for virt-builder	Medium	Not started	Resourcing
Fedora dockerfiles support for Beta	Medium	Not started	Resourcing
ostree mirroring	Medium	Not started	Resourcing
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	In Progress	None	Tweaking setup to reflect requirements
Public hardware for AutoCloud	Low	In Progress	None	Hardware is in hand, pending powering up and handing off to Autocloud folks
Bodhi: Design for supporting for containers	Low	Not started	Resourcing
Automate 2 week Power ostree releases & testing	Low	Not started	Resourcing
Ansible: Atomic 2 week build/test/release workflow automated	Low	Not started	Resourcing	Will not begin until Docker ansible automation is complete

Other

* 3 F25 deliverables have been carried over to F26 as they were not fully completed on the F25 schedule
  ** OSBS Scale out
  ** Container automation
  ** Taskotron automated testing for containers
* Note that status updates for modularity work will not be reported here

07NOV2016

Committed F25 Changes

Change	Status	Risks/Open Issues	Next
Layered image builds	Done	None	Pending 1 bugfix release next week for additional container testing by Cockpit owners
Koji: Koji hub consolidation groundwork	Done	None
New Releng Deliverable: Windows & OSX tools in Fedora	Done	None	Additional requests for improved support after Beta have been filed
Content signing can be automated	Done	None
Container Signing in Sigul	Done	None
Bodhi release 2.3	Done	None
OSBS has redundancy and support for multi-master / multi-node	In progress	None	Multi node support deployed in stage; pending firewall updates and then release to prod after F25 GA Freeze
Taskotron	Delayed	Remaining work	Dev update live; prod update pending decision about storing tests in dist-git and fixing fedmsg events
Ansible: Docker 2 week build/test/release workflow automated	Blocked	Blocked by what tool to use for automation

In Progress F26 Changes

Change	Status	Risks/Open Issues	Next
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	In Progress	None	Tweaking setup to reflect requirements
Deploy koji signed repo support - scope work for Pungi & Bodhi to use it	Blocked	Need PR revisions due to the amount of elapsed time Also need to support DNF in PR ongoing delays on the merge will impact our ability to scope work for Pungi & Bodhi
Kerberos support in koji, fedpkg, OSBS	In Progress	None	Live in stage & prod, will be requesting testing this week
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	In Progress	Merge introduced a number of issues that are mostly fixed now
Public hardware for AutoCloud	In Progress	None	Hardware is in hand, pending powering up and handing off to Autocloud folks

Other

• New Deliverables (not covered above)
  • Atomic for Power - Blocked; builds are not completing successfully
• Testing koji updates (1, 2) in stage
• Atomic ostree repo management is in progress again; autocloud testing has been updated to run per compose rather than per build; will move on to revising the 2 week script to deliver these requirements next
• I have added a section for F26 deliverables that have started this month and will add the full set of committed changes next month

13OCT2016

Committed Changes

Change	Status	Risks/Open Issues	Next
Layered image builds	Done	None	Pending 1 bugfix release next week for additional container testing by Cockpit owners
Koji: Koji hub consolidation groundwork	Done	None
New Releng Deliverable: Windows & OSX tools in Fedora	Done	None	Additional requests for improved support after Beta have been filed
Content signing can be automated	In Progress	None	Pending updates / bugfix release planned for this week after beta freeze
Container Signing in Sigul	In Progress	None	Pending final release this week after beta freeze
Taskotron	Delayed	Remaining work	Dev update live; prod update pending decision about storing tests in dist-git
Ansible: Docker 2 week build/test/release workflow automated	Blocked	Blocked by what tool to use for automation & where to run playbooks
OSBS has redundancy and support for multi-master / multi-node	Blocked	Blocked by where to run playbooks	Run playbooks on deployed VMs
Bodhi release 2.2; 2.3 (?)	2.2.4 is in production	2.3 is in progress and could realistically happen in October

Other

• New Deliverables (not covered above)
  • WS OSTree - In Rawhide and 25 Nightlies
  • Atomic for Power - In progress; planned for Beta
• Koji #48 has a proposed patch and is pending testing in stage - this was a very nice to have for Koji Hub Consolidation
• OSBS: Fedora Docker Registry has been delayed for F26 as we assess the impact of signing & decide on final design
• Atomic ostree repo management
• Beginning F26 planning with the team - if you have input, please contact us - plan to share these priorities in our next status update

13SEP2016

Committed Changes

Change	Status	Risks/Open Issues	Next
Layered image builds	complete but working on 1 remaining bug	No path to resolution of nss bug	Dominik Perpeet will lead getting Cockpit builds into the Fedora toolchain week of Sept 12
Taskotron	Completion delayed	Remaining work	-
Ansible: Docker 2 week build/test/release workflow automated	Loopabull ready for use POC release playbook created	Need decision from RCM about Pungi Docker support Need sign off & input from infra team on running playbooks somewhere At risk of missing beta delivery date	Create playbooks
Content signing can be automated	Code complete, deployed to Stage	Awaiting getting yubikeys in place in Prod	Deploy to Prod approx Sept 14
OSBS: Fedora Docker Registry	Blocked (short term registry solution in place for F25)	Image signing may break our planned design	Meeting with signing stakeholders to determine if we need to revise our plans
Koji: Koji hub consolidation groundwork	Moonshot chassis is functionally deployed, aarch64 merged & imported last week	Koji #48 will be needed soon creating aarch64 disk images in LMC is a blocker - updates were made to support this but have not yet been tested	LMC Disk Images for aarch64 & armv7
OSBS has redundancy and support for multi-master / multi-node	Working through multi-host ansible now	No requests for extra VMs submitted yet At risk of missing beta delivery date	Apply multi host updates to existing infra
New Releng Deliverable: Windows & OSX tools in Fedora	Dev complete from engineering; OSX Beta will be built outside of koji / Windows will be built in koji Signing key for Windows purchased Initial builds have completed but testing is ongoing	Need signing keys for OSX No testing of signed images has been done yet Resourcing for OSX work is now up in the air again - discussion ongoing with managers	Need to have initial images built by week of Sept 13
Upcoming Bodhi 2.2.0 release, beta deployed to stg	Waiting one week for it to go through stg testing

Other

• New Deliverables (not covered above)
  • WS OSTree - In Rawhide; pending addition to compose for Beta
  • Atomic for Power - In progress; planned for Beta
• Atomic ostree repo management - feedback period on proposal has closed; work targeted to be live for one of the October 2 week updates
• Pungi got better validation for configuration.

   $ pungi-config-validate fedora.conf
   Unrecognized config option: link_files. Did you mean link_type?
   Unrecognized config option: notification_script.
   Deprecated config option: pkgset_koji_url; use koji_profile instead.
   Failed validation in ostree_installer.0.1.x86_64:
       Additional properties are not allowed ('filename' was unexpected)