ReleaseEngineering/StatusReport

Monthly Status for Priority Pipeline Projects

17OCT2017

Committed F27 Changes

Change	Priority	Status	Now	Next
Modularity	High	Blocked	#1900 Masher is currently being integrated with Bodhi to ship modules. #7074 Compose configuration changes complete for modular Fedora. #7054 Koji upgraded to 1.14.0. #60 Modular server replicated to mirrors correctly. #7056 Information on modular composes in PDC.	#7057, #7058 Blocked on compose request for setup and generating torrents which is the remaining known rel eng work for this effort.
Flatpaks	Medium	Blocked	Blocked on modularity work - in particularly #6192 ODCS.	OCI support for registry.fedoraproject.org; an index for registry.fedoraproject.org; OSBS support.
Shipping without an alpha	Medium	At risk	Greenwave to give results on the gating criteria.	Investigation on implementing policy change.
Porting to Python 3- rpkg and fedpkg	Medium	Off target	rpkg and fedpkg currently being ported in parallel, but will not be complete in F27 cycle. No major impact of miss b/c python 2 will be EOL in 2019.	Continue to push for implementation in F28 cycle.
Porting to Python 3- rel eng scripts	Medium	Off target	Critical rel eng scripts have been ported to Python 3. Not all rel eng scripts will be ported in F27. No major impact of miss b/c python 2 will be EOL in 2019.	Continue to push for implementation in F28 cycle.
Porting to Python 3- rel eng scripts to DNF	Medium	Off target	Critical rel eng scripts have been ported to use DNF. Not all rel eng scripts will be ported to DNF in F27. No major impact of miss b/c python 2 will be EOL in 2019.	Continue to push for implementation in F28 cycle.
Porting to Python 3- pungi	Medium	Blocked	Waiting for Kobo release for work to complete. No major impact of miss b/c python 2 will be EOL in 2019.
#300 Support for accelerating the Fedora Atomic Host: Consolidate ostree generation process for pre-GA and post-GA OSTree generation / support for multi arch ostrees / better versioning of ostrees	High	Complete	Differences in Bodhi mash file hierarchy and Pungi repo hierarchy have been resolved; support for bodhi-backend01to run Pungi composes completed; Mirrormanager now handle changes to updates repos file hierarchy output by Bodhi
Koji hub consolidation for s390	Low	Complete
compose IDs closely match	Low	Complete

Other

15AUG2017

Committed F27 Changes

Change	Priority	Status	Risks/Open Issues	Next
#300 Support for accelerating the Fedora Atomic Host: Consolidate ostree generation process for pre-GA and post-GA OSTree generation / support for multi arch ostrees / better versioning of ostrees	High	On target	Almost there; bodhi+pungi running in stage	#6192 Need support for bodhi-backend01 to run pungi composes and #6193 to resolve differences in bodhi mash file hierarchy and pungi repo hierarchy
Modularity	High	On target	Bodhi is waiting on PR #1697 masher which is underway; still on target for release
Flatpaks	Medium	Some risk	Wrangling code into upstream projects; marking this at risk since a good amount of work is left to complete here	#6182 The On Demand Compose Server (ODCS) needs to be deployed. Once the changes to the OSBS components (atomic-reactor, osbs-client, koji-containerbuild) have landed, they need to be updated within Fedora. [https://github.com/docker/distribution/pull/2076, #2076 needs to land. We need to allow people to easily install the built flatpaks, we need to generate a static downloadable index of registry.fedoraproject.org.
Shipping without an alpha	Medium	On target	Need to make composes gating	Rel eng working to write configuration to do so
Porting to Python 3- rpkg and fedpkg	Medium	Off target	rpkg and fedpkg currently being ported in parallel	Major goal is to complete this work in next one or two versions after rpkg-1.50 and fedpkg-1.29.
Porting to Python 3- rel eng scripts	Medium	Off target	Large amount of linting errors to work through	Will focus on trying to get most "critical" scripts (e.g. scripts that run compose) ported before beta freeze
Porting to Python 3- rel eng scripts to DNF	Medium	On schedule	Conversion to DNF scripts well underway; working on ancilliary tasks	Switching DNF configs to use Pungi
Porting to Python 3- pungi	Medium	Some risk	Currently have a branch that passes all unit tests on Python 3, but no full compose with that branch	Need for Kobo.
Koji hub consolidation for s390	Low	On target	Compose host to be setup with correct mounts
compose IDs closely match	Low	Complete

Other

14JUNE2017

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done
Public hardware for AutoCloud	Low	Done
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	Done
Taskotron	High	Done
Atomic release reorientation	High	Done
Fedora dockerfiles support for Beta	Medium	Done
Bodhi: Design for supporting for containers	Low	Done
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	Done
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Done		Kernel bug is causing image creation to fail but dev work complete
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	Done
Ansible: Docker 2 week build/test/release workflow automated	High	Blocked	Working the OSBS team to leverage existing mechanisms to build layered images to handle parent image inheritance and monitor the "image stream" of updates. Work is dependent on "orchestrator cluster" and "worker cluster" OSBS architecture which is still under active development. Deliverable will miss 6-27 date due to dependency on OSBS work.	Loopable is deployed in stage and planned to roll out after freeze is lifted.
OSBS has redundancy and support for multi-master / multi-node	High	In progress		Current multi-node environment can be scaled as necessary. Collaborating with the Fedora Infrastructure Team to request more resources for OSBS in order to scale the masters.
OSBS: Fedora Docker Registry	High	Delayed		Deployed to stage. Will be deployed to production after F26 beta freeze and before F26 final.

Other

• F27 planning is finalized; the major themes will be support for modularity, allowing Fedora to ship with alpha, accelerating the Fedora Atomic Host, and porting to Python 3

02MAY2017

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done
Public hardware for AutoCloud	Low	Done
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	Done
Taskotron	High	Done
Atomic release reorientation	High	Done
Fedora dockerfiles support for Beta	Medium	Done
Bodhi: Design for supporting for containers	Low	Done
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	Done
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Done		Kernel bug is causing image creation to fail but dev work complete
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	Done
Ansible: Docker 2 week build/test/release workflow automated	High	In-Progress	Final loopabull patch has been submitted.	Planning for stage deployment the week after summit (5/15)
OSBS has redundancy and support for multi-master / multi-node	High	In progress	Scale testing dropped	Remaining work is only to get metrics stood up using hawkular
OSBS: Fedora Docker Registry	High	In progress	no change here; awaiting stage testing & monitoring and a prod VM deploy	Now that the minimal registry is deployed, next step is scaling

Other

• Support for non x86_64 ostrees as part of standard build & release process moved to F27
• Move releng, mash & pungi from yum to dnf
  • The work for pungi is done, but not turned on in production; will be working to finalize this in F27
• F27 planning is just about finalized; the major themes will be support for modularity, allowing Fedora to ship with alpha, accelerating the Fedora Atomic Host, and porting to Python 3

20MAR2017

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done
Public hardware for AutoCloud	Low	Done
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	Done
Taskotron	High	Done
Atomic release reorientation	High	Done
Fedora dockerfiles support for Beta	Medium	Done
Bodhi: Design for supporting for containers	Low	Done
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	Done
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Done		Kernel bug is causing image creation to fail but dev work complete
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	In progress	Will be blocked on a new Koji release that has been planned for April 5	Update Koji in Stage after April 5 & Prod after freeze lifts
Move releng, mash & pungi from yum to dnf	High	In progress	Rust support requirements from advanced functionality in DNF are not included in this work but are being requested; need to route those to the right owner for decision making and prioritization; need to get some sort of "shadow" compose created to begin comparing dnf output to yum	releng scripts not started; Mash update is in progress; python-mutlilib ported to support DNF; Pungi support is almost done
Ansible: Docker 2 week build/test/release workflow automated	High	In-Progress	Rewriting some pieces for better future tooling integration based on feedback from releng	Build, test, release automation exist individually; next step is wiring them all together
OSBS has redundancy and support for multi-master / multi-node	High	In progress	None	Multi node infra is live in prod. Pending scale testing on multi node infra.
OSBS: Fedora Docker Registry	High	In progress	awaiting stage testing & monitoring and a prod VM deploy	Now that the minimal registry is deployed, next step is scaling
Support for non x86_64 ostrees as part of standard build & release process	Medium	Not started	Resourcing	Awaiting confirmation from Atomic WG that this is desirable & a priority

Other

• In the process of deploying a UI for our container registry - hoping to demo in April
• F27 planning will be scheduled in April

06MAR2017

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done		Live in stage & prod
Public hardware for AutoCloud	Low	Done
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	Done
Taskotron	High	Done		Feedback from users
Atomic release reorientation	High	Done
Fedora dockerfiles support for Beta	Medium	Done
Bodhi: Design for supporting for containers	Low	Done
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	Blocked	UPDATE 15MAR16 - temporarily unblocked but will be blocked on a new Koji release that has been planned for April 5	Bug identifed in Koji#306 that requires a release & update; blocked on deploying update in stage for testing
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Delayed	UPDATE 15MAR16 - image composes are working in ImageFactory except for one arch that has a kernel bug
Move releng, mash & pungi from yum to dnf	High	In progress	Rust support requirements from advanced functionality in DNF are not included in this work but are being requested; need to route those to the right owner for decision making and prioritization	releng scripts not started; Mash update is in progress; python-mutlilib ported to support DNF; Pungi support is almost done
Ansible: Docker 2 week build/test/release workflow automated	High	In-Progress	None	Build, test, release automation exist individually; next step is wiring them all together
OSBS has redundancy and support for multi-master / multi-node	High	In progress	None	Multi node infra is live in prod. Pending scale testing on multi node infra.
OSBS: Fedora Docker Registry	High	In progress	awaiting stage testing & monitoring and a prod VM deploy	Now that the minimal registry is deployed, next step is scaling
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	In Progress	None	Requirements for aarch64 & Power are complete; s390 import is running and may make it in for F26 also
Support for non x86_64 ostrees as part of standard build & release process	Medium	Not started	Resourcing	Awaiting confirmation from Atomic WG that this is desirable & a priority

Other

08FEB2017

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done	None	Live in stage & prod
Public hardware for AutoCloud	Low	Done	None
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	Done	None
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Delayed	Decision to focus on moving from appliance creator to Image Factory temporarily
Taskotron	High	Delayed	Remaining work	Stage update live; prod update pending decision about storing tests in dist-git and
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	Delayed	Koji updates deployed; bug identifed in Koji#306 that requires a release & update
Ansible: Docker 2 week build/test/release workflow automated	High	In-Progress	None	Automation Notes and Action Items Etherpad
Atomic release reorientation	High	In progress	Deployment is in progress this week
OSBS has redundancy and support for multi-master / multi-node	High	In progress	None	Multi node infra is live in prod. Pending scale testing on multi node infra.
Move releng, mash & pungi from yum to dnf	High	In progress	None	releng scripts & pungi port in progress; pending work to port python multi-lib to dnf instead of updating mash itself
OSBS: Fedora Docker Registry	High	In progress		Decision made to run registry in OSD; planning completed Dec 14; implementation in progress
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	In Progress	Pending hardware configuration for aarch64/Power; s390 prep on target & may make it in early
Fedora dockerfiles support for Beta	Medium	Not started	Planned for February activity day
Bodhi: Design for supporting for containers	Low	Not started	Resourcing
Support for non x86_64 ostrees as part of standard build & release process	Medium	Not started	Resourcing

Other

• Moved deliverables that are not yet started and not going to make it in for F26 out to the F27 candidates list
  • Koji support for disk images
  • Deploy koji signed repo support - scope work for Pungi & Bodhi to use it
  • add index.asc files for cloud images for virt-builder
  • ostree mirroring
  • Ansible: Atomic 2 week build/test/release workflow automated

14DEC2016

Committed F26 Changes

Change	Priority	Status	Risks/Open Issues	Next
Kerberos support in koji, fedpkg, OSBS	High	Done	None	Live in stage & prod
Atomic release reorientation	High	Blocked	No available resources to work on this issue; open as a high priority for a year	???
ARM Disk Images, aarch64 and PXE2Live are created using LMC	High	Blocked	Mohan's testing is failing - requires fixes from LMC team Live cd creator is very fragile and could break at any moment, moving off is crucial	Create problem report for LMC to review
Taskotron	High	Blocked	Remaining work	Dev update live; prod update pending decision about storing tests in dist-git and fixing fedmsg events
Ansible: Docker 2 week build/test/release workflow automated	High	In-Progress	None	Automation Notes and Action Items Etherpad
OSBS has redundancy and support for multi-master / multi-node	High	In progress	None	Multi node infra is live in prod. Scale testing on multi node infra.
Move releng, mash & pungi from yum to dnf	High	In progress	DGilmore is the only active contributor to Mash	Identify backup for Mash work; begin to update releng scripts
OSBS: Fedora Docker Registry	High	In progress		Decision made to run registry in OSD; planning completed Dec 14; implementation to begin before shut down
Koji support for disk images	High	Not started	Resourcing & prioritization from Brew	Pending functional disk image builds in LMC
Deploy koji signed repo support - scope work for Pungi & Bodhi to use it	Medium	Blocked	Need PR revisions due to the amount of elapsed time Also need to support DNF in PR ongoing delays on the merge will impact our ability to scope work for Pungi & Bodhi
Pungi changes to allow non-blocking artifacts to fail without failing the compose	Medium	Blocked	Need koji updates deployed to stage for testing and dev
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	Medium	In Progress	Need status update from Mohan
add index.asc files for cloud images for virt-builder	Medium	Not started	Resourcing
Fedora dockerfiles support for Beta	Medium	Not started	Resourcing
ostree mirroring	Medium	Not started	Resourcing
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	Low	In Progress	None	Tweaking setup to reflect requirements
Public hardware for AutoCloud	Low	In Progress	None	Hardware is in hand, pending powering up and handing off to Autocloud folks
Bodhi: Design for supporting for containers	Low	Not started	Resourcing
Automate 2 week Power ostree releases & testing	Low	Not started	Resourcing
Ansible: Atomic 2 week build/test/release workflow automated	Low	Not started	Resourcing	Will not begin until Docker ansible automation is complete

Other

* 3 F25 deliverables have been carried over to F26 as they were not fully completed on the F25 schedule
  ** OSBS Scale out
  ** Container automation
  ** Taskotron automated testing for containers
* Note that status updates for modularity work will not be reported here

07NOV2016

Committed F25 Changes

Change	Status	Risks/Open Issues	Next
Layered image builds	Done	None	Pending 1 bugfix release next week for additional container testing by Cockpit owners
Koji: Koji hub consolidation groundwork	Done	None
New Releng Deliverable: Windows & OSX tools in Fedora	Done	None	Additional requests for improved support after Beta have been filed
Content signing can be automated	Done	None
Container Signing in Sigul	Done	None
Bodhi release 2.3	Done	None
OSBS has redundancy and support for multi-master / multi-node	In progress	None	Multi node support deployed in stage; pending firewall updates and then release to prod after F25 GA Freeze
Taskotron	Delayed	Remaining work	Dev update live; prod update pending decision about storing tests in dist-git and fixing fedmsg events
Ansible: Docker 2 week build/test/release workflow automated	Blocked	Blocked by what tool to use for automation

In Progress F26 Changes

Change	Status	Risks/Open Issues	Next
Distgit: Check when people add Exclude/ExclusiveArch to packages, notify the arch teams	In Progress	None	Tweaking setup to reflect requirements
Deploy koji signed repo support - scope work for Pungi & Bodhi to use it	Blocked	Need PR revisions due to the amount of elapsed time Also need to support DNF in PR ongoing delays on the merge will impact our ability to scope work for Pungi & Bodhi
Kerberos support in koji, fedpkg, OSBS	In Progress	None	Live in stage & prod, will be requesting testing this week
F26 Hub consolidation: aarch64 & Power64 koji merged; prep to import s390x in F27	In Progress	Merge introduced a number of issues that are mostly fixed now
Public hardware for AutoCloud	In Progress	None	Hardware is in hand, pending powering up and handing off to Autocloud folks

Other

• New Deliverables (not covered above)
  • Atomic for Power - Blocked; builds are not completing successfully
• Testing koji updates (1, 2) in stage
• Atomic ostree repo management is in progress again; autocloud testing has been updated to run per compose rather than per build; will move on to revising the 2 week script to deliver these requirements next
• I have added a section for F26 deliverables that have started this month and will add the full set of committed changes next month

13OCT2016

Committed Changes

Change	Status	Risks/Open Issues	Next
Layered image builds	Done	None	Pending 1 bugfix release next week for additional container testing by Cockpit owners
Koji: Koji hub consolidation groundwork	Done	None
New Releng Deliverable: Windows & OSX tools in Fedora	Done	None	Additional requests for improved support after Beta have been filed
Content signing can be automated	In Progress	None	Pending updates / bugfix release planned for this week after beta freeze
Container Signing in Sigul	In Progress	None	Pending final release this week after beta freeze
Taskotron	Delayed	Remaining work	Dev update live; prod update pending decision about storing tests in dist-git
Ansible: Docker 2 week build/test/release workflow automated	Blocked	Blocked by what tool to use for automation & where to run playbooks
OSBS has redundancy and support for multi-master / multi-node	Blocked	Blocked by where to run playbooks	Run playbooks on deployed VMs
Bodhi release 2.2; 2.3 (?)	2.2.4 is in production	2.3 is in progress and could realistically happen in October

Other

• New Deliverables (not covered above)
  • WS OSTree - In Rawhide and 25 Nightlies
  • Atomic for Power - In progress; planned for Beta
• Koji #48 has a proposed patch and is pending testing in stage - this was a very nice to have for Koji Hub Consolidation
• OSBS: Fedora Docker Registry has been delayed for F26 as we assess the impact of signing & decide on final design
• Atomic ostree repo management
• Beginning F26 planning with the team - if you have input, please contact us - plan to share these priorities in our next status update

13SEP2016

Committed Changes

Change	Status	Risks/Open Issues	Next
Layered image builds	complete but working on 1 remaining bug	No path to resolution of nss bug	Dominik Perpeet will lead getting Cockpit builds into the Fedora toolchain week of Sept 12
Taskotron	Completion delayed	Remaining work	-
Ansible: Docker 2 week build/test/release workflow automated	Loopabull ready for use POC release playbook created	Need decision from RCM about Pungi Docker support Need sign off & input from infra team on running playbooks somewhere At risk of missing beta delivery date	Create playbooks
Content signing can be automated	Code complete, deployed to Stage	Awaiting getting yubikeys in place in Prod	Deploy to Prod approx Sept 14
OSBS: Fedora Docker Registry	Blocked (short term registry solution in place for F25)	Image signing may break our planned design	Meeting with signing stakeholders to determine if we need to revise our plans
Koji: Koji hub consolidation groundwork	Moonshot chassis is functionally deployed, aarch64 merged & imported last week	Koji #48 will be needed soon creating aarch64 disk images in LMC is a blocker - updates were made to support this but have not yet been tested	LMC Disk Images for aarch64 & armv7
OSBS has redundancy and support for multi-master / multi-node	Working through multi-host ansible now	No requests for extra VMs submitted yet At risk of missing beta delivery date	Apply multi host updates to existing infra
New Releng Deliverable: Windows & OSX tools in Fedora	Dev complete from engineering; OSX Beta will be built outside of koji / Windows will be built in koji Signing key for Windows purchased Initial builds have completed but testing is ongoing	Need signing keys for OSX No testing of signed images has been done yet Resourcing for OSX work is now up in the air again - discussion ongoing with managers	Need to have initial images built by week of Sept 13
Upcoming Bodhi 2.2.0 release, beta deployed to stg	Waiting one week for it to go through stg testing

Other

• New Deliverables (not covered above)
  • WS OSTree - In Rawhide; pending addition to compose for Beta
  • Atomic for Power - In progress; planned for Beta
• Atomic ostree repo management - feedback period on proposal has closed; work targeted to be live for one of the October 2 week updates
• Pungi got better validation for configuration.

   $ pungi-config-validate fedora.conf
   Unrecognized config option: link_files. Did you mean link_type?
   Unrecognized config option: notification_script.
   Deprecated config option: pkgset_koji_url; use koji_profile instead.
   Failed validation in ostree_installer.0.1.x86_64:
       Additional properties are not allowed ('filename' was unexpected)