Infrastructure 2020/Fedora Messaging

This is a shorthand list of items we are needing to cover for migrating from fedmsg to fedora-messaging in Fedora Infrastructure for the Fiscal Year 2020.

Point of contact: Aurelien Bompard

Why?

Fedmsg has a number of issues but the main one is that it is unreliable. Since critical elements of our CI pipeline (and general infrastructure) are depending on it, it is important to migrate to fedora-messaging to solve that issue.

This is also why the migration to fedora-messaging needs to happen before the Gating Rawhide initiative goes to production (development can start, though).

What?

About 30+ apps require changes. The most critical ones are:

1. bodhi
2. koji
3. greenwave
4. waiverdb
5. pagure
6. centos infrastructure CI
7. openqa
8. anitya & the-new-hotness

Migrating to fedora-messaging will also allow us to define schemas for our data, use versions on the topic to avoid breakage in the message consumers, and rethink the topic to make them more useful with the routing capabilities we have. However, these tasks are out of scope for this initiative, for time/resources reasons.

The other applications will be migrated at a slower pace throughout FY20.

When?

• Critical applications: by 2019-06-01
• Other applications: 2020-03-31

Who?

The people involved will be abompard and the owner of each app (be it for writing the migration or for reviewing and testing it).

How?

We will start with the application that are most critical (see above).

The changes from fedmsg to fedora-messaging are usually similar in each application, so it can be a great opportunity for pair programming between abompard and the app owners.

It is however more complex to migrate an app that has consumers, because a new systemd service file must be written (and maybe a new script). In fedmsg only a class had to be defined and the system's fedmsg-hub daemon would pick it up. It's no longer how consumers work in fedora-messaging.

The staging and production networks are ready for fedora-messaging, so the migration and testing can start immediately.

Since this is a significant change from a deployment point of view, app owners may decide to change their major version number, but this decision is left to them.

Status

https://github.com/orgs/fedora-infra/projects/2

Details per-app

Bodhi

Owners: Randy Barlow & Aurélien Bompard

The producing side on the server is already migrated to fedora-messaging. Some PRs are left to merge: consumers , message schemas.

The migration to Fedora Messaging should be available in Bodhi 4.0

Koji

Owner: Nobody, since the plugin was just sitting in our Ansible repo (Ralph, Patrick & Kevin have edited it in the past)

Koji sends fedmsgs via a plugin, this plugin should be migrated. I did not find any upstream source control for the koji plugin, unless I'm mistaken it only lives in the Infra Ansible repository.

i created a space on Pagure to have a proper git, tracker, releases, tags, etc. : https://pagure.io/koji-fedmsg-plugin/

Tasks:

• Replace the fedmsg_koji_instance template variable by a configuration file.
• Migrate to Fedora Messaging
• Create the proper python distribution boilerplate
• Publish on PyPI
• Add unit tests
• Add documentation
• Create an RPM (or choose an alternate deployment mode)
• Update the Infra Ansible
• Test on staging
• Deploy to production

Greenwave

Source code: https://pagure.io/greenwave

Owner: Ralph's team

Greenwave has 2 consumers (ResultsDB and WaiverDB) and produces one message topic. The fedmsg config is used to store the consumers' configuration. There are only 2 calls to fedmsg.publish.

Tasks:

• Write a fedora-messaging consumer class and two handling classes (or functions) for the two message types.
• Update the unit tests
• Create a systemd unit file
• Update the installation system (python setup.py, spec file, Dockerfile, ansible infra)
• Update the documentation
• Test on staging
• Deploy to production

WaiverDB

Source code: https://pagure.io/waiverdb

Owner: Ralph's team

WaiverDB only produces one message topic. It does not use the fedmsg configuration to store its own configuration. There's only one call to fedmsg.publish.

Tasks:

• Migrate to Fedora Messaging
• Update the unit tests
• Update the installation system (spec file, Infra Ansible)
• Update the documentation
• Test on staging
• Deploy to production.

ResultsDB

Source code: https://pagure.io/taskotron/resultsdb

Owner: Tim Flink

ResultsDB only produces one message topic. There are only 2 calls to fedmsg.publish.

Tasks:

• Migrate to Fedora Messaging
• Update the unit tests
• Update the installation system (spec file, Infra Ansible)
• Update the documentation
• Test on staging
• Deploy to production.

Status tracking: https://pagure.io/taskotron/resultsdb/issue/125

Pagure

Source code: https://pagure.io/pagure/

Owner: Pierre-Yves Chibon

Pagure only produces messages, calls to fedmsg.publish are centralized, and fedmsg.config is not used to store app configuration.

Tasks:

• Migrate to Fedora Messaging
• Update the unit tests
• Update the installation system (spec file, Infra Ansible)
• Update the documentation
• Test on staging
• Deploy to production.

CentOS Infrastructure CI

Owner: Brian Stinson

CentOS CI is publishing via a JMS plugin that has ActiveMQ and Fedmsg backends.

Source code: https://github.com/jenkinsci/jms-messaging-plugin/tree/master/src/main/java/com/redhat/jenkins/plugins/ci/messaging

TODO: make sure the CentOS CI is not consuming messages.

Tasks:

• Write a Fedora Messaging backend using a Java AMQP library.
• Decide if we want a federated broker in the CentOS infra, if we give them access to ours via a VPN, or if we create a public read-only broker with a special read-write access for CentOS.
• Deploy that broker or VPN
• Test the code and deploy it.

Status: contacted one of the authors, Scott Herbert, with some details on the plan and how we deal with AMQP in Fedora Messaging. Awaiting answer.

OpenQA

Owner: Adam Williamson

Source code:

• Server: https://github.com/os-autoinst/openQA/blob/master/lib/OpenQA/WebAPI/Plugin/Fedmsg.pm
• Scheduler: https://pagure.io/fedora-qa/fedora_openqa

The server is an openQA plugin (written in... Perl!) that emits a fedmsg on events, using fedmsg-logger.

The scheduler is a consumers for the folowing topics:

• org.fedoraproject.*.openqa.job.done
• org.fedoraproject.*.pungi.compose.status.change
• org.fedoraproject.*.bodhi.update.request.testing
• org.fedoraproject.*.bodhi.update.edit

The fedmsg configuration is only used to decide which consumer to enable.

Tasks for the scheduler:

• Write a fedora-messaging consumer class and as many handling classes (or functions) as necessary.
• Update the unit tests
• Create a systemd unit file
• Update the installation system (spec file, ansible infra)
• Update the documentation
• Test on staging
• Deploy to production

Status tracking: https://pagure.io/fedora-qa/fedora_openqa/issue/73

Tasks for the server plugin:

Not sure yet. We need to decide if it's best to use an AMQP binding in Perl or to write an equivalent of fedmsg-logger (like in this PR)

Adam note: openQA actually has an existing AMQP publisher plugin. I haven't tested it yet, but we could likely use that to publish to fedora-messaging. There is a wrinkle, though. The fedmsg publisher plugin (which I wrote and maintain) publishes two different sets of messages. There's the older format, where it basically just takes openQA's internal events and republishes them. Those are the messages on the 'openqa' topic - see the datagrepper log. It also publishes another stream of messages in a format which is intended to follow the CI messaging spec. Here is an example of such a message, which have topics under the 'ci' category, e.g. ci.fedora-update.test.complete. The AMQP plugin I believe would publish messages very similar to the older 'openqa' ones; it has no ability to publish the 'ci' type messages. I will have to send a PR to enhance it to do this.

Anitya

Owner: Michal Konečný

Source code: https://github.com/release-monitoring/anitya

In current state Anitya is consuming and publishing various topics. Publishing only in future (See bellow).

PR with first draft of publishing migration to fedora messaging https://github.com/release-monitoring/anitya/pull/570. This PR will be merged shortly.

Anitya consumes libraries.io topics. However this will be removed and the SSE consumer will be part of Anitya instead of standalone application (It is the only application that is actually listening to libraries.io publisher). After this change Anitya will not consume any topics.

The-New-Hotness

Owner: Michal Konečný

Source code: https://github.com/fedora-infra/the-new-hotness

The-new-hotness is consuming and publishing few topics.