From Fedora Project Wiki

No edit summary
No edit summary
Line 20: Line 20:
We have been in a planning stage of the SIG for many years: bringing interested people together to flesh out the who, what, where, when, and how.
We have been in a planning stage of the SIG for many years: bringing interested people together to flesh out the who, what, where, when, and how.


We started earnest development in Spring of 2012 around [http://github.com/ralphbean/fedmsg fedmsg].  See [http://fedmsg.readthedocs.org/en/latest/status.html this table] for integration status.
We started earnest development in Spring of 2012 around [http://github.com/ralphbean/fedmsg fedmsg].  See [http://fedmsg.readthedocs.org/en/latest/status this table] for integration status and [http://fedmsg.readthedocs.org/en/latest/topology this diagram] for the system-wide view.


== Communication ==
== Communication ==
Line 28: Line 28:


== Schedule ==
== Schedule ==
None yet, need to work out a timeline for implementing a Messaging system.


=== Brainstorming ===
The [http://fedmsg.readthedocs.org/en/latest/#rough-outline-of-stages-of-development-deployment rough outline from the fedmsg docs] break things down into the following 5 major phases:
Things to add to schedule in order to implement this


* setup a qpid server in publictest infrastructure
1) Write fedmsg core routines for sending and receiving.  (DONE, in maintenance mode)
* Figure out how QMF fits into our plans
2) Start modifying or instrumenting our apps to use fedmsg to send messages.  (Mostly done, see [http://fedmsg.readthedocs.org/en/latest/status the status table] for what is done and what is left).
* Design notification server to interact with the bus, receiving messages and sending notifications out
3) Consume messages for statistics.  (Done, via [https://github.com/ralphbean/datanommer datnommer].)
** lmacken mentioned that a moksha plugin might be suitable for this
4) Consume messages for user experience.  (Done, via the #fedora-fedmsg irc bot, [https://github.com/lmacken/fedmsg-notify fedmsg-notify], and [http://apps.fedoraproject.org/busmon busmon].
** Figure out how people can change their notification settings -- should the settings be in the apps and then the notification server polls the apps for updates? Or should the apps redirect people to the notification server?
5) Consume messages for interoperability.  (Not started. Although, [https://fedoraproject.org/wiki/User:Herlo herlo] started work on a consumer to update fama trac tickets in response to FAS fedmsg messages.)
* setup notification server on publictest infrastructure
* migrate qpid and notification server to stg.
* deploy
* port apps to use notification server
** koji plugin
** pkgdb change
** fas?  (Or is this essential enough that it should stay in fas?)
** bodhi
** cvs commit mail
* What should the API look like?


=== Proposals ===
== Brainstorming ==


* [[Messaging_SIG/PublishSubscribeNotificationProposal|A proposal for implementing a publish/subscribe notification infrastructure for Fedora]]
Here's a list of things you could work on if you want to help!  It is by no means exhaustive, but is just to get the ball rolling. Add things here if you have an idea.
* [https://github.com/ralphbean/fedmsg/blob/develop/doc/proposal.rst A 0mq proposal]


== Tasks ==
* Work on centralizing user notifications preferences by helping with/taking charge of [http://github.com/rossdylan/FUSS FUSS].
* Bring people into the SIG
* Add hooks to *any* project that doesn't already appear as completed on the [http://fedmsg.readthedocs.org/en/latest/status/ status table].
* Discuss high level goals of the SIG
* Work with RH to get bugzilla messages.
* Create a schedule
** They'll be providing them via AMQP from qpidd.  We'll need to build a fedmsg service called "fedmsg-amqp-bridge" that receives amqp messages and rebroadcasts them to our zeromq bus.
* Define an object & event messaging model
* Improve the documentation anywhere you see fit!  Is it unclear anywhere?
* ???
* Help write a policy for consumers that are acceptable for being hosted in Fedora Infrastructure.
* Profit
* Start building consumers that respond and react to events on the bus.  Automate our ecosystem!


=== Triggered Mirroring ===


== Use Cases ==
[https://fedoraproject.org/wiki/User:Ausil dgilmore] mentioned that he knows somebody working on this.
At FUDcon Toronto 2009 we came up with this list of needs for each of the services that may end up on the bus, also noted is what services would then be sending that data out on the bus.


[[File:amqp_cases.png]]
For this the following wedges would need to be made to place the services on the bus:
===Shims===
*NetApp
*Compose
*AutoQA
*PkgDB
*Koji
*Bodi
*SCM
*[[Messaging_SIG/Bugzilla_Shim|Bugzilla]]
*Zabbix
=== Triggered Mirroring ===
# Message when the NetApps snapmirror goes out of sync, due to a write to the master.  Tier 1 mirrors subscribe, and don't pull until next message.
# Message when the NetApps snapmirror goes out of sync, due to a write to the master.  Tier 1 mirrors subscribe, and don't pull until next message.
# Message when the NetApps snapmirror is back in sync.  Tier 1 mirrors subscribe, and should pull now.
# Message when the NetApps snapmirror is back in sync.  Tier 1 mirrors subscribe, and should pull now.
Line 88: Line 59:


In all cases, having a clue as to which part of the tree has just changed would be useful.  Even a 'directories from this point downward are likely to have changed' would be of benefit.
In all cases, having a clue as to which part of the tree has just changed would be useful.  Even a 'directories from this point downward are likely to have changed' would be of benefit.
==Naming Scheme==
==Naming Scheme==
We need to define what this will look like, perhaps something like:
*org.fedoraproject.*
*org.fedoraproject.fas
== Resources ==
Links to documentation and project pages for the various tools we'll be using.


* [http://qpid.apache.org/ QPID project home page]
See [http://fedmsg.readthedocs.org/en/latest/overview/#namespace-considerations this section on topic namespace] from the [http://fedmsg.rtfd.org fedmsg docs]. [http://fedmsg.readthedocs.org/en/latest/publishing/ This document on sending messages] might also be helpful.
* [http://qpid.apache.org/qmf-python-console-tutorial.html QMF Python Console Tutorial]
* [http://moksha.fedorahosted.org Moksha]


[[Category:SIGs]]
[[Category:SIGs]]
[[Category:Messaging]]
[[Category:Messaging]]
[[Category:Fedora special-interest groups]]
[[Category:Fedora special-interest groups]]

Revision as of 01:50, 9 October 2012

Mission

Create a Messaging infrastructure within the Fedora Project to facilitate communication, interaction, and integration between services within the Fedora Infrastructure.

Members


Status

We have been in a planning stage of the SIG for many years: bringing interested people together to flesh out the who, what, where, when, and how.

We started earnest development in Spring of 2012 around fedmsg. See this table for integration status and this diagram for the system-wide view.

Communication

Schedule

The rough outline from the fedmsg docs break things down into the following 5 major phases:

1) Write fedmsg core routines for sending and receiving. (DONE, in maintenance mode) 2) Start modifying or instrumenting our apps to use fedmsg to send messages. (Mostly done, see the status table for what is done and what is left). 3) Consume messages for statistics. (Done, via datnommer.) 4) Consume messages for user experience. (Done, via the #fedora-fedmsg irc bot, fedmsg-notify, and busmon. 5) Consume messages for interoperability. (Not started. Although, herlo started work on a consumer to update fama trac tickets in response to FAS fedmsg messages.)

Brainstorming

Here's a list of things you could work on if you want to help! It is by no means exhaustive, but is just to get the ball rolling. Add things here if you have an idea.

  • Work on centralizing user notifications preferences by helping with/taking charge of FUSS.
  • Add hooks to *any* project that doesn't already appear as completed on the status table.
  • Work with RH to get bugzilla messages.
    • They'll be providing them via AMQP from qpidd. We'll need to build a fedmsg service called "fedmsg-amqp-bridge" that receives amqp messages and rebroadcasts them to our zeromq bus.
  • Improve the documentation anywhere you see fit! Is it unclear anywhere?
  • Help write a policy for consumers that are acceptable for being hosted in Fedora Infrastructure.
  • Start building consumers that respond and react to events on the bus. Automate our ecosystem!

Triggered Mirroring

dgilmore mentioned that he knows somebody working on this.

  1. Message when the NetApps snapmirror goes out of sync, due to a write to the master. Tier 1 mirrors subscribe, and don't pull until next message.
  2. Message when the NetApps snapmirror is back in sync. Tier 1 mirrors subscribe, and should pull now.
  3. Message when each Tier 1 mirror has started syncing. Tier 2 mirrors subscribe, and don't pull until the next message.
  4. Message when each Tier 1 mirror has finished syncing. Tier 2 mirrors subscribe, and should pull from their Tier 1 mirror now.

In all cases, having a clue as to which part of the tree has just changed would be useful. Even a 'directories from this point downward are likely to have changed' would be of benefit.

Naming Scheme

See this section on topic namespace from the fedmsg docs. This document on sending messages might also be helpful.

Retrieved from "https://fedoraproject.org/w/index.php?title=Messaging_SIG&oldid=306036"