Publish Subscribe Notification Proposal

Publish/Subscribe Notification for Fedora Infrastructure

NOTE: Portions of this document are being reworked to integrate the QMF framework

Authors

John (J5) Palmieri

Abstract

This proposal outlines the publish/subscribe infrastructure for notification within the Fedora Infrastructure pieces using the AMQP protocol, MRG Messaging services and the QMF object framework. Where possible library interaction will be described as a generic publish/subscribe layer over the complexities of AMQP/QMF. While the QMF protocol gives suggestions on naming conventions, this document will attempt to solidify best practices when dealing with namespaces inside the Fedora Infrastructure universe.

Definitions

• address - internet URI for connecting to broker
• broker - a simplification of the AMQP exchange/binding/queue concept. A broker routes and relays messages to their intended recipients. Intermediate brokers can be used to relay messages to specific security domains. For instance bodhi and koji wouldn't send messages directly to subscribers, they would send messages to only routers they trust such as the fedora community broker. In turn applications running in fedora community would trust the FComm router is sending them messages from koji and bodhi and not some other source. A similar broker can be run for desktop apps which would require individual FAS tokens to be able to monitor resource usage.
• callback - a function that is called in response to an incoming message
• client - any application publishing or consuming messages through AMQP
• event - identifier for the message being sent, e.g. koji would send build_started, build_finished events
• package - identifier for specific publishers, e.g. koji would identify itself as the org.fedoraproject.koji package
• payload - data QMF format sent with the signal
• publisher - any client sending messages in AMQP
• subscriber - any client consuming messages in AMQP

Physical Layout

Because we need to be able to control access in order to not overload our servers/bandwidth and to provide security the AMQP routers need to be split into different security domains, each with their own access rules and purpose.

Fedora Infrastructure AMQP Broker

This is the service that all Fedora Infrastructure applications publish notifications through. It is internal and only allows internal services publish and subscribe on it. It can do this through several methods including, an ip/domain white list, shared secrets (aka passwords), certificates and physical isolation from external networks. Everything published on this broker will be trusted by the downstream brokers as being correct. Downstream brokers will not be able to publish, only subscribe to messages on this broker.

Fedora Community AMQP Broker

This broker listens to messages from the Fedora Infrastructure Broker (as well as any other notification sources such as upstream) and translates the message to JSON. It then sends this message over an Orbit socket to any Fedora Community web app subscribed. Browser domain security ensures that only apps coming from the Fedora Community server can create a connection. This Broker is internal and relies on an Orbit bridge to communicate with external browsers. Applications are not yet allowed to publish messages to Fedora Community (they have to use the normal REST interfaces) though it might be interesting to see if we can export the REST interfaces over this bridge.

FAS User AMQP Broker

This broker listens to messages from the Fedora Infrastructure Broker (as well as any other notification sources such as upstream) and allows external applications to become AMQP clients. This is an external broker and is locked down via FAS user and group authentication. This allows fine grained control over access. For instance if someone is abusing the connection and taking up huge amounts of bandwith we can turn off their access in FAS. Uses include native applications which notify users when events happen. Things like notification icons and bubbles can for instance tell a user that their build is done without them having to load up a web page. We can eventually open this up to trusted external entities so that they can route Fedora messages to their users without taking up too much of our bandwidth (e.g. OLPC/Sugar Labs would provide notifications to their users instead of everyone subscribing to the Fedora Servers).

Other Brokers

Other brokers can be setup to spider the notifications for load balancing or setting up different security profiles. For instance we may want to have a broker for new and third party Fedora Infrastructure pieces where their interfaces may change significantly and in general can not be trusted as much as the established infrastructure pieces. As the piece becomes more stable it can be moved to the main Infrastructure broker.

Fedora AMQP Standards

Naming

Since we are using the publish/subscribe routing for notification naming becomes important for avoiding name clashes. All message routing keys will be use this naming standard:

<domain>.<signal>

Domains

A domain identifies a service. All domains must be unique and application should not trample on other applications domains. This is mostly avoided by the physical setup and can be locked down using different authentication tokens for each service, (or we can just implicitly trust internal services to be configured correctly and not trample on other's name spaces). All domains originating from Fedora Infrastructure should be prefixed with org.fedoraproject and end with their service name and any optional sub domains. For example:

org.fedoraproject.koji.builds

In the above example koji is the service name and builds is the sub domain. It should be noted that subdomains are optional and are only included for services which may want to organize their functionality. In the future we may use subdomains to restrict which signals routers and clients can connect to.

Events

Event names can not contain any period characters. They are used by the client to route to the correct callback handler. For example:

successful_build

In the above example koji is sending the successful_build signal.

Subscribing

Since we want to leave most of the AMQP complexity to a library and bring more standardization for our particular use case, here is a conceptual API for subscribing to a signal:

router = messsaging_connect("fcomm.messaging.fedoraproject.org", security_token)
result = router.subscribe("org.fedoraproject.koji",
                          "successful_build", process_successful_build,
                          "failed_build", process_failed_build)

As you notice we first connect to the router we are interested in and provide a security token (could be user/pass or a session_id, etc.). We then subscribe to the domain org.fedoraproject.koji.

In AMQP Publish/Subscribe terms we would be binding a queue to the routing_key org.fedoraproject.koji.# (# being a wild card). When the topic org.fedoraproject.koji.successful_build comes over the wire our API will process it and route the payload to the process_successful_build callback.

Publishing

Again we want to remove most of the complexity from the API and make it easy to publish standard formatted signals. Here is a conceptual API for publishing a signal:

router = messsaging_connect("fcomm.messaging.fedoraproject.org", security_token)
domain = router.request_domain("org.fedoraproject.koji")
msg = domain.create_signal("successful_build", payload)
msg.publish()

Here we connect to the router with our security token again. We then request a domain. If the domain is taken or we are not authorized to send signals an error will occur. Then whenever we wish to send a signal we use that domain and send the signal with the payload.

Payload

The payload should be a serialized JSON string similar to what is returned when requesting a particular record using REST or XML-RPC. The subscribe API should automatically deserialize the payload in the binding's preferred format. The only piece of data absolutely required is a unique primary key which can be used to obtain the record using the REST/XML-RPC API's and also identify the difference between a state change and a new record. We should use this opportunity standardize on using the identifier primary_uid for this purpose (we don't use id/uid to avoid clashing with already existing usage).

Further Routing Options

Sometimes we don't want to listen to all signals from a particular domain. For instance if I am just looking to be notified of builds I started or builds on a particular package. AMQP allows us to route based on arbitrary header key value pairs as well as the payload content. The API should allow services to enter key/value pairs for future use. For instance:

key_value_headers = {"built_by": "johnp"}

msg.update_headers(key_value_headers)
msg.publish()

This would give us the option to use the fast key/value queuing or write a json router down the road. Right now domains are sufficient for routing but it is easy enough to have the services adding these key/value pairs so that we get used to using them and they are there when we need them. I don't recommend writing a json router since the extra deserialization required could lead to lazy code causing performance issues but it does have the advantage of anything we send being able to be used for routing without duplicating it in the headers.