Event Queue

What is the Event Queue?

The Event Queue is a fast and scalable distributed message queueing service. It is designed to allow partners to react to changes to resources in the User, Rights and Subscription systems and is intended as a temporary buffer for messages awaiting processing.

When data in the Global Backend (GBE) changes, these changes can trigger events that should be processed by integration partners. Common events are UserAnnounce (triggered when a new user signs up) or ServiceAnnounce (triggered when the user starts using a service which has a service name).

The Event Queue can be thought of as a mailbox for messages from the system about something that happened. Unlike a normal mailbox, the Event Queue really wants to make sure that your message has been received. The Event Queue keeps a copy of all messages until notified that the message has been read and acknowledged.

Typical event flow

The normal event flow works as follows:

Typical Event Queue data flow

First, the client requests a message from the Event Queue. This message is generally the next message that was produced by the system. However, due to the distributed nature of the underlying messaging system this may not always be the case. In any case, a message is returned to the client. The message is marked as "InFlight" in the Event Queue to signal that a client has the message but that it has not been fully processed.

At this point the client should process the message. Processing should generally be quick and should roughly complete within 5 seconds. If more time is needed, the client should make its own persistent copy of the message (e.g. in an internal queue). Once the message has been processed or stored for later processing, the client sends an acknowledgement to the Event Queue. At this point the Event Queue will delete the message. The client should then try to fetch the next message.

Ensuring delivery

The Event Queue will not delete a message before it has been acknowledged by the client. This case looks something like the following sequence:

Event Queue and unacknowledged messages

First, the client requests a message from the Event Queue as normal. The Event Queue responds with the next available message as normal. Then some component in the system fails and the event is not acknowledged. This could happen for any number of reasons:

  • The client fails
  • The server fails
  • The acknowledgement is lost due to network failure

In any case the Event Queue does not receive an acknowledgement within the timeout window (typically 30 seconds) and returns the message to the queue for a subsequent client request to process. The Event Queue is guaranteeing that the message will be processed by the client at least once.

The implications of this error is that the client may receive messages out-of-order or receive the same message multiple times and should be implemented defensively to handle these cases.

What are best practices for handling events?

The following are some best-practices with regard to message processing:

  1. Program defensively -- Errors can occur in many places and may not always be in your control. Assume that any network call can fail and ensure that failures are safely handled. Also make sure that the client can handle messages in any order and duplicate messages.

  2. Scale out to increase throughput -- There is a limit to the number of requests that a single thread can make and still handle the messages that are received. To increase throughput, increase the number of threads, clients and machines that are making requests. Use the queue status endpoint to find the balance.

  3. Keep the queue empty -- Messages that have not been read for a certain amount of time, currently four days, will be deleted.

  4. Familiarize yourself with the API -- Documentation for the API is located here.