[OOZIE-599] Enable Oozie to be a subscriber to a topic on a message bus topic for job end notifications

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Craig Peters
• Created: 04/Nov/11 18:07
• Updated: 14/Nov/11 23:25

Description

Oozie should be able to listen to a user configurable topic on a message bus via JMS for job completion notifications. This requires use a general inter-component communication system using the publish/subscribe model provided by ActiveMQ. http://activemq.apache.org/ The value for Oozie and Hadoop is to enable a less brittle communication because Oozie would not poll the Job Tracker for job completion in the absence of a URL or a timeout.

Hadoop currently implements a call back URL to notify clients including Oozie that a job has completed. See https://issues.apache.org/jira/browse/MAPREDUCE-3028. This implementation has a number of challenges including that only one client can get notification easily and the implementation blocks the thread so that a bad URL blocks job tracker execution. This requires Map Reduce to enable a way for a job creator to specify which topic to which to publish the job end notification message.

Issue Links

• is related to: HADOOP-7821 Hadoop event notification system