Uploaded image for project: 'OODT'
  1. OODT
  2. OODT-946

Implement Kafka Messaging System Within OODT

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • 2.0
    • None
    • Committer (Medium) - This is for regular contributors/committers

    Description

      Under the hood of Apache OODT we make use of XMLRPC and/or Apache Avro to help with messaging and communications.

      Our proposal for Apache OODT 2.0 is to replace some of this underlying communication trafficking with Apache Kafka. In doing so this will allow multiple components to subscribe to streams and register their interest. It also allows us to track messages flowing throughout the system allowing for much improved auditability.

      The initial work would be done in the file manager, replacing some of the internal communications with Kafka based messaging and also then providing an external interface that other OODT components can subscribe to allowing for event registration when files are ingested.

      Secondly we would look to replace the messaging within the Workflow Manager with Kafka there by allowing easier tracabilty of the data flowing through it and again allowing other components to register themselves against it. This will also make integration of 3rd party applications easier.

      Part 3 would introduce Kafka based messaging into the resource manager, allowing components to be aware of the resources available to them in real time.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            bugg_tb Tom Barber
            bugg_tb Tom Barber

            Dates

              Created:
              Updated:

              Issue deployment