Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1111

Job completion notification to a job configured URL

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.12.2
    • 0.13.0
    • None
    • None
    • all

    Description

      Currently clients have to poll the JobTracker to find if a job has completed or not.

      When invoking Hadoop from other systems is desirable to have a notification mechanism on job completion.

      The notification approach simplifies the client waiting for completion and removes load from the JobTracker as polling can be avoided.

      Proposed solution:

      When the JobTracker processes the completion of a job (success and failure) if the job configuration has a jobEnd.notificationUrl property it will make a HTTP GET request to the specified URL.

      The jobEnd.notificationUrl property may include 2 variables in it '${jobId}' and '${jobStatus}'. if they are present, they will be replaced with tehe job ID and status of the job and the URL will be invoked.

      Two additional properties, 'jobEnd.retries' and 'jobEnd.retryInterval', will indicate retry behavior.

      Not to delay the JobTracker processing while doing notifications, a ConsumerProducer Queue will be used to queue up job notification upon completion.

      A daemon thread will consume job notifications from the above Queue and will make the URL invocation.

      On notification failure, the job notification is queue up again on the notification queue.

      The queue will be a java.util.concurrent.DelayQueue. This will make job notifications (on retries) to be avaiable on the consumer side only when the retry time is up.

      The changes will be done in the JobTracker and in the LocalJobRunner.

      Attachments

        1. patch-1111.txt
          23 kB
          Alejandro Abdelnur
        2. patch-1111.txt
          23 kB
          Alejandro Abdelnur
        3. patch-1111.txt
          23 kB
          Alejandro Abdelnur
        4. patch-1111.txt
          21 kB
          Alejandro Abdelnur
        5. patch-1111.txt
          21 kB
          Alejandro Abdelnur
        6. patch-1111.txt
          11 kB
          Ruchir
        7. patch-1111.txt
          11 kB
          Ruchir
        8. patch-1111.txt
          10 kB
          Alejandro Abdelnur
        9. patch-1111.txt
          10 kB
          Alejandro Abdelnur
        10. patch-1111.txt
          10 kB
          Alejandro Abdelnur
        11. patch-1111.txt
          11 kB
          Alejandro Abdelnur

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            tucu00 Alejandro Abdelnur
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment