Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5066

JobTracker should set a timeout when calling into job.end.notification.url


    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1-win, 2.0.3-alpha, 1.3.0
    • Fix Version/s: 1.2.0, 1-win, 2.1.0-beta
    • Component/s: None
    • Labels:


      In current code, timeout is not specified when JobTracker (JobEndNotifier) calls into the notification URL. When the given URL points to a server that will not respond for a long time, job notifications are completely stuck (given that we have only a single thread processing all notifications). We've seen this cause noticeable delays in job execution in components that rely on job end notifications (like Oozie workflows).

      I propose we introduce a configurable timeout option and set a default to a reasonably small value.

      If we want, we can also introduce a configurable number of workers processing the notification queue (not sure if this is needed though at this point).

      I will prepare a patch soon. Please comment back.

      1. MAPREDUCE-5066.branch-1-win.patch
        16 kB
        Ivan Mitic
      2. MAPREDUCE-5066.branch-1-win.2.patch
        16 kB
        Ivan Mitic
      3. MAPREDUCE-5066.branch-1-win.3.patch
        16 kB
        Ivan Mitic
      4. MAPREDUCE-5066.patch
        7 kB
        Ivan Mitic
      5. MAPREDUCE-5066.branch-1-win.4.patch
        17 kB
        Ivan Mitic
      6. MAPREDUCE-5066.2.patch
        8 kB
        Ivan Mitic
      7. MAPREDUCE-5066.branch-1-win.5.patch
        17 kB
        Ivan Mitic
      8. MAPREDUCE-5066.3.patch
        17 kB
        Ivan Mitic

        Issue Links


          No work has yet been logged on this issue.


            • Assignee:
              Ivan Mitic
              Ivan Mitic
            • Votes:
              0 Vote for this issue
              7 Start watching this issue


              • Created: