Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5066

JobTracker should set a timeout when calling into job.end.notification.url

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1-win, 2.0.3-alpha, 1.3.0
    • Fix Version/s: 1.2.0, 1-win, 2.1.0-beta
    • Component/s: None
    • Labels:
      None

      Description

      In current code, timeout is not specified when JobTracker (JobEndNotifier) calls into the notification URL. When the given URL points to a server that will not respond for a long time, job notifications are completely stuck (given that we have only a single thread processing all notifications). We've seen this cause noticeable delays in job execution in components that rely on job end notifications (like Oozie workflows).

      I propose we introduce a configurable timeout option and set a default to a reasonably small value.

      If we want, we can also introduce a configurable number of workers processing the notification queue (not sure if this is needed though at this point).

      I will prepare a patch soon. Please comment back.

        Attachments

        1. MAPREDUCE-5066.branch-1-win.patch
          16 kB
          Ivan Mitic
        2. MAPREDUCE-5066.branch-1-win.2.patch
          16 kB
          Ivan Mitic
        3. MAPREDUCE-5066.branch-1-win.3.patch
          16 kB
          Ivan Mitic
        4. MAPREDUCE-5066.patch
          7 kB
          Ivan Mitic
        5. MAPREDUCE-5066.branch-1-win.4.patch
          17 kB
          Ivan Mitic
        6. MAPREDUCE-5066.2.patch
          8 kB
          Ivan Mitic
        7. MAPREDUCE-5066.branch-1-win.5.patch
          17 kB
          Ivan Mitic
        8. MAPREDUCE-5066.3.patch
          17 kB
          Ivan Mitic

          Issue Links

            Activity

              People

              • Assignee:
                ivanmi Ivan Mitic
                Reporter:
                ivanmi Ivan Mitic
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: