Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3612

Task.TaskReporter.done method blocked for some time when task is finishing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      We recently have done some tests to evaluate performances of different Hadoop versions(1.0, 0.23, Baidu internal version), and found some weird results. One of them is in 1.0 Task.TaskReporter.done() takes too much time, about 2s, this is bad for small tasks. After reviewing source code and add some log, the following code block Task.TaskReporter.done

      src/mapred/org/apache/hadoop/mapred/Task.java
       658           try {
       659             Thread.sleep(PROGRESS_INTERVAL);
       660           }
      
      
       723     public void stopCommunicationThread() throws InterruptedException {
       724       // Updating resources specified in ResourceCalculatorPlugin
       725       if (pingThread != null) {
       726         synchronized(lock) {
       727           while(!done) {
       728             lock.wait();
       729           }
       730         }
       731         pingThread.interrupt();
       732         pingThread.join();
       733       }
       734     }
      

      Originally line 724-730 don't exists, I don't know why it is added. If it is needed, we can replace Thread.sleep with Object.wait(timeout) and Object.notify instead, so it won't block.

        Attachments

        1. MAPREDUCE-3612.patch
          1 kB
          Binglin Chang

          Issue Links

            Activity

              People

              • Assignee:
                decster Binglin Chang
                Reporter:
                decster Binglin Chang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: