Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The progress of the jobs are not being correctly updated on the client side. The map progress halts at 66% and both map/reduce progress % does not display 100 when the job completes.

      1. MAPREDUCE-2737.patch
        6 kB
        Mahadev konar
      2. MR2737_2.patch
        1 kB
        Siddharth Seth
      3. MAPREDUCE-2737.patch
        6 kB
        Mahadev konar
      4. MR2737_1.patch
        1 kB
        Siddharth Seth
      5. MR2737.patch
        1 kB
        Siddharth Seth

        Activity

        Hide
        Ramya Sunil added a comment -

        The progress of a sample job on 0.20.x is being displayed as:

         
        INFO mapred.JobClient:  map 0% reduce 0%
        INFO mapred.JobClient:  map 10% reduce 0%
        INFO mapred.JobClient:  map 70% reduce 0%
        INFO mapred.JobClient:  map 90% reduce 0%
        INFO mapred.JobClient:  map 100% reduce 5%
        INFO mapred.JobClient:  map 100% reduce 10%
        INFO mapred.JobClient:  map 100% reduce 14%
        INFO mapred.JobClient:  map 100% reduce 35%
        INFO mapred.JobClient:  map 100% reduce 42%
        INFO mapred.JobClient:  map 100% reduce 49%
        INFO mapred.JobClient:  map 100% reduce 59%
        INFO mapred.JobClient:  map 100% reduce 73%
        INFO mapred.JobClient:  map 100% reduce 83%
        INFO mapred.JobClient:  map 100% reduce 93%
        INFO mapred.JobClient:  map 100% reduce 100%
        INFO mapred.JobClient: Job complete: job_201105270127_0021
        

        Whereas the same job progress on mrv2 is being displayed as:

         
        INFO mapreduce.Job:  map 0% reduce 0%
        INFO mapreduce.Job:  map 60% reduce 0%
        INFO mapreduce.Job:  map 66% reduce 12%
        INFO mapreduce.Job:  map 66% reduce 32%
        INFO mapreduce.Job: Job job_1311726209366_0214 completed successfully
        
        Show
        Ramya Sunil added a comment - The progress of a sample job on 0.20.x is being displayed as: INFO mapred.JobClient: map 0% reduce 0% INFO mapred.JobClient: map 10% reduce 0% INFO mapred.JobClient: map 70% reduce 0% INFO mapred.JobClient: map 90% reduce 0% INFO mapred.JobClient: map 100% reduce 5% INFO mapred.JobClient: map 100% reduce 10% INFO mapred.JobClient: map 100% reduce 14% INFO mapred.JobClient: map 100% reduce 35% INFO mapred.JobClient: map 100% reduce 42% INFO mapred.JobClient: map 100% reduce 49% INFO mapred.JobClient: map 100% reduce 59% INFO mapred.JobClient: map 100% reduce 73% INFO mapred.JobClient: map 100% reduce 83% INFO mapred.JobClient: map 100% reduce 93% INFO mapred.JobClient: map 100% reduce 100% INFO mapred.JobClient: Job complete: job_201105270127_0021 Whereas the same job progress on mrv2 is being displayed as: INFO mapreduce.Job: map 0% reduce 0% INFO mapreduce.Job: map 60% reduce 0% INFO mapreduce.Job: map 66% reduce 12% INFO mapreduce.Job: map 66% reduce 32% INFO mapreduce.Job: Job job_1311726209366_0214 completed successfully
        Hide
        Siddharth Seth added a comment -

        Changed monitorAndPrintJob to print one additional report after the job completes.

        Show
        Siddharth Seth added a comment - Changed monitorAndPrintJob to print one additional report after the job completes.
        Hide
        Mahadev konar added a comment -

        Sid, there is a spelling mistake in reportedafterCompletion.

        Show
        Mahadev konar added a comment - Sid, there is a spelling mistake in reportedafterCompletion.
        Hide
        Siddharth Seth added a comment -

        Thanks for taking a look Mahadev. Fixed the spelling mistake.
        The patch does require MR2800 as well (otherwise Map progress will stay stuck at 66%)

        Show
        Siddharth Seth added a comment - Thanks for taking a look Mahadev. Fixed the spelling mistake. The patch does require MR2800 as well (otherwise Map progress will stay stuck at 66%)
        Hide
        Mahadev konar added a comment -

        Sid, Is this the right fix? I thought the output lines "completed successfully" in monitorAndPrintJob needs to be changed? no?

        Show
        Mahadev konar added a comment - Sid, Is this the right fix? I thought the output lines "completed successfully" in monitorAndPrintJob needs to be changed? no?
        Hide
        Mahadev konar added a comment -

        Disregard my last comment, was a little confused. Ill attach a testcase with the patch.

        Show
        Mahadev konar added a comment - Disregard my last comment, was a little confused. Ill attach a testcase with the patch.
        Hide
        Mahadev konar added a comment -

        Updated patch with a added a test case. The test case fails without the patch and passes with the patch.

        Show
        Mahadev konar added a comment - Updated patch with a added a test case. The test case fails without the patch and passes with the patch.
        Hide
        Mahadev konar added a comment -

        Also, Sid please make sure you grant the apache licence in your attachments! Its very important!

        Show
        Mahadev konar added a comment - Also, Sid please make sure you grant the apache licence in your attachments! Its very important!
        Hide
        Siddharth Seth added a comment -

        Same as 2737_1, grant license selected.

        Show
        Siddharth Seth added a comment - Same as 2737_1, grant license selected.
        Hide
        Mahadev konar added a comment -

        Patch with necessary changes with the new dir layout.

        Show
        Mahadev konar added a comment - Patch with necessary changes with the new dir layout.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12491724/MAPREDUCE-2737.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        -1 javadoc. The javadoc tool appears to have generated 73 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12491724/MAPREDUCE-2737.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 73 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/530//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        I just pushed this. Thanks Sid.

        Show
        Mahadev konar added a comment - I just pushed this. Thanks Sid.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #806 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/806/)
        MAPREDUCE-2737. Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #806 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/806/ ) MAPREDUCE-2737 . Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #883 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/883/)
        MAPREDUCE-2737. Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #883 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/883/ ) MAPREDUCE-2737 . Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #816 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/816/)
        MAPREDUCE-2737. Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #816 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/816/ ) MAPREDUCE-2737 . Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #776 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/776/)
        MAPREDUCE-2737. Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #776 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/776/ ) MAPREDUCE-2737 . Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #801 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/801/)
        MAPREDUCE-2737. Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #801 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/801/ ) MAPREDUCE-2737 . Update the progress of jobs on client side. (Siddharth Seth and Mahadev Konar via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162979 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJobMonitorAndPrint.java

          People

          • Assignee:
            Siddharth Seth
            Reporter:
            Ramya Sunil
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development