Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4786

Job End Notification retry interval is 5 milliseconds by default

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.0.2-alpha, 0.23.4
    • Fix Version/s: 3.0.0, 2.0.3-alpha, 0.23.5
    • Component/s: mrv2
    • Labels:
      None

      Description

      Courtesy Steven Willis and Harsh J

      From: Harsh J
      I believe the configs of the latter of both of the above
      classifications were meant to be added in as replacement names, but
      the property names got added in wrong (as the former/older named ones)
      in the XML.

      the word "seconds" in the description of retries? The code in MR2's
      JobEndNotifier seems to expect seconds but uses it directly in
      Thread.sleep(…) without making it milliseconds, which may be a bug we
      need to fix as well, perhaps in a same issue as the configs ones.

      On Fri, Nov 9, 2012 at 11:21 PM, Steven Willis <swillis@compete.com> wrote:
      > And I noticed that there are some duplicate properties with different values and different descriptions:

      1. MAPREDUCE-4786.patch
        11 kB
        Ravi Prakash
      2. MAPREDUCE-4786.branch-2.patch
        11 kB
        Ravi Prakash
      3. MAPREDUCE-4786.branch-2.patch
        11 kB
        Ravi Prakash
      4. MAPREDUCE-4786.patch
        11 kB
        Ravi Prakash
      5. MAPREDUCE-4786.patch
        11 kB
        Ravi Prakash

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1256 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1256/)
        MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408444)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408444
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1256 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1256/ ) MAPREDUCE-4786 . Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408444) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408444 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1225 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1225/)
        MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408444)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408444
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1225 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1225/ ) MAPREDUCE-4786 . Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408444) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408444 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #434 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/434/)
        svn merge -c 1408444 FIXES: MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408456)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408456
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #434 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/434/ ) svn merge -c 1408444 FIXES: MAPREDUCE-4786 . Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408456) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408456 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #35 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/35/)
        MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408444)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408444
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #35 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/35/ ) MAPREDUCE-4786 . Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408444) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408444 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Robert Joseph Evans added a comment -

        Thanks Ravi,

        I pulled this into trunk, branch-2, and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks Ravi, I pulled this into trunk, branch-2, and branch-0.23
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3003 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3003/)
        MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408444)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408444
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3003 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3003/ ) MAPREDUCE-4786 . Job End Notification retry interval is 5 milliseconds by default (Ravi Prakash via bobby) (Revision 1408444) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1408444 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Robert Joseph Evans added a comment -

        Oh I forgot to mention that I ran the tests that failed manually and they passed for me. I am not really sure what happened there. They look like there are some races in those tests.

        Show
        Robert Joseph Evans added a comment - Oh I forgot to mention that I ran the tests that failed manually and they passed for me. I am not really sure what happened there. They look like there are some races in those tests.
        Hide
        Robert Joseph Evans added a comment -

        Better, but the value of mapreduce.job.end-notification.url needs to be removed or commented out. I can do that when I check it in. +1

        Show
        Robert Joseph Evans added a comment - Better, but the value of mapreduce.job.end-notification.url needs to be removed or commented out. I can do that when I check it in. +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12553161/MAPREDUCE-4786.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:

        org.apache.hadoop.mapreduce.v2.app.TestRecovery
        org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3018//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3018//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553161/MAPREDUCE-4786.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: org.apache.hadoop.mapreduce.v2.app.TestRecovery org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3018//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3018//console This message is automatically generated.
        Hide
        Ravi Prakash added a comment -

        Same as patch as before except with --no-prefix

        Show
        Ravi Prakash added a comment - Same as patch as before except with --no-prefix
        Hide
        Ravi Prakash added a comment -

        All right Bobby!
        Here you go

        Show
        Ravi Prakash added a comment - All right Bobby! Here you go
        Hide
        Robert Joseph Evans added a comment -

        The change looks fine to me. My only comment is that I would prefer to see mapreduce.job.end-notification.url have no default value in mapred-default.xml instead of having it be commented out. When it is commented out then it will not appear in the generated documentation for the file.

        Show
        Robert Joseph Evans added a comment - The change looks fine to me. My only comment is that I would prefer to see mapreduce.job.end-notification.url have no default value in mapred-default.xml instead of having it be commented out. When it is commented out then it will not appear in the generated documentation for the file.
        Hide
        Ravi Prakash added a comment -

        The test failure is unrelated to this patch. It fails without the patch as well. I've filed https://issues.apache.org/jira/browse/MAPREDUCE-4787

        Show
        Ravi Prakash added a comment - The test failure is unrelated to this patch. It fails without the patch as well. I've filed https://issues.apache.org/jira/browse/MAPREDUCE-4787
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12553080/MAPREDUCE-4786.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:

        org.apache.hadoop.mapreduce.TestJobMonitorAndPrint

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3010//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3010//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553080/MAPREDUCE-4786.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: org.apache.hadoop.mapreduce.TestJobMonitorAndPrint +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3010//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3010//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12553084/MAPREDUCE-4786.branch-2.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3011//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553084/MAPREDUCE-4786.branch-2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3011//console This message is automatically generated.
        Hide
        Ravi Prakash added a comment -

        For branch-2 and branch-0.23

        Show
        Ravi Prakash added a comment - For branch-2 and branch-0.23
        Hide
        Ravi Prakash added a comment -

        Changes retry interval to 5000 milliseconds and adds to existing test to check the default setting. Also fixes the description in mapred-default.xml

        Show
        Ravi Prakash added a comment - Changes retry interval to 5000 milliseconds and adds to existing test to check the default setting. Also fixes the description in mapred-default.xml

          People

          • Assignee:
            Ravi Prakash
            Reporter:
            Ravi Prakash
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development