Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      Oozie primarily depends on the job end notification to determine when the job finishes. In the current version, job end notification is implemented in job tracker. Since job tracker will be removed in the upcoming hadoop release (.next), we wander where this support will move. I think this best effort notification could be implemented in the new Application Manager as one of the last step of job completion.

      Whatever implementation will it be, Oozie badly needs this feature to be continued in next releases as well.

      1. RMContainerAllocator.diff
        2 kB
        Hitesh Shah
      2. MAPREDUCE-3028.patch
        20 kB
        Arun C Murthy
      3. MAPREDUCE-3028.patch
        21 kB
        Ravi Prakash
      4. MAPREDUCE-3028.patch
        24 kB
        Ravi Prakash
      5. MAPREDUCE-3028.patch
        24 kB
        Ravi Prakash
      6. MAPREDUCE-3028.patch
        24 kB
        Ravi Prakash
      7. MAPREDUCE-3028.branch-0.23.patch
        24 kB
        Ravi Prakash

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #872 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/872/)
          MAPREDUCE-3263. Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #872 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/872/ ) MAPREDUCE-3263 . Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #844 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/844/)
          MAPREDUCE-3263. Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #844 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/844/ ) MAPREDUCE-3263 . Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1167/)
          MAPREDUCE-3263. Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1167/ ) MAPREDUCE-3263 . Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1231 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1231/)
          MAPREDUCE-3263. Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1231 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1231/ ) MAPREDUCE-3263 . Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1153 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1153/)
          MAPREDUCE-3263. Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1153 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1153/ ) MAPREDUCE-3263 . Fixed the MAPREDUCE-3028 commit which broke MR1. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188997 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/NotificationTestCase.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #842 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/842/)
          MAPREDUCE-3028. Added job-end notification support. Contributed by Ravi Prakash.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #842 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/842/ ) MAPREDUCE-3028 . Added job-end notification support. Contributed by Ravi Prakash. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #62 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/62/)
          Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #62 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/62/ ) Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #871 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/871/)
          MAPREDUCE-3028. Added job-end notification support. Contributed by Ravi Prakash.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #871 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/871/ ) MAPREDUCE-3028 . Added job-end notification support. Contributed by Ravi Prakash. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #50 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/50/)
          Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #50 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/50/ ) Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1161 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1161/)
          MAPREDUCE-3028. Added job-end notification support. Contributed by Ravi Prakash.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1161 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1161/ ) MAPREDUCE-3028 . Added job-end notification support. Contributed by Ravi Prakash. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #46 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/46/)
          Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #46 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/46/ ) Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #47 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/47/)
          Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #47 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/47/ ) Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #46 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/46/)
          Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #46 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/46/ ) Merge -c 1188377 from trunk to branch-0.23 to complete fix for MAPREDUCE-3028 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188378 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1146 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1146/)
          MAPREDUCE-3028. Added job-end notification support. Contributed by Ravi Prakash.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1146 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1146/ ) MAPREDUCE-3028 . Added job-end notification support. Contributed by Ravi Prakash. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1224 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1224/)
          MAPREDUCE-3028. Added job-end notification support. Contributed by Ravi Prakash.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1224 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1224/ ) MAPREDUCE-3028 . Added job-end notification support. Contributed by Ravi Prakash. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188377 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Ravi!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Ravi!
          Hide
          Ravi Prakash added a comment -

          Hitesh! I came back home. Nopes. I get intermittent failures despite the RMContainerAllocator.diff patch.

          Show
          Ravi Prakash added a comment - Hitesh! I came back home. Nopes. I get intermittent failures despite the RMContainerAllocator.diff patch.
          Hide
          Ravi Prakash added a comment -

          On my home computer which is a Fedora 14 with OpenJDK, TestRMContainerAllocator#testBlackListedNodes fails intermittently. On my work computer which is a RHEL 5, Sun JAVA SDK, the test passes all the time. The test passes even after I apply the MAPREDUCE-3028.patch. I don't know what is the reason for the difference in behavior (could be a number of issues)

          @Hitesh: I'll try out your patch when I get back home.

          But in the mean while. Is this going to be a blocker for MAPREDUCE-3028.patch?

          Show
          Ravi Prakash added a comment - On my home computer which is a Fedora 14 with OpenJDK, TestRMContainerAllocator#testBlackListedNodes fails intermittently. On my work computer which is a RHEL 5, Sun JAVA SDK, the test passes all the time. The test passes even after I apply the MAPREDUCE-3028 .patch. I don't know what is the reason for the difference in behavior (could be a number of issues) @Hitesh: I'll try out your patch when I get back home. But in the mean while. Is this going to be a blocker for MAPREDUCE-3028 .patch?
          Hide
          Hitesh Shah added a comment -

          @Ravi, could you try re-running your tests after applying the patch attached ( RMContainerAllocator.diff ). Thanks.

          Show
          Hitesh Shah added a comment - @Ravi, could you try re-running your tests after applying the patch attached ( RMContainerAllocator.diff ). Thanks.
          Hide
          Hitesh Shah added a comment -

          @Ravi, could you give more details on the blacklist node failures and file a new jira if its reproducible with the latest code. I could not reproduce this locally when run multiple times.

          Show
          Hitesh Shah added a comment - @Ravi, could you give more details on the blacklist node failures and file a new jira if its reproducible with the latest code. I could not reproduce this locally when run multiple times.
          Hide
          Mahadev konar added a comment -

          Mohammad,
          The key that you are using is deprecated but Oozie should work as it is without any changes. You will just get a deprecated warning.

          If you are interested, take a look at ConfigUtil.java

           Configuration.addDeprecation("job.end.notification.url", 
                new String[] {MRJobConfig.END_NOTIFICATION_URL});
          
          Show
          Mahadev konar added a comment - Mohammad, The key that you are using is deprecated but Oozie should work as it is without any changes. You will just get a deprecated warning. If you are interested, take a look at ConfigUtil.java Configuration.addDeprecation( "job.end.notification.url" , new String [] {MRJobConfig.END_NOTIFICATION_URL});
          Hide
          Mohammad Kamrul Islam added a comment -

          @Ravi
          We use the property name "job.end.notification.url" for specifying the URL.

          In one patch, I could see a property named mapreduce.job.end-notification.urls. Is it the replacement of the existing one ("job.end.notification.url")? Or, this is a new property for different purpose while the existing one will be supported as well.

          The code snippets of a patch...

          public static final String MR_JOB_END_NOTIFICATION_URLS = "mapreduce.job.end-notification.urls";

          Show
          Mohammad Kamrul Islam added a comment - @Ravi We use the property name "job.end.notification.url" for specifying the URL. In one patch, I could see a property named mapreduce.job.end-notification.urls. Is it the replacement of the existing one ("job.end.notification.url")? Or, this is a new property for different purpose while the existing one will be supported as well. The code snippets of a patch... public static final String MR_JOB_END_NOTIFICATION_URLS = "mapreduce.job.end-notification.urls";
          Hide
          Ravi Prakash added a comment -

          Oh and btw, this is before I put the patch. The patch is clearly unrelated to the test.

          Show
          Ravi Prakash added a comment - Oh and btw, this is before I put the patch. The patch is clearly unrelated to the test.
          Hide
          Ravi Prakash added a comment -

          Arun. Seems like an intermittent failure.

          [raviprak@localhost hadoop-mapreduce-client-app]$ ~/Code/hadoop/runTest.sh TestRMContainerAllocator#testBlackListedNodes
          Test run #1. Test failed. So far, test passed 0 and failed 1 out of 1 attempts
          Test run #2. Test failed. So far, test passed 0 and failed 2 out of 2 attempts
          Test run #3. Test passed. So far, test passed 1 and failed 2 out of 3 attempts
          Test run #4. Test failed. So far, test passed 1 and failed 3 out of 4 attempts
          Test run #5. Test passed. So far, test passed 2 and failed 3 out of 5 attempts
          Test run #6. Test failed. So far, test passed 2 and failed 4 out of 6 attempts
          Test run #7. Test passed. So far, test passed 3 and failed 4 out of 7 attempts
          Test run #8. Test passed. So far, test passed 4 and failed 4 out of 8 attempts
          Test run #9. Test failed. So far, test passed 4 and failed 5 out of 9 attempts
          Test run #10. Test failed. So far, test passed 4 and failed 6 out of 10 attempts
          Test run #11. Test failed. So far, test passed 4 and failed 7 out of 11 attempts
          Test run #12. Test failed. So far, test passed 4 and failed 8 out of 12 attempts
          Test run #13. Test passed. So far, test passed 5 and failed 8 out of 13 attempts
          Test run #14. Test passed. So far, test passed 6 and failed 8 out of 14 attempts
          Test run #15. Test failed. So far, test passed 6 and failed 9 out of 15 attempts
          Test run #16. Test failed. So far, test passed 6 and failed 10 out of 16 attempts
          Test run #17. Test failed. So far, test passed 6 and failed 11 out of 17 attempts
          Test run #18. Test failed. So far, test passed 6 and failed 12 out of 18 attempts
          Test run #19. Test failed. So far, test passed 6 and failed 13 out of 19 attempts
          Test run #20. Test failed. So far, test passed 6 and failed 14 out of 20 attempts
          Test run #21. Test failed. So far, test passed 6 and failed 15 out of 21 attempts
          Test run #22. Test failed. So far, test passed 6 and failed 16 out of 22 attempts
          

          I did clean

          Show
          Ravi Prakash added a comment - Arun. Seems like an intermittent failure. [raviprak@localhost hadoop-mapreduce-client-app]$ ~/Code/hadoop/runTest.sh TestRMContainerAllocator#testBlackListedNodes Test run #1. Test failed. So far, test passed 0 and failed 1 out of 1 attempts Test run #2. Test failed. So far, test passed 0 and failed 2 out of 2 attempts Test run #3. Test passed. So far, test passed 1 and failed 2 out of 3 attempts Test run #4. Test failed. So far, test passed 1 and failed 3 out of 4 attempts Test run #5. Test passed. So far, test passed 2 and failed 3 out of 5 attempts Test run #6. Test failed. So far, test passed 2 and failed 4 out of 6 attempts Test run #7. Test passed. So far, test passed 3 and failed 4 out of 7 attempts Test run #8. Test passed. So far, test passed 4 and failed 4 out of 8 attempts Test run #9. Test failed. So far, test passed 4 and failed 5 out of 9 attempts Test run #10. Test failed. So far, test passed 4 and failed 6 out of 10 attempts Test run #11. Test failed. So far, test passed 4 and failed 7 out of 11 attempts Test run #12. Test failed. So far, test passed 4 and failed 8 out of 12 attempts Test run #13. Test passed. So far, test passed 5 and failed 8 out of 13 attempts Test run #14. Test passed. So far, test passed 6 and failed 8 out of 14 attempts Test run #15. Test failed. So far, test passed 6 and failed 9 out of 15 attempts Test run #16. Test failed. So far, test passed 6 and failed 10 out of 16 attempts Test run #17. Test failed. So far, test passed 6 and failed 11 out of 17 attempts Test run #18. Test failed. So far, test passed 6 and failed 12 out of 18 attempts Test run #19. Test failed. So far, test passed 6 and failed 13 out of 19 attempts Test run #20. Test failed. So far, test passed 6 and failed 14 out of 20 attempts Test run #21. Test failed. So far, test passed 6 and failed 15 out of 21 attempts Test run #22. Test failed. So far, test passed 6 and failed 16 out of 22 attempts I did clean
          Hide
          Arun C Murthy added a comment -

          Try 'mvn clean test -Dtest=TestRMContainerAllocator'.

          Show
          Arun C Murthy added a comment - Try 'mvn clean test -Dtest=TestRMContainerAllocator'.
          Hide
          Arun C Murthy added a comment -

          No, unpatched trunk works for me - TestRMContainerAllocator runs fine?

          Show
          Arun C Murthy added a comment - No, unpatched trunk works for me - TestRMContainerAllocator runs fine?
          Hide
          Ravi Prakash added a comment -

          On an unpatched trunk TestRMContainerAllocator#testBlackListedNodes is failing. The test failure is unrelated to this patch. Since the trunk compilation was broken, I wasn't able to run test-patch to check the javadoc -1.

          Show
          Ravi Prakash added a comment - On an unpatched trunk TestRMContainerAllocator#testBlackListedNodes is failing. The test failure is unrelated to this patch. Since the trunk compilation was broken, I wasn't able to run test-patch to check the javadoc -1.
          Hide
          Arun C Murthy added a comment -

          Also, seems like log4j logs aren't showing up for any unit tests... something to do with MAPREDUCE-3199?

          Show
          Arun C Murthy added a comment - Also, seems like log4j logs aren't showing up for any unit tests... something to do with MAPREDUCE-3199 ?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12500340/MAPREDUCE-3028.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 162 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500340/MAPREDUCE-3028.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 162 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1116//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Fixed the nits.

          Show
          Arun C Murthy added a comment - Fixed the nits.
          Hide
          Arun C Murthy added a comment -

          Ravi, the patch looks good.

          Some nits:

          1. Please use a consistent coding style from Hadoop.
          2. No need for configuring sentinels, we can just $jobId, $jobStatus similar to 0.20
          3. Don't do 'null checks' for success/fail as in notifyUrlOnce, just return boolean.
          4. Use setConf rather than parseConf.
          Show
          Arun C Murthy added a comment - Ravi, the patch looks good. Some nits: Please use a consistent coding style from Hadoop. No need for configuring sentinels, we can just $jobId, $jobStatus similar to 0.20 Don't do 'null checks' for success/fail as in notifyUrlOnce, just return boolean. Use setConf rather than parseConf.
          Hide
          Ravi Prakash added a comment -

          Findbugs warning are not related to this patch. Please ignore them

          Show
          Ravi Prakash added a comment - Findbugs warning are not related to this patch. Please ignore them
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12500226/MAPREDUCE-3028.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500226/MAPREDUCE-3028.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1108//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          Submitting Patch to trigger Hadoop QA

          Show
          Ravi Prakash added a comment - Submitting Patch to trigger Hadoop QA
          Hide
          Ravi Prakash added a comment -

          Thanks for the reviews and comments. The latest patch removes multiple URL notification. I'm sorry for adding it in. I should have gathered requirements publicly on this JIRA before I started coding. Please review and commit the patch.

          Show
          Ravi Prakash added a comment - Thanks for the reviews and comments. The latest patch removes multiple URL notification. I'm sorry for adding it in. I should have gathered requirements publicly on this JIRA before I started coding. Please review and commit the patch.
          Hide
          Ravi Prakash added a comment -

          Thanks for the reviews folks. Uploading a new patch which keeps things backwards compatible. Please review and commit it.

          Please note that the $jobId and $jobStatus sentinels must be explicitly specified. I've incorporated them in mapred-default.xml as such. If you want me to incorporate them as default in code, please let me know.

          Show
          Ravi Prakash added a comment - Thanks for the reviews folks. Uploading a new patch which keeps things backwards compatible. Please review and commit it. Please note that the $jobId and $jobStatus sentinels must be explicitly specified. I've incorporated them in mapred-default.xml as such. If you want me to incorporate them as default in code, please let me know.
          Hide
          Arun C Murthy added a comment -

          Ravi, apologies for being anal about this.

          But I really don't think we should add features like multiple URLs. Please make it a single URL. Thanks.

          Show
          Arun C Murthy added a comment - Ravi, apologies for being anal about this. But I really don't think we should add features like multiple URLs. Please make it a single URL. Thanks.
          Hide
          Robert Joseph Evans added a comment -

          Oops, I guess I was wrong. The config key did change. It now has an s on the end, so it is not backwards compatible. That s really should be removed.

          Show
          Robert Joseph Evans added a comment - Oops, I guess I was wrong. The config key did change. It now has an s on the end, so it is not backwards compatible. That s really should be removed.
          Hide
          Mahadev konar added a comment -

          Wait,
          I take that back, looks like in MRJobConfig:

          END_NOTIFICATION_URL -> MR_JOB_END_NOTIFICATION_URLS
          and
          mapreduce.job.end-notification.url ->mapreduce.job.end-notification.urls

          So doesnt look like they wont require any change. We should probably keep this consistent so that downstream projects using it do not break.

          Show
          Mahadev konar added a comment - Wait, I take that back, looks like in MRJobConfig: END_NOTIFICATION_URL -> MR_JOB_END_NOTIFICATION_URLS and mapreduce.job.end-notification.url ->mapreduce.job.end-notification.urls So doesnt look like they wont require any change. We should probably keep this consistent so that downstream projects using it do not break.
          Hide
          Mahadev konar added a comment -

          ahh, missed that. Thanks for pointing out robert.

          Show
          Mahadev konar added a comment - ahh, missed that. Thanks for pointing out robert.
          Hide
          Robert Joseph Evans added a comment -

          From what I can read in the code the multiple URLs functionality is backwards compatible. If the config mapreduce.job.end-notification.url.separator.regex is "", which is the default value, then the config holding the URL is treated as a single URL just like it is now.

          Show
          Robert Joseph Evans added a comment - From what I can read in the code the multiple URLs functionality is backwards compatible. If the config mapreduce.job.end-notification.url.separator.regex is "", which is the default value, then the config holding the URL is treated as a single URL just like it is now.
          Hide
          Santhosh Srinivasan added a comment -

          With respect to the multiple urls, the only concern is that its not backwards compatible. Are Oozie folks ok with that?

          The support for multiple URLs should NOT require changes to the existing mechanism. As long as the current configuration works as it stands today, Oozie should be fine with any other enhancements.

          Show
          Santhosh Srinivasan added a comment - With respect to the multiple urls, the only concern is that its not backwards compatible. Are Oozie folks ok with that? The support for multiple URLs should NOT require changes to the existing mechanism. As long as the current configuration works as it stands today, Oozie should be fine with any other enhancements.
          Hide
          Mahadev konar added a comment -

          With respect to the multiple urls, the only concern is that its not backwards compatible. Are Oozie folks ok with that?

          Show
          Mahadev konar added a comment - With respect to the multiple urls, the only concern is that its not backwards compatible. Are Oozie folks ok with that?
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-10-20 21:33:43, Robert Evans wrote:

          > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java, line 34

          > <https://reviews.apache.org/r/2468/diff/2/?file=51833#file51833line34>

          >

          > It is not that common of a practice to have your test extend what it is testing. I don't think it is bad, I just think it would be more accepted, if you made the values you want to access package, instead of protected, or if you added in getters for those values.

          Ravi Prakash wrote:

          The values I wanted to test should be private to the class since I don't want other objects to mess with them. I chose protected so that they'd be easier to test

          That is why I suggested the getters. That way they are read only to the outside world. Also You could add in the getters yourself in a separate sub class. I think your tests are fine. I was just commenting that they are different from all of the other tests, and it may get shot down because of it.

          On 2011-10-20 21:33:43, Robert Evans wrote:

          > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java, line 134

          > <https://reviews.apache.org/r/2468/diff/2/?file=51833#file51833line134>

          >

          > Just a note that this can be done with mockito and spy instead of overriding this directly, so you would not have to subclass the original class.

          Ravi Prakash wrote:

          It was just easier this way, don't you think?

          It may have been easier to write but it was a bit more difficult to understand what was happening. If you do need to subclass it I would prefer to see it happen as a separate sub class from the test, but that is just me.

          • Robert

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/#review2727
          -----------------------------------------------------------

          On 2011-10-20 23:35:05, Ravi Prakash wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2468/

          -----------------------------------------------------------

          (Updated 2011-10-20 23:35:05)

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Summary

          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          Diffs

          -----

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing

          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-10-20 21:33:43, Robert Evans wrote: > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java, line 34 > < https://reviews.apache.org/r/2468/diff/2/?file=51833#file51833line34 > > > It is not that common of a practice to have your test extend what it is testing. I don't think it is bad, I just think it would be more accepted, if you made the values you want to access package, instead of protected, or if you added in getters for those values. Ravi Prakash wrote: The values I wanted to test should be private to the class since I don't want other objects to mess with them. I chose protected so that they'd be easier to test That is why I suggested the getters. That way they are read only to the outside world. Also You could add in the getters yourself in a separate sub class. I think your tests are fine. I was just commenting that they are different from all of the other tests, and it may get shot down because of it. On 2011-10-20 21:33:43, Robert Evans wrote: > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java, line 134 > < https://reviews.apache.org/r/2468/diff/2/?file=51833#file51833line134 > > > Just a note that this can be done with mockito and spy instead of overriding this directly, so you would not have to subclass the original class. Ravi Prakash wrote: It was just easier this way, don't you think? It may have been easier to write but it was a bit more difficult to understand what was happening. If you do need to subclass it I would prefer to see it happen as a separate sub class from the test, but that is just me. Robert ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/#review2727 ----------------------------------------------------------- On 2011-10-20 23:35:05, Ravi Prakash wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- (Updated 2011-10-20 23:35:05) Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) Diffs ----- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          Ravi Prakash added a comment -

          @Arun: Same reply as Hitesh. One reason I could think of is, when there are multiple kinds of jobs (hosted on different servers) waiting on the output of this job, its easier to send multiple notifications instead of chaining together notifications or having a mediator who receives the one single notification and then sends out multiple ones.

          Show
          Ravi Prakash added a comment - @Arun: Same reply as Hitesh. One reason I could think of is, when there are multiple kinds of jobs (hosted on different servers) waiting on the output of this job, its easier to send multiple notifications instead of chaining together notifications or having a mediator who receives the one single notification and then sends out multiple ones.
          Hide
          Ravi Prakash added a comment -

          @Hitesh: I gathered requirements from a couple of folks at Yahoo! This was one dream / wish. I'll ask them again where it could be helpful if you want. In any case, if you don't want to use it (which is the default), you only have to not specify "mapreduce.job.end-notification.url.separator.regex".

          The openConnection only returns a URLConnection object. I agree its ambiguously named. In fact that is the only way to get a URLConnection object I know. Quoting http://download.oracle.com/javase/6/docs/api/java/net/URLConnection.html

          1. The connection object is created by invoking the openConnection method on a URL.
          2. The setup parameters and general request properties are manipulated.
          3. The actual connection to the remote object is made, using the connect method.
          4. The remote object becomes available. The header fields and the contents of the remote object can be accessed.
          Show
          Ravi Prakash added a comment - @Hitesh: I gathered requirements from a couple of folks at Yahoo! This was one dream / wish. I'll ask them again where it could be helpful if you want. In any case, if you don't want to use it (which is the default), you only have to not specify "mapreduce.job.end-notification.url.separator.regex". The openConnection only returns a URLConnection object. I agree its ambiguously named. In fact that is the only way to get a URLConnection object I know. Quoting http://download.oracle.com/javase/6/docs/api/java/net/URLConnection.html The connection object is created by invoking the openConnection method on a URL. The setup parameters and general request properties are manipulated. The actual connection to the remote object is made, using the connect method. The remote object becomes available. The header fields and the contents of the remote object can be accessed.
          Hide
          Arun C Murthy added a comment -

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          What is the reason for this?

          Show
          Arun C Murthy added a comment - The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) What is the reason for this?
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/
          -----------------------------------------------------------

          (Updated 2011-10-20 23:35:05.016675)

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Changes
          -------

          Corrected minor javadoc

          Summary
          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          Diffs (updated)


          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing
          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- (Updated 2011-10-20 23:35:05.016675) Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Changes ------- Corrected minor javadoc Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) Diffs (updated) branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-10-20 21:33:43, Robert Evans wrote:

          >

          Thanks Bobby!

          On 2011-10-20 21:33:43, Robert Evans wrote:

          > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java, line 54

          > <https://reviews.apache.org/r/2468/diff/2/?file=51831#file51831line54>

          >

          > The Javadocs is not completely correct, it does not return the URLs (This is really minor)

          I've fixed the javadoc.

          On 2011-10-20 21:33:43, Robert Evans wrote:

          > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java, line 34

          > <https://reviews.apache.org/r/2468/diff/2/?file=51833#file51833line34>

          >

          > It is not that common of a practice to have your test extend what it is testing. I don't think it is bad, I just think it would be more accepted, if you made the values you want to access package, instead of protected, or if you added in getters for those values.

          The values I wanted to test should be private to the class since I don't want other objects to mess with them. I chose protected so that they'd be easier to test

          On 2011-10-20 21:33:43, Robert Evans wrote:

          > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java, line 134

          > <https://reviews.apache.org/r/2468/diff/2/?file=51833#file51833line134>

          >

          > Just a note that this can be done with mockito and spy instead of overriding this directly, so you would not have to subclass the original class.

          It was just easier this way, don't you think?

          On 2011-10-20 21:33:43, Robert Evans wrote:

          > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java, line 488

          > <https://reviews.apache.org/r/2468/diff/2/?file=51836#file51836line488>

          >

          > Removing/Renaming these values breaks binary compatibility and source compatibility. Even if the name is not that clean it is best to leave it alone. If you must rename it then @Deprecate the old ones, have them point to the new ones, and then file a JIRA to remove the old ones in a release or two, to give people time to update their code.

          The variables were public but the whole interface is @InterfaceAudience.Private, @InterfaceStability.Evolving . I also couldn't find these variables in 0.20-security (so they were new). Hence removing them should not break binary compatibility. Please correct me if I'm wrong.

          • Ravi

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/#review2727
          -----------------------------------------------------------

          On 2011-10-20 23:35:05, Ravi Prakash wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2468/

          -----------------------------------------------------------

          (Updated 2011-10-20 23:35:05)

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Summary

          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          Diffs

          -----

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing

          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-10-20 21:33:43, Robert Evans wrote: > Thanks Bobby! On 2011-10-20 21:33:43, Robert Evans wrote: > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java, line 54 > < https://reviews.apache.org/r/2468/diff/2/?file=51831#file51831line54 > > > The Javadocs is not completely correct, it does not return the URLs (This is really minor) I've fixed the javadoc. On 2011-10-20 21:33:43, Robert Evans wrote: > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java, line 34 > < https://reviews.apache.org/r/2468/diff/2/?file=51833#file51833line34 > > > It is not that common of a practice to have your test extend what it is testing. I don't think it is bad, I just think it would be more accepted, if you made the values you want to access package, instead of protected, or if you added in getters for those values. The values I wanted to test should be private to the class since I don't want other objects to mess with them. I chose protected so that they'd be easier to test On 2011-10-20 21:33:43, Robert Evans wrote: > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java, line 134 > < https://reviews.apache.org/r/2468/diff/2/?file=51833#file51833line134 > > > Just a note that this can be done with mockito and spy instead of overriding this directly, so you would not have to subclass the original class. It was just easier this way, don't you think? On 2011-10-20 21:33:43, Robert Evans wrote: > branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java, line 488 > < https://reviews.apache.org/r/2468/diff/2/?file=51836#file51836line488 > > > Removing/Renaming these values breaks binary compatibility and source compatibility. Even if the name is not that clean it is best to leave it alone. If you must rename it then @Deprecate the old ones, have them point to the new ones, and then file a JIRA to remove the old ones in a release or two, to give people time to update their code. The variables were public but the whole interface is @InterfaceAudience.Private, @InterfaceStability.Evolving . I also couldn't find these variables in 0.20-security (so they were new). Hence removing them should not break binary compatibility. Please correct me if I'm wrong. Ravi ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/#review2727 ----------------------------------------------------------- On 2011-10-20 23:35:05, Ravi Prakash wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- (Updated 2011-10-20 23:35:05) Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) Diffs ----- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          Ravi Prakash added a comment -

          Corrected minor javadoc

          Show
          Ravi Prakash added a comment - Corrected minor javadoc
          Hide
          Hitesh Shah added a comment -

          @Ravi, could you clarify in what scenarios you would need to notify multiple urls of a job completion?

          URLConnection conn = url.openConnection();
          conn.setConnectTimeout(5*1000);
          conn.setReadTimeout(5*1000);

          The above will probably not work as you are setting the connect timeout after the connection is created.

          Show
          Hitesh Shah added a comment - @Ravi, could you clarify in what scenarios you would need to notify multiple urls of a job completion? URLConnection conn = url.openConnection(); conn.setConnectTimeout(5*1000); conn.setReadTimeout(5*1000); The above will probably not work as you are setting the connect timeout after the connection is created.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12499954/MAPREDUCE-3028.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499954/MAPREDUCE-3028.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1093//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          The Findbug warnings linked on the Pre-commit build result are not related to this patch. They are mostly in MRProtos.java which is not touched by this patch.

          Show
          Ravi Prakash added a comment - The Findbug warnings linked on the Pre-commit build result are not related to this patch. They are mostly in MRProtos.java which is not touched by this patch.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/#review2727
          -----------------------------------------------------------

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          <https://reviews.apache.org/r/2468/#comment6133>

          The Javadocs is not completely correct, it does not return the URLs (This is really minor)

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          <https://reviews.apache.org/r/2468/#comment6134>

          It is not that common of a practice to have your test extend what it is testing. I don't think it is bad, I just think it would be more accepted, if you made the values you want to access package, instead of protected, or if you added in getters for those values.

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          <https://reviews.apache.org/r/2468/#comment6135>

          Just a note that this can be done with mockito and spy instead of overriding this directly, so you would not have to subclass the original class.

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          <https://reviews.apache.org/r/2468/#comment6136>

          Removing/Renaming these values breaks binary compatibility and source compatibility. Even if the name is not that clean it is best to leave it alone. If you must rename it then @Deprecate the old ones, have them point to the new ones, and then file a JIRA to remove the old ones in a release or two, to give people time to update their code.

          • Robert

          On 2011-10-20 21:09:33, Ravi Prakash wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2468/

          -----------------------------------------------------------

          (Updated 2011-10-20 21:09:33)

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Summary

          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          Diffs

          -----

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing

          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/#review2727 ----------------------------------------------------------- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java < https://reviews.apache.org/r/2468/#comment6133 > The Javadocs is not completely correct, it does not return the URLs (This is really minor) branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java < https://reviews.apache.org/r/2468/#comment6134 > It is not that common of a practice to have your test extend what it is testing. I don't think it is bad, I just think it would be more accepted, if you made the values you want to access package, instead of protected, or if you added in getters for those values. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java < https://reviews.apache.org/r/2468/#comment6135 > Just a note that this can be done with mockito and spy instead of overriding this directly, so you would not have to subclass the original class. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java < https://reviews.apache.org/r/2468/#comment6136 > Removing/Renaming these values breaks binary compatibility and source compatibility. Even if the name is not that clean it is best to leave it alone. If you must rename it then @Deprecate the old ones, have them point to the new ones, and then file a JIRA to remove the old ones in a release or two, to give people time to update their code. Robert On 2011-10-20 21:09:33, Ravi Prakash wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- (Updated 2011-10-20 21:09:33) Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) Diffs ----- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12499939/MAPREDUCE-3028.branch-0.23.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499939/MAPREDUCE-3028.branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1089//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          Renamed patch and removed extraneous whitespace

          Show
          Ravi Prakash added a comment - Renamed patch and removed extraneous whitespace
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/
          -----------------------------------------------------------

          (Updated 2011-10-20 21:09:33.246936)

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Changes
          -------

          • Incorporated review comments. Thanks Bobby and Mark.
          • Added unit tests
          • Removed JIRA number so as not to spam the JIRA

          Summary
          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          Diffs (updated)


          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing
          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- (Updated 2011-10-20 21:09:33.246936) Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Changes ------- Incorporated review comments. Thanks Bobby and Mark. Added unit tests Removed JIRA number so as not to spam the JIRA Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) Diffs (updated) branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-10-20 16:05:27, Robert Evans wrote:

          > Over all I would say the patch looks good. There are a few minor nits though.

          Thanks Bobby! I've incorporated both your suggestions.

          • Ravi

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/#review2701
          -----------------------------------------------------------

          On 2011-10-20 21:09:33, Ravi Prakash wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2468/

          -----------------------------------------------------------

          (Updated 2011-10-20 21:09:33)

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Summary

          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          Diffs

          -----

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing

          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-10-20 16:05:27, Robert Evans wrote: > Over all I would say the patch looks good. There are a few minor nits though. Thanks Bobby! I've incorporated both your suggestions. Ravi ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/#review2701 ----------------------------------------------------------- On 2011-10-20 21:09:33, Ravi Prakash wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- (Updated 2011-10-20 21:09:33) Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) Diffs ----- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-10-20 16:35:56, Mark Holderbaugh wrote:

          >

          Thanks Mark! I've incorporated all the changes you suggested in the new patch.

          • Ravi

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/#review2702
          -----------------------------------------------------------

          On 2011-10-20 21:09:33, Ravi Prakash wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2468/

          -----------------------------------------------------------

          (Updated 2011-10-20 21:09:33)

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Summary

          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          Diffs

          -----

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing

          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-10-20 16:35:56, Mark Holderbaugh wrote: > Thanks Mark! I've incorporated all the changes you suggested in the new patch. Ravi ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/#review2702 ----------------------------------------------------------- On 2011-10-20 21:09:33, Ravi Prakash wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- (Updated 2011-10-20 21:09:33) Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) Diffs ----- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          Ravi Prakash added a comment -

          The same patch also applies to trunk.

          Show
          Ravi Prakash added a comment - The same patch also applies to trunk.
          Hide
          Ravi Prakash added a comment -

          Please review job end notification. This patch applies on branch-0.23 .

          Show
          Ravi Prakash added a comment - Please review job end notification. This patch applies on branch-0.23 .
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/#review2702
          -----------------------------------------------------------

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          <https://reviews.apache.org/r/2468/#comment6092>

          Do we want a log message here that no URLS notified or maybe not set?

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          <https://reviews.apache.org/r/2468/#comment6093>

          Do we need check for Null or "" here? think they put back to back separators?

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          <https://reviews.apache.org/r/2468/#comment6090>

          Are these still valid: END_NOTIFICATION_URL, END_NOTIFICATION_RETRIES, END_NOTIFICATION_RETRIE_INTERVAL

          Should we move the new ones below closer to here?

          • Mark

          On 2011-10-20 15:36:27, Ravi Prakash wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2468/

          -----------------------------------------------------------

          (Updated 2011-10-20 15:36:27)

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Summary

          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          This addresses bug MAPREDUCE-3028.

          https://issues.apache.org/jira/browse/MAPREDUCE-3028

          Diffs

          -----

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing

          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/#review2702 ----------------------------------------------------------- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java < https://reviews.apache.org/r/2468/#comment6092 > Do we want a log message here that no URLS notified or maybe not set? branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java < https://reviews.apache.org/r/2468/#comment6093 > Do we need check for Null or "" here? think they put back to back separators? branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java < https://reviews.apache.org/r/2468/#comment6090 > Are these still valid: END_NOTIFICATION_URL, END_NOTIFICATION_RETRIES, END_NOTIFICATION_RETRIE_INTERVAL Should we move the new ones below closer to here? Mark On 2011-10-20 15:36:27, Ravi Prakash wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- (Updated 2011-10-20 15:36:27) Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) This addresses bug MAPREDUCE-3028 . https://issues.apache.org/jira/browse/MAPREDUCE-3028 Diffs ----- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/#review2701
          -----------------------------------------------------------

          Over all I would say the patch looks good. There are a few minor nits though.

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          <https://reviews.apache.org/r/2468/#comment6086>

          Need to add in the Apache License.

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java
          <https://reviews.apache.org/r/2468/#comment6089>

          Could you please put default values and documentation for the new configs in mapred-default.xml. This is mostly for documentation of the new configs, then it is for default values.

          • Robert

          On 2011-10-20 15:36:27, Ravi Prakash wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2468/

          -----------------------------------------------------------

          (Updated 2011-10-20 15:36:27)

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Summary

          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          This addresses bug MAPREDUCE-3028.

          https://issues.apache.org/jira/browse/MAPREDUCE-3028

          Diffs

          -----

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838

          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing

          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/#review2701 ----------------------------------------------------------- Over all I would say the patch looks good. There are a few minor nits though. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java < https://reviews.apache.org/r/2468/#comment6086 > Need to add in the Apache License. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java < https://reviews.apache.org/r/2468/#comment6089 > Could you please put default values and documentation for the new configs in mapred-default.xml. This is mostly for documentation of the new configs, then it is for default values. Robert On 2011-10-20 15:36:27, Ravi Prakash wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- (Updated 2011-10-20 15:36:27) Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) This addresses bug MAPREDUCE-3028 . https://issues.apache.org/jira/browse/MAPREDUCE-3028 Diffs ----- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2468/
          -----------------------------------------------------------

          Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh.

          Summary
          -------

          Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code.

          The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex)

          This addresses bug MAPREDUCE-3028.
          https://issues.apache.org/jira/browse/MAPREDUCE-3028

          Diffs


          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838
          branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838

          Diff: https://reviews.apache.org/r/2468/diff

          Testing
          -------

          Manually tested changes on a single node cluster. Going to add unit tests.

          Thanks,

          Ravi

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2468/ ----------------------------------------------------------- Review request for Tom Graves, Robert Evans, Jonathan Eagles, and Mark Holderbaugh. Summary ------- Job end notification for 0.23 / next. I'm going to work on unit tests while you folks review the code. The only new thing added to 0.20 is that multiple recipients can be configured to receive the job-end notification by supplying multiple URLs (one parameter separated by a regex) This addresses bug MAPREDUCE-3028 . https://issues.apache.org/jira/browse/MAPREDUCE-3028 Diffs branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java PRE-CREATION branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java 1186838 branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java 1186838 Diff: https://reviews.apache.org/r/2468/diff Testing ------- Manually tested changes on a single node cluster. Going to add unit tests. Thanks, Ravi
          Hide
          Arun C Murthy added a comment -

          Agreed. Let's keep it simple:

          1. AM, at the end of the job, tries a small number of times (3/5).
          2. Each retry has a short, aggressive timeout.
          Show
          Arun C Murthy added a comment - Agreed. Let's keep it simple: AM, at the end of the job, tries a small number of times (3/5). Each retry has a short, aggressive timeout.
          Hide
          Mahadev konar added a comment -

          Agree with Robert here. Lets just do the notification in AM for now. We can look for other enhancements/generalization/improvements later.

          Show
          Mahadev konar added a comment - Agree with Robert here. Lets just do the notification in AM for now. We can look for other enhancements/generalization/improvements later.
          Hide
          Robert Joseph Evans added a comment -

          I don't like the job history server doing it because launching the job history server is some what optional (The default start scripts do not launch it currently). Also it will add in a new delay, and a lot more churn on the Job history server. Currently the JobHistory server polls HDFS for new jobs that have finished. If the job has finished it will move it to its own directory, and then cache some info about the app based only on the name of the jhist file. If we want this to do notification also then we need to either store the URL in the jhist file name, or force the server to read in the config file every time it pulls in a new Application.

          I agree that having applications poll for status with something like a very low latency RSS is the preferable solution and as such I would like to see the minimal done to support backwards compatibility, which would be to put the notification in the ApplicationMaster. But it is Ravi that is doing the fix and if he wants to do more then that and it is not something that is going to cause more problems then it fixes then I would say let him do it.

          What I mean by cause more problems then it fixes is that we have had lots of issues in production with having the notification be in a single thread blocking other notifications. What ever solution it is it cannot allow other notifications to become blocked or severely delayed by a single bad URL. (Or our case of a remote server crashing in the middle of the request and some how causing the socket be in a psudo state where the read blocks indefinitely and there is no one on the other side that is ever going to write)

          Show
          Robert Joseph Evans added a comment - I don't like the job history server doing it because launching the job history server is some what optional (The default start scripts do not launch it currently). Also it will add in a new delay, and a lot more churn on the Job history server. Currently the JobHistory server polls HDFS for new jobs that have finished. If the job has finished it will move it to its own directory, and then cache some info about the app based only on the name of the jhist file. If we want this to do notification also then we need to either store the URL in the jhist file name, or force the server to read in the config file every time it pulls in a new Application. I agree that having applications poll for status with something like a very low latency RSS is the preferable solution and as such I would like to see the minimal done to support backwards compatibility, which would be to put the notification in the ApplicationMaster. But it is Ravi that is doing the fix and if he wants to do more then that and it is not something that is going to cause more problems then it fixes then I would say let him do it. What I mean by cause more problems then it fixes is that we have had lots of issues in production with having the notification be in a single thread blocking other notifications. What ever solution it is it cannot allow other notifications to become blocked or severely delayed by a single bad URL. (Or our case of a remote server crashing in the middle of the request and some how causing the socket be in a psudo state where the read blocks indefinitely and there is no one on the other side that is ever going to write)
          Hide
          Alejandro Abdelnur added a comment -

          Wouldn't make more sense to have the jobhistory service to do the notification? Then regardless of the AM implementation it would be done. And, as I've suggested before I think we should change from push to pull (a la RSS feed).

          Show
          Alejandro Abdelnur added a comment - Wouldn't make more sense to have the jobhistory service to do the notification? Then regardless of the AM implementation it would be done. And, as I've suggested before I think we should change from push to pull (a la RSS feed).
          Hide
          Vinod Kumar Vavilapalli added a comment -

          No, I'd not put this in YARN (i.e Nodemanager). The notification is currently a MapReduce only feature (atleast it seems to be for now). I can imagine a generic application finish notification from YARN itself (RM or NM), but I'd postpone any discussion about that if we can pull it off in MapReduce AM for now.

          Also 30secs seems way too much, even if it for that one job. Make it something in the order of 5s of seconds?

          Show
          Vinod Kumar Vavilapalli added a comment - No, I'd not put this in YARN (i.e Nodemanager). The notification is currently a MapReduce only feature (atleast it seems to be for now). I can imagine a generic application finish notification from YARN itself (RM or NM), but I'd postpone any discussion about that if we can pull it off in MapReduce AM for now. Also 30secs seems way too much, even if it for that one job. Make it something in the order of 5s of seconds?
          Hide
          Mahadev konar added a comment -

          Ravi,
          Why do you want to include the job end notification as part of container cleanup? Job end notification should probably be just a Map Reduce feature. As soon as the AM tell the RM that its done running the job (look at the RMCommunicator:unregister()), you should be able to make the http callback to say if the job is successful or not. Thoughts?

          Show
          Mahadev konar added a comment - Ravi, Why do you want to include the job end notification as part of container cleanup? Job end notification should probably be just a Map Reduce feature. As soon as the AM tell the RM that its done running the job (look at the RMCommunicator:unregister()), you should be able to make the http callback to say if the job is successful or not. Thoughts?
          Hide
          Ravi Prakash added a comment -

          I have been trying to see if I cannot include job-end notification as a part of YARN. So that the Node Manager, as part of maybe the AM Container's cleanup would issue the callback. Would you be opposed to that?

          Show
          Ravi Prakash added a comment - I have been trying to see if I cannot include job-end notification as a part of YARN. So that the Node Manager, as part of maybe the AM Container's cleanup would issue the callback. Would you be opposed to that?
          Hide
          Ravi Prakash added a comment -

          Hi Arun! Yes I've been working on this.

          From your and Amol's discussions:
          Spec is as such (details to be borrowed from current JT). Please use same config variable names to avoid compatibility.
          If the name has JT (or jobtracker)then lets discuss. The spec is as such

          • AM should trigger a http call upon job completion
          • The http call should be fire and forget by default.
          • If user specifies number of tries then the following should be true
            1. Allow hadoop config to be set to ignore this in case becomes a problem (too many retries)
            2. Set max retries for all AM; i.e. users cannot specify a number above this. If Hadoop config does not specify it
            the value should be 1 (i.e fire and forget)
            3. Lets say that the Hadoop config on max retries is set to 3. Then AM should try 3 times with timeout (again a
            Hadoop variable). If timeout is not specified, the value should be 30 seconds.

          The aim is to let AM get off quickly. Since unlike JT there is one AM per job there would not bottleneck and no hanging
          of queue. So 30 seconds is ok.

          Show
          Ravi Prakash added a comment - Hi Arun! Yes I've been working on this. From your and Amol's discussions: Spec is as such (details to be borrowed from current JT). Please use same config variable names to avoid compatibility. If the name has JT (or jobtracker)then lets discuss. The spec is as such AM should trigger a http call upon job completion The http call should be fire and forget by default. If user specifies number of tries then the following should be true 1. Allow hadoop config to be set to ignore this in case becomes a problem (too many retries) 2. Set max retries for all AM; i.e. users cannot specify a number above this. If Hadoop config does not specify it the value should be 1 (i.e fire and forget) 3. Lets say that the Hadoop config on max retries is set to 3. Then AM should try 3 times with timeout (again a Hadoop variable). If timeout is not specified, the value should be 30 seconds. The aim is to let AM get off quickly. Since unlike JT there is one AM per job there would not bottleneck and no hanging of queue. So 30 seconds is ok.
          Hide
          Arun C Murthy added a comment -

          Ravi, any progress on this? Thanks.

          Show
          Arun C Murthy added a comment - Ravi, any progress on this? Thanks.
          Hide
          Alejandro Abdelnur added a comment -

          Yes, makes sense to deprecate but leave the push functionality for a while.

          Regarding single-threaded, that was on purpose to avoid using multiple JT threads for this.

          Show
          Alejandro Abdelnur added a comment - Yes, makes sense to deprecate but leave the push functionality for a while. Regarding single-threaded, that was on purpose to avoid using multiple JT threads for this.
          Hide
          Robert Joseph Evans added a comment -

          @Alejandro, OK I agree completely that an RSS like feed for reporting which jobs have completed is preferable, so long as we can make it so there is relatively little load on the system even when polling multiple times a second.

          But if we remove the notification all together there may be other applications relying on this functionality that fail. I would prefer to see both go in, and we deprecate the push. This would give users time to move off of it.

          Also the delay is not that bad if it is the application master that is sending the notification. The issue with the delay before was primarily caused by the notification being single threaded. If one application failed with a 60sec time out then the other notifications started to pile up behind it. This would still be bad for a map/reduce job to pause 60 seconds for the notification to go through, especially with retry, but it would not impact any other applications not connected to it.

          Show
          Robert Joseph Evans added a comment - @Alejandro, OK I agree completely that an RSS like feed for reporting which jobs have completed is preferable, so long as we can make it so there is relatively little load on the system even when polling multiple times a second. But if we remove the notification all together there may be other applications relying on this functionality that fail. I would prefer to see both go in, and we deprecate the push. This would give users time to move off of it. Also the delay is not that bad if it is the application master that is sending the notification. The issue with the delay before was primarily caused by the notification being single threaded. If one application failed with a 60sec time out then the other notifications started to pile up behind it. This would still be bad for a map/reduce job to pause 60 seconds for the notification to go through, especially with retry, but it would not impact any other applications not connected to it.
          Hide
          Alejandro Abdelnur added a comment -

          @Robert, with MR1 we are doing push. A lesson learned from that is that if the receivers of the call are slow or not avail the push notifications pile up and there is a long delay before they are delivered (at 60 secs HTTP timeout per bad notification, this is a lot in a heavy loaded system). Using a RSS (like) feed the amount of memory used in the JT is about the same (I would think even less) and there is not contention & delay. Plus the fact that more than one party can 'poll' for the status of a job as opposed to a single party 'push'. Using something like size-bound concurrent list to store & serve the job status would not create contention in the JT. Clients could poll often (30 to 60 secs) which seems like a minimal delay.

          Show
          Alejandro Abdelnur added a comment - @Robert, with MR1 we are doing push. A lesson learned from that is that if the receivers of the call are slow or not avail the push notifications pile up and there is a long delay before they are delivered (at 60 secs HTTP timeout per bad notification, this is a lot in a heavy loaded system). Using a RSS (like) feed the amount of memory used in the JT is about the same (I would think even less) and there is not contention & delay. Plus the fact that more than one party can 'poll' for the status of a job as opposed to a single party 'push'. Using something like size-bound concurrent list to store & serve the job status would not create contention in the JT. Clients could poll often (30 to 60 secs) which seems like a minimal delay.
          Hide
          Robert Joseph Evans added a comment -

          The problem with the RSS feed for Oozie is that Oozie wants a push notification so that it can start other jobs as quickly as possible and avoid polling to know when a job has finished. Currently it supports polling in addition to the push but I believe the default polling interval is set to 10 min to reduce the load on the task tracker.

          There also may be other applications that depend on the push notification just like we had before. So for feature parity we probably still need the end notification.

          Show
          Robert Joseph Evans added a comment - The problem with the RSS feed for Oozie is that Oozie wants a push notification so that it can start other jobs as quickly as possible and avoid polling to know when a job has finished. Currently it supports polling in addition to the push but I believe the default polling interval is set to 10 min to reduce the load on the task tracker. There also may be other applications that depend on the push notification just like we had before. So for feature parity we probably still need the end notification.
          Hide
          Alejandro Abdelnur added a comment -

          Another possibility is to do an RSS like feed that provides info about jobs ended after a given timestamp up to a maximum (configurable) age.

          Show
          Alejandro Abdelnur added a comment - Another possibility is to do an RSS like feed that provides info about jobs ended after a given timestamp up to a maximum (configurable) age.

            People

            • Assignee:
              Ravi Prakash
              Reporter:
              Mohammad Kamrul Islam
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development