Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3154

Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 0.24.0
    • Fix Version/s: 0.23.0
    • Component/s: client, mrv2
    • Labels:
      None
    • Environment:

      mrv2

      Description

      Presently the output specification is validated after getting new JobId from ClientRMService, Copying the job jar, Configuration file, archives etc.

      Instead of that move following Job Output specification validation call to the begining of JobSubmitter.submitJobInternal(Job, Cluster) method.

      checkSpecs(job);
      

      This will avoid unnecessary work in case of invalid output specs.

      1. MAPREDUCE-3154.patch
        1 kB
        Abhijit Suresh Shingate

        Issue Links

          Activity

          Hide
          Jane Chen added a comment -

          No new diff introduced by the proposed change on branch-1.

          The reported issue does not occur on trunk. Attached patch mapreduce-3377.patch includes only the unit test and is generated on trunk.

          The other attachments shows that the added unit test passes after the proposed fix and no other diff is introduced.

          out-before: stdout running "ant test" before the change.
          err-before: stderr running "ant test" before the change.
          out-after: stdout running "ant test" after the change.
          err-after: stderr running "ant test" after the change.

          Show
          Jane Chen added a comment - No new diff introduced by the proposed change on branch-1. The reported issue does not occur on trunk. Attached patch mapreduce-3377.patch includes only the unit test and is generated on trunk. The other attachments shows that the added unit test passes after the proposed fix and no other diff is introduced. out-before: stdout running "ant test" before the change. err-before: stderr running "ant test" before the change. out-after: stdout running "ant test" after the change. err-after: stderr running "ant test" after the change.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #856 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/)
          MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #856 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/856/ ) MAPREDUCE-3154 . Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #42 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/)
          Merge -c 1180774 from trunk to branch-0.23 to fix MAPREDUCE-3154.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180775
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #42 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/42/ ) Merge -c 1180774 from trunk to branch-0.23 to fix MAPREDUCE-3154 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180775 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #35 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/)
          Merge -c 1180774 from trunk to branch-0.23 to fix MAPREDUCE-3154.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180775
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #35 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/35/ ) Merge -c 1180774 from trunk to branch-0.23 to fix MAPREDUCE-3154 . acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180775 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #826 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/)
          MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #826 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/826/ ) MAPREDUCE-3154 . Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Abhijit Suresh Shingate added a comment -

          Thanks Arun!!

          Show
          Abhijit Suresh Shingate added a comment - Thanks Arun!!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1069 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1069/)
          MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1069 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1069/ ) MAPREDUCE-3154 . Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1127 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1127/)
          MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1127 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1127/ ) MAPREDUCE-3154 . Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1049 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1049/)
          MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate.

          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1049 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1049/ ) MAPREDUCE-3154 . Fix JobSubmitter to check for output specs before copying job submission files to fail fast. Contributed by Abhijit Suresh Shingate. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180774 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Abhijit!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Abhijit!
          Hide
          Abhijit Suresh Shingate added a comment -

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          Only checkSpecs(job); call is moved to beginning of JobSubmitter.submitJobInternal(Job, Cluster) method.

          Show
          Abhijit Suresh Shingate added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. Only checkSpecs(job); call is moved to beginning of JobSubmitter.submitJobInternal(Job, Cluster) method.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12498193/MAPREDUCE-3154.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/967//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/967//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498193/MAPREDUCE-3154.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/967//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/967//console This message is automatically generated.

            People

            • Assignee:
              Abhijit Suresh Shingate
              Reporter:
              Abhijit Suresh Shingate
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development