Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6267

Refactor JobSubmitter#copyAndConfigureFiles into it's own class

    Details

      Description

      Refactor the uploading logic in JobSubmitter#copyAndConfigureFiles into it's own class. This makes the JobSubmitter class more readable and isolates the logic that is actually uploading the job resources to HDFS.

        Issue Links

          Activity

          Hide
          ctrezzo Chris Trezzo added a comment -

          V1 patch attached. Submitting patch for QA run.

          Show
          ctrezzo Chris Trezzo added a comment - V1 patch attached. Submitting patch for QA run.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12701144/MAPREDUCE-6267-trunk-v1.patch
          against trunk revision dce8b9c.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5226//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5226//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12701144/MAPREDUCE-6267-trunk-v1.patch against trunk revision dce8b9c. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5226//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5226//console This message is automatically generated.
          Hide
          ctrezzo Chris Trezzo added a comment -

          Note, didn't include a new unit test because coverage already exists with current unit tests. This patch does not make any functional changes and is purely a refactor.

          Show
          ctrezzo Chris Trezzo added a comment - Note, didn't include a new unit test because coverage already exists with current unit tests. This patch does not make any functional changes and is purely a refactor.
          Hide
          kasha Karthik Kambatla added a comment -

          Looks good. +1.

          Committing this.

          Show
          kasha Karthik Kambatla added a comment - Looks good. +1. Committing this.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for the patch, Chris.

          Committed to trunk and branch-2.

          Show
          kasha Karthik Kambatla added a comment - Thanks for the patch, Chris. Committed to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7257 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7257/)
          MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7257 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7257/ ) MAPREDUCE-6267 . Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #123 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/123/)
          MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #123 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/123/ ) MAPREDUCE-6267 . Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #857 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/857/)
          MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #857 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/857/ ) MAPREDUCE-6267 . Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2055 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2055/)
          MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2055 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2055/ ) MAPREDUCE-6267 . Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #114 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/114/)
          MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #114 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/114/ ) MAPREDUCE-6267 . Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #123 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/123/)
          MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #123 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/123/ ) MAPREDUCE-6267 . Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2073 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2073/)
          MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2073 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2073/ ) MAPREDUCE-6267 . Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) (kasha: rev c66c3ac6bf9f63177279feec3f2917e4b882e2bc) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobResourceUploader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Added this to 2.6.1 as a dependency for MAPREDUCE-6238.

          Pulled this into 2.6.1, the patch had conflicts in Job.java and JobSumitter.java which I fixed.

          Ran compilation before the push. Patch applied cleanly.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Added this to 2.6.1 as a dependency for MAPREDUCE-6238 . Pulled this into 2.6.1, the patch had conflicts in Job.java and JobSumitter.java which I fixed. Ran compilation before the push. Patch applied cleanly.

            People

            • Assignee:
              ctrezzo Chris Trezzo
              Reporter:
              ctrezzo Chris Trezzo
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development