Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9584

NPE in distcp when ssl configuration file does not exist in class path.

    Details

    • Hadoop Flags:
      Reviewed

      Description

      ./hadoop distcp -mapredSslConf ssl-distcp.xml hftp://x.x.x.x:25003/history hdfs://x.x.x.X:25008/history

      if ssl-distcp.xml file not exist in class path, distcp will throw NullPointerException.

      java.lang.NullPointerException
              at org.apache.hadoop.tools.DistCp.setupSSLConfig(DistCp.java:266)
              at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:250)
              at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:175)
              at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
              at org.apache.hadoop.tools.DistCp.run(DistCp.java:127)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
              at org.apache.hadoop.tools.DistCp.main(DistCp.java:431)
      
      1. HDFS-9584.patch
        1 kB
        Surendra Singh Lilhore
      2. HDFS-9584.patch
        1 kB
        Surendra Singh Lilhore
      3. HDFS-9584.001.patch
        1 kB
        Surendra Singh Lilhore

        Issue Links

          Activity

          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Attached patch, Please review ..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Attached patch, Please review ..
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Surendra Singh Lilhore for working on this. Patch LGTM.
          +1 pending Jenkins.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Surendra Singh Lilhore for working on this. Patch LGTM. +1 pending Jenkins.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Xiaoyu Yao for review..
          Resubmitting the patch to trigger Jenkins..

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Xiaoyu Yao for review.. Resubmitting the patch to trigger Jenkins..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks Surendra Singh Lilhore for reporting...I think, it should be Move to Hadoop-common..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks Surendra Singh Lilhore for reporting...I think, it should be Move to Hadoop-common..?
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Resubmitting patch to trigger jenkins build.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Resubmitting patch to trigger jenkins build.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 25s trunk passed
          +1 compile 0m 16s trunk passed with JDK v1.8.0_66
          +1 compile 0m 18s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 9s trunk passed
          +1 mvnsite 0m 26s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 0m 14s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 16s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.8.0_66
          +1 javac 0m 15s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.7.0_91
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 0m 42s hadoop-tools/hadoop-distcp introduced 2 new FindBugs issues.
          +1 javadoc 0m 13s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 15s the patch passed with JDK v1.7.0_91
          +1 unit 7m 25s hadoop-distcp in the patch passed with JDK v1.8.0_66.
          +1 unit 7m 15s hadoop-distcp in the patch passed with JDK v1.7.0_91.
          -1 asflicense 0m 25s Patch generated 1 ASF License warnings.
          29m 52s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-distcp
            Possible null pointer dereference of sslFileUrl in org.apache.hadoop.tools.DistCp.setupSSLConfig(Job) Dereferenced at DistCp.java:sslFileUrl in org.apache.hadoop.tools.DistCp.setupSSLConfig(Job) Dereferenced at DistCp.java:[line 281]
            new java.io.IOException(String) not thrown in org.apache.hadoop.tools.DistCp.setupSSLConfig(Job) At DistCp.java:org.apache.hadoop.tools.DistCp.setupSSLConfig(Job) At DistCp.java:[line 277]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779275/HDFS-9584.patch
          JIRA Issue HDFS-9584
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b9d52168715b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bb5df27
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13985/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-distcp.html
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13985/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13985/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13985/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 25s trunk passed +1 compile 0m 16s trunk passed with JDK v1.8.0_66 +1 compile 0m 18s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 9s trunk passed +1 mvnsite 0m 26s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 16s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 21s the patch passed +1 compile 0m 15s the patch passed with JDK v1.8.0_66 +1 javac 0m 15s the patch passed +1 compile 0m 17s the patch passed with JDK v1.7.0_91 +1 javac 0m 17s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 0m 42s hadoop-tools/hadoop-distcp introduced 2 new FindBugs issues. +1 javadoc 0m 13s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_91 +1 unit 7m 25s hadoop-distcp in the patch passed with JDK v1.8.0_66. +1 unit 7m 15s hadoop-distcp in the patch passed with JDK v1.7.0_91. -1 asflicense 0m 25s Patch generated 1 ASF License warnings. 29m 52s Reason Tests FindBugs module:hadoop-tools/hadoop-distcp   Possible null pointer dereference of sslFileUrl in org.apache.hadoop.tools.DistCp.setupSSLConfig(Job) Dereferenced at DistCp.java:sslFileUrl in org.apache.hadoop.tools.DistCp.setupSSLConfig(Job) Dereferenced at DistCp.java: [line 281]   new java.io.IOException(String) not thrown in org.apache.hadoop.tools.DistCp.setupSSLConfig(Job) At DistCp.java:org.apache.hadoop.tools.DistCp.setupSSLConfig(Job) At DistCp.java: [line 277] Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779275/HDFS-9584.patch JIRA Issue HDFS-9584 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b9d52168715b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bb5df27 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/13985/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-distcp.html JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13985/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13985/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13985/console This message was automatically generated.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Attached new patch, Fixed findbugs warnings..
          Please review.

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Attached new patch, Fixed findbugs warnings.. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 0s trunk passed
          +1 compile 0m 15s trunk passed with JDK v1.8.0_66
          +1 compile 0m 17s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 9s trunk passed
          +1 mvnsite 0m 24s trunk passed
          +1 mvneclipse 0m 22s trunk passed
          +1 findbugs 0m 30s trunk passed
          +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 17s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 23s the patch passed
          +1 compile 0m 15s the patch passed with JDK v1.8.0_66
          +1 javac 0m 15s the patch passed
          +1 compile 0m 19s the patch passed with JDK v1.7.0_91
          +1 javac 0m 19s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 39s the patch passed
          +1 javadoc 0m 14s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 17s the patch passed with JDK v1.7.0_91
          +1 unit 7m 12s hadoop-distcp in the patch passed with JDK v1.8.0_66.
          +1 unit 7m 1s hadoop-distcp in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 22s Patch does not generate ASF License warnings.
          29m 11s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779439/HDFS-9584.001.patch
          JIRA Issue HDFS-9584
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5bc632921dc5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6553784
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13990/testReport/
          modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13990/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 0s trunk passed +1 compile 0m 15s trunk passed with JDK v1.8.0_66 +1 compile 0m 17s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 9s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 22s trunk passed +1 findbugs 0m 30s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 23s the patch passed +1 compile 0m 15s the patch passed with JDK v1.8.0_66 +1 javac 0m 15s the patch passed +1 compile 0m 19s the patch passed with JDK v1.7.0_91 +1 javac 0m 19s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 17s the patch passed with JDK v1.7.0_91 +1 unit 7m 12s hadoop-distcp in the patch passed with JDK v1.8.0_66. +1 unit 7m 1s hadoop-distcp in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 29m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779439/HDFS-9584.001.patch JIRA Issue HDFS-9584 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5bc632921dc5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6553784 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13990/testReport/ modules C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13990/console This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks for working on this! Surendra Singh Lilhore
          I have also hit by this issue, and I saw it occured as early as Hadoop 2.5.x

          The patch looks good to me +1 (non-binding)

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks for working on this! Surendra Singh Lilhore I have also hit by this issue, and I saw it occured as early as Hadoop 2.5.x The patch looks good to me +1 (non-binding)
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Surendra Singh Lilhore for the contribution and all for the reviews. I've commit the change to trunk, branch-2 and branch-2.8.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Surendra Singh Lilhore for the contribution and all for the reviews. I've commit the change to trunk, branch-2 and branch-2.8.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks Xiaoyu Yao for the commit. I noticed that the commit message is a bit misleading:
          "HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore."

          It should be HDFS-9584 instead.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks Xiaoyu Yao for the commit. I noticed that the commit message is a bit misleading: " HDFS-8584 . NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore." It should be HDFS-9584 instead.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Wei-Chiu Chuang! I've corrected the commit message.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Wei-Chiu Chuang ! I've corrected the commit message.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9088 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9088/)
          Correct commit message for HDFS-9584 (xyao: rev 103d3cfc4ee1ac21970fd6bbca54bb085ab771ba)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9088 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9088/ ) Correct commit message for HDFS-9584 (xyao: rev 103d3cfc4ee1ac21970fd6bbca54bb085ab771ba) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Xiaoyu Yao for review and commit..
          Thanks Wei-Chiu Chuang for review...

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Xiaoyu Yao for review and commit.. Thanks Wei-Chiu Chuang for review...

            People

            • Assignee:
              surendrasingh Surendra Singh Lilhore
              Reporter:
              surendrasingh Surendra Singh Lilhore
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development