Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4099

HFTP interface compatibility with older releases broken

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Using current trunk (Revision 692556) build and trying to distcp from a 0.18.0 cluster via HFTP, got the following NullPointerException. Seems like on line 165 in HftpFileSystem.java, a potential null return value isn't checked before using.

      coursemud-lm:bin kan$ ./hadoop distcp hftp://ucdev18.inktomisearch.com:50070/output/part-00000 /copied/part-00001
      08/09/05 19:00:10 INFO tools.DistCp: srcPaths=[hftp://ucdev18.inktomisearch.com:50070/output/part-00000]
      08/09/05 19:00:10 INFO tools.DistCp: destPath=/copied/part-00001
      With failures, global counters are inaccurate; consider running with -i
      Copy failed: java.lang.NullPointerException
      at java.text.SimpleDateFormat.parse(SimpleDateFormat.java:1215)
      at java.text.DateFormat.parse(DateFormat.java:335)
      at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.startElement(HftpFileSystem.java:165)
      at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:501)
      at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:179)
      at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:377)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2740)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:647)
      at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:508)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:807)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
      at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)
      at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
      at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:194)
      at org.apache.hadoop.hdfs.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:205)
      at org.apache.hadoop.hdfs.HftpFileSystem.getFileStatus(HftpFileSystem.java:234)
      at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:649)
      at org.apache.hadoop.tools.DistCp.checkSrcPath(DistCp.java:614)
      at org.apache.hadoop.tools.DistCp.copy(DistCp.java:631)
      at org.apache.hadoop.tools.DistCp.run(DistCp.java:838)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.tools.DistCp.main(DistCp.java:865)

      1. hftpAccessTime.patch
        0.7 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          rangadi Raghu Angadi added a comment -

          Dhruba, could you take a quick look? Looks like hsftp should handle missing accesstime field in xml.

          Show
          rangadi Raghu Angadi added a comment - Dhruba, could you take a quick look? Looks like hsftp should handle missing accesstime field in xml.
          Hide
          dhruba dhruba borthakur added a comment -

          Ok, looking at it...

          Show
          dhruba dhruba borthakur added a comment - Ok, looking at it...
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Setting this to a 0.19 blocker.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Setting this to a 0.19 blocker.
          Hide
          dhruba dhruba borthakur added a comment -

          Use the access time only if present.

          Show
          dhruba dhruba borthakur added a comment - Use the access time only if present.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12389693/hftpAccessTime.patch
          against trunk revision 693048.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3210/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3210/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3210/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3210/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12389693/hftpAccessTime.patch against trunk revision 693048. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3210/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3210/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3210/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3210/console This message is automatically generated.
          Hide
          dhruba dhruba borthakur added a comment -

          @Nicholas: can you pl review this one-line patch?

          Show
          dhruba dhruba borthakur added a comment - @Nicholas: can you pl review this one-line patch?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          dhruba dhruba borthakur added a comment -

          The test failures are not related to this patch. Resubmitting patch to trigger another round of HadoopQA tests

          Show
          dhruba dhruba borthakur added a comment - The test failures are not related to this patch. Resubmitting patch to trigger another round of HadoopQA tests
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12389693/hftpAccessTime.patch
          against trunk revision 693545.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no tests are needed for this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3221/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3221/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3221/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3221/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12389693/hftpAccessTime.patch against trunk revision 693545. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3221/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3221/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3221/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3221/console This message is automatically generated.
          Hide
          owen.omalley Owen O'Malley added a comment -

          I just committed this. Thanks, Dhruba!

          Show
          owen.omalley Owen O'Malley added a comment - I just committed this. Thanks, Dhruba!

            People

            • Assignee:
              dhruba dhruba borthakur
              Reporter:
              kzhang Kan Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development