Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1170

Add more assertions to TestLargeDirectoryDelete

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This issue is for a patch which will only add some more asserts to the TestLargeDirectoryDelete test case, to help identify the causes of HDFS-615 and HDFS-816.

      Proposed

      1. split the chained operations into one per line for better stack trace diagnostics
      2. add assertNotNull assertions for better diagnostics.
      1. HDFS-1170.patch
        5 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          I've just committed this. Thanks Steve!

          Show
          Tom White added a comment - I've just committed this. Thanks Steve!
          Hide
          Tom White added a comment -

          +1

          I think this can be committed (to 0.21 and trunk). I ran TestLargeDirectoryDelete 10 times and got 8 NullPointerExceptions. With this patch I got no failures and no NPEs in 10 runs, so it fixes HDFS-1170 for me (this is on a Mac).

          Show
          Tom White added a comment - +1 I think this can be committed (to 0.21 and trunk). I ran TestLargeDirectoryDelete 10 times and got 8 NullPointerExceptions. With this patch I got no failures and no NPEs in 10 runs, so it fixes HDFS-1170 for me (this is on a Mac).
          Hide
          Steve Loughran added a comment -

          seeing what happens this week

          Show
          Steve Loughran added a comment - seeing what happens this week
          Hide
          Steve Loughran added a comment -

          contrib tests are going to be unrelated; this patch only patches core tests. but the log says that the #of NN tests went down by two, which isn't good and which I don't know whether or not is related.

          Show
          Steve Loughran added a comment - contrib tests are going to be unrelated; this patch only patches core tests. but the log says that the #of NN tests went down by two, which isn't good and which I don't know whether or not is related.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445182/HDFS-1170.patch
          against trunk revision 951178.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445182/HDFS-1170.patch against trunk revision 951178. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/console This message is automatically generated.
          Hide
          Steve Loughran added a comment -

          test failures don't appear related, but something is up on hudson as in't parsing a config file with the usual JDK helpful error messages:

          org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testTruncatedBlockReport (from TestDatanodeBlockScanner)
          Failing for the past 1 build (Since Failed#373 )
          Took 2 ms.
          add description
          Error Message
          
          java.util.zip.ZipException: error reading zip file
          
          Stacktrace
          
          java.lang.RuntimeException: java.util.zip.ZipException: error reading zip file
          at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1509)
          at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1374)
          at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1320)
          at org.apache.hadoop.conf.Configuration.set(Configuration.java:573)
          at org.apache.hadoop.fs.FileSystem.setDefaultUri(FileSystem.java:143)
          at org.apache.hadoop.fs.FileSystem.setDefaultUri(FileSystem.java:151)
          at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
          at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:138)
          at org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testTruncatedBlockReport(TestDatanodeBlockScanner.java:388)
          Caused by: java.util.zip.ZipException: error reading zip file
          at java.util.zip.ZipFile.read(Native Method)
          at java.util.zip.ZipFile.access$1200(ZipFile.java:29)
          at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:447)
          at java.util.zip.ZipFile$1.fill(ZipFile.java:230)
          at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
          at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105)
          at java.io.FilterInputStream.read(FilterInputStream.java:66)
          at com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2910)
          at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:704)
          at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186)
          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
          at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)
          at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:225)
          at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:283)
          at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
          at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1423)
          
          Show
          Steve Loughran added a comment - test failures don't appear related, but something is up on hudson as in't parsing a config file with the usual JDK helpful error messages: org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testTruncatedBlockReport (from TestDatanodeBlockScanner) Failing for the past 1 build (Since Failed#373 ) Took 2 ms. add description Error Message java.util.zip.ZipException: error reading zip file Stacktrace java.lang.RuntimeException: java.util.zip.ZipException: error reading zip file at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1509) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1374) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1320) at org.apache.hadoop.conf.Configuration.set(Configuration.java:573) at org.apache.hadoop.fs.FileSystem.setDefaultUri(FileSystem.java:143) at org.apache.hadoop.fs.FileSystem.setDefaultUri(FileSystem.java:151) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:138) at org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testTruncatedBlockReport(TestDatanodeBlockScanner.java:388) Caused by: java.util.zip.ZipException: error reading zip file at java.util.zip.ZipFile.read(Native Method) at java.util.zip.ZipFile.access$1200(ZipFile.java:29) at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:447) at java.util.zip.ZipFile$1.fill(ZipFile.java:230) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105) at java.io.FilterInputStream.read(FilterInputStream.java:66) at com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2910) at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:704) at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107) at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:225) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:283) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1423)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445182/HDFS-1170.patch
          against trunk revision 946488.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445182/HDFS-1170.patch against trunk revision 946488. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/console This message is automatically generated.
          Hide
          Steve Loughran added a comment -

          patch with better thread shutdown as well as exception catch rethrow and more assertions. This should shut down better, and if not, the asserts help identify problems more reliably.

          Show
          Steve Loughran added a comment - patch with better thread shutdown as well as exception catch rethrow and more assertions. This should shut down better, and if not, the asserts help identify problems more reliably.
          Hide
          Steve Loughran added a comment -

          And catch exceptions in child threads, relay them up, have a better way of stopping the worker threads than just interrupt(), which isn't checked on every operation.

          Show
          Steve Loughran added a comment - And catch exceptions in child threads, relay them up, have a better way of stopping the worker threads than just interrupt(), which isn't checked on every operation.

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development