Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1170

Add more assertions to TestLargeDirectoryDelete

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This issue is for a patch which will only add some more asserts to the TestLargeDirectoryDelete test case, to help identify the causes of HDFS-615 and HDFS-816.

      Proposed

      1. split the chained operations into one per line for better stack trace diagnostics
      2. add assertNotNull assertions for better diagnostics.
      1. HDFS-1170.patch
        5 kB
        Steve Loughran

        Issue Links

          Activity

          Steve Loughran created issue -
          Steve Loughran made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Link This issue is related to HDFS-615 [ HDFS-615 ]
          Hide
          Steve Loughran added a comment -

          And catch exceptions in child threads, relay them up, have a better way of stopping the worker threads than just interrupt(), which isn't checked on every operation.

          Show
          Steve Loughran added a comment - And catch exceptions in child threads, relay them up, have a better way of stopping the worker threads than just interrupt(), which isn't checked on every operation.
          Hide
          Steve Loughran added a comment -

          patch with better thread shutdown as well as exception catch rethrow and more assertions. This should shut down better, and if not, the asserts help identify problems more reliably.

          Show
          Steve Loughran added a comment - patch with better thread shutdown as well as exception catch rethrow and more assertions. This should shut down better, and if not, the asserts help identify problems more reliably.
          Steve Loughran made changes -
          Attachment HDFS-1170.patch [ 12445182 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445182/HDFS-1170.patch
          against trunk revision 946488.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445182/HDFS-1170.patch against trunk revision 946488. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/373/console This message is automatically generated.
          Hide
          Steve Loughran added a comment -

          test failures don't appear related, but something is up on hudson as in't parsing a config file with the usual JDK helpful error messages:

          org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testTruncatedBlockReport (from TestDatanodeBlockScanner)
          Failing for the past 1 build (Since Failed#373 )
          Took 2 ms.
          add description
          Error Message
          
          java.util.zip.ZipException: error reading zip file
          
          Stacktrace
          
          java.lang.RuntimeException: java.util.zip.ZipException: error reading zip file
          at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1509)
          at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1374)
          at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1320)
          at org.apache.hadoop.conf.Configuration.set(Configuration.java:573)
          at org.apache.hadoop.fs.FileSystem.setDefaultUri(FileSystem.java:143)
          at org.apache.hadoop.fs.FileSystem.setDefaultUri(FileSystem.java:151)
          at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280)
          at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:138)
          at org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testTruncatedBlockReport(TestDatanodeBlockScanner.java:388)
          Caused by: java.util.zip.ZipException: error reading zip file
          at java.util.zip.ZipFile.read(Native Method)
          at java.util.zip.ZipFile.access$1200(ZipFile.java:29)
          at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:447)
          at java.util.zip.ZipFile$1.fill(ZipFile.java:230)
          at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
          at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105)
          at java.io.FilterInputStream.read(FilterInputStream.java:66)
          at com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2910)
          at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:704)
          at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186)
          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
          at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
          at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)
          at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:225)
          at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:283)
          at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
          at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1423)
          
          Show
          Steve Loughran added a comment - test failures don't appear related, but something is up on hudson as in't parsing a config file with the usual JDK helpful error messages: org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testTruncatedBlockReport (from TestDatanodeBlockScanner) Failing for the past 1 build (Since Failed#373 ) Took 2 ms. add description Error Message java.util.zip.ZipException: error reading zip file Stacktrace java.lang.RuntimeException: java.util.zip.ZipException: error reading zip file at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1509) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1374) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1320) at org.apache.hadoop.conf.Configuration.set(Configuration.java:573) at org.apache.hadoop.fs.FileSystem.setDefaultUri(FileSystem.java:143) at org.apache.hadoop.fs.FileSystem.setDefaultUri(FileSystem.java:151) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:138) at org.apache.hadoop.hdfs.TestDatanodeBlockScanner.testTruncatedBlockReport(TestDatanodeBlockScanner.java:388) Caused by: java.util.zip.ZipException: error reading zip file at java.util.zip.ZipFile.read(Native Method) at java.util.zip.ZipFile.access$1200(ZipFile.java:29) at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:447) at java.util.zip.ZipFile$1.fill(ZipFile.java:230) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:105) at java.io.FilterInputStream.read(FilterInputStream.java:66) at com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2910) at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:704) at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107) at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:225) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:283) at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1423)
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12445182/HDFS-1170.patch
          against trunk revision 951178.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445182/HDFS-1170.patch against trunk revision 951178. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/396/console This message is automatically generated.
          Hide
          Steve Loughran added a comment -

          contrib tests are going to be unrelated; this patch only patches core tests. but the log says that the #of NN tests went down by two, which isn't good and which I don't know whether or not is related.

          Show
          Steve Loughran added a comment - contrib tests are going to be unrelated; this patch only patches core tests. but the log says that the #of NN tests went down by two, which isn't good and which I don't know whether or not is related.
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Steve Loughran added a comment -

          seeing what happens this week

          Show
          Steve Loughran added a comment - seeing what happens this week
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Tom White added a comment -

          +1

          I think this can be committed (to 0.21 and trunk). I ran TestLargeDirectoryDelete 10 times and got 8 NullPointerExceptions. With this patch I got no failures and no NPEs in 10 runs, so it fixes HDFS-1170 for me (this is on a Mac).

          Show
          Tom White added a comment - +1 I think this can be committed (to 0.21 and trunk). I ran TestLargeDirectoryDelete 10 times and got 8 NullPointerExceptions. With this patch I got no failures and no NPEs in 10 runs, so it fixes HDFS-1170 for me (this is on a Mac).
          Hide
          Tom White added a comment -

          I've just committed this. Thanks Steve!

          Show
          Tom White added a comment - I've just committed this. Thanks Steve!
          Tom White made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12314046 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Resolution Fixed [ 1 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          4s 1 Steve Loughran 21/May/10 17:09
          In Progress In Progress Patch Available Patch Available
          1h 27m 1 Steve Loughran 21/May/10 18:36
          Patch Available Patch Available Open Open
          20d 17h 21m 2 Steve Loughran 11/Jun/10 11:58
          Open Open Patch Available Patch Available
          23s 2 Steve Loughran 11/Jun/10 11:58
          Patch Available Patch Available Resolved Resolved
          4d 5h 24m 1 Tom White 15/Jun/10 17:22
          Resolved Resolved Closed Closed
          70d 4h 30m 1 Tom White 24/Aug/10 21:52

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development