Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1031

Enhance the webUi to list a few of the corrupted files in HDFS

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The existing webUI displays something like this:
      WARNING : There are about 12 missing blocks. Please check the log or run fsck.

      It would be nice if we can display the filenames that have missing blocks.

      1. hdfs-1031_aoriani_5.patch
        8 kB
        André Oriani
      2. hdfs-1031_aoriani_4.patch
        8 kB
        André Oriani
      3. ASF.LICENSE.NOT.GRANTED--hdfs-1031_aoriani_3.patch
        8 kB
        André Oriani
      4. ASF.LICENSE.NOT.GRANTED--hdfs-1031_aoriani_2.patch
        8 kB
        André Oriani
      5. hdfs-1031_aoriani.patch
        3 kB
        André Oriani

        Issue Links

          Activity

          Hide
          dhruba borthakur added a comment -

          I just committed this. Thanks Andre.

          Show
          dhruba borthakur added a comment - I just committed this. Thanks Andre.
          Hide
          André Oriani added a comment -

          No further comments.

          Show
          André Oriani added a comment - No further comments.
          Hide
          dhruba borthakur added a comment -

          Thanks guys. I am going to commit this soon, unless anybody else has any other review comments.

          Show
          dhruba borthakur added a comment - Thanks guys. I am going to commit this soon, unless anybody else has any other review comments.
          Hide
          Rodrigo Schmidt added a comment -

          +1. Patch looks good. Thanks André!

          Show
          Rodrigo Schmidt added a comment - +1. Patch looks good. Thanks André!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12442542/hdfs-1031_aoriani_5.patch
          against trunk revision 936473.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/322/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/322/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/322/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/322/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442542/hdfs-1031_aoriani_5.patch against trunk revision 936473. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/322/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/322/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/322/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/322/console This message is automatically generated.
          Hide
          André Oriani added a comment -

          Minor fix made

          Show
          André Oriani added a comment - Minor fix made
          Hide
          Rodrigo Schmidt added a comment -

          The patch looks good. Just one minor nitpick: Can you add a final "." to the sentence "Please run fsck for a thorough health analysis"?

          Thanks!

          Show
          Rodrigo Schmidt added a comment - The patch looks good. Just one minor nitpick: Can you add a final "." to the sentence "Please run fsck for a thorough health analysis"? Thanks!
          Hide
          André Oriani added a comment -

          Failed core test

          java.lang.IllegalMonitorStateException
          at java.lang.Object.notifyAll(Native Method)
          at org.apache.hadoop.ipc.Server.stop(Server.java:1469)
          at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:473)
          at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:619)
          at org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs.testNameEditsConfigs(TestNameEditsConfigs.java:228)

          Failed contributed test

          [exec] test-junit:
          [exec] [junit] WARNING: multiple versions of ant detected in path for junit
          [exec] [junit] jar:file:/homes/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class
          [exec] [junit] and jar:file:/homes/hudson/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
          [exec] [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
          [exec] [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 37.468 sec
          [exec] [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED
          [exec]
          [exec] BUILD FAILED
          [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:582: The following error occurred while executing this line:
          [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/build.xml:48: The following error occurred while executing this line:
          [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/hdfsproxy/build.xml:260: Tests failed!

          Both unrelated to this patch

          Show
          André Oriani added a comment - Failed core test java.lang.IllegalMonitorStateException at java.lang.Object.notifyAll(Native Method) at org.apache.hadoop.ipc.Server.stop(Server.java:1469) at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:473) at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:619) at org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs.testNameEditsConfigs(TestNameEditsConfigs.java:228) Failed contributed test [exec] test-junit: [exec] [junit] WARNING: multiple versions of ant detected in path for junit [exec] [junit] jar: file:/homes/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class [exec] [junit] and jar: file:/homes/hudson/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [exec] [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy [exec] [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 37.468 sec [exec] [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED [exec] [exec] BUILD FAILED [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:582: The following error occurred while executing this line: [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/build.xml:48: The following error occurred while executing this line: [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/hdfsproxy/build.xml:260: Tests failed! Both unrelated to this patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12442391/hdfs-1031_aoriani_4.patch
          against trunk revision 936132.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442391/hdfs-1031_aoriani_4.patch against trunk revision 936132. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321/console This message is automatically generated.
          Hide
          André Oriani added a comment -

          In case hudson is still not adding test result to Jira, the build is http://hudson.zones.apache.org/hudson/view/Hdfs/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321

          Show
          André Oriani added a comment - In case hudson is still not adding test result to Jira, the build is http://hudson.zones.apache.org/hudson/view/Hdfs/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/321
          Hide
          André Oriani added a comment -

          Suggestions applied
          File list sorted
          Some changes made due semantic issues ( listed files are not potentially corrupt, but in fact corrupt. The list is potentially incomplete)

          Show
          André Oriani added a comment - Suggestions applied File list sorted Some changes made due semantic issues ( listed files are not potentially corrupt, but in fact corrupt. The list is potentially incomplete)
          Hide
          Rodrigo Schmidt added a comment -

          Side effect in my opinion!

          Show
          Rodrigo Schmidt added a comment - Side effect in my opinion!
          Hide
          André Oriani added a comment -

          One doubt regarding sorting the output. On my work, l10n is a daily concern, so I pay attention to that when coding.
          We're going to compare Paths. Path.compareTo method delegates its implementation to java.net.URI.compareTo, which uses String.compareTo for path elements. That method of String class does not uses collation.

          So for the program:

          import java.net.URI;
          import java.net.URISyntaxException;
          import java.util.Arrays;
          
          public class PathComparisons {
          	public static void main(String[] args) {
          		try {
          			URI paths[] = { new URI("file:///b/a"), new URI("file:///a/c"),
          					new URI("file:///a/b"), new URI("file:///b/z"),
          					new URI("file:///a/á"), new URI("file:///b/ç"),
          					new URI("file:///a/b/c/d/e")};
          			Arrays.sort(paths);
          			for (URI path : paths) {System.out.println(path.toString());}
          		} catch (URISyntaxException e) {
          		}
          	}
          }
          

          The output is

          file:///a/b
          file:///a/b/c/d/e
          file:///a/c
          file:///a/á
          file:///b/a
          file:///b/z
          file:///b/ç
          

          I mean , the character order is

          {a,b,z,á,ç}

          instead of an expected

          {a,á,b,c,ç,z}

          Should I handle this or treat it as a minor side-effect ?

          Show
          André Oriani added a comment - One doubt regarding sorting the output. On my work, l10n is a daily concern, so I pay attention to that when coding. We're going to compare Paths. Path.compareTo method delegates its implementation to java.net.URI.compareTo, which uses String.compareTo for path elements. That method of String class does not uses collation. So for the program: import java.net.URI; import java.net.URISyntaxException; import java.util.Arrays; public class PathComparisons { public static void main( String [] args) { try { URI paths[] = { new URI( "file: ///b/a" ), new URI( "file:///a/c" ), new URI( "file: ///a/b" ), new URI( "file:///b/z" ), new URI( "file: ///a/á" ), new URI( "file:///b/ç" ), new URI( "file: ///a/b/c/d/e" )}; Arrays.sort(paths); for (URI path : paths) { System .out.println(path.toString());} } catch (URISyntaxException e) { } } } The output is file:///a/b file:///a/b/c/d/e file:///a/c file:///a/á file:///b/a file:///b/z file:///b/ç I mean , the character order is {a,b,z,á,ç} instead of an expected {a,á,b,c,ç,z} Should I handle this or treat it as a minor side-effect ?
          Hide
          Rodrigo Schmidt added a comment -

          +1 to "List corrupt files"

          Show
          Rodrigo Schmidt added a comment - +1 to "List corrupt files"
          Hide
          dhruba borthakur added a comment -

          +1 to sorting the output.

          Also, in dfshealth.jsp, can we change "See corrupt files" to "List corrupt files"

          Show
          dhruba borthakur added a comment - +1 to sorting the output. Also, in dfshealth.jsp, can we change "See corrupt files" to "List corrupt files"
          Hide
          Rodrigo Schmidt added a comment -

          Printing the output in lexicographical order seems to be best thing.

          Show
          Rodrigo Schmidt added a comment - Printing the output in lexicographical order seems to be best thing.
          Hide
          André Oriani added a comment -

          So should we stick to Rodrigo's message for the zero case? The output is not is not sorted. Considering there is a ceiling of 500 files, should it be sorted?

          Show
          André Oriani added a comment - So should we stick to Rodrigo's message for the zero case? The output is not is not sorted. Considering there is a ceiling of 500 files, should it be sorted?
          Hide
          Eli Collins added a comment -

          An "Unable to locate any" style message sounds good to me, I just thought "No corrupt files yet" looked odd.

          Show
          Eli Collins added a comment - An "Unable to locate any" style message sounds good to me, I just thought "No corrupt files yet" looked odd.
          Hide
          Rodrigo Schmidt added a comment -

          I'm sorry I'm a little late on this discussion, but I don't agree with Eli's comment on the message when corruptFileCount == 0. The API nn.getCorruptFiles() is based on the list of blocks that need replication and might not be up-to-date (HDFS-729). For instance, I think it returns 0 if the namenode is on safe mode. We had a similar discussion on HDFS-1032 and we decided to go for the message André had before ("unable to locate any"-style). Maybe we should use something in between "100% guaranteed it's safe" and "Please don't trust me".

          What about "No missing blocks found at the moment. Please run fsck for a thorough health analysis." ?

          Show
          Rodrigo Schmidt added a comment - I'm sorry I'm a little late on this discussion, but I don't agree with Eli's comment on the message when corruptFileCount == 0. The API nn.getCorruptFiles() is based on the list of blocks that need replication and might not be up-to-date ( HDFS-729 ). For instance, I think it returns 0 if the namenode is on safe mode. We had a similar discussion on HDFS-1032 and we decided to go for the message André had before ("unable to locate any"-style). Maybe we should use something in between "100% guaranteed it's safe" and "Please don't trust me". What about "No missing blocks found at the moment. Please run fsck for a thorough health analysis." ?
          Hide
          dhruba borthakur added a comment -

          if u run metasave on a namenode that ahs anywhere close to 50 million files, the NN hangs for about 8 minutes Not really useful, isn't it?

          Show
          dhruba borthakur added a comment - if u run metasave on a namenode that ahs anywhere close to 50 million files, the NN hangs for about 8 minutes Not really useful, isn't it?
          Hide
          Suresh Srinivas added a comment -

          HDFS has metasave command that provides this information. Do we still need this on web UI?

          Show
          Suresh Srinivas added a comment - HDFS has metasave command that provides this information. Do we still need this on web UI?
          Hide
          André Oriani added a comment -

          Adding hudson results, including the log for failed test. Same result from previous build. Hdfsproxy test still failing but it is not related to this code

               [exec] compile-test:
               [exec]      [echo] contrib: hdfsproxy
               [exec]     [javac] Compiling 9 source files to /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build/contrib/hdfsproxy/test
               [exec] 
               [exec] test-junit:
               [exec]     [junit] WARNING: multiple versions of ant detected in path for junit 
               [exec]     [junit]          jar:file:/homes/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class
               [exec]     [junit]      and jar:file:/homes/hudson/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
               [exec]     [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
               [exec]     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 3.17 sec
               [exec]     [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED
               [exec] 
               [exec] BUILD FAILED
               [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:582: The following error occurred while executing this line:
               [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/build.xml:48: The following error occurred while executing this line:
               [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/hdfsproxy/build.xml:260: Tests failed!
               [exec] 
               [exec] Total time: 12 seconds
               [exec] 
               [exec] 
               [exec] 
               [exec] 
               [exec] -1 overall.  Here are the results of testing the latest attachment 
               [exec]   http://issues.apache.org/jira/secure/attachment/12441789/hdfs-1031_aoriani_3.patch
               [exec]   against trunk revision 934196.
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     +1 core tests.  The patch passed core unit tests.
               [exec] 
               [exec]     -1 contrib tests.  The patch failed contrib unit tests.
               [exec] 
               [exec] Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/314/testReport/
               [exec] Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/314/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
               [exec] Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/314/artifact/trunk/build/test/checkstyle-errors.html
               [exec] Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/314/console
               [exec] 
               [exec] This message is automatically generated.
          
          Show
          André Oriani added a comment - Adding hudson results, including the log for failed test. Same result from previous build. Hdfsproxy test still failing but it is not related to this code [exec] compile-test: [exec] [echo] contrib: hdfsproxy [exec] [javac] Compiling 9 source files to /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build/contrib/hdfsproxy/test [exec] [exec] test-junit: [exec] [junit] WARNING: multiple versions of ant detected in path for junit [exec] [junit] jar:file:/homes/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class [exec] [junit] and jar:file:/homes/hudson/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [exec] [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy [exec] [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 3.17 sec [exec] [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED [exec] [exec] BUILD FAILED [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:582: The following error occurred while executing this line: [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/build.xml:48: The following error occurred while executing this line: [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/hdfsproxy/build.xml:260: Tests failed! [exec] [exec] Total time: 12 seconds [exec] [exec] [exec] [exec] [exec] -1 overall. Here are the results of testing the latest attachment [exec] http: //issues.apache.org/jira/secure/attachment/12441789/hdfs-1031_aoriani_3.patch [exec] against trunk revision 934196. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 2 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] -1 contrib tests. The patch failed contrib unit tests. [exec] [exec] Test results: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/314/testReport/ [exec] Findbugs warnings: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/314/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Checkstyle results: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/314/artifact/trunk/build/test/checkstyle-errors.html [exec] Console output: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/314/console [exec] [exec] This message is automatically generated.
          Hide
          André Oriani added a comment -

          Upmerged to trunk
          Applied suggestions from comment
          Test was modified to use JUnit4 styte use the @Test annotation and static import of assertion methods. I did not write any setup or tearDown method because there is only method, the subtests are not independent. and I don't foresee the reutilization of setup routine.

          Hudson is currently not publishing the build results. Once the build ends I will publish the results here.

          Show
          André Oriani added a comment - Upmerged to trunk Applied suggestions from comment Test was modified to use JUnit4 styte use the @Test annotation and static import of assertion methods. I did not write any setup or tearDown method because there is only method, the subtests are not independent. and I don't foresee the reutilization of setup routine. Hudson is currently not publishing the build results. Once the build ends I will publish the results here.
          Hide
          Eli Collins added a comment -

          Thanks for contributing! Minor comments follow:

          • corruptFileCount == 0 could indicate there are no corrupt files, ie not just that it wasn't unable to locate any. Would change the message to something like "There are no corrupt files found. You can run fsck to verify that filesystem is healthy."
          • Please convert the uinit test to junit 4 style (eg don't extend TestCase, add setup, teardown, use @Test etc. see FileContextSymlinkBaseTest.java for an example).
          Show
          Eli Collins added a comment - Thanks for contributing! Minor comments follow: corruptFileCount == 0 could indicate there are no corrupt files, ie not just that it wasn't unable to locate any. Would change the message to something like "There are no corrupt files found. You can run fsck to verify that filesystem is healthy." Please convert the uinit test to junit 4 style (eg don't extend TestCase, add setup, teardown, use @Test etc. see FileContextSymlinkBaseTest.java for an example).
          Hide
          André Oriani added a comment -

          Oops! Log for failed test here :

               [exec] compile:
               [exec]      [echo] contrib: hdfsproxy
               [exec]
               [exec] compile-examples:
               [exec]
               [exec] compile-test:
               [exec]      [echo] contrib: hdfsproxy
               [exec]     [javac] Compiling 9 source files to /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build/contrib/hdfsproxy/test
               [exec]
               [exec] test-junit:
               [exec]     [junit] WARNING: multiple versions of ant detected in path for junit
               [exec]     [junit]          jar:file:/homes/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class
               [exec]     [junit]      and jar:file:/homes/hudson/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
               [exec]     [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy
               [exec]     [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 32.246 sec
               [exec]     [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED
               [exec]
               [exec] BUILD FAILED
               [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:582: The following error occurred while executing this line:
               [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/build.xml:48: The following error occurred while executing this line:
               [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/hdfsproxy/build.xml:260: Tests failed!
               [exec]
               [exec] Total time: 47 seconds
          
          Show
          André Oriani added a comment - Oops! Log for failed test here : [exec] compile: [exec] [echo] contrib: hdfsproxy [exec] [exec] compile-examples: [exec] [exec] compile-test: [exec] [echo] contrib: hdfsproxy [exec] [javac] Compiling 9 source files to /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build/contrib/hdfsproxy/test [exec] [exec] test-junit: [exec] [junit] WARNING: multiple versions of ant detected in path for junit [exec] [junit] jar:file:/homes/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class [exec] [junit] and jar:file:/homes/hudson/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [exec] [junit] Running org.apache.hadoop.hdfsproxy.TestHdfsProxy [exec] [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 32.246 sec [exec] [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED [exec] [exec] BUILD FAILED [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:582: The following error occurred while executing this line: [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/build.xml:48: The following error occurred while executing this line: [exec] /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/src/contrib/hdfsproxy/build.xml:260: Tests failed! [exec] [exec] Total time: 47 seconds
          Hide
          André Oriani added a comment -

          Hudson still cannot add comments to Jira. So I am manually add them

          http://hudson.zones.apache.org/hudson/view/Hdfs/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312

          [exec] -1 overall.  Here are the results of testing the latest attachment 
               [exec]   http://issues.apache.org/jira/secure/attachment/12441369/hdfs-1031_aoriani_2.patch
               [exec]   against trunk revision 933428.
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     +1 core tests.  The patch passed core unit tests.
               [exec] 
               [exec]     -1 contrib tests.  The patch failed contrib unit tests.
               [exec] 
               [exec] Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/testReport/
               [exec] Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
               [exec] Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/artifact/trunk/build/test/checkstyle-errors.html
               [exec] Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/console
          
          

          The only failed test is a hdfsproxy test, not related t this bug

          [exec] -1 overall.  Here are the results of testing the latest attachment 
               [exec]   http://issues.apache.org/jira/secure/attachment/12441369/hdfs-1031_aoriani_2.patch
               [exec]   against trunk revision 933428.
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     +1 core tests.  The patch passed core unit tests.
               [exec] 
               [exec]     -1 contrib tests.  The patch failed contrib unit tests.
               [exec] 
               [exec] Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/testReport/
               [exec] Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
               [exec] Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/artifact/trunk/build/test/checkstyle-errors.html
               [exec] Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/console
          
          Show
          André Oriani added a comment - Hudson still cannot add comments to Jira. So I am manually add them http://hudson.zones.apache.org/hudson/view/Hdfs/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312 [exec] -1 overall. Here are the results of testing the latest attachment [exec] http: //issues.apache.org/jira/secure/attachment/12441369/hdfs-1031_aoriani_2.patch [exec] against trunk revision 933428. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 2 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] -1 contrib tests. The patch failed contrib unit tests. [exec] [exec] Test results: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/testReport/ [exec] Findbugs warnings: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Checkstyle results: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/artifact/trunk/build/test/checkstyle-errors.html [exec] Console output: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/console The only failed test is a hdfsproxy test, not related t this bug [exec] -1 overall. Here are the results of testing the latest attachment [exec] http: //issues.apache.org/jira/secure/attachment/12441369/hdfs-1031_aoriani_2.patch [exec] against trunk revision 933428. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 2 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] -1 contrib tests. The patch failed contrib unit tests. [exec] [exec] Test results: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/testReport/ [exec] Findbugs warnings: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Checkstyle results: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/artifact/trunk/build/test/checkstyle-errors.html [exec] Console output: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/312/console
          Hide
          André Oriani added a comment -

          The test report (http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/testReport/) shows only one test failed, not related to code introduced by my patch

          Regression

          org.apache.hadoop.hdfs.TestFiHFlush.hFlushFi01_a (from TestFiHFlush)

          Stacktrace

          junit.framework.AssertionFailedError:
          at org.apache.hadoop.hdfs.TestFiHFlush.runDiskErrorTest(TestFiHFlush.java:56)
          at org.apache.hadoop.hdfs.TestFiHFlush.hFlushFi01_a(TestFiHFlush.java:72)

          Show
          André Oriani added a comment - The test report ( http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/testReport/ ) shows only one test failed, not related to code introduced by my patch Regression org.apache.hadoop.hdfs.TestFiHFlush.hFlushFi01_a (from TestFiHFlush) Stacktrace junit.framework.AssertionFailedError: at org.apache.hadoop.hdfs.TestFiHFlush.runDiskErrorTest(TestFiHFlush.java:56) at org.apache.hadoop.hdfs.TestFiHFlush.hFlushFi01_a(TestFiHFlush.java:72)
          Hide
          André Oriani added a comment -

          Hudson failed to add comment to JIRA , so I am reproducing the output of build http://hudson.zones.apache.org/hudson/view/Hdfs/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/ here:

           [exec] -1 overall.  Here are the results of testing the latest attachment 
               [exec]   http://issues.apache.org/jira/secure/attachment/12441369/hdfs-1031_aoriani_2.patch
               [exec]   against trunk revision 932036.
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] 
               [exec]     -1 core tests.  The patch failed core unit tests.
               [exec] 
               [exec]     -1 contrib tests.  The patch failed contrib unit tests.
               [exec] 
               [exec] Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/testReport/
               [exec] Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
               [exec] Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/artifact/trunk/build/test/checkstyle-errors.html
               [exec] Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/console
               [exec] 
               [exec] This message is automatically generated.
               [exec] 
               [exec] 
               [exec] ======================================================================
               [exec] ======================================================================
               [exec]     Adding comment to Jira.
               [exec] ======================================================================
               [exec] ======================================================================
               [exec] 
               [exec] 
               [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
               [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
               [exec] Failed to connect to: http://issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl
               [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
               [exec]                                  Dload  Upload   Total   Spent    Left  Speed
               [exec] 
               [exec]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
               [exec] 
               [exec] 
               [exec] ======================================================================
               [exec] ======================================================================
               [exec]     Finished build.
               [exec] ======================================================================
               [exec] ======================================================================
               [exec] 
               [exec] 
          
          BUILD FAILED
          /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:1198: exec returned: 2
          
          Total time: 210 minutes 25 seconds
          Archiving artifacts
          Recording test results
          Description set: HDFS-1031
          Error updating JIRA issues. Saving issues for next build.
          javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: subject/issuer name chaining check failed
          Sending e-mails to: gkesavan@apache.org
          Finished: FAILURE
          
          
          Show
          André Oriani added a comment - Hudson failed to add comment to JIRA , so I am reproducing the output of build http://hudson.zones.apache.org/hudson/view/Hdfs/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/ here: [exec] -1 overall. Here are the results of testing the latest attachment [exec] http: //issues.apache.org/jira/secure/attachment/12441369/hdfs-1031_aoriani_2.patch [exec] against trunk revision 932036. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 2 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 core tests. The patch failed core unit tests. [exec] [exec] -1 contrib tests. The patch failed contrib unit tests. [exec] [exec] Test results: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/testReport/ [exec] Findbugs warnings: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html [exec] Checkstyle results: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/artifact/trunk/build/test/checkstyle-errors.html [exec] Console output: http: //hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/308/console [exec] [exec] This message is automatically generated. [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Adding comment to Jira. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec] [exec] Failed to connect to: http: //issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl [exec] Failed to connect to: http: //issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl [exec] Failed to connect to: http: //issues.apache.org/jira/rpc/soap/jirasoapservice-v2?wsdl [exec] % Total % Received % Xferd Average Speed Time Time Time Current [exec] Dload Upload Total Spent Left Speed [exec] [exec] 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec] BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hdfs-Patch-h5.grid.sp2.yahoo.net/trunk/build.xml:1198: exec returned: 2 Total time: 210 minutes 25 seconds Archiving artifacts Recording test results Description set: HDFS-1031 Error updating JIRA issues. Saving issues for next build. javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: subject/issuer name chaining check failed Sending e-mails to: gkesavan@apache.org Finished: FAILURE
          Hide
          André Oriani added a comment -

          Upmerged to trunk
          Rewrite based on HDFS-1032 resolution
          Tests added
          Asking for review

          Show
          André Oriani added a comment - Upmerged to trunk Rewrite based on HDFS-1032 resolution Tests added Asking for review
          Hide
          André Oriani added a comment -

          That's the prototype of my solution to this Jira in order to gather requirements and feedback for a final patch.

          Show
          André Oriani added a comment - That's the prototype of my solution to this Jira in order to gather requirements and feedback for a final patch.
          Hide
          Todd Lipcon added a comment -

          +1 on this idea, also agree with Allen that it should be a linked page.

          Show
          Todd Lipcon added a comment - +1 on this idea, also agree with Allen that it should be a linked page.
          Hide
          Allen Wittenauer added a comment -

          I worry about cluttering up the front page with this info, especially if there are a lot of files that are broken. I think it would be good to put this on a secondary page. [Make it clickable from the error message?]

          Show
          Allen Wittenauer added a comment - I worry about cluttering up the front page with this info, especially if there are a lot of files that are broken. I think it would be good to put this on a secondary page. [Make it clickable from the error message?]

            People

            • Assignee:
              André Oriani
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development