Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1307

Add start time, end time and total time taken for FSCK to FSCK report

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode, tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      FSCK is a long running operation and makes namenode very busy when it runs. Adding information such as start time, end time and time taken helps in determining when the FSCK was run and the impact of that on Namenode.

      1. HDFS-1307.1.patch
        1 kB
        Suresh Srinivas
      2. HDFS-1307.2.patch
        2 kB
        Suresh Srinivas
      3. HDFS-1307.3.patch
        2 kB
        Suresh Srinivas
      4. HDFS-1307.3.y20.patch
        2 kB
        Suresh Srinivas
      5. HDFS-1307.patch
        1 kB
        Suresh Srinivas

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        3d 18h 11m 3 Suresh Srinivas 20/Jul/10 17:03
        Open Open Patch Available Patch Available
        1h 13m 4 Suresh Srinivas 20/Jul/10 17:03
        Patch Available Patch Available Resolved Resolved
        4h 53m 1 Suresh Srinivas 20/Jul/10 21:57
        Resolved Resolved Closed Closed
        509d 8h 22m 1 Konstantin Shvachko 12/Dec/11 06:19
        Konstantin Shvachko made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #348 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/348/)
        HDFS-1307 Add start time, end time and total time taken for FSCK to FSCK report. Contributed by Suresh Srinvias.

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #348 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/348/ ) HDFS-1307 Add start time, end time and total time taken for FSCK to FSCK report. Contributed by Suresh Srinvias.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 the y20 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 the y20 patch looks good.
        Suresh Srinivas made changes -
        Attachment HDFS-1307.3.y20.patch [ 12449992 ]
        Hide
        Suresh Srinivas added a comment -

        New y20 version of the patch.

        Show
        Suresh Srinivas added a comment - New y20 version of the patch.
        Suresh Srinivas made changes -
        Attachment HDFS-1307.y20.patch [ 12449983 ]
        Suresh Srinivas made changes -
        Attachment HDFS-1307.y20.patch [ 12449730 ]
        Suresh Srinivas made changes -
        Attachment HDFS-1307.y20.patch [ 12449983 ]
        Hide
        Suresh Srinivas added a comment -

        y20 version of the new patch.

        Show
        Suresh Srinivas added a comment - y20 version of the new patch.
        Suresh Srinivas made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Suresh Srinivas added a comment -

        I committed the patch.

        Show
        Suresh Srinivas added a comment - I committed the patch.
        Hide
        Suresh Srinivas added a comment -

        The new test results link, I am not able to open. For the previous test results, the following tests failed, unrelated to this patch:
        TestJspHelper tracked in HDFS-1285
        TestBlockToken tracked in HDFS-1284
        TestFileAppend4 tracked in HDFS-1306
        TestBlockReport tracked in HDFS-733
        TestFiHFlush tracked in HDFS-1206
        TestFileConcurrentReader tracked in HDFS-1310

        Show
        Suresh Srinivas added a comment - The new test results link, I am not able to open. For the previous test results, the following tests failed, unrelated to this patch: TestJspHelper tracked in HDFS-1285 TestBlockToken tracked in HDFS-1284 TestFileAppend4 tracked in HDFS-1306 TestBlockReport tracked in HDFS-733 TestFiHFlush tracked in HDFS-1206 TestFileConcurrentReader tracked in HDFS-1310
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449874/HDFS-1307.3.patch
        against trunk revision 965697.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449874/HDFS-1307.3.patch against trunk revision 965697. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/console This message is automatically generated.
        Suresh Srinivas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Suresh Srinivas made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449874/HDFS-1307.3.patch
        against trunk revision 965621.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449874/HDFS-1307.3.patch against trunk revision 965621. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/console This message is automatically generated.
        Suresh Srinivas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Suresh Srinivas made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 the new patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 the new patch looks good.
        Suresh Srinivas made changes -
        Attachment HDFS-1307.3.patch [ 12449874 ]
        Hide
        Suresh Srinivas added a comment -

        New patch with minor modification...

        Show
        Suresh Srinivas added a comment - New patch with minor modification...
        Suresh Srinivas made changes -
        Attachment HDFS-1307.2.patch [ 12449871 ]
        Hide
        Suresh Srinivas added a comment -

        Looks like the tests depend on the last line of the output printing DFS status. TestFsck failed because the patch prints Fsck end time and total time as the last line. New patch restores the last line as expected.

        Show
        Suresh Srinivas added a comment - Looks like the tests depend on the last line of the output printing DFS status. TestFsck failed because the patch prints Fsck end time and total time as the last line. New patch restores the last line as expected.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449761/HDFS-1307.1.patch
        against trunk revision 964947.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449761/HDFS-1307.1.patch against trunk revision 964947. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/console This message is automatically generated.
        Suresh Srinivas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Suresh Srinivas made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Suresh Srinivas made changes -
        Attachment HDFS-1307.1.patch [ 12449761 ]
        Hide
        Suresh Srinivas added a comment -

        Attaching the trunk patch again for hudson to pickup for testing intead of y20 version.

        Show
        Suresh Srinivas added a comment - Attaching the trunk patch again for hudson to pickup for testing intead of y20 version.
        Suresh Srinivas made changes -
        Attachment HDFS-1307.1.patch [ 12449726 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449730/HDFS-1307.y20.patch
        against trunk revision 964947.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/217/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449730/HDFS-1307.y20.patch against trunk revision 964947. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/217/console This message is automatically generated.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 for the y20 patch.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 for the y20 patch.
        Suresh Srinivas made changes -
        Attachment HDFS-1307.y20.patch [ 12449730 ]
        Hide
        Suresh Srinivas added a comment -

        y20 version of the patch.

        Show
        Suresh Srinivas added a comment - y20 version of the patch.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Suresh Srinivas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.22.0 [ 12314241 ]
        Suresh Srinivas made changes -
        Attachment HDFS-1307.1.patch [ 12449726 ]
        Hide
        Suresh Srinivas added a comment -

        Yes

        New patch with println at the end.

        Show
        Suresh Srinivas added a comment - Yes New patch with println at the end.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Do you want to do println at the end?

        Show
        Tsz Wo Nicholas Sze added a comment - Do you want to do println at the end?
        Suresh Srinivas made changes -
        Field Original Value New Value
        Attachment HDFS-1307.patch [ 12449715 ]
        Hide
        Suresh Srinivas added a comment -

        Attached patch prints additional information to FSCK report.

        Show
        Suresh Srinivas added a comment - Attached patch prints additional information to FSCK report.
        Suresh Srinivas created issue -

          People

          • Assignee:
            Suresh Srinivas
            Reporter:
            Suresh Srinivas
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development