Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1307

Add start time, end time and total time taken for FSCK to FSCK report

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode, tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      FSCK is a long running operation and makes namenode very busy when it runs. Adding information such as start time, end time and time taken helps in determining when the FSCK was run and the impact of that on Namenode.

      1. HDFS-1307.patch
        1 kB
        Suresh Srinivas
      2. HDFS-1307.1.patch
        1 kB
        Suresh Srinivas
      3. HDFS-1307.2.patch
        2 kB
        Suresh Srinivas
      4. HDFS-1307.3.patch
        2 kB
        Suresh Srinivas
      5. HDFS-1307.3.y20.patch
        2 kB
        Suresh Srinivas

        Activity

        Hide
        Suresh Srinivas added a comment -

        Attached patch prints additional information to FSCK report.

        Show
        Suresh Srinivas added a comment - Attached patch prints additional information to FSCK report.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Do you want to do println at the end?

        Show
        Tsz Wo Nicholas Sze added a comment - Do you want to do println at the end?
        Hide
        Suresh Srinivas added a comment -

        Yes

        New patch with println at the end.

        Show
        Suresh Srinivas added a comment - Yes New patch with println at the end.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        Suresh Srinivas added a comment -

        y20 version of the patch.

        Show
        Suresh Srinivas added a comment - y20 version of the patch.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 for the y20 patch.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 for the y20 patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449730/HDFS-1307.y20.patch
        against trunk revision 964947.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/217/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449730/HDFS-1307.y20.patch against trunk revision 964947. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/217/console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        Attaching the trunk patch again for hudson to pickup for testing intead of y20 version.

        Show
        Suresh Srinivas added a comment - Attaching the trunk patch again for hudson to pickup for testing intead of y20 version.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449761/HDFS-1307.1.patch
        against trunk revision 964947.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449761/HDFS-1307.1.patch against trunk revision 964947. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/438/console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        Looks like the tests depend on the last line of the output printing DFS status. TestFsck failed because the patch prints Fsck end time and total time as the last line. New patch restores the last line as expected.

        Show
        Suresh Srinivas added a comment - Looks like the tests depend on the last line of the output printing DFS status. TestFsck failed because the patch prints Fsck end time and total time as the last line. New patch restores the last line as expected.
        Hide
        Suresh Srinivas added a comment -

        New patch with minor modification...

        Show
        Suresh Srinivas added a comment - New patch with minor modification...
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 the new patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 the new patch looks good.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449874/HDFS-1307.3.patch
        against trunk revision 965621.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449874/HDFS-1307.3.patch against trunk revision 965621. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/218/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449874/HDFS-1307.3.patch
        against trunk revision 965697.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449874/HDFS-1307.3.patch against trunk revision 965697. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/441/console This message is automatically generated.
        Hide
        Suresh Srinivas added a comment -

        The new test results link, I am not able to open. For the previous test results, the following tests failed, unrelated to this patch:
        TestJspHelper tracked in HDFS-1285
        TestBlockToken tracked in HDFS-1284
        TestFileAppend4 tracked in HDFS-1306
        TestBlockReport tracked in HDFS-733
        TestFiHFlush tracked in HDFS-1206
        TestFileConcurrentReader tracked in HDFS-1310

        Show
        Suresh Srinivas added a comment - The new test results link, I am not able to open. For the previous test results, the following tests failed, unrelated to this patch: TestJspHelper tracked in HDFS-1285 TestBlockToken tracked in HDFS-1284 TestFileAppend4 tracked in HDFS-1306 TestBlockReport tracked in HDFS-733 TestFiHFlush tracked in HDFS-1206 TestFileConcurrentReader tracked in HDFS-1310
        Hide
        Suresh Srinivas added a comment -

        I committed the patch.

        Show
        Suresh Srinivas added a comment - I committed the patch.
        Hide
        Suresh Srinivas added a comment -

        y20 version of the new patch.

        Show
        Suresh Srinivas added a comment - y20 version of the new patch.
        Hide
        Suresh Srinivas added a comment -

        New y20 version of the patch.

        Show
        Suresh Srinivas added a comment - New y20 version of the patch.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 the y20 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 the y20 patch looks good.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #348 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/348/)
        HDFS-1307 Add start time, end time and total time taken for FSCK to FSCK report. Contributed by Suresh Srinvias.

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #348 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/348/ ) HDFS-1307 Add start time, end time and total time taken for FSCK to FSCK report. Contributed by Suresh Srinvias.

          People

          • Assignee:
            Suresh Srinivas
            Reporter:
            Suresh Srinivas
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development