Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1298

Add support in HDFS to update statistics that tracks number of file system operations in FileSystem

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      See HADOOP-6859 for the new statistics.

      1. HDFS-1298.patch
        15 kB
        Suresh Srinivas
      2. HDFS-1298.y20.patch
        12 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -
          1. Following operations increment read operations count:
            • getFileStatus(), getFileBlockLocations(), open(), getContentSummary(), listStatus(), checkStatistics(), getFileChecksum()
          2. Following operations increment write operations count:
            • mkdirs(), create(), setReplication(), rename(), setPermission(), setTimes(), setOwner(), delete()
          3. Large read operation count is incremented when performing listStatus() on a large directory, for every iterative step.
          Show
          Suresh Srinivas added a comment - Following operations increment read operations count: getFileStatus(), getFileBlockLocations(), open(), getContentSummary(), listStatus(), checkStatistics(), getFileChecksum() Following operations increment write operations count: mkdirs(), create(), setReplication(), rename(), setPermission(), setTimes(), setOwner(), delete() Large read operation count is incremented when performing listStatus() on a large directory, for every iterative step.
          Hide
          Suresh Srinivas added a comment -

          Attached patch makes the following changes:

          1. Change to update statistics and tests for the new functionality
          2. Converted TestDistributedFileSystem to junit4 test
          3. Removed unused methods in TestFileCreation
          Show
          Suresh Srinivas added a comment - Attached patch makes the following changes: Change to update statistics and tests for the new functionality Converted TestDistributedFileSystem to junit4 test Removed unused methods in TestFileCreation
          Hide
          Konstantin Shvachko added a comment -

          +1 Both patches look good to me.

          Show
          Konstantin Shvachko added a comment - +1 Both patches look good to me.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12449392/HDFS-1298.patch
          against trunk revision 964266.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/434/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/434/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/434/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/434/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449392/HDFS-1298.patch against trunk revision 964266. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/434/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/434/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/434/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/434/console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12449392/HDFS-1298.patch
          against trunk revision 964266.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/435/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/435/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/435/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/435/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449392/HDFS-1298.patch against trunk revision 964266. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/435/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/435/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/435/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/435/console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Test failures are unrelated to this patch. Following jiras track the failure:
          TestJspHelper tracked in HDFS-1285
          TestBlockToken tracked in HDFS-1284
          TestFileAppend4 tracked in HDFS-1306

          Show
          Suresh Srinivas added a comment - Test failures are unrelated to this patch. Following jiras track the failure: TestJspHelper tracked in HDFS-1285 TestBlockToken tracked in HDFS-1284 TestFileAppend4 tracked in HDFS-1306
          Hide
          Suresh Srinivas added a comment -

          y20 version of the patch

          Show
          Suresh Srinivas added a comment - y20 version of the patch
          Hide
          Suresh Srinivas added a comment -

          I committed the patch.

          Show
          Suresh Srinivas added a comment - I committed the patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #344 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/344/)
          HDFS-1298 - Add support in HDFS for new statistics added in FileSystem to track the file system operations. Contributed by Suresh Srinivas.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #344 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/344/ ) HDFS-1298 - Add support in HDFS for new statistics added in FileSystem to track the file system operations. Contributed by Suresh Srinivas.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          How about the following failed tests?

          • org.apache.hadoop.hdfs.server.datanode.TestBlockReport.blockReport_09
          • org.apache.hadoop.hdfs.server.datanode.TestDatanodeRestart.testFinalizedReplicas
          • org.apache.hadoop.hdfs.server.datanode.TestDatanodeRestart.testRecoverReplicas
          Show
          Tsz Wo Nicholas Sze added a comment - How about the following failed tests? org.apache.hadoop.hdfs.server.datanode.TestBlockReport.blockReport_09 org.apache.hadoop.hdfs.server.datanode.TestDatanodeRestart.testFinalizedReplicas org.apache.hadoop.hdfs.server.datanode.TestDatanodeRestart.testRecoverReplicas
          Hide
          Suresh Srinivas added a comment -

          Nicholas, between the two hudson tests, only one has these tests failing. Also I could not duplicate this on my local machine.

          Show
          Suresh Srinivas added a comment - Nicholas, between the two hudson tests, only one has these tests failing. Also I could not duplicate this on my local machine.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          They are probably not related to your patch but I think you might have overlooked them. Just want to make sure.

          Show
          Tsz Wo Nicholas Sze added a comment - They are probably not related to your patch but I think you might have overlooked them. Just want to make sure.

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development