Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-204

Revive number of files listed metrics

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When namenode becomes unresponsive by HADOOP-4693 (large filelist calls), metrics has been helpful in finding out the cause.
      When gc time hikes, "FileListed" metrics also hiked.

      In 0.18, after we fixed "FileListed" metrics so that it shows number of operations instead of number of files listed (HADOOP-3683), I stopped seeing this relationship graph.

      Can we bring back "NumbverOfFilesListed" metrics?

      1. getFileNum-yahoo20.patch
        5 kB
        Hairong Kuang
      2. HDFS-204-2.patch
        5 kB
        Suresh Srinivas
      3. HDFS-204-2.patch
        5 kB
        Suresh Srinivas
      4. HDFS-204.patch
        5 kB
        Jitendra Nath Pandey
      5. HDFS-204.patch
        4 kB
        Jitendra Nath Pandey

        Activity

        Koji Noguchi created issue -
        Owen O'Malley made changes -
        Field Original Value New Value
        Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
        Key HADOOP-4694 HDFS-204
        Affects Version/s 0.18.0 [ 12312972 ]
        Component/s dfs [ 12310710 ]
        Component/s metrics [ 12310971 ]
        Hide
        Jitendra Nath Pandey added a comment -

        Files Changed
        1. hdfs/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        2. hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java

        New file added for unit test
        1. test/hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/TestNNMetricFilesInGetListingOps.java

        Show
        Jitendra Nath Pandey added a comment - Files Changed 1. hdfs/org/apache/hadoop/hdfs/server/namenode/NameNode.java 2. hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java New file added for unit test 1. test/hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/TestNNMetricFilesInGetListingOps.java
        Jitendra Nath Pandey made changes -
        Attachment HDFS-204.patch [ 12411574 ]
        Hide
        Suresh Srinivas added a comment -
        1. Patch might not be created right. It does not seem to have the full kkkpath for the file added/modified.
        2. Nit - Create files with 1 replica. With that starting just 1 datanode should be sufficient in MiniDFSCluster
        3. No need to explicitly call teardown(). It is automatically called by junit framework
        Show
        Suresh Srinivas added a comment - Patch might not be created right. It does not seem to have the full kkkpath for the file added/modified. Nit - Create files with 1 replica. With that starting just 1 datanode should be sufficient in MiniDFSCluster No need to explicitly call teardown() . It is automatically called by junit framework
        Hide
        Jitendra Nath Pandey added a comment -

        1. This patch shows full paths.
        2. Removed explicit tear down in unit test.
        3. Starting only one datanode in the unit test.

        Show
        Jitendra Nath Pandey added a comment - 1. This patch shows full paths. 2. Removed explicit tear down in unit test. 3. Starting only one datanode in the unit test.
        Jitendra Nath Pandey made changes -
        Attachment HDFS-204.patch [ 12411675 ]
        Hide
        Suresh Srinivas added a comment -

        +1. Made a minor change in indentation (from tab to space) in TestNNMetricFilesInGetListingOps.setup().

        Show
        Suresh Srinivas added a comment - +1. Made a minor change in indentation (from tab to space) in TestNNMetricFilesInGetListingOps.setup().
        Suresh Srinivas made changes -
        Attachment HDFS-204-2.patch [ 12411678 ]
        Hide
        Suresh Srinivas added a comment -

        Uploading the patch for the new HDFS directory structure after the split

        Show
        Suresh Srinivas added a comment - Uploading the patch for the new HDFS directory structure after the split
        Suresh Srinivas made changes -
        Attachment HDFS-204-2.patch [ 12411679 ]
        Jitendra Nath Pandey made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.21.0 [ 12314046 ]
        Jitendra Nath Pandey made changes -
        Assignee Jitendra Nath Pandey [ jnp ]
        Hide
        Jitendra Nath Pandey added a comment -

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        Jitendra Nath Pandey added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Jitendra Nath Pandey added a comment -

        I ran ant test after applying this patch, it completed without any failures.

        Show
        Jitendra Nath Pandey added a comment - I ran ant test after applying this patch, it completed without any failures.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Jitendra!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Jitendra!
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.21.0 [ 12314046 ]
        Resolution Fixed [ 1 ]
        Tsz Wo Nicholas Sze made changes -
        Component/s name-node [ 12312926 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #11 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/11/)
        . Add a new metrics FilesInGetListingOps to the Namenode. Contributed by Jitendra Nath Pandey

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #11 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/11/ ) . Add a new metrics FilesInGetListingOps to the Namenode. Contributed by Jitendra Nath Pandey
        Hide
        Hairong Kuang added a comment -

        This patch ported the feature to Yahoo 20 branch. In addition, it fixed the bug that NPE will be thrown when getListing on a non-existent path and I also added two more test cases, one is listing a non-existent path and one is listing path represented a file.

        Show
        Hairong Kuang added a comment - This patch ported the feature to Yahoo 20 branch. In addition, it fixed the bug that NPE will be thrown when getListing on a non-existent path and I also added two more test cases, one is listing a non-existent path and one is listing path represented a file.
        Hairong Kuang made changes -
        Attachment getFileNum-yahoo20.patch [ 12437576 ]
        Hide
        Jitendra Nath Pandey added a comment -

        +1. 20 patch looks good.

        Show
        Jitendra Nath Pandey added a comment - +1. 20 patch looks good.
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jitendra Nath Pandey
            Reporter:
            Koji Noguchi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development