Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When namenode becomes unresponsive by HADOOP-4693 (large filelist calls), metrics has been helpful in finding out the cause.
      When gc time hikes, "FileListed" metrics also hiked.

      In 0.18, after we fixed "FileListed" metrics so that it shows number of operations instead of number of files listed (HADOOP-3683), I stopped seeing this relationship graph.

      Can we bring back "NumbverOfFilesListed" metrics?

      1. HDFS-204-2.patch
        5 kB
        Suresh Srinivas
      2. HDFS-204-2.patch
        5 kB
        Suresh Srinivas
      3. HDFS-204.patch
        4 kB
        Jitendra Nath Pandey
      4. HDFS-204.patch
        5 kB
        Jitendra Nath Pandey
      5. getFileNum-yahoo20.patch
        5 kB
        Hairong Kuang

        Activity

        Hide
        jnp Jitendra Nath Pandey added a comment -

        +1. 20 patch looks good.

        Show
        jnp Jitendra Nath Pandey added a comment - +1. 20 patch looks good.
        Hide
        hairong Hairong Kuang added a comment -

        This patch ported the feature to Yahoo 20 branch. In addition, it fixed the bug that NPE will be thrown when getListing on a non-existent path and I also added two more test cases, one is listing a non-existent path and one is listing path represented a file.

        Show
        hairong Hairong Kuang added a comment - This patch ported the feature to Yahoo 20 branch. In addition, it fixed the bug that NPE will be thrown when getListing on a non-existent path and I also added two more test cases, one is listing a non-existent path and one is listing path represented a file.
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #11 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/11/)
        . Add a new metrics FilesInGetListingOps to the Namenode. Contributed by Jitendra Nath Pandey

        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #11 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/11/ ) . Add a new metrics FilesInGetListingOps to the Namenode. Contributed by Jitendra Nath Pandey
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Jitendra!

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Jitendra!
        Hide
        jnp Jitendra Nath Pandey added a comment -

        I ran ant test after applying this patch, it completed without any failures.

        Show
        jnp Jitendra Nath Pandey added a comment - I ran ant test after applying this patch, it completed without any failures.
        Hide
        jnp Jitendra Nath Pandey added a comment -

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        jnp Jitendra Nath Pandey added a comment - [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        sureshms Suresh Srinivas added a comment -

        Uploading the patch for the new HDFS directory structure after the split

        Show
        sureshms Suresh Srinivas added a comment - Uploading the patch for the new HDFS directory structure after the split
        Hide
        sureshms Suresh Srinivas added a comment -

        +1. Made a minor change in indentation (from tab to space) in TestNNMetricFilesInGetListingOps.setup().

        Show
        sureshms Suresh Srinivas added a comment - +1. Made a minor change in indentation (from tab to space) in TestNNMetricFilesInGetListingOps.setup().
        Hide
        jnp Jitendra Nath Pandey added a comment -

        1. This patch shows full paths.
        2. Removed explicit tear down in unit test.
        3. Starting only one datanode in the unit test.

        Show
        jnp Jitendra Nath Pandey added a comment - 1. This patch shows full paths. 2. Removed explicit tear down in unit test. 3. Starting only one datanode in the unit test.
        Hide
        sureshms Suresh Srinivas added a comment -
        1. Patch might not be created right. It does not seem to have the full kkkpath for the file added/modified.
        2. Nit - Create files with 1 replica. With that starting just 1 datanode should be sufficient in MiniDFSCluster
        3. No need to explicitly call teardown(). It is automatically called by junit framework
        Show
        sureshms Suresh Srinivas added a comment - Patch might not be created right. It does not seem to have the full kkkpath for the file added/modified. Nit - Create files with 1 replica. With that starting just 1 datanode should be sufficient in MiniDFSCluster No need to explicitly call teardown() . It is automatically called by junit framework
        Hide
        jnp Jitendra Nath Pandey added a comment -

        Files Changed
        1. hdfs/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        2. hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java

        New file added for unit test
        1. test/hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/TestNNMetricFilesInGetListingOps.java

        Show
        jnp Jitendra Nath Pandey added a comment - Files Changed 1. hdfs/org/apache/hadoop/hdfs/server/namenode/NameNode.java 2. hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java New file added for unit test 1. test/hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/TestNNMetricFilesInGetListingOps.java

          People

          • Assignee:
            jnp Jitendra Nath Pandey
            Reporter:
            knoguchi Koji Noguchi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development