Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3647

Backport HDFS-2868 (Add number of active transfer threads to the DataNode status) to branch-1

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 1.2.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Not sure if this is in a newer version of Hadoop, but in CDH3u3 it isn't there.

      There is a lot of mystery surrounding how large to set dfs.datanode.max.xcievers. Most people say to just up it to 4096, but given that exceeding this will cause an HBase RegionServer shutdown (see Lars' blog post here: http://www.larsgeorge.com/2012/03/hadoop-hbase-and-xceivers.html), it would be nice if we could expose the current count via the built-in metrics framework (most likely under dfs). In this way we could watch it to see if we have it set too high, too low, time to bump it up, etc.

      Thoughts?

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          I believe I've already done this for Hadoop 0.23+ (Now 2.x), via HDFS-2868. Perhaps we can backport that onto 1.x as well, for which we can re-purpose this JIRA.

          For CDH requests though, this is the wrong place and the right open channel to use is https://issues.cloudera.org/browse/DISTRO or its mailing lists.

          Show
          Harsh J added a comment - I believe I've already done this for Hadoop 0.23+ (Now 2.x), via HDFS-2868 . Perhaps we can backport that onto 1.x as well, for which we can re-purpose this JIRA. For CDH requests though, this is the wrong place and the right open channel to use is https://issues.cloudera.org/browse/DISTRO or its mailing lists.
          Hide
          Steve Hoffman added a comment -

          Yea, like I said, wasn't sure if this had already been implemented.

          I'll open a CDH DISTRO jira asking for the back port, but as you point out, could be useful in the general 1.X sense.

          Show
          Steve Hoffman added a comment - Yea, like I said, wasn't sure if this had already been implemented. I'll open a CDH DISTRO jira asking for the back port, but as you point out, could be useful in the general 1.X sense.
          Hide
          Steve Hoffman added a comment -

          https://issues.cloudera.org/browse/DISTRO-414 opened with cloudera. Thx.

          I'll leave it to you guys if you want to use this to track a 1.X apache back port.

          Show
          Steve Hoffman added a comment - https://issues.cloudera.org/browse/DISTRO-414 opened with cloudera. Thx. I'll leave it to you guys if you want to use this to track a 1.X apache back port.
          Hide
          Harsh J added a comment -

          Straight forward backport with fixed paths and conflicts. Test-patch results will arrive shortly.

          Show
          Harsh J added a comment - Straight forward backport with fixed paths and conflicts. Test-patch results will arrive shortly.
          Hide
          Harsh J added a comment -

          Screenshot of jconsole mxbean browser showing the metric, after application to branch-1 and some few fs -cat commands.

          Show
          Harsh J added a comment - Screenshot of jconsole mxbean browser showing the metric, after application to branch-1 and some few fs -cat commands.
          Hide
          Harsh J added a comment -

          test-patch results:

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec] 
               [exec]     -1 findbugs.  The patch appears to introduce 218 new Findbugs (version 2.0.1-rc3) warnings.
          

          The 218 is consistent with patch-less findbugs test, with the version 2.0.1-rc3 findbugs.

          Show
          Harsh J added a comment - test-patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 218 new Findbugs (version 2.0.1-rc3) warnings. The 218 is consistent with patch-less findbugs test, with the version 2.0.1-rc3 findbugs.
          Hide
          Todd Lipcon added a comment -

          +1 on the backport, thx Harsh.

          Show
          Todd Lipcon added a comment - +1 on the backport, thx Harsh.
          Hide
          Harsh J added a comment -

          Thanks Todd, I've committed this backport to branch-1.

          Show
          Harsh J added a comment - Thanks Todd, I've committed this backport to branch-1.
          Hide
          Suresh Srinivas added a comment -

          why does the component have performance?

          Show
          Suresh Srinivas added a comment - why does the component have performance?
          Hide
          Suresh Srinivas added a comment -

          Additional comment - Lets not create a new jira and attach the patch for backporting to the same jira. A separate jira should be created if the patch does not apply to older release and requires substantial work.

          Show
          Suresh Srinivas added a comment - Additional comment - Lets not create a new jira and attach the patch for backporting to the same jira. A separate jira should be created if the patch does not apply to older release and requires substantial work.
          Hide
          Suresh Srinivas added a comment -

          BTW removing performance from the component - add it if you disagree.

          Show
          Suresh Srinivas added a comment - BTW removing performance from the component - add it if you disagree.
          Hide
          Harsh J added a comment -

          Suresh - Sorry, added it cause it was a perf-tuning measure. Its OK to remove that.

          I didn't do it on the old JIRA cause it was marked Closed. If its acceptable to edit Closed issues, I'll do that in the future. Apologies for the inconvenience

          Show
          Harsh J added a comment - Suresh - Sorry, added it cause it was a perf-tuning measure. Its OK to remove that. I didn't do it on the old JIRA cause it was marked Closed. If its acceptable to edit Closed issues, I'll do that in the future. Apologies for the inconvenience
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.

            People

            • Assignee:
              Harsh J
              Reporter:
              Steve Hoffman
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development