Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3647

Backport HDFS-2868 (Add number of active transfer threads to the DataNode status) to branch-1

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 1.2.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Not sure if this is in a newer version of Hadoop, but in CDH3u3 it isn't there.

      There is a lot of mystery surrounding how large to set dfs.datanode.max.xcievers. Most people say to just up it to 4096, but given that exceeding this will cause an HBase RegionServer shutdown (see Lars' blog post here: http://www.larsgeorge.com/2012/03/hadoop-hbase-and-xceivers.html), it would be nice if we could expose the current count via the built-in metrics framework (most likely under dfs). In this way we could watch it to see if we have it set too high, too low, time to bump it up, etc.

      Thoughts?

        Issue Links

          Activity

          Steve Hoffman created issue -
          Steve Hoffman made changes -
          Field Original Value New Value
          Summary Expose dfs.datanode.max.xcievers as metric Expose current xcievers count as metric
          Steve Hoffman made changes -
          Description Not sure if this is in a newer version of Hadoop, but in CDH3u3 it isn't there.

          There is a lot of mystery surrounding how large to set dfs.datanode.max.xcievers. Most people say to just up it to 4096, but given that exceeding this will cause an HBase RegionServer shutdown (see Lars' blog post here (http://www.larsgeorge.com/2012/03/hadoop-hbase-and-xceivers.html), it would be nice if we could expose the current count via the built-in metrics framework (most likely under dfs). In this way we could watch it to see if we have it set too high, too low, time to bump it up, etc.

          Thoughts?
          Not sure if this is in a newer version of Hadoop, but in CDH3u3 it isn't there.

          There is a lot of mystery surrounding how large to set dfs.datanode.max.xcievers. Most people say to just up it to 4096, but given that exceeding this will cause an HBase RegionServer shutdown (see Lars' blog post here: http://www.larsgeorge.com/2012/03/hadoop-hbase-and-xceivers.html), it would be nice if we could expose the current count via the built-in metrics framework (most likely under dfs). In this way we could watch it to see if we have it set too high, too low, time to bump it up, etc.

          Thoughts?
          Eli Collins made changes -
          Summary Expose current xcievers count as metric Backport HDFS-2868 (Add number of active transfer threads to the DataNode status) to branch-1
          Target Version/s 1.2.0 [ 12321657 ]
          Harsh J made changes -
          Assignee Harsh J [ qwertymaniac ]
          Harsh J made changes -
          Link This issue is superceded by HDFS-2868 [ HDFS-2868 ]
          Harsh J made changes -
          Attachment HDFS-3647.patch [ 12536437 ]
          Harsh J made changes -
          Harsh J made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 1.2.0 [ 12321657 ]
          Fix Version/s 1.2.0 [ 12321657 ]
          Resolution Fixed [ 1 ]
          Suresh Srinivas made changes -
          Component/s performance [ 12316501 ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Harsh J
              Reporter:
              Steve Hoffman
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development