Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1861

Rename dfs.datanode.max.xcievers and bump its default value

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Reasonably sized jobs and HBase easily exhaust the current default for dfs.datanode.max.xcievers. 4096 works better in practice.

      Let's also deprecate it in favor of a more intuitive name, eg dfs.datanode.max.receiver.threads.

      1. hdfs-1861-1.patch
        4 kB
        Eli Collins
      2. hdfs-1861-2.patch
        4 kB
        Eli Collins

        Activity

        Hide
        Eli Collins added a comment -

        Patch attached.

        Show
        Eli Collins added a comment - Patch attached.
        Hide
        dhruba borthakur added a comment -

        maybe do it in a backward-compatible way for a release or two?

        Show
        dhruba borthakur added a comment - maybe do it in a backward-compatible way for a release or two?
        Hide
        Eli Collins added a comment -

        Isn't the attached patch backwards-compatible? It adds dfs.datanode.max.xcievers as a deprecated key, does not remove it.

        Show
        Eli Collins added a comment - Isn't the attached patch backwards-compatible? It adds dfs.datanode.max.xcievers as a deprecated key, does not remove it.
        Hide
        dhruba borthakur added a comment -

        apologies, patch looks good. +1

        Show
        dhruba borthakur added a comment - apologies, patch looks good. +1
        Hide
        Todd Lipcon added a comment -

        I don't think "receiver.threads" is correct - these threads are used for both writing and reading blocks. max.transceiver.threads is probably a better name.

        Show
        Todd Lipcon added a comment - I don't think "receiver.threads" is correct - these threads are used for both writing and reading blocks. max.transceiver.threads is probably a better name.
        Hide
        Eli Collins added a comment -

        How about "dfs.datanode.max.transfer.threads"? The words receiver and transceiver refer to the implementation (pipe-lined repl) and therefore probably make less sense to users.

        Show
        Eli Collins added a comment - How about "dfs.datanode.max.transfer.threads"? The words receiver and transceiver refer to the implementation (pipe-lined repl) and therefore probably make less sense to users.
        Hide
        Todd Lipcon added a comment -

        how about "dfs.datanode.max.xfer.threds?"

        Just kidding. Yours is good.

        Show
        Todd Lipcon added a comment - how about "dfs.datanode.max.xfer.threds?" Just kidding. Yours is good.
        Hide
        Eli Collins added a comment -

        Patch attached. Same as the last one just s/"receiver"/"transfer" in DFSConfigKeys.

        Show
        Eli Collins added a comment - Patch attached. Same as the last one just s/"receiver"/"transfer" in DFSConfigKeys.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12477337/hdfs-1861-2.patch
        against trunk revision 1096010.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.hdfs.TestFileAppend4
        org.apache.hadoop.hdfs.TestLargeBlock
        org.apache.hadoop.hdfs.TestWriteConfigurationToDFS

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/415//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/415//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/415//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477337/hdfs-1861-2.patch against trunk revision 1096010. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.TestWriteConfigurationToDFS +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/415//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/415//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/415//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        Test failures are unrelated.

        Show
        Eli Collins added a comment - Test failures are unrelated.
        Hide
        Eli Collins added a comment -

        I've committed this to branch 22 and trunk. Thanks Dhruba and Todd!

        Show
        Eli Collins added a comment - I've committed this to branch 22 and trunk. Thanks Dhruba and Todd!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #604 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/604/)
        HDFS-1861. Rename dfs.datanode.max.xcievers and bump its default value. Contributed by Eli Collins

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #604 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/604/ ) HDFS-1861 . Rename dfs.datanode.max.xcievers and bump its default value. Contributed by Eli Collins
        Hide
        Aaron T. Myers added a comment -

        Apologies for showing up to this JIRA late, but this JIRA seems like a good opportunity to have added this config to hdfs-default.xml, with a helpful description. Changing the name of it is good, but I believe there's still no documentation about it.

        Show
        Aaron T. Myers added a comment - Apologies for showing up to this JIRA late, but this JIRA seems like a good opportunity to have added this config to hdfs-default.xml , with a helpful description. Changing the name of it is good, but I believe there's still no documentation about it.
        Hide
        Eli Collins added a comment -

        Filed HDFS-1866.

        Show
        Eli Collins added a comment - Filed HDFS-1866 .
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #650 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/650/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #650 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/650/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-22-branch #41 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/41/)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-22-branch #41 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/41/ )

          People

          • Assignee:
            Eli Collins
            Reporter:
            Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development