Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3990

NN's health report has severe performance problems

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: namenode
    • Labels:
      None

      Description

      The dfshealth page will place a read lock on the namespace while it does a dns lookup for every DN. On a multi-thousand node cluster, this often results in 10s+ load time for the health page. 10 concurrent requests were found to cause 7m+ load times during which time write operations blocked.

      1. HDFS-3990.branch-0.23.patch
        12 kB
        Daryn Sharp
      2. HDFS-3990.branch-0.23.patch
        13 kB
        Daryn Sharp
      3. HDFS-3990.patch
        9 kB
        Daryn Sharp
      4. HDFS-3990.patch
        9 kB
        Daryn Sharp
      5. HDFS-3990.patch
        10 kB
        Daryn Sharp
      6. HDFS-3990.patch
        12 kB
        Daryn Sharp
      7. HDFS-3990.patch
        11 kB
        Daryn Sharp
      8. HDFS-3990.patch
        11 kB
        Daryn Sharp
      9. HDFS-3990.patch
        13 kB
        Daryn Sharp
      10. HDFS-3990.patch
        10 kB
        Daryn Sharp
      11. hdfs-3990.txt
        13 kB
        Eli Collins
      12. hdfs-3990.txt
        4 kB
        Eli Collins

        Issue Links

          Activity

          Hide
          Daryn Sharp added a comment -

          Enabling a nscd host cache helped mitigate the issue by reducing load times to a few seconds. However the namespace read lock is highly undesirable, and the repeated dns lookups are questionable.

          Show
          Daryn Sharp added a comment - Enabling a nscd host cache helped mitigate the issue by reducing load times to a few seconds. However the namespace read lock is highly undesirable, and the repeated dns lookups are questionable.
          Hide
          Daryn Sharp added a comment -

          Arun, please update the target version if you want to defer the fix to a later 2.x release.

          Show
          Daryn Sharp added a comment - Arun, please update the target version if you want to defer the fix to a later 2.x release.
          Hide
          Daryn Sharp added a comment -

          This is an incremental improvement that caches resolved datanode addrs to prevent multiple unnecessary dns lookups. It actually affects more than just the dfshealth page so this should provide much improved performance.

          I will file another jira for investigating how to get the web page execution out of the namesystem lock.

          Show
          Daryn Sharp added a comment - This is an incremental improvement that caches resolved datanode addrs to prevent multiple unnecessary dns lookups. It actually affects more than just the dfshealth page so this should provide much improved performance. I will file another jira for investigating how to get the web page execution out of the namesystem lock.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12548945/HDFS-3990.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3323//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3323//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3323//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548945/HDFS-3990.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3323//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3323//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3323//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          I'm sorry I've been out of the loop, but why would caching be the solution?
          If we want to reassign the IP addresse to hostname for a single node, would it require a restart of the NN? Is there a timeout with the caching? Even with a timeout I would have my reservations.
          Do nodes have hadoop generated unique IDs that we can leverage and match with IP addresses that we have cached?

          Show
          Ravi Prakash added a comment - I'm sorry I've been out of the loop, but why would caching be the solution? If we want to reassign the IP addresse to hostname for a single node, would it require a restart of the NN? Is there a timeout with the caching? Even with a timeout I would have my reservations. Do nodes have hadoop generated unique IDs that we can leverage and match with IP addresses that we have cached?
          Hide
          Daryn Sharp added a comment -

          The caching is to prevent the unnecessary dns lookups that are a multiple of the number of datanodes - typically just to view a jsp or query json, or for other internal operations as well. Every time a node is checked against the include/exclude lists, it generates dns queries of 2X the datanodes. Counting the number of nodes causes a dns query for every datanode.

          Reassigning an ip should require no restart of the NN. The DN's are tracked by their ip and storage id. If a DN registers with a previously known ip or storage id, the existing node is updated with the fields in the new node id which contain a refreshed lookup.

          Show
          Daryn Sharp added a comment - The caching is to prevent the unnecessary dns lookups that are a multiple of the number of datanodes - typically just to view a jsp or query json, or for other internal operations as well. Every time a node is checked against the include/exclude lists, it generates dns queries of 2X the datanodes. Counting the number of nodes causes a dns query for every datanode. Reassigning an ip should require no restart of the NN. The DN's are tracked by their ip and storage id. If a DN registers with a previously known ip or storage id, the existing node is updated with the fields in the new node id which contain a refreshed lookup.
          Hide
          Daryn Sharp added a comment -

          I didn't think I needed sync since the setter is only called once on new objects, but added it since findbugs complained.

          The throughput benchmark failed because it was using dummy port numbers starting at 100,000 which are invalid. It did this so the xfer addrs would lexicographically sort. I changed the test to use valid dummy port numbers and changed the comparator so they still sort correctly. Also made a small tweak to hadoop.tmp.dir so the test runs in eclipse.

          Show
          Daryn Sharp added a comment - I didn't think I needed sync since the setter is only called once on new objects, but added it since findbugs complained. The throughput benchmark failed because it was using dummy port numbers starting at 100,000 which are invalid. It did this so the xfer addrs would lexicographically sort. I changed the test to use valid dummy port numbers and changed the comparator so they still sort correctly. Also made a small tweak to hadoop.tmp.dir so the test runs in eclipse.
          Hide
          Daryn Sharp added a comment -

          Pre-commit build is clean, but it failed to connect to jira:
          https://builds.apache.org/job/PreCommit-HDFS-Build/3331/consoleText

          Show
          Daryn Sharp added a comment - Pre-commit build is clean, but it failed to connect to jira: https://builds.apache.org/job/PreCommit-HDFS-Build/3331/consoleText
          Hide
          Eli Collins added a comment -

          Why not use the DatanodeID hostName field instead of calling and caching InetAddress#getByName in the NN? The DN has already done the lookup (modulo the tests which use dfs.datanode.hostname) and this way we don't have to worry about inconsistency between the nodeAddr and the ipAddr/hostName fields. For sanity the NN could do a lookup when the DN registers and compare it's value to the DN reported one.

          Comments on this patch:

          • In registerDatanode why is OK to no longer update the registration info with the reported IP?
          • The comments in DatanodeManager ("Mostly called inside an RPC.".. and "Update the IP to the address of the RPC request"..) are no longer accurate after your change.
          Show
          Eli Collins added a comment - Why not use the DatanodeID hostName field instead of calling and caching InetAddress#getByName in the NN? The DN has already done the lookup (modulo the tests which use dfs.datanode.hostname) and this way we don't have to worry about inconsistency between the nodeAddr and the ipAddr/hostName fields. For sanity the NN could do a lookup when the DN registers and compare it's value to the DN reported one. Comments on this patch: In registerDatanode why is OK to no longer update the registration info with the reported IP? The comments in DatanodeManager ("Mostly called inside an RPC.".. and "Update the IP to the address of the RPC request"..) are no longer accurate after your change.
          Hide
          Daryn Sharp added a comment -

          As best I can tell, the DatanodeID's hostname is what the DN claims to be in the registration. The existing include/exclude list checks use the DN's ip and "real" hostname, not the one the node claimed to be in the registration. I'm trying to preserve existing behavior by just caching the socket's peer name at registration, so that resolved socket addr can be reused when checking the include/exclude lists.

          In registerDatanode why is OK to no longer update the registration info with the reported IP?

          The ip actually is updated when setNodeAddr is called with the socket's peer.

          My bad on the comments. I'm not sure how I lost that change.

          I know the approach isn't perfect, and many of the fields could likely be folded together into the socket addr, but I'm trying to make the minimalist change to avoid a slew of dns queries that are having an adverse performance impact on multi-thousand node clusters.

          Show
          Daryn Sharp added a comment - As best I can tell, the DatanodeID 's hostname is what the DN claims to be in the registration. The existing include/exclude list checks use the DN's ip and "real" hostname, not the one the node claimed to be in the registration. I'm trying to preserve existing behavior by just caching the socket's peer name at registration, so that resolved socket addr can be reused when checking the include/exclude lists. In registerDatanode why is OK to no longer update the registration info with the reported IP? The ip actually is updated when setNodeAddr is called with the socket's peer. My bad on the comments. I'm not sure how I lost that change. I know the approach isn't perfect, and many of the fields could likely be folded together into the socket addr, but I'm trying to make the minimalist change to avoid a slew of dns queries that are having an adverse performance impact on multi-thousand node clusters.
          Hide
          Daryn Sharp added a comment -

          I'm also handling the case where a transient dns error may have occurred at the time a socket connected. The patch will attempt another lookup when the nodeAddr is requested.

          Show
          Daryn Sharp added a comment - I'm also handling the case where a transient dns error may have occurred at the time a socket connected. The patch will attempt another lookup when the nodeAddr is requested.
          Hide
          Ravi Prakash added a comment -

          Thanks for your explanations Daryn! The src/main code looks reasonable to me.

          Show
          Ravi Prakash added a comment - Thanks for your explanations Daryn! The src/main code looks reasonable to me.
          Hide
          Eli Collins added a comment -

          Maintaining both an ipAddr/hostName plus nodeAddr with the same information, which can become inconsistent is error prone. For example what do you do when the ipAddr and the nodeAddr disagree? The ipAddr field for a DataNode ID should never change because it (and the xferPort) are the unique key for a DataNode. We also now have to worry about the state where we're both resolved and unresolved. Given that the crux of the problem is that we want to cache the DNS lookup for the ipAddr of a DN, it seems simplest to just do that.

          What do you think of the attached patch? It sets the DatanodeID hostname field at registration time (like the IP addr) using the same lookup we do today and replaces the two problematic lookups with uses of this field.

          This breaks dfs.datanode.hostname but this config is only used by the tests and we can fix those up. I'm happy to do that in another rev of this patch if you like the approach. I think a better approach would be to just use the lookup on the DN side (ie have the NN use the DN reported value) but that's a more risky change.

          Show
          Eli Collins added a comment - Maintaining both an ipAddr/hostName plus nodeAddr with the same information, which can become inconsistent is error prone. For example what do you do when the ipAddr and the nodeAddr disagree? The ipAddr field for a DataNode ID should never change because it (and the xferPort) are the unique key for a DataNode. We also now have to worry about the state where we're both resolved and unresolved. Given that the crux of the problem is that we want to cache the DNS lookup for the ipAddr of a DN, it seems simplest to just do that. What do you think of the attached patch? It sets the DatanodeID hostname field at registration time (like the IP addr) using the same lookup we do today and replaces the two problematic lookups with uses of this field. This breaks dfs.datanode.hostname but this config is only used by the tests and we can fix those up. I'm happy to do that in another rev of this patch if you like the approach. I think a better approach would be to just use the lookup on the DN side (ie have the NN use the DN reported value) but that's a more risky change.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12549228/hdfs-3990.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.TestMiniDFSCluster
          org.apache.hadoop.hdfs.TestReplication
          org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3338//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3338//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549228/hdfs-3990.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.TestMiniDFSCluster org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3338//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3338//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Maintaining both an ipAddr/hostName plus nodeAddr with the same information, which can become inconsistent is error prone. For example what do you do when the ipAddr and the nodeAddr disagree?

          They should never disagree because the nodeAddr is based on the ipAddr, and when the nodeAddr is changed, so is the ipAddr.

          The ipAddr field for a DataNode ID should never change because it (and the xferPort) are the unique key for a DataNode.

          They will change when a pre-existing node, say one with the same storage id, is updated with the new info.

          We also now have to worry about the state where we're both resolved and unresolved.

          We need to worry about that case just like the code did before. Let's say the exclude list has hostnames. A node registration occurs but there's a dns hiccup so all we have is its ip. Your proposed patch may let the node in whereas the existing code (and my patch) will block the node.

          What do you think of the attached patch? It sets the DatanodeID hostname field at registration time (like the IP addr) ...

          The patch appears to change the way the include and exclude work by trusting who the datanode claims to be. What if a datanode "lies" about who it is? Or if a dns hiccup occurs when the datanode is going to register? It sends its name as an ip, but the exclude list only has hosts. There are a number of scenarios where a datanode could bypass the include/exclude list, which is why we should never trust the client.

          ... using the same lookup we do today and replaces the two problematic lookups with uses of this field.

          Unless I've overlooked something, there's only one lookup that occurs?

          I'll post a minor rev for consideration that should further ensure the fields never go out of sync.

          Show
          Daryn Sharp added a comment - Maintaining both an ipAddr/hostName plus nodeAddr with the same information, which can become inconsistent is error prone. For example what do you do when the ipAddr and the nodeAddr disagree? They should never disagree because the nodeAddr is based on the ipAddr, and when the nodeAddr is changed, so is the ipAddr. The ipAddr field for a DataNode ID should never change because it (and the xferPort) are the unique key for a DataNode. They will change when a pre-existing node, say one with the same storage id, is updated with the new info. We also now have to worry about the state where we're both resolved and unresolved. We need to worry about that case just like the code did before. Let's say the exclude list has hostnames. A node registration occurs but there's a dns hiccup so all we have is its ip. Your proposed patch may let the node in whereas the existing code (and my patch) will block the node. What do you think of the attached patch? It sets the DatanodeID hostname field at registration time (like the IP addr) ... The patch appears to change the way the include and exclude work by trusting who the datanode claims to be. What if a datanode "lies" about who it is? Or if a dns hiccup occurs when the datanode is going to register? It sends its name as an ip, but the exclude list only has hosts. There are a number of scenarios where a datanode could bypass the include/exclude list, which is why we should never trust the client. ... using the same lookup we do today and replaces the two problematic lookups with uses of this field. Unless I've overlooked something, there's only one lookup that occurs? I'll post a minor rev for consideration that should further ensure the fields never go out of sync.
          Hide
          Eli Collins added a comment -

          They will change when a pre-existing node, say one with the same storage id, is updated with the new info.

          I'm not sure re-registering with a new IP and the same storage ID actually works today.

          The patch appears to change the way the include and exclude work by trusting who the datanode claims to be. What if a datanode "lies" about who it is? Or if a dns hiccup occurs when the datanode is going to register? It sends its name as an ip, but the exclude list only has hosts. There are a number of scenarios where a datanode could bypass the include/exclude list, which is why we should never trust the client.

          Take another look at the patch, the NN is doing the lookup not the DN, just at registration time. How about we reject the DN registration in case of a DNS hiccup (rather than use the DN value which the patch currently does in this case)? The DN will retry until it succeeds. When working on HDFS-3171 I considered removing the ability for the DN to override the hostname, and have just one lookup per DN (ie currently both the NN and DN resolve the DN hostname). We could open a separate jira for that, might be easier to layer this one atop it.

          I'm against having DatanodeID fields that duplicates the other fields since I think we can solve the problem here and avoid doing so. My experience from HDFS-3144 indicates we will introduce bugs and it's hard to correctly untangle later.

          Show
          Eli Collins added a comment - They will change when a pre-existing node, say one with the same storage id, is updated with the new info. I'm not sure re-registering with a new IP and the same storage ID actually works today. The patch appears to change the way the include and exclude work by trusting who the datanode claims to be. What if a datanode "lies" about who it is? Or if a dns hiccup occurs when the datanode is going to register? It sends its name as an ip, but the exclude list only has hosts. There are a number of scenarios where a datanode could bypass the include/exclude list, which is why we should never trust the client. Take another look at the patch, the NN is doing the lookup not the DN, just at registration time. How about we reject the DN registration in case of a DNS hiccup (rather than use the DN value which the patch currently does in this case)? The DN will retry until it succeeds. When working on HDFS-3171 I considered removing the ability for the DN to override the hostname, and have just one lookup per DN (ie currently both the NN and DN resolve the DN hostname). We could open a separate jira for that, might be easier to layer this one atop it. I'm against having DatanodeID fields that duplicates the other fields since I think we can solve the problem here and avoid doing so. My experience from HDFS-3144 indicates we will introduce bugs and it's hard to correctly untangle later.
          Hide
          Daryn Sharp added a comment -

          I'm not sure re-registering with a new IP and the same storage ID actually works today.

          Jason Lowe recently finished a jira to make that work.

          How about we reject the DN registration in case of a DNS hiccup (rather than use the DN value which the patch currently does in this case)?

          I think I'm fine with that, so long as we are more strictly ruling out the ability to run a cluster in a dns-less or dns error-tolerant environment. I was considering a second jira that would first scan the include/exclude for the ip, and if not found, would return include=false or exclude=true if the ip is unresolved instead of flat out rejecting the node.

          Ignoring the name the dn declares is a trivial enough change that do you think we can just do it in this patch? I was trying to avoid any functional change with this patch (because who knows what will break!) but I'll post a revised patch that rejects unresolved and ignores the dn's declared name if that's ok with you?

          Show
          Daryn Sharp added a comment - I'm not sure re-registering with a new IP and the same storage ID actually works today. Jason Lowe recently finished a jira to make that work. How about we reject the DN registration in case of a DNS hiccup (rather than use the DN value which the patch currently does in this case)? I think I'm fine with that, so long as we are more strictly ruling out the ability to run a cluster in a dns-less or dns error-tolerant environment. I was considering a second jira that would first scan the include/exclude for the ip, and if not found, would return include=false or exclude=true if the ip is unresolved instead of flat out rejecting the node. Ignoring the name the dn declares is a trivial enough change that do you think we can just do it in this patch? I was trying to avoid any functional change with this patch (because who knows what will break!) but I'll post a revised patch that rejects unresolved and ignores the dn's declared name if that's ok with you?
          Hide
          Daryn Sharp added a comment -

          Uses the resolved hostname of the ipc connection and rejects unresolved hosts.

          Show
          Daryn Sharp added a comment - Uses the resolved hostname of the ipc connection and rejects unresolved hosts.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12549353/HDFS-3990.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.TestMiniDFSCluster
          org.apache.hadoop.cli.TestHDFSCLI
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.hdfs.TestReplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3347//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3347//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3347//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549353/HDFS-3990.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.TestMiniDFSCluster org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.TestReplication +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3347//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3347//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3347//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Ignoring the hostname the datanode claims to be is blowing up tests that are checking rack placement. Those tests need to use spoofed hostnames for the rack mapping.

          Prior to the patch, only the include/exclude lists checked the real hostname. Using the datanode's claimed hostname for the include/exclude checks creates a security issue, and ignoring the claimed hostname causes tests to fail. I was fearful that any functional change would break the code, so I'll toss up another variant of the original patch that keeps the two names separate.

          We really need this dns fix, so I think we'll need to break the unified and proper handling of the dn hostnames to another jira. Agree?

          Show
          Daryn Sharp added a comment - Ignoring the hostname the datanode claims to be is blowing up tests that are checking rack placement. Those tests need to use spoofed hostnames for the rack mapping. Prior to the patch, only the include/exclude lists checked the real hostname. Using the datanode's claimed hostname for the include/exclude checks creates a security issue, and ignoring the claimed hostname causes tests to fail. I was fearful that any functional change would break the code, so I'll toss up another variant of the original patch that keeps the two names separate. We really need this dns fix, so I think we'll need to break the unified and proper handling of the dn hostnames to another jira. Agree?
          Hide
          Eli Collins added a comment -

          Yea, that's what I meant above by "This breaks dfs.datanode.hostname but this config is only used by the tests and we can fix those up". How about I fix up my previous patch to unconditionally set the hostname and fix the tests?

          Btw the latest patch has some changes like changing DatanodeID#hashCode to ignore the IP addr, I don't think that's correct.

          Show
          Eli Collins added a comment - Yea, that's what I meant above by "This breaks dfs.datanode.hostname but this config is only used by the tests and we can fix those up". How about I fix up my previous patch to unconditionally set the hostname and fix the tests? Btw the latest patch has some changes like changing DatanodeID#hashCode to ignore the IP addr, I don't think that's correct.
          Hide
          Daryn Sharp added a comment -

          Track the peer name from the socket separately.

          Show
          Daryn Sharp added a comment - Track the peer name from the socket separately.
          Hide
          Daryn Sharp added a comment -

          Btw the latest patch has some changes like changing DatanodeID#hashCode to ignore the IP addr, I don't think that's correct.

          The ip is mutable, so it can't be part of the hashCode. When a datanode registers with an existing storage id, the ip & port will be updated which will affect a node in a collection. The storage id is immutable and unique so basing the hashCode off of it should be sufficient?

          Show
          Daryn Sharp added a comment - Btw the latest patch has some changes like changing DatanodeID#hashCode to ignore the IP addr, I don't think that's correct. The ip is mutable, so it can't be part of the hashCode . When a datanode registers with an existing storage id, the ip & port will be updated which will affect a node in a collection. The storage id is immutable and unique so basing the hashCode off of it should be sufficient?
          Hide
          Daryn Sharp added a comment -

          I forgot to mention that I think you'll find fixing/updating the rack placement tests will be exceedingly difficult w/o doing something very hacky. Everything looks like "localhost" to a minicluster. At least Kihwal and I couldn't find a clean way to update the tests...

          Show
          Daryn Sharp added a comment - I forgot to mention that I think you'll find fixing/updating the rack placement tests will be exceedingly difficult w/o doing something very hacky. Everything looks like "localhost" to a minicluster. At least Kihwal and I couldn't find a clean way to update the tests...
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12549399/HDFS-3990.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestReplication
          org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement
          org.apache.hadoop.hdfs.TestMiniDFSCluster
          org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
          org.apache.hadoop.cli.TestHDFSCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3349//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3349//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3349//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549399/HDFS-3990.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.TestMiniDFSCluster org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3349//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/3349//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3349//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Pulled cleanup out to HDFS-4068.

          Show
          Eli Collins added a comment - Pulled cleanup out to HDFS-4068 .
          Hide
          Eli Collins added a comment -

          What do you think of the attached patch?

          1. The NN looks up the DN hostname when it registers and updates DatanodeID. If the lookup fails we throw DatanodeRegistrationException for the DN to sleep and retry later
          2. The NN uses the DatanodeID hostname field rather than doing lookups
          Show
          Eli Collins added a comment - What do you think of the attached patch? The NN looks up the DN hostname when it registers and updates DatanodeID. If the lookup fails we throw DatanodeRegistrationException for the DN to sleep and retry later The NN uses the DatanodeID hostname field rather than doing lookups
          Hide
          Daryn Sharp added a comment -

          In your patch, it's not necessary for the NN to do another lookup of the DN's hostname. It's already available in the InetAddress returned by Server.getRemoteIp(). Passing this InetAddress to updateNodeAddr, rather than individually update the hostname and ip ensures the host and ip are always updated in tandem to avoid your concern about the fields going out of sync.

          If we do change the datanode manager to ignore the hostname in the node registration, do you think it's possible to update all the tests that check rack placement? I'm not sure how we can do that in a timely manner, so would you be willing to have a separate jira for that functional change to expedite this compatible one?

          Show
          Daryn Sharp added a comment - In your patch, it's not necessary for the NN to do another lookup of the DN's hostname. It's already available in the InetAddress returned by Server.getRemoteIp() . Passing this InetAddress to updateNodeAddr , rather than individually update the hostname and ip ensures the host and ip are always updated in tandem to avoid your concern about the fields going out of sync. If we do change the datanode manager to ignore the hostname in the node registration, do you think it's possible to update all the tests that check rack placement? I'm not sure how we can do that in a timely manner, so would you be willing to have a separate jira for that functional change to expedite this compatible one?
          Hide
          Daryn Sharp added a comment -

          Sorry, I accidentally re-submitted the wrong patch (same as prior) yesterday. Here's the real one that is compatible with current behavior by tracking that doesn't functionally alter the way hostnames are tracked.

          Show
          Daryn Sharp added a comment - Sorry, I accidentally re-submitted the wrong patch (same as prior) yesterday. Here's the real one that is compatible with current behavior by tracking that doesn't functionally alter the way hostnames are tracked.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12549515/HDFS-3990.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3355//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3355//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549515/HDFS-3990.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3355//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3355//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          The balancer test seems to randomly fail. It passes for me.

          Show
          Daryn Sharp added a comment - The balancer test seems to randomly fail. It passes for me.
          Hide
          Eli Collins added a comment -

          Think the approach in the latest patch should work. Once HDFS-4068 you can rebase on it and remove all the cleanup.

          Comments:

          • We can remove the dnAddress check for null now that it looks like NNThroughputBenchmark always uses RPC
          • Rename getNodeNames something more explicit like getNodeNamesForHostFiltering?
          • Rather than have updateNodeAddr let's use the two setters explicitly, easier to follow the registration behavior (ie we explicitly clobber the ip and peer hostname). Hopefully we'll eventually be able to make DatanodeID immutable so we don't update it in place.
          • Let's update getNodeNames to include the DN hostname since that is the current behavior, and file a separate jira for removing the use of the DN reported hostname here (or perhaps removing the reported DN field entirely)
          • Let's update hashCode in a separate change. I think this will need some additional changes like modifying Host2NodesMap to use DataNodeID hashCode, it currently explicitly uses the IP addr for the hash and ignores DatanodeID#hashCode.
          • Add a javadoc to testDNSLookups indicating that we're testing that the NN does not do DN lookups after registration
          • Nit, I'd create the SM inline via "System.setSecurityManager(new SecurityManager() {" so it's clear it's only associated with this DNS tests (like TestDFSShell for eg)
          • Nit, rename "lookups" in the test to "initialLookups"
          Show
          Eli Collins added a comment - Think the approach in the latest patch should work. Once HDFS-4068 you can rebase on it and remove all the cleanup. Comments: We can remove the dnAddress check for null now that it looks like NNThroughputBenchmark always uses RPC Rename getNodeNames something more explicit like getNodeNamesForHostFiltering? Rather than have updateNodeAddr let's use the two setters explicitly, easier to follow the registration behavior (ie we explicitly clobber the ip and peer hostname). Hopefully we'll eventually be able to make DatanodeID immutable so we don't update it in place. Let's update getNodeNames to include the DN hostname since that is the current behavior, and file a separate jira for removing the use of the DN reported hostname here (or perhaps removing the reported DN field entirely) Let's update hashCode in a separate change. I think this will need some additional changes like modifying Host2NodesMap to use DataNodeID hashCode, it currently explicitly uses the IP addr for the hash and ignores DatanodeID#hashCode. Add a javadoc to testDNSLookups indicating that we're testing that the NN does not do DN lookups after registration Nit, I'd create the SM inline via "System.setSecurityManager(new SecurityManager() {" so it's clear it's only associated with this DNS tests (like TestDFSShell for eg) Nit, rename "lookups" in the test to "initialLookups"
          Hide
          Daryn Sharp added a comment -

          I'm making the changes, but I found that I cannot remove the null check for dnAddress in the nodemanager. Tests using the minicluster directly get the rpc server (the remote/internal one of the NN) so no rpc socket connection is formed.

          I also don't think I can inline the SecurityManager (I initially tried) otherwise I cannot get access to the count of lookups. Java won't recognize that the field is available, or let me call a getter method because it's not defined by the SM.

          Show
          Daryn Sharp added a comment - I'm making the changes, but I found that I cannot remove the null check for dnAddress in the nodemanager. Tests using the minicluster directly get the rpc server (the remote/internal one of the NN) so no rpc socket connection is formed. I also don't think I can inline the SecurityManager (I initially tried) otherwise I cannot get access to the count of lookups. Java won't recognize that the field is available, or let me call a getter method because it's not defined by the SM.
          Hide
          Daryn Sharp added a comment -

          Will upload 23 patch shortly.

          Show
          Daryn Sharp added a comment - Will upload 23 patch shortly.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12549589/HDFS-3990.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3358//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3358//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549589/HDFS-3990.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3358//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3358//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          23's host & ip handling is very bizarre but I think I preserved existing behavior.

          Show
          Daryn Sharp added a comment - 23's host & ip handling is very bizarre but I think I preserved existing behavior.
          Hide
          Eli Collins added a comment -

          Two small comments, +1 otherwise!

          • Let's remove this line and leave peerHostName as null since we're claiming the peerHostname is the "hostname from the actual connection". It's also useful to have something to check to indicate the peerHostName has not been determined.
            this.peerHostName = hostName; // will assume it's the given host for now
          • move the "// Update the IP to the address of the RPC request..." comment up with the setIpAddr call
          Show
          Eli Collins added a comment - Two small comments, +1 otherwise! Let's remove this line and leave peerHostName as null since we're claiming the peerHostname is the "hostname from the actual connection". It's also useful to have something to check to indicate the peerHostName has not been determined. this .peerHostName = hostName; // will assume it's the given host for now move the "// Update the IP to the address of the RPC request..." comment up with the setIpAddr call
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12549614/HDFS-3990.branch-0.23.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3360//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12549614/HDFS-3990.branch-0.23.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3360//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Let's remove this line and leave peerHostName as null since we're claiming the peerHostname is the "hostname from the actual connection". It's also useful to have something to check to indicate the peerHostName has not been determined.

          The known case where the peerHostName will not be set is when the minicluster tests directly register a dn. If the assignment is removed, then I'm not sure where the null check should be and what it should do? It could either be in DatanodeID#getPeerHostName and return the hostName field? Or it could return null and DatanodeManager#getNodeNamesForHostFiltering will not return the peerHostName if null? I'm a bit concerned that tests, such as include/exclude list checks, might again break... Or I could update the comment to indicate it's either the remote RPC host or the dn reg's hostname?

          Show
          Daryn Sharp added a comment - Let's remove this line and leave peerHostName as null since we're claiming the peerHostname is the "hostname from the actual connection". It's also useful to have something to check to indicate the peerHostName has not been determined. The known case where the peerHostName will not be set is when the minicluster tests directly register a dn. If the assignment is removed, then I'm not sure where the null check should be and what it should do? It could either be in DatanodeID#getPeerHostName and return the hostName field? Or it could return null and DatanodeManager#getNodeNamesForHostFiltering will not return the peerHostName if null? I'm a bit concerned that tests, such as include/exclude list checks, might again break... Or I could update the comment to indicate it's either the remote RPC host or the dn reg's hostname?
          Hide
          Eli Collins added a comment -

          A null peerHostname just means you don't match, since we also check "hostName" which reported by the DataNode which the mini cluster explicitly sets we should be good, that's the current behavior after all right? Ie the only thing we're adding here is an additional hostname field to check, which is null and we won't check in the tests. Related, would be good to make the minicluster match real cluster behavior here.

          Show
          Eli Collins added a comment - A null peerHostname just means you don't match, since we also check "hostName" which reported by the DataNode which the mini cluster explicitly sets we should be good, that's the current behavior after all right? Ie the only thing we're adding here is an additional hostname field to check, which is null and we won't check in the tests. Related, would be good to make the minicluster match real cluster behavior here.
          Hide
          Daryn Sharp added a comment -

          No longer init peerHostName to the DN's registration hostname. Check for null when building list of node names to filter.

          I again looked into removing the null check on Server.getRemoteAddress. The tests that call directly into the rpc server object, rather than via a connection, appear to be passing mock dn registrations. So the majority of functional tests are matching real cluster behavior.

          I tried having the rpc server set the ip/peerHostName but some of the tests are verifying the layout and version checks work. So I tried to push those down into the FSNamesystem#registerDatanode but that method isn't exposed for the tests to call.

          If this patch is ok, I'll update the 23 patch.

          Show
          Daryn Sharp added a comment - No longer init peerHostName to the DN's registration hostname. Check for null when building list of node names to filter. I again looked into removing the null check on Server.getRemoteAddress . The tests that call directly into the rpc server object, rather than via a connection, appear to be passing mock dn registrations. So the majority of functional tests are matching real cluster behavior. I tried having the rpc server set the ip/peerHostName but some of the tests are verifying the layout and version checks work. So I tried to push those down into the FSNamesystem#registerDatanode but that method isn't exposed for the tests to call. If this patch is ok, I'll update the 23 patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550340/HDFS-3990.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3378//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3378//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550340/HDFS-3990.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3378//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3378//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          Now that you're switching getNodeNamesForHostFiltering from using an array to a List, I'd use an ImmutableList. +1 otherwise

          Show
          Eli Collins added a comment - Now that you're switching getNodeNamesForHostFiltering from using an array to a List, I'd use an ImmutableList. +1 otherwise
          Hide
          Daryn Sharp added a comment -

          I reverted method signature to returning an array instead of a list.

          Show
          Daryn Sharp added a comment - I reverted method signature to returning an array instead of a list.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12551382/HDFS-3990.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3428//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3428//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12551382/HDFS-3990.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3428//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3428//console This message is automatically generated.
          Hide
          Eli Collins added a comment -

          I missed that you switched to a List because we're conditionally adding items so hard to use an ImmutableList, I think using a List is better than the latest patch where you convert the List to an array, so +1 to the Oct 22nd patch

          Show
          Eli Collins added a comment - I missed that you switched to a List because we're conditionally adding items so hard to use an ImmutableList, I think using a List is better than the latest patch where you convert the List to an array, so +1 to the Oct 22nd patch
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #2985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2985/)
          HDFS-3990. NN's health report has severe performance problems (daryn) (Revision 1407333)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407333
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #2985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/2985/ ) HDFS-3990 . NN's health report has severe performance problems (daryn) (Revision 1407333) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407333 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #31 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/31/)
          HDFS-3990. NN's health report has severe performance problems (daryn) (Revision 1407333)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407333
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #31 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/31/ ) HDFS-3990 . NN's health report has severe performance problems (daryn) (Revision 1407333) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407333 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #430 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/430/)
          HDFS-3990. NN's health report has severe performance problems (daryn) (Revision 1407336)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407336
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #430 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/430/ ) HDFS-3990 . NN's health report has severe performance problems (daryn) (Revision 1407336) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407336 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1221 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1221/)
          HDFS-3990. NN's health report has severe performance problems (daryn) (Revision 1407333)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407333
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1221 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1221/ ) HDFS-3990 . NN's health report has severe performance problems (daryn) (Revision 1407333) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407333 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1251 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1251/)
          HDFS-3990. NN's health report has severe performance problems (daryn) (Revision 1407333)

          Result = FAILURE
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407333
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1251 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1251/ ) HDFS-3990 . NN's health report has severe performance problems (daryn) (Revision 1407333) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407333 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDatanodeRegistration.java
          Hide
          Chris Nauroth added a comment -

          Daryn and Eli, we merged this change to branch-trunk-win on Friday, 11/30. Unfortunately, this had an unintended side effect of breaking on Windows, at least for single-node developer setups, because of the code change to reject registration of an unresolved data node:

            public void registerDatanode(DatanodeRegistration nodeReg)
                throws DisallowedDatanodeException {
              InetAddress dnAddress = Server.getRemoteIp();
              if (dnAddress != null) {
                // Mostly called inside an RPC, update ip and peer hostname
                String hostname = dnAddress.getHostName();
                String ip = dnAddress.getHostAddress();
                if (hostname.equals(ip)) {
                  LOG.warn("Unresolved datanode registration from " + ip);
                  throw new DisallowedDatanodeException(nodeReg);
                }
          

          On Windows, 127.0.0.1 does not resolve to localhost. It reports host name as "127.0.0.1". Therefore, on Windows, running pseudo-distributed mode or MiniDFSCluster-based tests always rejects the datanode registrations. (See HADOOP-8414 for more discussion of the particulars of resolving 127.0.0.1 on Windows.)

          Potential fixes I can think of:

          1. Add special case logic to allow registration if ip.equals("127.0.0.1"). This is the quick fix I applied to my dev environment to unblock myself last Friday.
          2. Add a check against NetUtils.getStaticResolution and register it with NetUtils.addStaticResolution("127.0.0.1", "localhost") somewhere at initialization time.

          Do you have an opinion on the best way to fix it? I have a Windows VM ready to go, so I can code the patch and test.

          Show
          Chris Nauroth added a comment - Daryn and Eli, we merged this change to branch-trunk-win on Friday, 11/30. Unfortunately, this had an unintended side effect of breaking on Windows, at least for single-node developer setups, because of the code change to reject registration of an unresolved data node: public void registerDatanode(DatanodeRegistration nodeReg) throws DisallowedDatanodeException { InetAddress dnAddress = Server.getRemoteIp(); if (dnAddress != null ) { // Mostly called inside an RPC, update ip and peer hostname String hostname = dnAddress.getHostName(); String ip = dnAddress.getHostAddress(); if (hostname.equals(ip)) { LOG.warn( "Unresolved datanode registration from " + ip); throw new DisallowedDatanodeException(nodeReg); } On Windows, 127.0.0.1 does not resolve to localhost. It reports host name as "127.0.0.1". Therefore, on Windows, running pseudo-distributed mode or MiniDFSCluster-based tests always rejects the datanode registrations. (See HADOOP-8414 for more discussion of the particulars of resolving 127.0.0.1 on Windows.) Potential fixes I can think of: Add special case logic to allow registration if ip.equals("127.0.0.1"). This is the quick fix I applied to my dev environment to unblock myself last Friday. Add a check against NetUtils.getStaticResolution and register it with NetUtils.addStaticResolution("127.0.0.1", "localhost") somewhere at initialization time. Do you have an opinion on the best way to fix it? I have a Windows VM ready to go, so I can code the patch and test.
          Hide
          Daryn Sharp added a comment -

          The check was floated up out of DatanodeManager.checkInList which rejected unresolvable nodes. Is it that InetAddress.getByName on windows doesn't resolve 127.0.0.1 and doesn't throw UnknownHostException, which makes it appear it didn't resolve? I seem to have vague recollection of a similar issue before.

          Show
          Daryn Sharp added a comment - The check was floated up out of DatanodeManager.checkInList which rejected unresolvable nodes. Is it that InetAddress.getByName on windows doesn't resolve 127.0.0.1 and doesn't throw UnknownHostException , which makes it appear it didn't resolve? I seem to have vague recollection of a similar issue before.
          Hide
          Chris Nauroth added a comment -

          The problem I observe is that a server accepts a client socket connection, gets the connection's InetAddress, and then getHostName returns "127.0.0.1". Below is a short code sample that demonstrates the problem. This is a very rough approximation of the IPC Server/Connection and DatanodeManager logic. When I run this server on Mac, it prints "connection from hostName = localhost, hostAddress = 127.0.0.1, canonicalHostName = localhost" for any client connection. On Windows, it prints "connection from hostName = 127.0.0.1, hostAddress = 127.0.0.1, canonicalHostName = 127.0.0.1".

          package cnauroth;
          
          import java.io.PrintWriter;
          import java.net.InetAddress;
          import java.net.InetSocketAddress;
          import java.net.ServerSocket;
          import java.net.Socket;
          import java.nio.channels.ServerSocketChannel;
          
          class Main {
            public static void main(String[] args) throws Exception {
              ServerSocket ss = ServerSocketChannel.open().socket();
              ss.bind(new InetSocketAddress("localhost", 1234), 0);
              System.out.println("ss = " + ss);
              for (;;) {
                Socket s = ss.accept();
                InetAddress addr = s.getInetAddress();
                System.out.println("connection from hostName = " + addr.getHostName() + ", hostAddress = " + addr.getHostAddress() + ", canonicalHostName = " + addr.getCanonicalHostName());
                PrintWriter pw = new PrintWriter(s.getOutputStream());
                pw.println("hello");
                pw.close();
                s.close();
              }
            }
          }
          
          Show
          Chris Nauroth added a comment - The problem I observe is that a server accepts a client socket connection, gets the connection's InetAddress, and then getHostName returns "127.0.0.1". Below is a short code sample that demonstrates the problem. This is a very rough approximation of the IPC Server/Connection and DatanodeManager logic. When I run this server on Mac, it prints "connection from hostName = localhost, hostAddress = 127.0.0.1, canonicalHostName = localhost" for any client connection. On Windows, it prints "connection from hostName = 127.0.0.1, hostAddress = 127.0.0.1, canonicalHostName = 127.0.0.1". package cnauroth; import java.io.PrintWriter; import java.net.InetAddress; import java.net.InetSocketAddress; import java.net.ServerSocket; import java.net.Socket; import java.nio.channels.ServerSocketChannel; class Main { public static void main( String [] args) throws Exception { ServerSocket ss = ServerSocketChannel.open().socket(); ss.bind( new InetSocketAddress( "localhost" , 1234), 0); System .out.println( "ss = " + ss); for (;;) { Socket s = ss.accept(); InetAddress addr = s.getInetAddress(); System .out.println( "connection from hostName = " + addr.getHostName() + ", hostAddress = " + addr.getHostAddress() + ", canonicalHostName = " + addr.getCanonicalHostName()); PrintWriter pw = new PrintWriter(s.getOutputStream()); pw.println( "hello" ); pw.close(); s.close(); } } }
          Hide
          Chris Nauroth added a comment -

          I have moved discussion of the new issue on Windows to HDFS-4269. Thank you.

          Show
          Chris Nauroth added a comment - I have moved discussion of the new issue on Windows to HDFS-4269 . Thank you.
          Hide
          Liang Xie added a comment -

          we hit the same issue like Chris Nauroth on linux + CDH4.1.1 modified version, only different is "0.0.0.0" , not "127.0.0.1".
          so i changed the registerDatanode code snippet based the final patch:

                if (hostname.equals(ip)) {
                  try {
                    hostname = InetAddress.getByName(Server.getRemoteAddress()).
                        getHostName();
                  } catch (UnknownHostException e) {
                    LOG.warn("Unable to lookup hostname for DataNode " +
                        ip + " which registered with hostname " +
                        nodeReg.getHostName());
                    throw new DisallowedDatanodeException(nodeReg);
                  }
                }
          

          maybe it helpful for somebody else who hit the same issue.

          Show
          Liang Xie added a comment - we hit the same issue like Chris Nauroth on linux + CDH4.1.1 modified version, only different is "0.0.0.0" , not "127.0.0.1". so i changed the registerDatanode code snippet based the final patch: if (hostname.equals(ip)) { try { hostname = InetAddress.getByName(Server.getRemoteAddress()). getHostName(); } catch (UnknownHostException e) { LOG.warn( "Unable to lookup hostname for DataNode " + ip + " which registered with hostname " + nodeReg.getHostName()); throw new DisallowedDatanodeException(nodeReg); } } maybe it helpful for somebody else who hit the same issue.
          Hide
          Jagane Sundar added a comment -

          Here is another instance of this problem.

          Here is what I am trying to do:

          Create a single VM developer environment that runs all daemons in a VM. The VM gets a DHCP IP address, but there is no hostname associated with the IP address. I configure hadoop using the DHCP IP address (e.g. 192.168.1.94) instead of the hostname, or 'localhost' or '127.0.0.1]'. Datanode registration fails because of this check.

          HDFS-4269 creates an escape hatch just for 127.0.0.1. That does not solve my problem because I want to use the DHCP address 192.168.1.94. I want to use 192.168.1.94 because I want to be able to access this VM from my host OS, or from other machines in the network (if I use bridged networking in the virtual NIC configuration).

          I don't quite follow the original reasoning behind this check. Is there some fundamental reason why HDFS cannot operate in an environment where the IP address of the host cannot be resolved to a hostname?

          Show
          Jagane Sundar added a comment - Here is another instance of this problem. Here is what I am trying to do: Create a single VM developer environment that runs all daemons in a VM. The VM gets a DHCP IP address, but there is no hostname associated with the IP address. I configure hadoop using the DHCP IP address (e.g. 192.168.1.94) instead of the hostname, or 'localhost' or '127.0.0.1]'. Datanode registration fails because of this check. HDFS-4269 creates an escape hatch just for 127.0.0.1. That does not solve my problem because I want to use the DHCP address 192.168.1.94. I want to use 192.168.1.94 because I want to be able to access this VM from my host OS, or from other machines in the network (if I use bridged networking in the virtual NIC configuration). I don't quite follow the original reasoning behind this check. Is there some fundamental reason why HDFS cannot operate in an environment where the IP address of the host cannot be resolved to a hostname?
          Hide
          Chris Nauroth added a comment -

          Hello, Jagane Sundar. I noticed that you commented on both this and HDFS-4269. I'm going to focus the response on HDFS-4269, so please see my comment there. Thanks!

          Show
          Chris Nauroth added a comment - Hello, Jagane Sundar . I noticed that you commented on both this and HDFS-4269 . I'm going to focus the response on HDFS-4269 , so please see my comment there. Thanks!
          Hide
          Chris Nauroth added a comment -

          I filed HDFS-4702 to investigate removing the namesystem lock from this code path.

          Show
          Chris Nauroth added a comment - I filed HDFS-4702 to investigate removing the namesystem lock from this code path.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The hostname.equals(ip) check added to DatanodeManager.registerDatanode(..) may not work for existing clusters which do not have reverse DNS lookup support; filed HDFS-5338.

          Show
          Tsz Wo Nicholas Sze added a comment - The hostname.equals(ip) check added to DatanodeManager.registerDatanode(..) may not work for existing clusters which do not have reverse DNS lookup support; filed HDFS-5338 .

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development