Hadoop Common
  1. Hadoop Common
  2. HADOOP-3620

Namenode should synchronously resolve a datanode's network location when the datanode registers

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Release 0.18.0 removes the rpc timeout. So the namenode is ok to resolve a datanode's network location when the datanode registers. This could remove quite a lot of unnecessary code in both datanode and namenode to handle asynchronous network location resolution and avoid many potential bugs.

      1. netResolution.patch
        17 kB
        Hairong Kuang
      2. netResolution1.patch
        23 kB
        Hairong Kuang
      3. netResolution2.patch
        22 kB
        Hairong Kuang
      4. netResolution3.patch
        27 kB
        Hairong Kuang
      5. netResolution4.patch
        27 kB
        Hairong Kuang
      6. netResolution5.patch
        32 kB
        Hairong Kuang
      7. netResolution6.patch
        33 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #581 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/ )
          Hide
          Amareshwari Sriramadasu added a comment -

          I noticed that the new files are not committed. Trunk doesnt compile

          Show
          Amareshwari Sriramadasu added a comment - I noticed that the new files are not committed. Trunk doesnt compile
          Hide
          Hairong Kuang added a comment -

          I've just committed this.

          Show
          Hairong Kuang added a comment - I've just committed this.
          Hide
          Devaraj Das added a comment -

          +1

          Show
          Devaraj Das added a comment - +1
          Hide
          Raghu Angadi added a comment -

          But this new patch reduces all possible calls to DNS resolution. It has all the following changes: [...]

          +1 for the improvements.

          I talked with Raghu and understood his concern is that the number of calls to DNS resolution might impact the performance of network location resolution performance. From my experiment, this seems not a big concern.

          But, I don't think DNS resultions is non-issue. see HADOOP-3694 for e.g. Its good that the latest patch reduces DNS resolutions.

          Show
          Raghu Angadi added a comment - But this new patch reduces all possible calls to DNS resolution. It has all the following changes: [...] +1 for the improvements. I talked with Raghu and understood his concern is that the number of calls to DNS resolution might impact the performance of network location resolution performance. From my experiment, this seems not a big concern. But, I don't think DNS resultions is non-issue. see HADOOP-3694 for e.g. Its good that the latest patch reduces DNS resolutions.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12387066/netResolution6.patch
          against trunk revision 680577.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2965/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2965/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2965/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2965/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12387066/netResolution6.patch against trunk revision 680577. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2965/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2965/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2965/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2965/console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          Fixed the failed unit tests.

          Show
          Hairong Kuang added a comment - Fixed the failed unit tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12386770/netResolution5.patch
          against trunk revision 679286.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2936/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2936/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2936/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2936/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12386770/netResolution5.patch against trunk revision 679286. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2936/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2936/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2936/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2936/console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          I talked with Raghu and understood his concern is that the number of calls to DNS resolution might impact the performance of network location resolution performance. From my experiment, this seems not a big concern. Instead, reducing the number of calls to the script would greatly improve the resolution performance.

          But this new patch reduces all possible calls to DNS resolution. It has all the following changes:
          1. Increase maxArg of ScriptBasedMapping from 20 to 100;
          2. CachedDNSToSwitchMap maps IP addresses to the network location;
          3. It allows include/exclude host files to contain ip addresses.

          Show
          Hairong Kuang added a comment - I talked with Raghu and understood his concern is that the number of calls to DNS resolution might impact the performance of network location resolution performance. From my experiment, this seems not a big concern. Instead, reducing the number of calls to the script would greatly improve the resolution performance. But this new patch reduces all possible calls to DNS resolution. It has all the following changes: 1. Increase maxArg of ScriptBasedMapping from 20 to 100; 2. CachedDNSToSwitchMap maps IP addresses to the network location; 3. It allows include/exclude host files to contain ip addresses.
          Hide
          Raghu Angadi added a comment - - edited

          > So network resolution in the front could be an overhead.
          this may not be a problem since a DataNode would not re-register unless there is a real problem/bug. Not sure if we need to optimize that. Even if we want to, we can make 'internalRegisterDatanode()' throw an exception to indicate that netwo needs to be resolved before calling it.

          I think doing this way will simplify the code and patch even further.

          > Pre-resolving should help since it resolves network locations in batch and therefore reducing the number of calls to the rack script dramatically.
          Only if the script can do the resolutions in parallel. Does the default script make use of this? Also there are 2 DNS resolutions done for each host serially inside namenode to 'normalize' the host names, right? In addition, many installations may not specify include hosts.

          Show
          Raghu Angadi added a comment - - edited > So network resolution in the front could be an overhead. this may not be a problem since a DataNode would not re-register unless there is a real problem/bug. Not sure if we need to optimize that. Even if we want to, we can make 'internalRegisterDatanode()' throw an exception to indicate that netwo needs to be resolved before calling it. I think doing this way will simplify the code and patch even further. > Pre-resolving should help since it resolves network locations in batch and therefore reducing the number of calls to the rack script dramatically. Only if the script can do the resolutions in parallel. Does the default script make use of this? Also there are 2 DNS resolutions done for each host serially inside namenode to 'normalize' the host names, right? In addition, many installations may not specify include hosts.
          Hide
          Hairong Kuang added a comment -

          > Does the above work?
          I thought about this solution too. But if you look at the registration code, sometimes there is no need to resolve a node's network location if the node has already registered. So network resolution in the front could be an overhead.

          > Note that pre-resolving hosts in include file might not help start up.
          Pre-resolving should help since it resolves network locations in batch and therefore reducing the number of calls to the rack script dramatically.

          Show
          Hairong Kuang added a comment - > Does the above work? I thought about this solution too. But if you look at the registration code, sometimes there is no need to resolve a node's network location if the node has already registered. So network resolution in the front could be an overhead. > Note that pre-resolving hosts in include file might not help start up. Pre-resolving should help since it resolves network locations in batch and therefore reducing the number of calls to the rack script dramatically.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12386366/netResolution4.patch
          against trunk revision 677839.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2899/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2899/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2899/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2899/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12386366/netResolution4.patch against trunk revision 677839. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2899/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2899/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2899/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2899/console This message is automatically generated.
          Hide
          Raghu Angadi added a comment - - edited

          One way to move the resolution out of global lock :

          resolveNetworkLocation does not strictly need DatanodeDescriptor.
          So regeisterDatanode() could look something like :

           public void registerDatanode(DatanodeRegistration nodeReg) throws IOException {
               String networkLocation = resolveNeworklocation(nodeReg.getHost());
               internalRegisterDatanode(nodeReg, networkLocation); //holds global lock.
          }
          

          Does the above work?

          Note that pre-resolving hosts in include file might not help start up since

          • resolving serially at the beginning still increases the start up time.
          • "normalizing host names" does multiple DNS resolves.
          Show
          Raghu Angadi added a comment - - edited One way to move the resolution out of global lock : resolveNetworkLocation does not strictly need DatanodeDescriptor. So regeisterDatanode() could look something like : public void registerDatanode(DatanodeRegistration nodeReg) throws IOException { String networkLocation = resolveNeworklocation(nodeReg.getHost()); internalRegisterDatanode(nodeReg, networkLocation); //holds global lock. } Does the above work? Note that pre-resolving hosts in include file might not help start up since resolving serially at the beginning still increases the start up time. "normalizing host names" does multiple DNS resolves.
          Hide
          Hairong Kuang added a comment -

          This patch fixed the javadoc warning.

          Show
          Hairong Kuang added a comment - This patch fixed the javadoc warning.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12385685/netResolution3.patch
          against trunk revision 676069.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2850/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2850/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2850/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2850/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12385685/netResolution3.patch against trunk revision 676069. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2850/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2850/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2850/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2850/console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          It's very hard to release the global lock during network resolution without breaking the data structure consistency. So what I did is to pre-resolve the network locations of datanodes that are in the include file. So the resolution during the registration is going to be fast.

          Show
          Hairong Kuang added a comment - It's very hard to release the global lock during network resolution without breaking the data structure consistency. So what I did is to pre-resolve the network locations of datanodes that are in the include file. So the resolution during the registration is going to be fast.
          Hide
          Devaraj Das added a comment -

          As far as I can see, the resolution happens with the global FSNamesystem lock held. Is that true? Dhruba had a concern there and I don't see a nice way to handle that either. The patch looks good otherwise (subject to hudson passing it and successfully running MapReduce jobs with this patch).

          Show
          Devaraj Das added a comment - As far as I can see, the resolution happens with the global FSNamesystem lock held. Is that true? Dhruba had a concern there and I don't see a nice way to handle that either. The patch looks good otherwise (subject to hudson passing it and successfully running MapReduce jobs with this patch).
          Hide
          Hairong Kuang added a comment -

          This patch applies to trunk. Additionally
          1. It provides a cached implementation of DNSToSwitchMapping, which caches all resolved DNS to switch mapping. Any implementation could extend CachedDNSToSwitchMapping if this helps improve its performance.
          2. ScriptBasedMapping is CachedDNSToSwitchMapping while RawScriptBasedMapping resolves a location only by running a script.
          3. Name node pre-resolves the network location of all datanodes in the include list only if the configured DNSToSwitchMapping is cached.

          Show
          Hairong Kuang added a comment - This patch applies to trunk. Additionally 1. It provides a cached implementation of DNSToSwitchMapping, which caches all resolved DNS to switch mapping. Any implementation could extend CachedDNSToSwitchMapping if this helps improve its performance. 2. ScriptBasedMapping is CachedDNSToSwitchMapping while RawScriptBasedMapping resolves a location only by running a script. 3. Name node pre-resolves the network location of all datanodes in the include list only if the configured DNSToSwitchMapping is cached.
          Hide
          Hairong Kuang added a comment -

          Ok, this patch still keeps the cache in SriptBasedMapping. But pre-resolve datanodes' network locations only when script based mapping is used.

          Show
          Hairong Kuang added a comment - Ok, this patch still keeps the cache in SriptBasedMapping. But pre-resolve datanodes' network locations only when script based mapping is used.
          Hide
          Devaraj Das added a comment -

          I haven't gone through the patch in detail but one thing which struck me is that you removed the cache from ScriptBasedMapping and moved it to the dfs part of the framework. That might be a problem for MR part of the framework.

          Show
          Devaraj Das added a comment - I haven't gone through the patch in detail but one thing which struck me is that you removed the cache from ScriptBasedMapping and moved it to the dfs part of the framework. That might be a problem for MR part of the framework.
          Hide
          Hairong Kuang added a comment -

          This patch additionally improves the performance of network resolution during registration by prersolving the network locations of every included host and storing them in a cache.

          Show
          Hairong Kuang added a comment - This patch additionally improves the performance of network resolution during registration by prersolving the network locations of every included host and storing them in a cache.
          Hide
          Owen O'Malley added a comment -

          I'm +1 to doing all of the resolution synchronously in both the namenode and jobtracker.

          Show
          Owen O'Malley added a comment - I'm +1 to doing all of the resolution synchronously in both the namenode and jobtracker.
          Hide
          Amar Kamat added a comment -

          I think it also makes sense to do the same in the JobTracker. There too the resolution is async. With HADOOP-3590 getting fixed, the problem is that the tasks will be scheduled randomly until the node gets resolved. Some of the test cases assume that the all the TTs are resolved before the job gets submitted which might not be true always. Thoughts?

          Show
          Amar Kamat added a comment - I think it also makes sense to do the same in the JobTracker. There too the resolution is async. With HADOOP-3590 getting fixed, the problem is that the tasks will be scheduled randomly until the node gets resolved. Some of the test cases assume that the all the TTs are resolved before the job gets submitted which might not be true always. Thoughts?
          Hide
          Hairong Kuang added a comment - - edited

          Dhruba's comment makes sense. The attached inital patch still let the thread hold the global lock while resolving a network location. I need to figure out how not to hold the lock without any risk of data structure inconsistency.

          The attached patch resolves a data node's network location when it registers. It also lets a data node to randomly back off its block report when the data node starts up.

          Show
          Hairong Kuang added a comment - - edited Dhruba's comment makes sense. The attached inital patch still let the thread hold the global lock while resolving a network location. I need to figure out how not to hold the lock without any risk of data structure inconsistency. The attached patch resolves a data node's network location when it registers. It also lets a data node to randomly back off its block report when the data node starts up.
          Hide
          dhruba borthakur added a comment -

          It would be nice if the code can be organized in such a way that the FSnamesystem global lock is not held when the datanode's network location is resolved. Otherwise, cluster restart times could potentially take much much longer.

          Show
          dhruba borthakur added a comment - It would be nice if the code can be organized in such a way that the FSnamesystem global lock is not held when the datanode's network location is resolved. Otherwise, cluster restart times could potentially take much much longer.

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development