Hadoop Common
  1. Hadoop Common
  2. HADOOP-7030

Add TableMapping topology implementation to read host to rack mapping from a file

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.20.2, 0.21.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.

      This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.

      1. topology.patch
        22 kB
        Patrick Angeles
      2. HADOOP-7030-branch-1.patch
        11 kB
        Tom White
      3. HADOOP-7030-2.patch
        29 kB
        Patrick Angeles
      4. HADOOP-7030.patch
        23 kB
        Patrick Angeles
      5. HADOOP-7030.patch
        12 kB
        Tom White
      6. HADOOP-7030.patch
        13 kB
        Tom White
      7. HADOOP-7030.patch
        13 kB
        Tom White

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12459424/topology.patch
          against trunk revision 1034184.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 16 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 4 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/95//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/95//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/95//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459424/topology.patch against trunk revision 1034184. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 4 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/95//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/95//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/95//console This message is automatically generated.
          Hide
          Patrick Angeles added a comment -

          Reworked to fix 3 of the 4 findbugs issues. 4th one is harmless.

          Show
          Patrick Angeles added a comment - Reworked to fix 3 of the 4 findbugs issues. 4th one is harmless.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12459427/HADOOP-7030.patch
          against trunk revision 1034184.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 16 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/96//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/96//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/96//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459427/HADOOP-7030.patch against trunk revision 1034184. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/96//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/96//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/96//console This message is automatically generated.
          Hide
          Patrick Angeles added a comment -

          Updated patch that introduces the RefreshableDNSToSwitchMapping interface which adds a 'refresh' method to DNSToSwitchMapping. This can be used later as a hook for 'hadoop (dfs|mr)admin -refreshNodes' to update the topology without restarting the cluster.

          Show
          Patrick Angeles added a comment - Updated patch that introduces the RefreshableDNSToSwitchMapping interface which adds a 'refresh' method to DNSToSwitchMapping. This can be used later as a hook for 'hadoop (dfs|mr)admin -refreshNodes' to update the topology without restarting the cluster.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12469308/HADOOP-7030-2.patch
          against trunk revision 1063090.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/197//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/197//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/197//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12469308/HADOOP-7030-2.patch against trunk revision 1063090. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/197//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/197//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/197//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12469308/HADOOP-7030-2.patch
          against trunk revision 1071364.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/252//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/252//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/252//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12469308/HADOOP-7030-2.patch against trunk revision 1071364. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/252//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/252//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/252//console This message is automatically generated.
          Hide
          Tom White added a comment -

          This looks like a useful addition. Here are my comments on the patch:

          • Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation?
          • Have you thought about InetAddress to avoid implementing IP address parsing logic? http://guava-libraries.googlecode.com/svn/tags/release08/javadoc/com/google/common/net/InetAddresses.html might be useful (there was talk of introducing Guava recently).
          • RefreshableDNSToSwitchMapping isn't hooked up yet, so perhaps it should go in a follow on JIRA.
          • The name "TableMapping" is a bit general. How about "FileBasedMapping", or similar?
          • The configuration keys should go in CommonConfigurationKeysPublic.
          • Primes are not needed in hashCode implementations. For Ip4 Arrays.hashCode(value) is sufficient.
          • The tests swallow exceptions - there should at least be a comment saying that this is expected. Also, fail() with a message is preferable to assertTrue(false).
          • The tests should be JUnit 4 style.
          Show
          Tom White added a comment - This looks like a useful addition. Here are my comments on the patch: Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation? Have you thought about InetAddress to avoid implementing IP address parsing logic? http://guava-libraries.googlecode.com/svn/tags/release08/javadoc/com/google/common/net/InetAddresses.html might be useful (there was talk of introducing Guava recently). RefreshableDNSToSwitchMapping isn't hooked up yet, so perhaps it should go in a follow on JIRA. The name "TableMapping" is a bit general. How about "FileBasedMapping", or similar? The configuration keys should go in CommonConfigurationKeysPublic. Primes are not needed in hashCode implementations. For Ip4 Arrays.hashCode(value) is sufficient. The tests swallow exceptions - there should at least be a comment saying that this is expected. Also, fail() with a message is preferable to assertTrue(false). The tests should be JUnit 4 style.
          Hide
          Patrick Angeles added a comment -

          Hey Tom,

          Thanks for the review. Here are some responses:

          Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation?

          • I'd prefer to keep them separate as the first two columns have completely different meanings when using one style (table lookup) over the other (IP-range).

          Have you thought about InetAddress to avoid implementing IP address parsing logic? http://guava-libraries.googlecode.com/svn/tags/release08/javadoc/com/google/common/net/InetAddresses.html might be useful (there was talk of introducing Guava recently).

          • I have not. Will look into this, although I'd rather keep this patch lightweight and not require the addition of another jar.

          RefreshableDNSToSwitchMapping isn't hooked up yet, so perhaps it should go in a follow on JIRA.

          • Yes, that is the intent. Those JIRAs would go into MAPREDUCE and HDFS.

          The name "TableMapping" is a bit general. How about "FileBasedMapping", or similar?

          • I'm willing to listen to suggestions, however I think FileBasedMapping is even more vague

          The configuration keys should go in CommonConfigurationKeysPublic.

          • Since this is a pluggable interface, I should not have to modify existing core code. That's better WRT separation of concerns and componentization. I'm willing to take your suggestion if the general consensus is that I should

          Primes are not needed in hashCode implementations. For Ip4 Arrays.hashCode(value) is sufficient.

          • Ok.

          The tests swallow exceptions - there should at least be a comment saying that this is expected. Also, fail() with a message is preferable to assertTrue(false).

          • Ok.

          The tests should be JUnit 4 style.

          • Ok.
          Show
          Patrick Angeles added a comment - Hey Tom, Thanks for the review. Here are some responses: Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation? I'd prefer to keep them separate as the first two columns have completely different meanings when using one style (table lookup) over the other (IP-range). Have you thought about InetAddress to avoid implementing IP address parsing logic? http://guava-libraries.googlecode.com/svn/tags/release08/javadoc/com/google/common/net/InetAddresses.html might be useful (there was talk of introducing Guava recently). I have not. Will look into this, although I'd rather keep this patch lightweight and not require the addition of another jar. RefreshableDNSToSwitchMapping isn't hooked up yet, so perhaps it should go in a follow on JIRA. Yes, that is the intent. Those JIRAs would go into MAPREDUCE and HDFS. The name "TableMapping" is a bit general. How about "FileBasedMapping", or similar? I'm willing to listen to suggestions, however I think FileBasedMapping is even more vague The configuration keys should go in CommonConfigurationKeysPublic. Since this is a pluggable interface, I should not have to modify existing core code. That's better WRT separation of concerns and componentization. I'm willing to take your suggestion if the general consensus is that I should Primes are not needed in hashCode implementations. For Ip4 Arrays.hashCode(value) is sufficient. Ok. The tests swallow exceptions - there should at least be a comment saying that this is expected. Also, fail() with a message is preferable to assertTrue(false). Ok. The tests should be JUnit 4 style. Ok.
          Hide
          Patrick Angeles added a comment -
          • Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation?
          • I'd prefer to keep them separate as the first two columns have completely different meanings when using one style (table lookup) over the other (IP-range).

          BTW, I don't think CIDR is appropriate here. For the table-based mapping, you can get either hosts or IPs, possibly depending on who (JT or NN) is requesting the rack ID. The docs are unclear here.

          In either case, servers within a rack rarely fall in units that are powers of two.

          Show
          Patrick Angeles added a comment - Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation? I'd prefer to keep them separate as the first two columns have completely different meanings when using one style (table lookup) over the other (IP-range). BTW, I don't think CIDR is appropriate here. For the table-based mapping, you can get either hosts or IPs, possibly depending on who (JT or NN) is requesting the rack ID. The docs are unclear here. In either case, servers within a rack rarely fall in units that are powers of two.
          Hide
          Steve Loughran added a comment -

          With the changes I've been doing to the topology classes (with an optional abstract base class), these classes would be best if
          RefreshableDNSToSwitchMapping was a subclass of the new AbstractDNSToSwitchMapping.

          On barrier to any dynamic/refreshable topology class is that the values get cached. In particular
          org.apache.hadoop.yarn.util.RackResolver checks to see if the mapping is an instance of CachedDNSToSwitchMapping -and if not, wraps it in one. the DNManager doesn't do anything so dramatic - it does do an instance check, but only to trigger a pre-emptive lookup of all known hosts.

          Unless we add flags to AbstractDNSToSwitchMapping to provide more info about the resolver (e.g. caching, dynamic, etc), it's probably safest to subclass CachedDNSToSwitchMapping and overwrite its resolve method (and anything else that is useful).

          Show
          Steve Loughran added a comment - With the changes I've been doing to the topology classes (with an optional abstract base class), these classes would be best if RefreshableDNSToSwitchMapping was a subclass of the new AbstractDNSToSwitchMapping . On barrier to any dynamic/refreshable topology class is that the values get cached. In particular org.apache.hadoop.yarn.util.RackResolver checks to see if the mapping is an instance of CachedDNSToSwitchMapping -and if not, wraps it in one. the DNManager doesn't do anything so dramatic - it does do an instance check, but only to trigger a pre-emptive lookup of all known hosts. Unless we add flags to AbstractDNSToSwitchMapping to provide more info about the resolver (e.g. caching, dynamic, etc), it's probably safest to subclass CachedDNSToSwitchMapping and overwrite its resolve method (and anything else that is useful).
          Hide
          Tom White added a comment -

          I've created a simplified patch which only has the two-column layout (we can add the range variant later as needed - also, now we have Guava the implementation will be simplified since we won't need to model IP addresses). I've also removed the refreshable interface, since as Steve notes it doesn't work well with the way that resolvers currently use the mapping interface (that should be revisited too).

          In addition to the unit tests provided by Patrick, I tested on a small cluster and observed that the rack mappings I specified were picked up correctly.

          Show
          Tom White added a comment - I've created a simplified patch which only has the two-column layout (we can add the range variant later as needed - also, now we have Guava the implementation will be simplified since we won't need to model IP addresses). I've also removed the refreshable interface, since as Steve notes it doesn't work well with the way that resolvers currently use the mapping interface (that should be revisited too). In addition to the unit tests provided by Patrick, I tested on a small cluster and observed that the rack mappings I specified were picked up correctly.
          Hide
          Patrick Angeles added a comment -

          +1 on the patch.

          Thanks for picking this one up, Tom!

          Show
          Patrick Angeles added a comment - +1 on the patch. Thanks for picking this one up, Tom!
          Hide
          Steve Loughran added a comment -

          Tom, could you have the class extend the cached mapping, the way the scripted one does?

          This would mean that

          1. the code to manage hash tables are in there
          2. on .23+ it will automatically pick up any extra features put into the base class I added. E.g permitting providing info to callers about whether the config is multi-switch, allowing callers to get/dump the topology map for diagnostics, etc -yet the same code will work for both branches, and be a better starting point for adding refresh operations.
          3. those bits of the code that check for the mapping being a cached mapping won't wrap the mapping in another cache, so making it easier to do updates &C. As you note, that bit of RackResolver should be fixed at some point -but subclassing the cached mapper would do this automatically.
          Show
          Steve Loughran added a comment - Tom, could you have the class extend the cached mapping, the way the scripted one does? This would mean that the code to manage hash tables are in there on .23+ it will automatically pick up any extra features put into the base class I added. E.g permitting providing info to callers about whether the config is multi-switch, allowing callers to get/dump the topology map for diagnostics, etc -yet the same code will work for both branches, and be a better starting point for adding refresh operations. those bits of the code that check for the mapping being a cached mapping won't wrap the mapping in another cache, so making it easier to do updates &C. As you note, that bit of RackResolver should be fixed at some point -but subclassing the cached mapper would do this automatically.
          Hide
          Tom White added a comment -

          Updated patch to use CachedDNSToSwitchMapping.

          on .23+ it will automatically pick up any extra features put into the base class I added. E.g permitting providing info to callers about whether the config is multi-switch, allowing callers to get/dump the topology map for diagnostics

          As far as I can tell nothing calls these new methods. We don't normally add new methods unless they are used.

          Show
          Tom White added a comment - Updated patch to use CachedDNSToSwitchMapping. on .23+ it will automatically pick up any extra features put into the base class I added. E.g permitting providing info to callers about whether the config is multi-switch, allowing callers to get/dump the topology map for diagnostics As far as I can tell nothing calls these new methods. We don't normally add new methods unless they are used.
          Hide
          Steve Loughran added a comment -

          #the accessors are good for testing.
          #I've been slowly putting the changes into the code, but gradually and trying to avoid any form of problem w/existing custom topologies.

          What I do want to do -and they arent a jiras, I should add them are

          • A command line entry point to take a list of hostnames, look at the current config, tell you what they map to and list the final map. This lets you preflight check the operations.
          • A way of getting the dump of the topo map in use by a service, such as through one of the JSP pages. That way people can see what's gone wrong.

          Being able to get into the map is what you need for these.

          Show
          Steve Loughran added a comment - #the accessors are good for testing. #I've been slowly putting the changes into the code, but gradually and trying to avoid any form of problem w/existing custom topologies. What I do want to do -and they arent a jiras, I should add them are A command line entry point to take a list of hostnames, look at the current config, tell you what they map to and list the final map. This lets you preflight check the operations. A way of getting the dump of the topo map in use by a service, such as through one of the JSP pages. That way people can see what's gone wrong. Being able to get into the map is what you need for these.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519484/HADOOP-7030.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/745//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/745//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519484/HADOOP-7030.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/745//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/745//console This message is automatically generated.
          Hide
          Tom White added a comment -

          Steve - these are good features, but it's better to introduce API changes at the time the feature is added so that you and reviewers are sure the API fits the use case.

          Show
          Tom White added a comment - Steve - these are good features, but it's better to introduce API changes at the time the feature is added so that you and reviewers are sure the API fits the use case.
          Hide
          Alejandro Abdelnur added a comment -

          +1 after a few minor corrections:

          1. Reader creation simplification:

                  File file = new File(filename);
                  reader = new BufferedReader(new InputStreamReader(new FileInputStream(
                      file)));
          

          should be:

                  File file = new File(filename);
                  reader = new BufferedReader(new FileReader(filename));
          

          2. non-comment & invalid lines (not 2 columns) should write a log warning instead failing silently.

          3. rename refresh() to load() or initialize(). refresh() gives the impression of well refresh() and I'm not sure this is the intention.

          Show
          Alejandro Abdelnur added a comment - +1 after a few minor corrections: 1. Reader creation simplification: File file = new File(filename); reader = new BufferedReader( new InputStreamReader( new FileInputStream( file))); should be: File file = new File(filename); reader = new BufferedReader( new FileReader(filename)); 2. non-comment & invalid lines (not 2 columns) should write a log warning instead failing silently. 3. rename refresh() to load() or initialize(). refresh() gives the impression of well refresh() and I'm not sure this is the intention.
          Hide
          Tom White added a comment -

          New patch addressing Alejandro's feedback.

          Show
          Tom White added a comment - New patch addressing Alejandro's feedback.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519688/HADOOP-7030.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.ha.TestHealthMonitor

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/756//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/756//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519688/HADOOP-7030.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.ha.TestHealthMonitor +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/756//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/756//console This message is automatically generated.
          Hide
          Tom White added a comment -

          I just committed this. Thanks Patrick!

          I opened HADOOP-8204 for the test failure, which was unrelated to the change.

          Show
          Tom White added a comment - I just committed this. Thanks Patrick! I opened HADOOP-8204 for the test failure, which was unrelated to the change.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #711 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/711/)
          Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #711 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/711/ ) Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #721 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/721/)
          Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #721 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/721/ ) Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1922 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1922/)
          HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1922 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1922/ ) HADOOP-7030 . Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1996 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1996/)
          HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1996 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1996/ ) HADOOP-7030 . Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1931 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1931/)
          HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)

          Result = ABORTED
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1931 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1931/ ) HADOOP-7030 . Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597) Result = ABORTED tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #729 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/729/)
          Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)

          Result = ABORTED
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #729 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/729/ ) Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599) Result = ABORTED tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Tom White added a comment -

          Attaching a backport for branch-1.

          Show
          Tom White added a comment - Attaching a backport for branch-1.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #994 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/994/)
          HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)

          Result = FAILURE
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #994 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/994/ ) HADOOP-7030 . Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #207 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/207/)
          Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)

          Result = UNSTABLE
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #207 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/207/ ) Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599) Result = UNSTABLE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #235 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/235/)
          Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)

          Result = FAILURE
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #235 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/235/ ) Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1029 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1029/)
          HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)

          Result = SUCCESS
          tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1029 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1029/ ) HADOOP-7030 . Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
          Hide
          Matt Foley added a comment - - edited

          Patch not committed in time for 1.1.0.

          Show
          Matt Foley added a comment - - edited Patch not committed in time for 1.1.0.
          Hide
          Kang Xiao added a comment -

          tow question about the patch

          1.If new rack and host added to the cluster, it will resolve rack name to DEFAULT_RACK. It maybe better to allow reload of the config file.

          2.The following code set initialized to be true before load(). It may cause the config file not fully loaded if some exception throwed by load().

          +    public synchronized List<String> resolve(List<String> names) {
          +      if (!initialized) {
          +        initialized = true;
          +        load();
          +      }
          
          Show
          Kang Xiao added a comment - tow question about the patch 1.If new rack and host added to the cluster, it will resolve rack name to DEFAULT_RACK. It maybe better to allow reload of the config file. 2.The following code set initialized to be true before load(). It may cause the config file not fully loaded if some exception throwed by load(). + public synchronized List< String > resolve(List< String > names) { + if (!initialized) { + initialized = true ; + load(); + }
          Hide
          Steve Loughran added a comment -

          1. -conf file reload is a bigger issue/need than just this file, as it really requires cache reset and remote calls, but I've had a thought there specifically for the use case of "new host added". Filed as HADOOP-8629

          2: filed as HADOOP-8628.

          Show
          Steve Loughran added a comment - 1. -conf file reload is a bigger issue/need than just this file, as it really requires cache reset and remote calls, but I've had a thought there specifically for the use case of "new host added". Filed as HADOOP-8629 2: filed as HADOOP-8628 .
          Hide
          Matt Foley added a comment -

          moved target version to 1.2.0 upon publishing 1.1.1 RC.

          Show
          Matt Foley added a comment - moved target version to 1.2.0 upon publishing 1.1.1 RC.

            People

            • Assignee:
              Tom White
              Reporter:
              Patrick Angeles
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development