Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4024

YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: resourcemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, YARN RM NodesListManager will resolve IP address every time when node doing heartbeat. When DNS server becomes slow, NM heartbeat will be blocked and cannot make progress.

      1. YARN-4024-draft.patch
        8 kB
        Hong Zhiguo
      2. YARN-4024-draft-v2.patch
        10 kB
        Hong Zhiguo
      3. YARN-4024-draft-v3.patch
        10 kB
        Hong Zhiguo
      4. YARN-4024-v4.patch
        12 kB
        Hong Zhiguo
      5. YARN-4024-v5.patch
        13 kB
        Hong Zhiguo
      6. YARN-4024-v6.patch
        14 kB
        Hong Zhiguo
      7. YARN-4024-v7.patch
        14 kB
        Hong Zhiguo

        Activity

        Hide
        leftnoteasy Wangda Tan added a comment -

        Actually, there're two problems we need to solve:
        1. DNS resolve hostname is under synchronized lock of hostsReader:
        In NodesListManager:

          public boolean isValidNode(String hostName) {
            synchronized (hostsReader) {
              Set<String> hostsList = hostsReader.getHosts();
              Set<String> excludeList = hostsReader.getExcludedHosts();
              String ip = NetUtils.normalizeHostName(hostName);
              return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList
                  .contains(ip))
                  && !(excludeList.contains(hostName) || excludeList.contains(ip));
            }
          }
        

        We should take NetUtils.normalizeHostName(hostName); out of the synchronized lock.

        2. We may need to cache IP address when NMs doing registering.
        If we assume IP address won't change after NM registered, we can cache resolved IP address for each host. We can add a switch to turn it on when we needed.

        Show
        leftnoteasy Wangda Tan added a comment - Actually, there're two problems we need to solve: 1. DNS resolve hostname is under synchronized lock of hostsReader: In NodesListManager: public boolean isValidNode( String hostName) { synchronized (hostsReader) { Set< String > hostsList = hostsReader.getHosts(); Set< String > excludeList = hostsReader.getExcludedHosts(); String ip = NetUtils.normalizeHostName(hostName); return (hostsList.isEmpty() || hostsList.contains(hostName) || hostsList .contains(ip)) && !(excludeList.contains(hostName) || excludeList.contains(ip)); } } We should take NetUtils.normalizeHostName(hostName); out of the synchronized lock. 2. We may need to cache IP address when NMs doing registering. If we assume IP address won't change after NM registered, we can cache resolved IP address for each host. We can add a switch to turn it on when we needed.
        Hide
        sunilg Sunil G added a comment -

        Hi Wangda Tan
        Thank you for bringing this, which is a potential area to cause delay n heartbeat.

        I feel we can cache IP addresses in RM side during NM registration, and can look into this storage as needed. Also apart from switch on/off this, cud we try to resolve IP if this cache lookup fails and can add to this list.
        So a basic lookup in cache during heartbeat, and upon failure we can resolve using existing way. A possible distant problem is that some one uses proxy server ahead of NM and which changes IP every time. I feel it is too negative case.

        How do you feel?

        Show
        sunilg Sunil G added a comment - Hi Wangda Tan Thank you for bringing this, which is a potential area to cause delay n heartbeat. I feel we can cache IP addresses in RM side during NM registration, and can look into this storage as needed. Also apart from switch on/off this, cud we try to resolve IP if this cache lookup fails and can add to this list. So a basic lookup in cache during heartbeat, and upon failure we can resolve using existing way. A possible distant problem is that some one uses proxy server ahead of NM and which changes IP every time. I feel it is too negative case. How do you feel?
        Hide
        zxu zhihai xu added a comment -

        Does this issue duplicate with YARN-4001?

        Show
        zxu zhihai xu added a comment - Does this issue duplicate with YARN-4001 ?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Sunil G,
        Thanks for comment, I'm not sure what's the "cache lookup fails".

        There're two different kinds of cache lookup fails. One is the IP doesn't in the cache, we definitely need to re-resolve the address. Another one is the resolved IP is not a valid host according to hostsReader, there're two different cases:

        1) If a host_a, has IP=IP1, IP1 is on whitelist. If we change the IP of host_a to IP2, IP2 is in blacklist. We won't do the re-resolve since the cached IP1 is on whitelist.
        2) If a host_a, has IP=IP1, IP1 is on blacklist. We may need to do re-resolve every time when the node doing heartbeat since it may change to its IP to a one not on the blacklist.

        So my thinking on this is: there should be a switch to control this, when a node's IP won't change OR there's no black/white node list, we should do caching, otherwise we need do resolving for every node heartbeat.

        Thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Sunil G , Thanks for comment, I'm not sure what's the "cache lookup fails". There're two different kinds of cache lookup fails. One is the IP doesn't in the cache, we definitely need to re-resolve the address. Another one is the resolved IP is not a valid host according to hostsReader, there're two different cases: 1) If a host_a, has IP=IP1, IP1 is on whitelist. If we change the IP of host_a to IP2, IP2 is in blacklist. We won't do the re-resolve since the cached IP1 is on whitelist. 2) If a host_a, has IP=IP1, IP1 is on blacklist. We may need to do re-resolve every time when the node doing heartbeat since it may change to its IP to a one not on the blacklist. So my thinking on this is: there should be a switch to control this, when a node's IP won't change OR there's no black/white node list, we should do caching, otherwise we need do resolving for every node heartbeat. Thoughts?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks for pointing YARN-4001, yes, they're same issue. Since this one has more discussion, I suggest to keep this one open and close YARN-4001. I'm making me unassigned.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks for pointing YARN-4001 , yes, they're same issue. Since this one has more discussion, I suggest to keep this one open and close YARN-4001 . I'm making me unassigned.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Hong Zhiguo, I noticed YARN-4001 after filed YARN-4024, since we have some discussions on YARN-4024 already. I suggest to close YARN-4001 and feel free to assign this one to you.

        Show
        leftnoteasy Wangda Tan added a comment - Hong Zhiguo , I noticed YARN-4001 after filed YARN-4024 , since we have some discussions on YARN-4024 already. I suggest to close YARN-4001 and feel free to assign this one to you.
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan for sharing detailed case.

        Yes, what I meant is a case where lookup fails and we need to resolve again. And further to this then we ll end up in having black/white level issue.
        I feel there will not be much of a problem to do only caching provided we need to clear cache from some external way once a threshold is reached. This is to save too much of invalid IPs as in your case. Will it be fine?

        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan for sharing detailed case. Yes, what I meant is a case where lookup fails and we need to resolve again. And further to this then we ll end up in having black/white level issue. I feel there will not be much of a problem to do only caching provided we need to clear cache from some external way once a threshold is reached. This is to save too much of invalid IPs as in your case. Will it be fine?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks for sharing your thoughts, Sunil G.
        I think we can clear cache every X seconds, it will be an option like max-ip-caching time, and -1 will disable cache. Sounds like a plan?

        Show
        leftnoteasy Wangda Tan added a comment - Thanks for sharing your thoughts, Sunil G . I think we can clear cache every X seconds, it will be an option like max-ip-caching time, and -1 will disable cache. Sounds like a plan?
        Hide
        sunilg Sunil G added a comment -

        +1. Yes. That will be fine. I feel we can keep unit as mins may be.

        Show
        sunilg Sunil G added a comment - +1. Yes. That will be fine. I feel we can keep unit as mins may be.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        We've did this one year ago in our 5k+ cluster. Can I take this issue?

        Show
        zhiguohong Hong Zhiguo added a comment - We've did this one year ago in our 5k+ cluster. Can I take this issue?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Hong Zhiguo, sure, please go ahead!

        Show
        leftnoteasy Wangda Tan added a comment - Hong Zhiguo , sure, please go ahead!
        Hide
        zhiguohong Hong Zhiguo added a comment -

        There's DNS cache in InetAddress. What's the benefit to have another layer of cache in memory? Maybe easier to control?

        Show
        zhiguohong Hong Zhiguo added a comment - There's DNS cache in InetAddress. What's the benefit to have another layer of cache in memory? Maybe easier to control?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Hong Zhiguo, the DNS cache is a global parameter for a JVM, correct? IMHO, we shouldn't use the global parameter, because RM may need to get latest IP address from DNS for other purpose. For example, RM needs to get latest address when NMs are registering (and also reconnect), but it may not need it when NMs is running.

        Thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Hong Zhiguo , the DNS cache is a global parameter for a JVM, correct? IMHO, we shouldn't use the global parameter, because RM may need to get latest IP address from DNS for other purpose. For example, RM needs to get latest address when NMs are registering (and also reconnect), but it may not need it when NMs is running. Thoughts?
        Hide
        zhiguohong Hong Zhiguo added a comment -

        That's a good reason to have this cache.
        Wangda Tan, in earlier comments, you said

        1) If a host_a, has IP=IP1, IP1 is on whitelist. If we change the IP of host_a to IP2, IP2 is in blacklist. We won't do the re-resolve since the cached IP1 is on whitelist.
        2) If a host_a, has IP=IP1, IP1 is on blacklist. We may need to do re-resolve every time when the node doing heartbeat since it may change to its IP to a one not on the blacklist.
        

        I think that's too complicated. The cache lookup is a part of resolving (name to address). And the check of IP whitelist/blacklist is just the following stage. I think cache with configurable expiration is enough, we'd better leave the 2 stages orthogonal, not to mix them up.

        BTW, I think it's not good to have "Name" in NodeId, but "Address" in whitelist/blacklist. Different layers of abstraction are mixed up. We'll don't have this issue if "Name" or "Address" is used for both NodeId and whitelist/blacklist.
        a better way is to have "Name" in whitelist/blacklist, instead of "Address".

        Show
        zhiguohong Hong Zhiguo added a comment - That's a good reason to have this cache. Wangda Tan , in earlier comments, you said 1) If a host_a, has IP=IP1, IP1 is on whitelist. If we change the IP of host_a to IP2, IP2 is in blacklist. We won't do the re-resolve since the cached IP1 is on whitelist. 2) If a host_a, has IP=IP1, IP1 is on blacklist. We may need to do re-resolve every time when the node doing heartbeat since it may change to its IP to a one not on the blacklist. I think that's too complicated. The cache lookup is a part of resolving (name to address). And the check of IP whitelist/blacklist is just the following stage. I think cache with configurable expiration is enough, we'd better leave the 2 stages orthogonal, not to mix them up. BTW, I think it's not good to have "Name" in NodeId, but "Address" in whitelist/blacklist. Different layers of abstraction are mixed up. We'll don't have this issue if "Name" or "Address" is used for both NodeId and whitelist/blacklist. a better way is to have "Name" in whitelist/blacklist, instead of "Address".
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Please ignore the last sentence "a better way is to have "Name" in whitelist/blacklist, instead of "Address"". Or could someone help to delete it.

        Show
        zhiguohong Hong Zhiguo added a comment - Please ignore the last sentence "a better way is to have "Name" in whitelist/blacklist, instead of "Address"". Or could someone help to delete it.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Add an configuration option "yarn.resourcemanager.node-ip-cache.expiry-interval-secs", while -1 disables caching.

        Show
        zhiguohong Hong Zhiguo added a comment - Add an configuration option "yarn.resourcemanager.node-ip-cache.expiry-interval-secs", while -1 disables caching.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        In this patch, both positive and negative lookup result is cached and has the same expiry interval.

        Show
        zhiguohong Hong Zhiguo added a comment - In this patch, both positive and negative lookup result is cached and has the same expiry interval.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 27s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 44s There were no new javac warning messages.
        +1 javadoc 9m 44s There were no new javadoc warning messages.
        +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 29s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 22s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 2m 59s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 0m 19s Tests failed in hadoop-yarn-api.
        -1 yarn tests 52m 58s Tests failed in hadoop-yarn-server-resourcemanager.
            95m 12s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager
        Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
          hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12750780/YARN-4024-draft.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 13604bd
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8861/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8861/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8861/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8861/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8861/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8861/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 27s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 29s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 2m 59s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. -1 yarn tests 0m 19s Tests failed in hadoop-yarn-api. -1 yarn tests 52m 58s Tests failed in hadoop-yarn-server-resourcemanager.     95m 12s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750780/YARN-4024-draft.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 13604bd checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8861/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8861/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8861/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8861/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8861/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8861/console This message was automatically generated.
        Hide
        wilsoncraft Allan Wilson added a comment -

        When a nodemanager is decommissioned, is the IP cached for that host flushed out of the cache? Normally when a host gets a new IP its because it gets moved or some other deliberate maintenance which would normally be preceded by a decommission. If the IP is flushed when decommissioned or a IP is always resolved from the host name when a new or recommissioned nodemanager is added to the cluster I think that would be adequate IMHO.

        Also, it may be worthwhile or adequate to expose the method in a yarn rmadin command to force a flush of the IP cache. Is this IP cache the same used for Rack Awareness by the RM?

        Thanks

        Show
        wilsoncraft Allan Wilson added a comment - When a nodemanager is decommissioned, is the IP cached for that host flushed out of the cache? Normally when a host gets a new IP its because it gets moved or some other deliberate maintenance which would normally be preceded by a decommission. If the IP is flushed when decommissioned or a IP is always resolved from the host name when a new or recommissioned nodemanager is added to the cluster I think that would be adequate IMHO. Also, it may be worthwhile or adequate to expose the method in a yarn rmadin command to force a flush of the IP cache. Is this IP cache the same used for Rack Awareness by the RM? Thanks
        Hide
        leftnoteasy Wangda Tan added a comment -

        Hi Hong Zhiguo,

        Thanks for working on this,
        For your comments:

        I think that's too complicated...

        Agree, I changed my idea, see https://issues.apache.org/jira/browse/YARN-4024?focusedCommentId=14660607&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14660607.

        Approach in your patch general looks good to me, few suggestions:
        1) When node becomes NODE_UNUSABLE/NODE_USABLE, I suggest remove them from the cache to force update its ip, since a node status change will (likely) update its ip. So this may require update the Resolver interface
        2) seconds -> something like normalizedHostnameCacheTimeout

        Allan Wilson,

        When a nodemanager is decommissioned, is the IP cached for that host flushed out of the cache? Normally when a host gets a new IP its because it gets moved or some other deliberate maintenance which would normally be preceded by a decommission. If the IP is flushed when decommissioned or a IP is always resolved from the host name when a new or recommissioned nodemanager is added to the cluster I think that would be adequate IMHO.

        I'm not quite sure about what did you mean, does my comment solve the problem you meantioned?

        1) When node becomes NODE_UNUSABLE/NODE_USABLE, I suggest remove them from the cache to force update its ip, since a node status change will (likely) update its ip. So this may require update the Resolver interface

        Also, it may be worthwhile or adequate to expose the method in a yarn rmadin command to force a flush of the IP cache. Is this IP cache the same used for Rack Awareness by the RM?

        I prefer keep this to be an internal behavior, this won't be used to determine rack IIUC.

        Please let me know your thoughts.

        Thanks,
        Wangda

        Show
        leftnoteasy Wangda Tan added a comment - Hi Hong Zhiguo , Thanks for working on this, For your comments: I think that's too complicated... Agree, I changed my idea, see https://issues.apache.org/jira/browse/YARN-4024?focusedCommentId=14660607&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14660607 . Approach in your patch general looks good to me, few suggestions: 1) When node becomes NODE_UNUSABLE/NODE_USABLE, I suggest remove them from the cache to force update its ip, since a node status change will (likely) update its ip. So this may require update the Resolver interface 2) seconds -> something like normalizedHostnameCacheTimeout Allan Wilson , When a nodemanager is decommissioned, is the IP cached for that host flushed out of the cache? Normally when a host gets a new IP its because it gets moved or some other deliberate maintenance which would normally be preceded by a decommission. If the IP is flushed when decommissioned or a IP is always resolved from the host name when a new or recommissioned nodemanager is added to the cluster I think that would be adequate IMHO. I'm not quite sure about what did you mean, does my comment solve the problem you meantioned? 1) When node becomes NODE_UNUSABLE/NODE_USABLE, I suggest remove them from the cache to force update its ip, since a node status change will (likely) update its ip. So this may require update the Resolver interface Also, it may be worthwhile or adequate to expose the method in a yarn rmadin command to force a flush of the IP cache. Is this IP cache the same used for Rack Awareness by the RM? I prefer keep this to be an internal behavior, this won't be used to determine rack IIUC. Please let me know your thoughts. Thanks, Wangda
        Hide
        zhiguohong Hong Zhiguo added a comment -

        updated the patch with flushing when node state is transiting between USABLE and UNUSABLE.

        Show
        zhiguohong Hong Zhiguo added a comment - updated the patch with flushing when node state is transiting between USABLE and UNUSABLE.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 23s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 0s There were no new javac warning messages.
        +1 javadoc 9m 55s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 38s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 26s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        -1 findbugs 3m 9s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 0m 21s Tests failed in hadoop-yarn-api.
        -1 yarn tests 53m 34s Tests failed in hadoop-yarn-server-resourcemanager.
            97m 42s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager
        Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
          hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12750953/YARN-4024-draft-v2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 71566e2
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8872/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8872/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 0s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 38s The applied patch generated 1 new checkstyle issues (total was 211, now 211). -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 3m 9s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. -1 yarn tests 0m 21s Tests failed in hadoop-yarn-api. -1 yarn tests 53m 34s Tests failed in hadoop-yarn-server-resourcemanager.     97m 42s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750953/YARN-4024-draft-v2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71566e2 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8872/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8872/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8872/console This message was automatically generated.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        YARN-4024-draft-v3.patch: fix the checkstyle warning and testcase failure

        Show
        zhiguohong Hong Zhiguo added a comment - YARN-4024 -draft-v3.patch: fix the checkstyle warning and testcase failure
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 1s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 46s There were no new javac warning messages.
        +1 javadoc 9m 40s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 23s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        -1 findbugs 4m 37s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
        -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.
        -1 yarn tests 57m 27s Tests failed in hadoop-yarn-server-resourcemanager.
            105m 51s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager
        Failed unit tests hadoop.yarn.util.TestRackResolver
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12751009/YARN-4024-draft-v3.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 71566e2
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8875/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8875/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 4m 37s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common. -1 yarn tests 57m 27s Tests failed in hadoop-yarn-server-resourcemanager.     105m 51s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.util.TestRackResolver   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751009/YARN-4024-draft-v3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71566e2 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8875/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8875/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8875/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Hi Hong Zhiguo,
        Thanks for update, some minor comments:
        1) I think we can limit the changes of remove cache in the NodesListManager, in the handle(..), we can do the flush(..), it will be as same as doing this in RMNodeImpl, and don't need expose an extra method, correct?

        2) I suggest to rename CachedResolver.flush to something like removeCache, flush is more like a file system concept to me.

        3) Add tests to see if NodesListManager can handle events correctly if you agree with 2).

        Show
        leftnoteasy Wangda Tan added a comment - Hi Hong Zhiguo , Thanks for update, some minor comments: 1) I think we can limit the changes of remove cache in the NodesListManager, in the handle(..), we can do the flush(..), it will be as same as doing this in RMNodeImpl, and don't need expose an extra method, correct? 2) I suggest to rename CachedResolver.flush to something like removeCache, flush is more like a file system concept to me. 3) Add tests to see if NodesListManager can handle events correctly if you agree with 2).
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Thanks for your comments, Wangda Tan. I didn't notice there's already such events. I updated the patch accordingly.

        Show
        zhiguohong Hong Zhiguo added a comment - Thanks for your comments, Wangda Tan . I didn't notice there's already such events. I updated the patch accordingly.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 17m 35s Findbugs (version ) appears to be broken on trunk.
        +1 @author 0m 1s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 58s There were no new javac warning messages.
        +1 javadoc 9m 57s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 29s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 26s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 4m 44s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
        -1 yarn tests 2m 0s Tests failed in hadoop-yarn-common.
        -1 yarn tests 53m 59s Tests failed in hadoop-yarn-server-resourcemanager.
            101m 2s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager
        Failed unit tests hadoop.yarn.util.TestRackResolver
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12751205/YARN-4024-v4.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 22dc5fc
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8883/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8883/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 35s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 1s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 29s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 4m 44s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 2m 0s Tests failed in hadoop-yarn-common. -1 yarn tests 53m 59s Tests failed in hadoop-yarn-server-resourcemanager.     101m 2s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.util.TestRackResolver   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751205/YARN-4024-v4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 22dc5fc checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8883/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8883/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8883/console This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Hi Hong Zhiguo
        Thanks for working on this. Some high level comments.

        • I feel ExpireChecker could extend Timer class and it can be started from serviceStart of NodesListManager, and can be cancelled from serviceStop. Pls share your opinion.
        • Since we use removeCache, I think update will be better suited as addToCache
        • Please add more comments and details about interface Resolver and its api. I feel the api resolve has to be UnStable and Public for now. may be we can separate the interface to another file.
        Show
        sunilg Sunil G added a comment - Hi Hong Zhiguo Thanks for working on this. Some high level comments. I feel ExpireChecker could extend Timer class and it can be started from serviceStart of NodesListManager, and can be cancelled from serviceStop. Pls share your opinion. Since we use removeCache , I think update will be better suited as addToCache Please add more comments and details about interface Resolver and its api. I feel the api resolve has to be UnStable and Public for now. may be we can separate the interface to another file.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks update, Hong Zhiguo and comments from Sunil G.

        Since we use removeCache, I think update will be better suited as addToCache

        +1, I think it's better to rename it to removeFromCache.

        Please add more comments and details about interface Resolver and its api. I feel the api resolve has to be UnStable and Public for now. may be we can separate the interface to another file.

        Since it's an internal only interface, I think you should remove public for them. And no need to add @Unstable and @public for such internal interfaces.

        LOG.debug("[" +...

        Should be wrapped by isDebugEnabled

        Regarding to tests, I think it maybe easier to expose a getResolver (just default accessibility is fine) and mark it to be @VisibleForTesting like other tests.

        And it may be important to add a test to make sure DirectResolver will be created to avoid future possible regression.

        And I think it's better to modify check if (nodeIpCacheTimeout == -1) { to be <= 0, since it doesn't make sense to have a timeout <= 0 for CachedResolver.

        Thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Thanks update, Hong Zhiguo and comments from Sunil G . Since we use removeCache, I think update will be better suited as addToCache +1, I think it's better to rename it to removeFromCache. Please add more comments and details about interface Resolver and its api. I feel the api resolve has to be UnStable and Public for now. may be we can separate the interface to another file. Since it's an internal only interface, I think you should remove public for them. And no need to add @Unstable and @public for such internal interfaces. LOG.debug("[" +... Should be wrapped by isDebugEnabled Regarding to tests, I think it maybe easier to expose a getResolver (just default accessibility is fine) and mark it to be @VisibleForTesting like other tests. And it may be important to add a test to make sure DirectResolver will be created to avoid future possible regression. And I think it's better to modify check if (nodeIpCacheTimeout == -1) { to be <= 0, since it doesn't make sense to have a timeout <= 0 for CachedResolver. Thoughts?
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Thanks for your comments, Sunil G and Wangda Tan. I updated patch v5 accordingly.
        Interface Resolver is left as public with @VisibleForTesting, because it's accessed from test cases.

        Show
        zhiguohong Hong Zhiguo added a comment - Thanks for your comments, Sunil G and Wangda Tan . I updated patch v5 accordingly. Interface Resolver is left as public with @VisibleForTesting, because it's accessed from test cases.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Hong Zhiguo, thanks for update, patch generally looks good, could you take a look at findbugs warning?

        FindBugs module:hadoop-yarn-server-resourcemanager

        Sometimes it shows 0 findbugs when you click at the findbugs report link, but it has some bugs. You can go to yarn-resourcemanager project to run "mvn clean findbugs:findbugs".

        Show
        leftnoteasy Wangda Tan added a comment - Hong Zhiguo , thanks for update, patch generally looks good, could you take a look at findbugs warning? FindBugs module:hadoop-yarn-server-resourcemanager Sometimes it shows 0 findbugs when you click at the findbugs report link, but it has some bugs. You can go to yarn-resourcemanager project to run "mvn clean findbugs:findbugs".
        Hide
        zhiguohong Hong Zhiguo added a comment -

        the findbugs warning is about "unchecked rawtypes" in AMLivelinessMonitor.java. I fixed it in the v6 patch.

        Show
        zhiguohong Hong Zhiguo added a comment - the findbugs warning is about "unchecked rawtypes" in AMLivelinessMonitor.java. I fixed it in the v6 patch.
        Hide
        leftnoteasy Wangda Tan added a comment -

        By some reason Jenkins wasn't triggerred, manual triggered Jenkins.

        Show
        leftnoteasy Wangda Tan added a comment - By some reason Jenkins wasn't triggerred, manual triggered Jenkins.
        Hide
        adhoot Anubhav Dhoot added a comment -

        Hi Hong Zhiguo the fix looks good.
        We can avoid adding a sleep to the test if we use Clock instead of System in CachedResolver#addToCache and ExpireChecker#run. That way you can use a ControlledClock in the test to manipulate time and verify expiry.
        Another minor nit, the assertEquals has an "" as first argument in many places in TestNodesListManager. You can remove this argument completely since the other overload does the same thing or replace with a proper string if you wish.

        Show
        adhoot Anubhav Dhoot added a comment - Hi Hong Zhiguo the fix looks good. We can avoid adding a sleep to the test if we use Clock instead of System in CachedResolver#addToCache and ExpireChecker#run. That way you can use a ControlledClock in the test to manipulate time and verify expiry. Another minor nit, the assertEquals has an "" as first argument in many places in TestNodesListManager. You can remove this argument completely since the other overload does the same thing or replace with a proper string if you wish.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 19s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 2s There were no new javac warning messages.
        +1 javadoc 9m 53s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 59s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 4m 39s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
        +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.
        -1 yarn tests 53m 32s Tests failed in hadoop-yarn-server-resourcemanager.
            102m 55s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager
        Failed unit tests hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
          hadoop.yarn.server.resourcemanager.TestRMAdminService
        Timed out tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12751636/YARN-4024-v6.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 22de7c1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8893/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8893/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 19s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 2s There were no new javac warning messages. +1 javadoc 9m 53s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 59s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 4m 39s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common. -1 yarn tests 53m 32s Tests failed in hadoop-yarn-server-resourcemanager.     102m 55s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.server.resourcemanager.security.TestAMRMTokens   hadoop.yarn.server.resourcemanager.TestRMAdminService Timed out tests org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751636/YARN-4024-v6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 22de7c1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8893/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8893/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8893/console This message was automatically generated.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Thanks for your comments, Anubhav Dhoot, updated the patch.

        Show
        zhiguohong Hong Zhiguo added a comment - Thanks for your comments, Anubhav Dhoot , updated the patch.
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Why jenkins doesn't run against the latest patch?

        Show
        zhiguohong Hong Zhiguo added a comment - Why jenkins doesn't run against the latest patch?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 7s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 41s There were no new javac warning messages.
        +1 javadoc 9m 45s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 2m 0s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 30s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 4m 40s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
        +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.
        +1 yarn tests 53m 33s Tests passed in hadoop-yarn-server-resourcemanager.
            102m 19s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12751913/YARN-4024-v7.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / e2c9b28
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8943/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8943/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 7s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 45s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 0s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 4m 40s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common. +1 yarn tests 53m 33s Tests passed in hadoop-yarn-server-resourcemanager.     102m 19s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751913/YARN-4024-v7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e2c9b28 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8943/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8943/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8943/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Latest patch LGTM, Anubhav Dhoot, any comments?

        Show
        leftnoteasy Wangda Tan added a comment - Latest patch LGTM, Anubhav Dhoot , any comments?
        Hide
        adhoot Anubhav Dhoot added a comment -

        LGTM. Thanks for removing those sleep

        Show
        adhoot Anubhav Dhoot added a comment - LGTM. Thanks for removing those sleep
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rekicked Jenkins, will commit once Jenkins get back.

        Show
        leftnoteasy Wangda Tan added a comment - Rekicked Jenkins, will commit once Jenkins get back.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 21m 41s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 46s There were no new javac warning messages.
        +1 javadoc 10m 3s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 56s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 2s The patch has no lines that end in whitespace.
        +1 install 1m 28s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 4m 38s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
        +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.
        -1 yarn tests 46m 59s Tests failed in hadoop-yarn-server-resourcemanager.
            98m 37s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager
        Failed unit tests hadoop.yarn.server.resourcemanager.TestRMRestart
        Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestAppManager
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12751913/YARN-4024-v7.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / c006c3a
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-resourcemanager.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8986/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8986/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 41s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 javadoc 10m 3s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 56s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 4m 38s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common. -1 yarn tests 46m 59s Tests failed in hadoop-yarn-server-resourcemanager.     98m 37s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.server.resourcemanager.TestRMRestart Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestAppManager   org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751913/YARN-4024-v7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c006c3a Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-resourcemanager.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8986/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8986/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8986/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Strange test failures, rekicked Jenkins again..

        Show
        leftnoteasy Wangda Tan added a comment - Strange test failures, rekicked Jenkins again..
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 pre-patch 19m 50s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 2s There were no new javac warning messages.
        +1 javadoc 10m 8s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
        +1 whitespace 0m 1s The patch has no lines that end in whitespace.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        -1 findbugs 4m 44s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
        +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common.
        +1 yarn tests 58m 16s Tests passed in hadoop-yarn-server-resourcemanager.
            108m 31s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12751913/YARN-4024-v7.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / d31a41c
        Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-resourcemanager.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8990/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8990/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 50s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 2s There were no new javac warning messages. +1 javadoc 10m 8s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 4m 44s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common. +1 yarn tests 58m 16s Tests passed in hadoop-yarn-server-resourcemanager.     108m 31s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751913/YARN-4024-v7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d31a41c Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-resourcemanager.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8990/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8990/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8990/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to trunk/branch-2, thanks Hong Zhiguo and review from Sunil G/Anubhav Dhoot!

        Show
        leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2, thanks Hong Zhiguo and review from Sunil G / Anubhav Dhoot !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8407 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8407/)
        YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8407 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8407/ ) YARN-4024 . YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #349 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/349/)
        YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #349 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/349/ ) YARN-4024 . YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #355 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/355/)
        YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #355 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/355/ ) YARN-4024 . YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1087 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1087/)
        YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1087 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1087/ ) YARN-4024 . YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #337 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/337/)
        YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #337 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/337/ ) YARN-4024 . YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2298 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2298/)
        YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2298 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2298/ ) YARN-4024 . YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2276 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2276/)
        YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2276 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2276/ ) YARN-4024 . YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) (wangda: rev bcc85e3bab78bcacd430eac23141774465b96ef9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/NodesListManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestNodesListManager.java
        Hide
        mingma Ming Ma added a comment -

        Thanks for the good improvement Wangda Tan, Hong Zhiguo, Sunil G, Anubhav Dhoot! For cache timeout interval, should we change the semantics of -1 as "cache forever" and 0 as "no cache" to be more consistent with JVM setting "networkaddress.cache.ttl"?

        Show
        mingma Ming Ma added a comment - Thanks for the good improvement Wangda Tan , Hong Zhiguo , Sunil G , Anubhav Dhoot ! For cache timeout interval, should we change the semantics of -1 as "cache forever" and 0 as "no cache" to be more consistent with JVM setting "networkaddress.cache.ttl"?
        Hide
        zhiguohong Hong Zhiguo added a comment -

        Thanks for your good point. Yes I can do it. Should I reopen this issue and post a new patch?

        Show
        zhiguohong Hong Zhiguo added a comment - Thanks for your good point. Yes I can do it. Should I reopen this issue and post a new patch?
        Hide
        varun_saxena Varun Saxena added a comment -

        Sorry assigned JIRA to myself by mistake. Assigned it back to Hong Zhiguo.

        Show
        varun_saxena Varun Saxena added a comment - Sorry assigned JIRA to myself by mistake. Assigned it back to Hong Zhiguo .

          People

          • Assignee:
            zhiguohong Hong Zhiguo
            Reporter:
            leftnoteasy Wangda Tan
          • Votes:
            1 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development