Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3497

ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: client
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      yarn-client's ContainerManagementProtocolProxy is updating ipc.client.connection.maxidletime in the conf passed in without making a copy of it. That modification "leaks" into other systems using the same conf and can cause them to setup RPC connections with a timeout of zero as well.

      1. YARN-3497.001.patch
        0.9 kB
        Jason Lowe
      2. YARN-3497.002.patch
        1 kB
        Jason Lowe

        Activity

        Hide
        jlowe Jason Lowe added a comment -

        This causes the Tez AM to reconnect to the RM on every AM heartbeat.

        Patch to create a copy of the configuration when initializing the proxy cache.

        Show
        jlowe Jason Lowe added a comment - This causes the Tez AM to reconnect to the RM on every AM heartbeat. Patch to create a copy of the configuration when initializing the proxy cache.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12725933/YARN-3497.001.patch
        against trunk revision 1fa8075.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7362//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7362//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725933/YARN-3497.001.patch against trunk revision 1fa8075. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7362//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7362//console This message is automatically generated.
        Hide
        jianhe Jian He added a comment -

        should we change below to use this.conf as well ?

        conf.setInt(
                  CommonConfigurationKeysPublic.IPC_CLIENT_CONNECTION_MAXIDLETIME_KEY,
                  0);
        
        Show
        jianhe Jian He added a comment - should we change below to use this.conf as well ? conf.setInt( CommonConfigurationKeysPublic.IPC_CLIENT_CONNECTION_MAXIDLETIME_KEY, 0);
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for the review, Jian! I updated the patch to use this.conf when setting the max idle time conf.

        Show
        jlowe Jason Lowe added a comment - Thanks for the review, Jian! I updated the patch to use this.conf when setting the max idle time conf.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12726573/YARN-3497.002.patch
        against trunk revision c17cd4f.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7397//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7397//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726573/YARN-3497.002.patch against trunk revision c17cd4f. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7397//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7397//console This message is automatically generated.
        Hide
        jianhe Jian He added a comment -

        looks good , + 1

        Show
        jianhe Jian He added a comment - looks good , + 1
        Hide
        jianhe Jian He added a comment -

        Committed to trunk, branch-2, branch-2,7. thanks Jason !

        Show
        jianhe Jian He added a comment - Committed to trunk, branch-2, branch-2,7. thanks Jason !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7617 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7617/)
        YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7617 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7617/ ) YARN-3497 . ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #170 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/170/)
        YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #170 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/170/ ) YARN-3497 . ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #904 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/904/)
        YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #904 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/904/ ) YARN-3497 . ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2102/)
        YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2102 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2102/ ) YARN-3497 . ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #161 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/161/)
        YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #161 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/161/ ) YARN-3497 . ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/171/)
        YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #171 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/171/ ) YARN-3497 . ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2120 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2120/)
        YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2120 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2120/ ) YARN-3497 . ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe (jianhe: rev f967fd2f21791c5c4a5a090cc14ee88d155d2e2b) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/ContainerManagementProtocolProxy.java hadoop-yarn-project/CHANGES.txt

          People

          • Assignee:
            jlowe Jason Lowe
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development