Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7053

Move curator transaction support to ZKCuratorManager

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore has some curator transaction support which can be reused, so this can be moved to ZKCuratorManager as well.

      1. YARN-7053.005.patch
        21 kB
        Jonathan Hung
      2. YARN-7053.004.patch
        27 kB
        Jonathan Hung
      3. YARN-7053.003.patch
        21 kB
        Jonathan Hung
      4. YARN-7053.002.patch
        18 kB
        Jonathan Hung
      5. YARN-7053.001.patch
        18 kB
        Jonathan Hung

        Issue Links

          Activity

          Hide
          jhung Jonathan Hung added a comment -

          Attached 001 patch which moves SafeTransaction and related methods from ZKRMStateStore to ZKCuratorManager.

          Inigo/Subru Krishnan, do you mind taking a look? Thanks!

          Show
          jhung Jonathan Hung added a comment - Attached 001 patch which moves SafeTransaction and related methods from ZKRMStateStore to ZKCuratorManager. Inigo / Subru Krishnan , do you mind taking a look? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          0 mvndep 1m 25s Maven dependency ordering for branch
          +1 mvninstall 13m 32s trunk passed
          +1 compile 13m 31s trunk passed
          +1 checkstyle 1m 54s trunk passed
          +1 mvnsite 2m 11s trunk passed
          +1 findbugs 2m 29s trunk passed
          +1 javadoc 1m 26s trunk passed
                Patch Compile Tests
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 1m 14s the patch passed
          +1 compile 10m 27s the patch passed
          +1 javac 10m 27s the patch passed
          -0 checkstyle 1m 47s root: The patch generated 6 new + 1 unchanged - 0 fixed = 7 total (was 1)
          +1 mvnsite 2m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 52s the patch passed
          +1 javadoc 1m 36s the patch passed
                Other Tests
          -1 unit 7m 7s hadoop-common in the patch failed.
          -1 unit 45m 13s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          133m 32s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-7053
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882697/YARN-7053.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a8258d022231 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8991f0b
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17002/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17002/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17002/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17002/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17002/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests 0 mvndep 1m 25s Maven dependency ordering for branch +1 mvninstall 13m 32s trunk passed +1 compile 13m 31s trunk passed +1 checkstyle 1m 54s trunk passed +1 mvnsite 2m 11s trunk passed +1 findbugs 2m 29s trunk passed +1 javadoc 1m 26s trunk passed       Patch Compile Tests 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 10m 27s the patch passed +1 javac 10m 27s the patch passed -0 checkstyle 1m 47s root: The patch generated 6 new + 1 unchanged - 0 fixed = 7 total (was 1) +1 mvnsite 2m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 52s the patch passed +1 javadoc 1m 36s the patch passed       Other Tests -1 unit 7m 7s hadoop-common in the patch failed. -1 unit 45m 13s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 133m 32s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-7053 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882697/YARN-7053.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a8258d022231 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8991f0b Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17002/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17002/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17002/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17002/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/17002/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jhung Jonathan Hung added a comment -

          002 patch fixes checkstyle

          Show
          jhung Jonathan Hung added a comment - 002 patch fixes checkstyle
          Hide
          subru Subru Krishnan added a comment -

          Thanks Jonathan Hung for working on this. The patch looks good, couple of minor comments:

          • Is there any test to move or add?
          • Can you update ZookeeperFederationStateStore to also use the refactored transactions?
          Show
          subru Subru Krishnan added a comment - Thanks Jonathan Hung for working on this. The patch looks good, couple of minor comments: Is there any test to move or add? Can you update ZookeeperFederationStateStore to also use the refactored transactions?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 13m 12s trunk passed
          +1 compile 13m 33s trunk passed
          +1 checkstyle 1m 55s trunk passed
          +1 mvnsite 2m 5s trunk passed
          +1 findbugs 2m 31s trunk passed
          +1 javadoc 1m 7s trunk passed
                Patch Compile Tests
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 9s the patch passed
          +1 compile 10m 21s the patch passed
          +1 javac 10m 21s the patch passed
          +1 checkstyle 1m 44s the patch passed
          +1 mvnsite 1m 58s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 39s the patch passed
          +1 javadoc 1m 12s the patch passed
                Other Tests
          -1 unit 7m 7s hadoop-common in the patch failed.
          -1 unit 44m 13s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          127m 56s



          Reason Tests
          Failed junit tests hadoop.net.TestDNS
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-7053
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882929/YARN-7053.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3c722b55dc47 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 913760c
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17034/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17034/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17034/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17034/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 13m 12s trunk passed +1 compile 13m 33s trunk passed +1 checkstyle 1m 55s trunk passed +1 mvnsite 2m 5s trunk passed +1 findbugs 2m 31s trunk passed +1 javadoc 1m 7s trunk passed       Patch Compile Tests 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 9s the patch passed +1 compile 10m 21s the patch passed +1 javac 10m 21s the patch passed +1 checkstyle 1m 44s the patch passed +1 mvnsite 1m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 39s the patch passed +1 javadoc 1m 12s the patch passed       Other Tests -1 unit 7m 7s hadoop-common in the patch failed. -1 unit 44m 13s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 127m 56s Reason Tests Failed junit tests hadoop.net.TestDNS   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-7053 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882929/YARN-7053.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3c722b55dc47 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 913760c Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/17034/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17034/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17034/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/17034/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jhung Jonathan Hung added a comment -

          Thanks Subru Krishnan for the review. Added a ZKCuratorManager test.

          Regarding ZookeeperFederationStateStore transactions, I didn't see any existing transactions in this class that could be refactored. And the only place adding transactions seemed applicable was in the put(String znode, byte[] data, boolean update) method. Did you want to change the put logic as part of this ticket?

          Show
          jhung Jonathan Hung added a comment - Thanks Subru Krishnan for the review. Added a ZKCuratorManager test. Regarding ZookeeperFederationStateStore transactions, I didn't see any existing transactions in this class that could be refactored. And the only place adding transactions seemed applicable was in the put(String znode, byte[] data, boolean update) method. Did you want to change the put logic as part of this ticket?
          Hide
          subru Subru Krishnan added a comment -

          Jonathan Hung, thanks for adding the test. Can you add a setData and delete also in the same test so that we cover all the transactions.

          Did you want to change the put logic as part of this ticket?

          Yes, if ZookeeperFederationStateStore::put is straightforward to update. If not, I am fine with doing it separately in future.

          Show
          subru Subru Krishnan added a comment - Jonathan Hung , thanks for adding the test. Can you add a setData and delete also in the same test so that we cover all the transactions. Did you want to change the put logic as part of this ticket? Yes, if ZookeeperFederationStateStore::put is straightforward to update. If not, I am fine with doing it separately in future.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          0 mvndep 1m 35s Maven dependency ordering for branch
          +1 mvninstall 15m 49s trunk passed
          +1 compile 15m 4s trunk passed
          +1 checkstyle 2m 0s trunk passed
          +1 mvnsite 2m 16s trunk passed
          +1 findbugs 3m 12s trunk passed
          +1 javadoc 1m 34s trunk passed
                Patch Compile Tests
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 1m 35s the patch passed
          +1 compile 14m 16s the patch passed
          +1 javac 14m 15s the patch passed
          +1 checkstyle 2m 24s the patch passed
          +1 mvnsite 2m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 48s the patch passed
          +1 javadoc 1m 41s the patch passed
                Other Tests
          -1 unit 10m 13s hadoop-common in the patch failed.
          -1 unit 45m 42s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          147m 14s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.security.TestKDiag
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-7053
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883003/YARN-7053.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6bfcc1ed3de3 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b6bfb2f
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17046/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17046/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17046/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17046/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 1m 35s Maven dependency ordering for branch +1 mvninstall 15m 49s trunk passed +1 compile 15m 4s trunk passed +1 checkstyle 2m 0s trunk passed +1 mvnsite 2m 16s trunk passed +1 findbugs 3m 12s trunk passed +1 javadoc 1m 34s trunk passed       Patch Compile Tests 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 1m 35s the patch passed +1 compile 14m 16s the patch passed +1 javac 14m 15s the patch passed +1 checkstyle 2m 24s the patch passed +1 mvnsite 2m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 48s the patch passed +1 javadoc 1m 41s the patch passed       Other Tests -1 unit 10m 13s hadoop-common in the patch failed. -1 unit 45m 42s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 147m 14s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.security.TestKDiag   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-7053 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883003/YARN-7053.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6bfcc1ed3de3 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b6bfb2f Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/17046/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17046/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17046/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/17046/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Jonathan Hung, I looked at ZookeeperFederationStateStore::put, it's fine to defer that. But can you fix the test failures.

          Show
          subru Subru Krishnan added a comment - Jonathan Hung , I looked at ZookeeperFederationStateStore::put , it's fine to defer that. But can you fix the test failures.
          Hide
          jhung Jonathan Hung added a comment -

          Thanks Subru Krishnan, actually I just uploaded 004 patch which addresses the ZookeeperFederationStateStore#put issue. Also added setData and delete to ZKCuratorManager tests.

          Not sure if the other test failures are related. Will look into it

          Show
          jhung Jonathan Hung added a comment - Thanks Subru Krishnan , actually I just uploaded 004 patch which addresses the ZookeeperFederationStateStore#put issue. Also added setData and delete to ZKCuratorManager tests. Not sure if the other test failures are related. Will look into it
          Hide
          jhung Jonathan Hung added a comment -

          TestZKFailoverController and TestSubmitApplicationWithRMHA pass locally for me. The TestKDiag tests don't seem to be related (kerberos-related failures). TestContainerAllocation fails with or without the change.

          Show
          jhung Jonathan Hung added a comment - TestZKFailoverController and TestSubmitApplicationWithRMHA pass locally for me. The TestKDiag tests don't seem to be related (kerberos-related failures). TestContainerAllocation fails with or without the change.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Jonathan Hung for updating the patch. Let's look at how the tests go for v4.

          We do not need fencing in ZookeeperFederationStateStore as we rely on RM for that. That's why after looking at the code I wanted to skip this. So you'll have to revert the changes to ZookeeperFederationStateStore (sorry).

          In TestZKCuratorManager::testTransaction, this should be false after deletion right?

          assertTrue(Arrays.equals(setData, curator.getData(node1)));

          Show
          subru Subru Krishnan added a comment - Thanks Jonathan Hung for updating the patch. Let's look at how the tests go for v4. We do not need fencing in ZookeeperFederationStateStore as we rely on RM for that. That's why after looking at the code I wanted to skip this. So you'll have to revert the changes to ZookeeperFederationStateStore (sorry). In TestZKCuratorManager::testTransaction , this should be false after deletion right? assertTrue(Arrays.equals(setData, curator.getData(node1)));
          Hide
          jhung Jonathan Hung added a comment -

          Oops, OK. Attached 005 which reverts these changes.

          Also for assertTrue(Arrays.equals(setData, curator.getData(node1)));, in this transaction I'm changing node1's data and deleting node2, so node1 still exists and I'm verifying the data was set properly.

          Show
          jhung Jonathan Hung added a comment - Oops, OK. Attached 005 which reverts these changes. Also for assertTrue(Arrays.equals(setData, curator.getData(node1))); , in this transaction I'm changing node1's data and deleting node2, so node1 still exists and I'm verifying the data was set properly.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 1m 36s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 14m 17s trunk passed
          +1 compile 14m 34s trunk passed
          +1 checkstyle 1m 51s trunk passed
          +1 mvnsite 2m 39s trunk passed
          +1 findbugs 3m 38s trunk passed
          +1 javadoc 1m 40s trunk passed
                Patch Compile Tests
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 38s the patch passed
          +1 compile 10m 50s the patch passed
          +1 javac 10m 50s the patch passed
          -0 checkstyle 1m 53s root: The patch generated 6 new + 1 unchanged - 0 fixed = 7 total (was 1)
          +1 mvnsite 2m 47s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 0s the patch passed
          +1 javadoc 1m 55s the patch passed
                Other Tests
          -1 unit 7m 20s hadoop-common in the patch failed.
          +1 unit 1m 54s hadoop-yarn-server-common in the patch passed.
          -1 unit 45m 2s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          140m 57s



          Reason Tests
          Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag
            hadoop.fs.shell.TestCopyFromLocal
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-7053
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883201/YARN-7053.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a2265463890c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c379310
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17080/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17080/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17080/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17080/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17080/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 36s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 14m 17s trunk passed +1 compile 14m 34s trunk passed +1 checkstyle 1m 51s trunk passed +1 mvnsite 2m 39s trunk passed +1 findbugs 3m 38s trunk passed +1 javadoc 1m 40s trunk passed       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 38s the patch passed +1 compile 10m 50s the patch passed +1 javac 10m 50s the patch passed -0 checkstyle 1m 53s root: The patch generated 6 new + 1 unchanged - 0 fixed = 7 total (was 1) +1 mvnsite 2m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 0s the patch passed +1 javadoc 1m 55s the patch passed       Other Tests -1 unit 7m 20s hadoop-common in the patch failed. +1 unit 1m 54s hadoop-yarn-server-common in the patch passed. -1 unit 45m 2s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 140m 57s Reason Tests Failed junit tests hadoop.fs.shell.TestCopyPreserveFlag   hadoop.fs.shell.TestCopyFromLocal   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-7053 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883201/YARN-7053.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a2265463890c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c379310 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17080/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17080/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17080/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17080/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/17080/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 13m 22s trunk passed
          +1 compile 13m 38s trunk passed
          +1 checkstyle 1m 47s trunk passed
          +1 mvnsite 2m 8s trunk passed
          +1 findbugs 2m 24s trunk passed
          +1 javadoc 1m 7s trunk passed
                Patch Compile Tests
          0 mvndep 0m 48s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 10m 42s the patch passed
          +1 javac 10m 42s the patch passed
          +1 checkstyle 1m 51s the patch passed
          +1 mvnsite 2m 3s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 54s the patch passed
          +1 javadoc 1m 15s the patch passed
                Other Tests
          -1 unit 7m 18s hadoop-common in the patch failed.
          -1 unit 47m 10s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          132m 22s



          Reason Tests
          Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem
            hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService
            hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-7053
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883210/YARN-7053.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 23caccc104a6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c379310
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17081/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17081/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17081/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17081/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 13m 22s trunk passed +1 compile 13m 38s trunk passed +1 checkstyle 1m 47s trunk passed +1 mvnsite 2m 8s trunk passed +1 findbugs 2m 24s trunk passed +1 javadoc 1m 7s trunk passed       Patch Compile Tests 0 mvndep 0m 48s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 10m 42s the patch passed +1 javac 10m 42s the patch passed +1 checkstyle 1m 51s the patch passed +1 mvnsite 2m 3s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 54s the patch passed +1 javadoc 1m 15s the patch passed       Other Tests -1 unit 7m 18s hadoop-common in the patch failed. -1 unit 47m 10s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 132m 22s Reason Tests Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem   hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService   hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-7053 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883210/YARN-7053.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 23caccc104a6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c379310 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/17081/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17081/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17081/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/17081/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jhung Jonathan Hung added a comment -

          TestSFTPFileSystem.testGetAccessTime, TestOpportunisticContainerAllocatorAMService.testContainerAutoUpdateContainer, TestSchedulingMonitor.testRMStarts, TestFSAppStarvation.testPreemptionEnabled pass locally for me. TestDelegationTokenRenewer.testCancelWithMultipleAppSubmissions
          and TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable fails with or without the change.

          Show
          jhung Jonathan Hung added a comment - TestSFTPFileSystem.testGetAccessTime , TestOpportunisticContainerAllocatorAMService.testContainerAutoUpdateContainer , TestSchedulingMonitor.testRMStarts , TestFSAppStarvation.testPreemptionEnabled pass locally for me. TestDelegationTokenRenewer.testCancelWithMultipleAppSubmissions and TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable fails with or without the change.
          Hide
          subru Subru Krishnan added a comment -

          +1 on the latest patch (rev 005). Thanks Jonathan Hung for the contribution, I have committed this to trunk/branch-2.

          Show
          subru Subru Krishnan added a comment - +1 on the latest patch (rev 005). Thanks Jonathan Hung for the contribution, I have committed this to trunk/branch-2.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12229 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12229/)
          YARN-7053. Move curator transaction support to ZKCuratorManager. (subru: rev 4249172e1419acdb2b69ae3db43dc59da2aa2e03)

          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestZKCuratorManager.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12229 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12229/ ) YARN-7053 . Move curator transaction support to ZKCuratorManager. (subru: rev 4249172e1419acdb2b69ae3db43dc59da2aa2e03) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/curator/TestZKCuratorManager.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/curator/ZKCuratorManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java
          Hide
          jhung Jonathan Hung added a comment -

          Thanks Subru Krishnan for the commit/reviews!

          Show
          jhung Jonathan Hung added a comment - Thanks Subru Krishnan for the commit/reviews!

            People

            • Assignee:
              jhung Jonathan Hung
              Reporter:
              jhung Jonathan Hung
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development