Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3145

ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.7.0
    • None
    • None
    • Reviewed

    Description

      ava.util.ConcurrentModificationException(java.util.ConcurrentModificationException
      at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1115)
      at java.util.TreeMap$KeyIterator.next(TreeMap.java:1169)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:347)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.getQueueUserAclInfo(ParentQueue.java:348)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getQueueUserAclInfo(CapacityScheduler.java:850)
      at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getQueueUserAcls(ClientRMService.java:844)
      at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueUserAcls(ApplicationClientProtocolPBServiceImpl.java:250)
      at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:335)
      

      Attachments

        1. YARN-3145.001.patch
          1 kB
          Tsuyoshi Ozawa
        2. YARN-3145.002.patch
          4 kB
          Tsuyoshi Ozawa

        Activity

          jianhe I am looking into problem,is it from trunk version? I could not see line number matching!!
          The scenario I think is refreshQueue has been triggered concurrently by removing childqueues. Is this is the case in your scenario?

          rohithsharma Rohith Sharma K S added a comment - jianhe I am looking into problem,is it from trunk version? I could not see line number matching!! The scenario I think is refreshQueue has been triggered concurrently by removing childqueues. Is this is the case in your scenario?
          ozawa Tsuyoshi Ozawa added a comment -

          The synchronization of childQueues looks not enough. Attaching a first patch.

          ozawa Tsuyoshi Ozawa added a comment - The synchronization of childQueues looks not enough. Attaching a first patch.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696681/YARN-3145.001.patch
          against trunk revision 0b567f4.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6517//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6517//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696681/YARN-3145.001.patch against trunk revision 0b567f4. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6517//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6517//console This message is automatically generated.
          ozawa Tsuyoshi Ozawa added a comment -

          rohithsharma Oh, I submitted a patch at the same time when you commented. Sorry for that.

          ozawa Tsuyoshi Ozawa added a comment - rohithsharma Oh, I submitted a patch at the same time when you commented. Sorry for that.

          Thats fine.. Even I was not known that you are updating it. Feel free to assign yourself.

          rohithsharma Rohith Sharma K S added a comment - Thats fine.. Even I was not known that you are updating it. Feel free to assign yourself.
          jianhe Jian He added a comment - - edited

          thanks for working on this!
          In ParentQueue#completedContainer, the following is not synchronized, I think this should be inside the previous queue synchronization block.

                if (sortQueues) {
                  // reinsert the updated queue
                  for (Iterator<CSQueue> iter=childQueues.iterator(); iter.hasNext();) {
                    CSQueue csqueue = iter.next();
                    if(csqueue.equals(completedChildQueue))
                    {
                      iter.remove();
                      LOG.info("Re-sorting completed queue: " + csqueue.getQueuePath() + 
                          " stats: " + csqueue);
                      childQueues.add(csqueue);
                      break;
                    }
                  }
                }
          
          jianhe Jian He added a comment - - edited thanks for working on this! In ParentQueue#completedContainer, the following is not synchronized, I think this should be inside the previous queue synchronization block. if (sortQueues) { // reinsert the updated queue for (Iterator<CSQueue> iter=childQueues.iterator(); iter.hasNext();) { CSQueue csqueue = iter.next(); if (csqueue.equals(completedChildQueue)) { iter.remove(); LOG.info( "Re-sorting completed queue: " + csqueue.getQueuePath() + " stats: " + csqueue); childQueues.add(csqueue); break ; } } }
          ozawa Tsuyoshi Ozawa added a comment -

          Thank you!

          ozawa Tsuyoshi Ozawa added a comment - Thank you!
          ozawa Tsuyoshi Ozawa added a comment -

          Thank you for the review, Jian. Good catch. Updating a patch to move the if block into synchronization block.

          ozawa Tsuyoshi Ozawa added a comment - Thank you for the review, Jian. Good catch. Updating a patch to move the if block into synchronization block.
          leftnoteasy Wangda Tan added a comment -

          Thanks ozawa working on this, patch LGTM, +1.

          leftnoteasy Wangda Tan added a comment - Thanks ozawa working on this, patch LGTM, +1.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12696843/YARN-3145.002.patch
          against trunk revision 276485e.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6525//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6525//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696843/YARN-3145.002.patch against trunk revision 276485e. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6525//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6525//console This message is automatically generated.
          jianhe Jian He added a comment -

          Committed to trunk and branch-2, thanks Tsuyoshi, Rohith, Wangda !

          jianhe Jian He added a comment - Committed to trunk and branch-2, thanks Tsuyoshi, Rohith, Wangda !
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7030 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7030/)
          YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7030 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7030/ ) YARN-3145 . Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Jian and Wangda for your review and thanks Rohith for taking this.

          ozawa Tsuyoshi Ozawa added a comment - Thanks Jian and Wangda for your review and thanks Rohith for taking this.
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/)
          YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          • hadoop-yarn-project/CHANGES.txt
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #96 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/96/ ) YARN-3145 . Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java hadoop-yarn-project/CHANGES.txt
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/830/)
          YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #830 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/830/ ) YARN-3145 . Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/)
          YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #93 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/93/ ) YARN-3145 . Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/)
          YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2028 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2028/ ) YARN-3145 . Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/)
          YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #97 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/97/ ) YARN-3145 . Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2047 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2047/)
          YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2047 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2047/ ) YARN-3145 . Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA (jianhe: rev 4641196fe02af5cab3d56a9f3c78875c495dbe03) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java

          People

            ozawa Tsuyoshi Ozawa
            jianhe Jian He
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: