Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.0, 0.23.1
    • Component/s: security
    • Labels:
      None

      Description

      Sporadic ConcurrentModificationExceptions are originating from UGI.getCurrentUser when it needs to create a new instance. The problem was specifically observed in a JT under heavy load when a post-job cleanup is accessing the UGI while a new job is being processed.

      1. HADOOP-7854.patch
        0.8 kB
        Daryn Sharp
      2. HADOOP-7854-trunk.patch
        0.9 kB
        Daryn Sharp

        Activity

        Hide
        Daryn Sharp added a comment -

        Simply added synchronized keyword. I'm not sure how I can write a unit test...

        Show
        Daryn Sharp added a comment - Simply added synchronized keyword. I'm not sure how I can write a unit test...
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504902/HADOOP-7854.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/399//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504902/HADOOP-7854.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/399//console This message is automatically generated.
        Hide
        Kihwal Lee added a comment -

        I think TRUNK has the same issue.

        Show
        Kihwal Lee added a comment - I think TRUNK has the same issue.
        Hide
        Todd Lipcon added a comment -

        I'm confused - what's throwing the CME? It's a static method and it doesn't touch any class variables...

        Show
        Todd Lipcon added a comment - I'm confused - what's throwing the CME? It's a static method and it doesn't touch any class variables...
        Hide
        Kihwal Lee added a comment -

        See HADOOP-7831 for the actual exception.

        Show
        Kihwal Lee added a comment - See HADOOP-7831 for the actual exception.
        Hide
        Daryn Sharp added a comment -

        Per Kihwal's comment, the exception occurs when two unsynchronized callers attempt to use the Subject from the current context.

        Show
        Daryn Sharp added a comment - Per Kihwal's comment, the exception occurs when two unsynchronized callers attempt to use the Subject from the current context.
        Hide
        Todd Lipcon added a comment -

        Should we also be reporting this as a JDK bug? Seems like it should be up to the JDK to synchronize this.

        Show
        Todd Lipcon added a comment - Should we also be reporting this as a JDK bug? Seems like it should be up to the JDK to synchronize this.
        Hide
        Jitendra Nath Pandey added a comment -

        It does seem like a JDK issue, do you have a stack trace for the CME?

        Show
        Jitendra Nath Pandey added a comment - It does seem like a JDK issue, do you have a stack trace for the CME?
        Hide
        Daryn Sharp added a comment -

        Good question. The Subject} class appears to generally be thread-safe. It's internally synchronizing and using instances such as {{Collections.synchronizedSet. The problem is due to iterators becoming very grumpy when their collection is modified, and Subject is full of collections that can be accessed directly. I can't find any documentation about whether Subject is even supposed to be thread-safe. I suspect it is once you lock it down as read-only, but we don't do that.

        Show
        Daryn Sharp added a comment - Good question. The Subject} class appears to generally be thread-safe. It's internally synchronizing and using instances such as {{Collections.synchronizedSet . The problem is due to iterators becoming very grumpy when their collection is modified, and Subject is full of collections that can be accessed directly. I can't find any documentation about whether Subject is even supposed to be thread-safe. I suspect it is once you lock it down as read-only, but we don't do that.
        Hide
        Daryn Sharp added a comment -

        Stack trace is in the bug Kihwal mentioned: HADOOP-7831

        Show
        Daryn Sharp added a comment - Stack trace is in the bug Kihwal mentioned: HADOOP-7831
        Hide
        Jitendra Nath Pandey added a comment -

        I think we should commit it. +1

        Show
        Jitendra Nath Pandey added a comment - I think we should commit it. +1
        Hide
        Matt Foley added a comment -

        Marking this fixed in 1.0.0 (0.20.205.1), because this patch will prevent the problem from recurring.

        If further disposition is required (such as opening bug against JDK), please do so.

        Should this be merged to trunk?

        Show
        Matt Foley added a comment - Marking this fixed in 1.0.0 (0.20.205.1), because this patch will prevent the problem from recurring. If further disposition is required (such as opening bug against JDK), please do so. Should this be merged to trunk?
        Hide
        Mahadev konar added a comment -

        Yes, please merge this to trunk as well.

        Show
        Mahadev konar added a comment - Yes, please merge this to trunk as well.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12505679/HADOOP-7854-trunk.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 15 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/417//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/417//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505679/HADOOP-7854-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 15 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/417//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/417//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        Can this be checked into 0.23 branch as well?

        Show
        Mahadev konar added a comment - Can this be checked into 0.23 branch as well?
        Hide
        Jitendra Nath Pandey added a comment -

        Committed to trunk and 23 as well. Thanks to Daryn.

        Show
        Jitendra Nath Pandey added a comment - Committed to trunk and 23 as well. Thanks to Daryn.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1422 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1422/)
        HADOOP-7854. UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1422 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1422/ ) HADOOP-7854 . UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1349 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1349/)
        HADOOP-7854. UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1349 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1349/ ) HADOOP-7854 . UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #236 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/236/)
        Merged r1208926 from trunk for HADOOP-7854.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #236 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/236/ ) Merged r1208926 from trunk for HADOOP-7854 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #238 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/238/)
        Merged r1208926 from trunk for HADOOP-7854.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #238 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/238/ ) Merged r1208926 from trunk for HADOOP-7854 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #111 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/111/)
        Merged r1208926 from trunk for HADOOP-7854.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #111 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/111/ ) Merged r1208926 from trunk for HADOOP-7854 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1372 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1372/)
        HADOOP-7854. UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1372 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1372/ ) HADOOP-7854 . UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #248 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/248/)
        Merged r1208926 from trunk for HADOOP-7854.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #248 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/248/ ) Merged r1208926 from trunk for HADOOP-7854 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Matt Foley added a comment -

        Corrected Fix Versions and Target Versions to correspond to actual commits.

        Show
        Matt Foley added a comment - Corrected Fix Versions and Target Versions to correspond to actual commits.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #94 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/94/)
        Merged r1208926 from trunk for HADOOP-7854.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #94 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/94/ ) Merged r1208926 from trunk for HADOOP-7854 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-HAbranch-build #2 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/2/)
        Merged r1208926 from trunk for HADOOP-7854.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-HAbranch-build #2 (See https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/2/ ) Merged r1208926 from trunk for HADOOP-7854 . jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208929 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #881 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/881/)
        HADOOP-7854. UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #881 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/881/ ) HADOOP-7854 . UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #914 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/914/)
        HADOOP-7854. UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp.

        jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #914 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/914/ ) HADOOP-7854 . UGI getCurrentUser is not synchronized. Contributed by Daryn Sharp. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208926 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
        Hide
        Matt Foley added a comment -

        Closed upon release of version 1.0.0.

        Show
        Matt Foley added a comment - Closed upon release of version 1.0.0.

          People

          • Assignee:
            Daryn Sharp
            Reporter:
            Daryn Sharp
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development