Hadoop Common
  1. Hadoop Common
  2. HADOOP-3010

ConcurrentModificationException from org.apache.hadoop.ipc.Server$Responder in JobTracker

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.1
    • Fix Version/s: 0.16.3
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I see lot of these in my JobTracker log

      WARN org.apache.hadoop.ipc.Server: Exception in Responder java.util.ConcurrentModificationException
              at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
              at java.util.HashMap$KeyIterator.next(HashMap.java:827)
              at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1010)
              at org.apache.hadoop.ipc.Server$Responder.run(Server.java:475)
      

        Activity

        Hide
        Koji Noguchi added a comment -

        As Nigel mentioned in HADOOP-3141, we are seeing this on namenode logs as well.

        Show
        Koji Noguchi added a comment - As Nigel mentioned in HADOOP-3141 , we are seeing this on namenode logs as well.
        Hide
        Raghu Angadi added a comment -

        I would recommend this for 0.16.3. The fact that it is currently not harmful is secondary, I think.

        Show
        Raghu Angadi added a comment - I would recommend this for 0.16.3. The fact that it is currently not harmful is secondary, I think.
        Hide
        Raghu Angadi added a comment -

        Setting the earliest next release 0.17. If there is 0.16.3, this should go there too.

        Show
        Raghu Angadi added a comment - Setting the earliest next release 0.17. If there is 0.16.3, this should go there too.
        Hide
        Raghu Angadi added a comment -

        Patch attached. Server synchronizes on selector.keys() when it iterates over it once every few seconds. JavaDoc for Selector says we need to synchronize on these sets while accessing them. In this perticular case, IPC handler could enqueue more connections (thus modifying selector.keys()) while Responder iterates over them.

        With some artificial sleeps I verified these exceptions show up before the patch and disappear after.

        Regd unit test, I think this was straight forward incorrect access of an object and we are correcting it. I don't think this requires a unit test, as such it is not easy to trigger this reliably.

        Show
        Raghu Angadi added a comment - Patch attached. Server synchronizes on selector.keys() when it iterates over it once every few seconds. JavaDoc for Selector says we need to synchronize on these sets while accessing them. In this perticular case, IPC handler could enqueue more connections (thus modifying selector.keys()) while Responder iterates over them. With some artificial sleeps I verified these exceptions show up before the patch and disappear after. Regd unit test, I think this was straight forward incorrect access of an object and we are correcting it. I don't think this requires a unit test, as such it is not easy to trigger this reliably.
        Hide
        Raghu Angadi added a comment -

        Could someone review this? Its a simple patch. Thanks.

        Show
        Raghu Angadi added a comment - Could someone review this? Its a simple patch. Thanks.
        Hide
        dhruba borthakur added a comment -

        +1. Code looks good.

        Show
        dhruba borthakur added a comment - +1. Code looks good.
        Hide
        Raghu Angadi added a comment -

        Thanks Dhruba.

        Show
        Raghu Angadi added a comment - Thanks Dhruba.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12379060/HADOOP-3010.patch
        against trunk revision 643282.

        @author +1. The patch does not contain any @author tags.

        tests included -1. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new javac compiler warnings.

        release audit +1. The applied patch does not generate any new release audit warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2137/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2137/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2137/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2137/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12379060/HADOOP-3010.patch against trunk revision 643282. @author +1. The patch does not contain any @author tags. tests included -1. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2137/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2137/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2137/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2137/console This message is automatically generated.
        Hide
        Raghu Angadi added a comment -

        I just commented this. Unit Test omission is explained in the 4th comment above.

        Show
        Raghu Angadi added a comment - I just commented this. Unit Test omission is explained in the 4th comment above.
        Hide
        Nigel Daley added a comment -

        I just merged this to 0.16 branch.

        Show
        Nigel Daley added a comment - I just merged this to 0.16 branch.
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #451 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/451/ )

          People

          • Assignee:
            Raghu Angadi
            Reporter:
            Amar Kamat
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development