Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2629

Class loading quirk prevents inner class method compilation

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: task
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      While profiling jobs like terasort and gridmix, I noticed that a
      method "org.apache.hadoop.mapreduce.task.ReduceContextImpl.access
      $000" is near the top. It turns out that this is because the
      ReduceContextImpl class has a member backupStore which is accessed
      from an inner class ReduceContextImpl$ValueIterator. Due to the way
      synthetic accessor methods work, every access of backupStore results
      in a call to access$000 to the outer class. For some portion of the
      run, backupStore is null and the BackupStore class has never been
      loaded by the reducer.

      Due to the way the Hotspot JVM inliner works, by default it will not
      inline a short method where the class of of the return value object
      is unloaded - if you use a debug JVM with -XX:+PrintCompilation you
      will see a failure reason message like "unloaded signature classes."
      This causes every call to ReduceContextImpl.access$000 to be executed
      in the interpreter for the handful of bytecodes to return the null
      backupStore.

      1. MAPREDUCE-2629.patch
        1 kB
        Eric Caspole
      2. mr-2629.txt
        2 kB
        Todd Lipcon

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #837 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/837/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #837 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/837/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #58 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/58/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #58 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/58/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #867 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/867/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #867 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/867/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #46 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/46/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #46 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/46/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1206 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1206/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1206 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1206/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #34 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/34/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #34 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/34/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1128 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1128/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1128 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1128/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #34 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/34/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #34 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/34/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #34 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/34/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #34 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/34/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187182 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1144 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1144/)
        MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1144 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1144/ ) MAPREDUCE-2629 . Workaround a JVM class loading quirk which prevents JIT compilation of inner classes methods in ReduceContextImpl. Contributed by Eric Caspole. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187183 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java
        Hide
        Todd Lipcon added a comment -

        Committed to 23 and trunk. Thanks, Eric!

        Show
        Todd Lipcon added a comment - Committed to 23 and trunk. Thanks, Eric!
        Hide
        Todd Lipcon added a comment -

        findbugs are bogus. No tests since this is covered by existing tests. I successfully ran several terasorts with this patch.

        Show
        Todd Lipcon added a comment - findbugs are bogus. No tests since this is covered by existing tests. I successfully ran several terasorts with this patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12499918/mr-2629.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499918/mr-2629.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1087//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        the CPU usage was bugging me so I rebased Eric's patch last night.

        Show
        Todd Lipcon added a comment - the CPU usage was bugging me so I rebased Eric's patch last night.
        Hide
        Todd Lipcon added a comment -

        Eric: Would you mind updating this against the current trunk? Profiling a running MRv2 terasort I'm seeing this use a significant amount of CPU

        Show
        Todd Lipcon added a comment - Eric: Would you mind updating this against the current trunk? Profiling a running MRv2 terasort I'm seeing this use a significant amount of CPU
        Hide
        Todd Lipcon added a comment -

        I'm still seeing this in MRv2

        Show
        Todd Lipcon added a comment - I'm still seeing this in MRv2
        Hide
        Eric Caspole added a comment -

        It seems MRv2 removed the classes where this problem was occurring. If I see this pattern arise again I'll file a new bug.

        Show
        Eric Caspole added a comment - It seems MRv2 removed the classes where this problem was occurring. If I see this pattern arise again I'll file a new bug.
        Hide
        Todd Lipcon added a comment -

        OK. I'm happy to +1 as is, but if you think there's an improvement, I'll hold off. Enjoy your vacation.

        Show
        Todd Lipcon added a comment - OK. I'm happy to +1 as is, but if you think there's an improvement, I'll hold off. Enjoy your vacation.
        Hide
        Eric Caspole added a comment -

        On my test system I see about 2% cpu time in that method, which is removed with various versions of the patch. But since I spend a lot of time in I/O wait I cannot see an improvement in the wall clock run time of, say, a 8gb terasort.

        I think this patch can be improved, I'll update it after I get back from vacation.

        Show
        Eric Caspole added a comment - On my test system I see about 2% cpu time in that method, which is removed with various versions of the patch. But since I spend a lot of time in I/O wait I cannot see an improvement in the wall clock run time of, say, a 8gb terasort. I think this patch can be improved, I'll update it after I get back from vacation.
        Hide
        Todd Lipcon added a comment -

        Seems reasonable, and thanks for teaching me something new about JVM performance Do you have some benchmark that shows a difference here?

        Show
        Todd Lipcon added a comment - Seems reasonable, and thanks for teaching me something new about JVM performance Do you have some benchmark that shows a difference here?
        Hide
        Eric Caspole added a comment -

        Hotspot has an XX option relates to this situation -XX:
        UseFastAccessorMethods, which is likely to be the default in future JDK
        versions. The attached patch allows the -UseFastAccessorMethods to
        correctly analyze the method and inline the call. The JVM analyses the bytecodes of the getter method and takes the correct action, which it cannot currently do for the access$000 call.

        Show
        Eric Caspole added a comment - Hotspot has an XX option relates to this situation -XX: UseFastAccessorMethods, which is likely to be the default in future JDK versions. The attached patch allows the -UseFastAccessorMethods to correctly analyze the method and inline the call. The JVM analyses the bytecodes of the getter method and takes the correct action, which it cannot currently do for the access$000 call.

          People

          • Assignee:
            Eric Caspole
            Reporter:
            Eric Caspole
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development