Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15686

Add override mechanism for the exempt classes when dynamically loading table coprocessor

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.1
    • Fix Version/s: 1.4.0, 0.98.20, 2.0.0
    • Component/s: Coprocessors
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      New coprocessor table descriptor attribute, hbase.coprocessor.classloader.included.classes, is added.
      User can specify class name prefixes (semicolon separated) which should be loaded by CoprocessorClassLoader through this attribute using the following syntax:
      {code}
        hbase> alter 't1', 'coprocessor'=>'hdfs:///foo.jar|com.foo.FooRegionObserver|1001|arg1=1,arg2=2'
      {code}
      Show
      New coprocessor table descriptor attribute, hbase.coprocessor.classloader.included.classes, is added. User can specify class name prefixes (semicolon separated) which should be loaded by CoprocessorClassLoader through this attribute using the following syntax: {code}   hbase> alter 't1', 'coprocessor'=>' hdfs:///foo.jar |com.foo.FooRegionObserver|1001|arg1=1,arg2=2' {code}

      Description

      As part of Hadoop's Timeline Service v.2 (YARN-2928), we're adding a table coprocessor (YARN-4062). However, we're finding that the coprocessor cannot be loaded dynamically. A relevant snippet for the exception:

      java.io.IOException: Class org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor cannot be loaded
          at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1329)
          at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1269)
          at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:398)
          at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42436)
          at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
          at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
          at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
          at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
          at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Class org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor cannot be loaded
          at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:324)
          at org.apache.hadoop.hbase.master.HMaster.checkClassLoading(HMaster.java:1483)
          at org.apache.hadoop.hbase.master.HMaster.sanityCheckTableDescriptor(HMaster.java:1327)
          ... 8 more
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunCoprocessor
          at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
          at org.apache.hadoop.hbase.util.CoprocessorClassLoader.loadClass(CoprocessorClassLoader.java:275)
          at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.testTableCoprocessorAttrs(RegionCoprocessorHost.java:322)
          ... 10 more
      

      We tracked it down to the fact that CoprocessorClassLoader regarding all hadoop classes as exempt from loading from the coprocessor jar. Since our coprocessor sits in the coprocessor jar, and yet the loading of this class is delegated to the parent which does not have this jar, the classloading fails.

      What would be nice is the ability to exclude certain classes from the exempt classes so that they can be loaded via table coprocessor classloader. See hadoop's ApplicationClassLoader for a similar feature.

      Is there any other way to load this coprocessor at the table scope?

        Attachments

        1. 15686.v6.txt
          5 kB
          Ted Yu
        2. 15686.v5.txt
          5 kB
          Ted Yu
        3. 15686.v4.txt
          5 kB
          Ted Yu
        4. 15686.v3.txt
          5 kB
          Ted Yu
        5. 15686.v2.txt
          5 kB
          Ted Yu
        6. 15686.wip
          3 kB
          Ted Yu

        Issue Links

          Activity

            People

            • Assignee:
              yuzhihong@gmail.com Ted Yu
              Reporter:
              sjlee0 Sangjin Lee

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment