Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-19769

IllegalAccessError on package-private Hadoop metrics2 classes in MapReduce jobs

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Client-side ZooKeeper metrics which were added to 2.0.0 alpha/beta releases cause issues when launching MapReduce jobs via {{yarn jar}} on the command line. This stems from ClassLoader separation issues that YARN implements. It was chosen that the easiest solution was to remove these ZooKeeper metrics entirely.
      Show
      Client-side ZooKeeper metrics which were added to 2.0.0 alpha/beta releases cause issues when launching MapReduce jobs via {{yarn jar}} on the command line. This stems from ClassLoader separation issues that YARN implements. It was chosen that the easiest solution was to remove these ZooKeeper metrics entirely.

      Description

      issues for context: HBASE-17170, HBASE-17448, TEZ-3299, HADOOP-10893

      Since Hadoop 2.6.0, the yarn jar entry point to submit a YARN job has been using a custom classloader to separate Hadoop dependencies from the user's JAR being run. A separate classloader is created for the user-provided jar, and then this classloader is set as the contextClassLoader before the Tool is executed by Hadoop's RunJar class. This has been (mostly?) fine for us to date because we don't try to access any Hadoop internal classes client-side.

      However, with the ZK metrics, clients are pushing ZK metrics to metrics2. The problem is that Hadoop metrics2 implementations which we reference from the same package are loaded by a different classloader than our HBase code is loaded from. This makes the expected package-private access of these Metrics2 classes (e.g. MetricsInfoImpl) fail with an IllegalAccessError.

      java.lang.RuntimeException: Could not create  interface org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource Is the hadoop compatibility jar on the classpath?
              at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:75)
              at org.apache.hadoop.hbase.zookeeper.ZKMetrics.<init>(ZKMetrics.java:36)
              at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:115)
              at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:139)
              at org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:128)
              at org.apache.hadoop.hbase.zookeeper.ZKWatcher.<init>(ZKWatcher.java:102)
              at org.apache.hadoop.hbase.security.token.TokenUtil.getAuthToken(TokenUtil.java:293)
              at org.apache.hadoop.hbase.security.token.TokenUtil.addTokenForJob(TokenUtil.java:259)
              at org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil.initCredentials(TableMapReduceUtil.java:535)
              at org.apache.phoenix.mapreduce.MultiHfileOutputFormat.configureIncrementalLoad(MultiHfileOutputFormat.java:712)
              at org.apache.phoenix.mapreduce.AbstractBulkLoadTool.submitJob(AbstractBulkLoadTool.java:300)
              at org.apache.phoenix.mapreduce.AbstractBulkLoadTool.loadData(AbstractBulkLoadTool.java:267)
              at org.apache.phoenix.mapreduce.AbstractBulkLoadTool.run(AbstractBulkLoadTool.java:180)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
              at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
              at org.apache.phoenix.mapreduce.CsvBulkLoadTool.main(CsvBulkLoadTool.java:109)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
      Caused by: java.util.ServiceConfigurationError: org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSource: Provider org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl could not be instantiated
              at java.util.ServiceLoader.fail(ServiceLoader.java:232)
              at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
              at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
              at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
              at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
              at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
              ... 21 more
      Caused by: java.lang.IllegalAccessError: tried to access class org.apache.hadoop.metrics2.lib.MetricsInfoImpl from class org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry
              at org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry.newGauge(DynamicMetricsRegistry.java:139)
              at org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:59)
              at org.apache.hadoop.hbase.zookeeper.MetricsZooKeeperSourceImpl.<init>(MetricsZooKeeperSourceImpl.java:51)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
              at java.lang.Class.newInstance(Class.java:442)
              at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
              ... 24 more
      

      There are two ways to fix this problem that I see now:

      1. Remove client-side ZK metrics and disallow any more client-side metrics that directly use metrics2 API
      2. Use public API for metrics2 only in HBase, or copy implementations from Hadoop to avoid the need to access the package-private classes from this goofy state.

        Attachments

          Activity

            People

            • Assignee:
              elserj Josh Elser
              Reporter:
              elserj Josh Elser
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: