HBase
  1. HBase
  2. HBASE-10078

Dynamic Filter - Not using DynamicClassLoader when using FilterList

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.94.13
    • Fix Version/s: 0.94.16
    • Component/s: Filters
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I've tried to use dynamic jar load (https://issues.apache.org/jira/browse/HBASE-1936) but seems to have an issue with FilterList.
      Here is some log from my app where i send a Get with a FilterList containing AFilter and other with BFilter.

      2013-12-02 13:55:42,564 DEBUG org.apache.hadoop.hbase.util.DynamicClassLoader: Class d.p.AFilter not found - using dynamical class loader
      2013-12-02 13:55:42,564 DEBUG org.apache.hadoop.hbase.util.DynamicClassLoader: Finding class: d.p.AFilter
      2013-12-02 13:55:42,564 DEBUG org.apache.hadoop.hbase.util.DynamicClassLoader: Loading new jar files, if any
      2013-12-02 13:55:42,677 DEBUG org.apache.hadoop.hbase.util.DynamicClassLoader: Finding class again: d.p.AFilter
      2013-12-02 13:55:43,004 ERROR org.apache.hadoop.hbase.io.HbaseObjectWritable: Can't find class d.p.BFilter
      java.lang.ClassNotFoundException: d.p.BFilter
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:247)
      	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
      	at org.apache.hadoop.hbase.io.HbaseObjectWritable.getClassByName(HbaseObjectWritable.java:792)
      	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:679)
      	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:594)
      	at org.apache.hadoop.hbase.filter.FilterList.readFields(FilterList.java:324)
      	at org.apache.hadoop.hbase.client.Get.readFields(Get.java:405)
      	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:690)
      	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:594)
      	at org.apache.hadoop.hbase.client.Action.readFields(Action.java:101)
      	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:690)
      	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:594)
      	at org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116)
      	at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:690)
      	at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126)
      	at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1311)
      	at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1226)
      	at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
      	at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
      	at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      

      AFilter is not found so it tries with DynamicClassLoader, but when it tries to load AFilter, it uses URLClassLoader and fails without checking out for dynamic jars.

      I think the issue is releated to FilterList#readFields

      FilterList.java
       
       public void readFields(final DataInput in) throws IOException {
          byte opByte = in.readByte();
          operator = Operator.values()[opByte];
          int size = in.readInt();
          if (size > 0) {
            filters = new ArrayList<Filter>(size);
            for (int i = 0; i < size; i++) {
              Filter filter = (Filter)HbaseObjectWritable.readObject(in, conf);
              filters.add(filter);
            }
          }
        }
      

      HbaseObjectWritable#readObject uses a conf (created by calling HBaseConfiguration.create()) which i suppose doesn't include a DynamicClassLoader instance.

      1. 0.94-10078.patch
        12 kB
        Jimmy Xiang
      2. hbase-10078.patch
        7 kB
        Jimmy Xiang
      3. 0.94-10078_v2.patch
        12 kB
        Jimmy Xiang

        Issue Links

          Activity

          Hide
          Jimmy Xiang added a comment -

          I looked into this issue. It is a problem in 0.94. In trunk and 0.96, it works fine, i.e. filters inside a FilterList are loaded using DynamicClassLoader. In 0.94, we can't fix it without breaking compatibility. So I think we won't fix this instead.

          Show
          Jimmy Xiang added a comment - I looked into this issue. It is a problem in 0.94. In trunk and 0.96, it works fine, i.e. filters inside a FilterList are loaded using DynamicClassLoader. In 0.94, we can't fix it without breaking compatibility. So I think we won't fix this instead.
          Hide
          Lars Hofhansl added a comment -

          Why would this break compatibility in 0.94? You mean wire compatibility?

          Show
          Lars Hofhansl added a comment - Why would this break compatibility in 0.94? You mean wire compatibility?
          Hide
          Jimmy Xiang added a comment -

          My bad. There is no wire compatibility issue in 0.94 actually. Let me post a patch soon.

          Trunk/0.96 doesn't have this problem.

          Show
          Jimmy Xiang added a comment - My bad. There is no wire compatibility issue in 0.94 actually. Let me post a patch soon. Trunk/0.96 doesn't have this problem.
          Hide
          Jimmy Xiang added a comment -

          The 0.94 patch fixed the issue in 0.94. The trunk patch just added tests for using DynamicClassLoader in FilterList.

          Show
          Jimmy Xiang added a comment - The 0.94 patch fixed the issue in 0.94. The trunk patch just added tests for using DynamicClassLoader in FilterList.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12621379/hbase-10078.patch
          against trunk revision .
          ATTACHMENT ID: 12621379

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop1.1. The patch compiles against the hadoop 1.1 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          -1 site. The patch appears to cause mvn site goal to fail.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12621379/hbase-10078.patch against trunk revision . ATTACHMENT ID: 12621379 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop1.1 . The patch compiles against the hadoop 1.1 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8335//console This message is automatically generated.
          Hide
          Jimmy Xiang added a comment -

          Minor change to the 0.94 patch

          Show
          Jimmy Xiang added a comment - Minor change to the 0.94 patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12621634/0.94-10078_v2.patch
          against trunk revision .
          ATTACHMENT ID: 12621634

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8347//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12621634/0.94-10078_v2.patch against trunk revision . ATTACHMENT ID: 12621634 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8347//console This message is automatically generated.
          Hide
          Jimmy Xiang added a comment -

          Lars Hofhansl, could you take a look the patch? Thanks.

          Show
          Jimmy Xiang added a comment - Lars Hofhansl , could you take a look the patch? Thanks.
          Hide
          Lars Hofhansl added a comment -

          Looking...

          Show
          Lars Hofhansl added a comment - Looking...
          Hide
          Lars Hofhansl added a comment -

          Took me a bit to grok the change. Looks good. +1

          Show
          Lars Hofhansl added a comment - Took me a bit to grok the change. Looks good. +1
          Hide
          Jimmy Xiang added a comment -

          Thanks a lot.

          Show
          Jimmy Xiang added a comment - Thanks a lot.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94-security #381 (See https://builds.apache.org/job/HBase-0.94-security/381/)
          HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556027)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #381 (See https://builds.apache.org/job/HBase-0.94-security/381/ ) HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556027) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Hide
          Lars Hofhansl added a comment -

          NP

          This change only went into 0.94, right? (removed the other fix tags)

          Show
          Lars Hofhansl added a comment - NP This change only went into 0.94, right? (removed the other fix tags)
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94 #1253 (See https://builds.apache.org/job/HBase-0.94/1253/)
          HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556027)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1253 (See https://builds.apache.org/job/HBase-0.94/1253/ ) HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556027) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Hide
          Jimmy Xiang added a comment -

          The fix went into 0.94 only. Also added some tests for other branches to cover FilterList, no other code change.

          Show
          Jimmy Xiang added a comment - The fix went into 0.94 only. Also added some tests for other branches to cover FilterList, no other code change.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-TRUNK #4794 (See https://builds.apache.org/job/HBase-TRUNK/4794/)
          HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556024)

          • /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4794 (See https://builds.apache.org/job/HBase-TRUNK/4794/ ) HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556024) /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Hide
          Hudson added a comment -

          ABORTED: Integrated in HBase-0.94-JDK7 #20 (See https://builds.apache.org/job/HBase-0.94-JDK7/20/)
          HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556027)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Show
          Hudson added a comment - ABORTED: Integrated in HBase-0.94-JDK7 #20 (See https://builds.apache.org/job/HBase-0.94-JDK7/20/ ) HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556027) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98 #60 (See https://builds.apache.org/job/HBase-0.98/60/)
          HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556025)

          • /hbase/branches/0.98/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #60 (See https://builds.apache.org/job/HBase-0.98/60/ ) HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556025) /hbase/branches/0.98/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in hbase-0.96 #251 (See https://builds.apache.org/job/hbase-0.96/251/)
          HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556026)

          • /hbase/branches/0.96/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Show
          Hudson added a comment - FAILURE: Integrated in hbase-0.96 #251 (See https://builds.apache.org/job/hbase-0.96/251/ ) HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556026) /hbase/branches/0.96/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in hbase-0.96-hadoop2 #170 (See https://builds.apache.org/job/hbase-0.96-hadoop2/170/)
          HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556026)

          • /hbase/branches/0.96/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Show
          Hudson added a comment - SUCCESS: Integrated in hbase-0.96-hadoop2 #170 (See https://builds.apache.org/job/hbase-0.96-hadoop2/170/ ) HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556026) /hbase/branches/0.96/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #55 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/55/)
          HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556025)

          • /hbase/branches/0.98/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #55 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/55/ ) HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556025) /hbase/branches/0.98/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #44 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/44/)
          HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556024)

          • /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #44 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/44/ ) HBASE-10078 Dynamic Filter - Not using DynamicClassLoader when using FilterList (jxiang: rev 1556024) /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java

            People

            • Assignee:
              Jimmy Xiang
              Reporter:
              Federico Gaule
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development