HBase
  1. HBase
  2. HBASE-1936

ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.98.0, 0.94.7, 0.95.1
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      With this patch, customer filters can be dropped into a pre-configured folder (hbase.dynamic.jars.dir), which can be in hdfs. Region servers can pick them up dynamically, no need to restart the cluster for the new filters to take effect.

      However, if a filter class is already loaded, it won't be un-loaded. Therefore, we can't load a new version of an existing class. Users have to have a proper way to do filter class versioning.
      Show
      With this patch, customer filters can be dropped into a pre-configured folder (hbase.dynamic.jars.dir), which can be in hdfs. Region servers can pick them up dynamically, no need to restart the cluster for the new filters to take effect. However, if a filter class is already loaded, it won't be un-loaded. Therefore, we can't load a new version of an existing class. Users have to have a proper way to do filter class versioning.
    1. 0.94-1936.patch
      29 kB
      Jimmy Xiang
    2. cp_from_hdfs.patch
      28 kB
      stack
    3. HBASE-1936-trunk(forReview).patch
      32 kB
      Jieshan Bean
    4. trunk-1936_v2.1.patch
      25 kB
      Jimmy Xiang
    5. trunk-1936_v2.2.patch
      23 kB
      Jimmy Xiang
    6. trunk-1936_v2.patch
      25 kB
      Jimmy Xiang
    7. trunk-1936_v3.patch
      147 kB
      Jimmy Xiang
    8. trunk-1936.patch
      24 kB
      Jimmy Xiang

      Issue Links

        Activity

        Hide
        stack added a comment -

        Here is start on a classloader that will add to local classpath the content of a folder up in hdfs.

        Show
        stack added a comment - Here is start on a classloader that will add to local classpath the content of a folder up in hdfs.
        Hide
        Jieshan Bean added a comment -

        @stack,
        can you look back to this new feature again? I think it's very useful either for hot deployment of Filter or dynamic coprocessor. Any potential reasons for hanging this task up?

        Thank you.

        Show
        Jieshan Bean added a comment - @stack, can you look back to this new feature again? I think it's very useful either for hot deployment of Filter or dynamic coprocessor. Any potential reasons for hanging this task up? Thank you.
        Hide
        ramkrishna.s.vasudevan added a comment -

        @Jieshan
        Thanks for bringing this up.

        Show
        ramkrishna.s.vasudevan added a comment - @Jieshan Thanks for bringing this up.
        Hide
        stack added a comment -

        @Jieshan Does this patch even apply any more?

        Show
        stack added a comment - @Jieshan Does this patch even apply any more?
        Hide
        Jieshan Bean added a comment -

        Yes.
        Regarding on how to add the new class into HDFS seems not mentioned in this patch. Should we also take care of this?

        Show
        Jieshan Bean added a comment - Yes. Regarding on how to add the new class into HDFS seems not mentioned in this patch. Should we also take care of this?
        Hide
        stack added a comment -

        Jieshan This patch is really old. Do whatever you want with it to make it work.

        Show
        stack added a comment - Jieshan This patch is really old. Do whatever you want with it to make it work.
        Hide
        Jieshan Bean added a comment -

        Thank you, stack.
        I will make a patch based on this one.

        Show
        Jieshan Bean added a comment - Thank you, stack. I will make a patch based on this one.
        Hide
        Lars Hofhansl added a comment -

        We have since solved this problem with coprocessors, right?
        It might make sense to look at how coprocessors handle loading from HDFS and (hopefully) reuse the bits.

        Thanks for picking this up again Jieshan.

        Show
        Lars Hofhansl added a comment - We have since solved this problem with coprocessors, right? It might make sense to look at how coprocessors handle loading from HDFS and (hopefully) reuse the bits. Thanks for picking this up again Jieshan.
        Hide
        Jieshan Bean added a comment -

        Thank you Lars. I will check

        Show
        Jieshan Bean added a comment - Thank you Lars. I will check
        Hide
        Jieshan Bean added a comment -

        This patch is just for review.

        I have checked the code in coprocessor, it can't support the dynamic class during running, though I can refer to the code of how to load jars from HDFS.
        Plz correct me if I'm wrong.

        In current approach, will use this classloader as the default one(Plz share your comment if you think different.).

        I'm still doing more tests for this patch.

        (I will upload the introductive document next monday. Thank you)

        Show
        Jieshan Bean added a comment - This patch is just for review. I have checked the code in coprocessor, it can't support the dynamic class during running, though I can refer to the code of how to load jars from HDFS. Plz correct me if I'm wrong. In current approach, will use this classloader as the default one(Plz share your comment if you think different.). I'm still doing more tests for this patch. (I will upload the introductive document next monday. Thank you)
        Hide
        Ted Yu added a comment -

        The patch is of decent size. Please upload to review board.

        Why does callIsATriggeringClass() use reflection to call isATriggeringClass() ?

        Can you add some tests ?

        Thanks

        Show
        Ted Yu added a comment - The patch is of decent size. Please upload to review board. Why does callIsATriggeringClass() use reflection to call isATriggeringClass() ? Can you add some tests ? Thanks
        Hide
        Andrew Purtell added a comment -

        I have checked the code in coprocessor, it can't support the dynamic class during running, though I can refer to the code of how to load jars from HDFS.

        Can you rephrase this?

        Ideally we should have one classloader that can load from HDFS, not more than one only slightly different from each other.

        Show
        Andrew Purtell added a comment - I have checked the code in coprocessor, it can't support the dynamic class during running, though I can refer to the code of how to load jars from HDFS. Can you rephrase this? Ideally we should have one classloader that can load from HDFS, not more than one only slightly different from each other.
        Hide
        Jieshan Bean added a comment -

        I will upload the patch to review board after more tests.
        Based on stack's original design, ClassPathLocalizer is a configurable class, that's why we use the refrection to call the method. I think I can change this. @stack, what do you think?
        I'm also thinking about how to add test case. Thank you, Ted.

        @Andrew, thank you. I will try to reuse the code. I agree with we should use one classloader. The classloader in CP is not a general one.

        Show
        Jieshan Bean added a comment - I will upload the patch to review board after more tests. Based on stack's original design, ClassPathLocalizer is a configurable class, that's why we use the refrection to call the method. I think I can change this. @stack, what do you think? I'm also thinking about how to add test case. Thank you, Ted. @Andrew, thank you. I will try to reuse the code. I agree with we should use one classloader. The classloader in CP is not a general one.
        Hide
        Jieshan Bean added a comment -

        I'm trying to unify the classLoader used in CP and Filter, it will take a few days. Sorry for the delay.

        Show
        Jieshan Bean added a comment - I'm trying to unify the classLoader used in CP and Filter, it will take a few days. Sorry for the delay.
        Hide
        Lars Hofhansl added a comment -

        Thank Jieshan. Yes, I think you should be able to reuse most of the CP classloading code. In the end you "only" need a classloader which can load a jar from HDFS. Filters (like coprocessors) then would be packed into a jar (together with any supporting classes needed).

        Show
        Lars Hofhansl added a comment - Thank Jieshan. Yes, I think you should be able to reuse most of the CP classloading code. In the end you "only" need a classloader which can load a jar from HDFS. Filters (like coprocessors) then would be packed into a jar (together with any supporting classes needed).
        Hide
        Lars Hofhansl added a comment -

        Any update on this, Jieshan?

        Show
        Lars Hofhansl added a comment - Any update on this, Jieshan?
        Hide
        Jieshan Bean added a comment -

        Sorry, some other higher priority tasks delayed my plan on this feature. I will do it at this weekend, how about that?

        Show
        Jieshan Bean added a comment - Sorry, some other higher priority tasks delayed my plan on this feature. I will do it at this weekend, how about that?
        Hide
        Lars Hofhansl added a comment -

        Hey Jieshan, you're taking long weekends

        Show
        Lars Hofhansl added a comment - Hey Jieshan, you're taking long weekends
        Hide
        Jimmy Xiang added a comment -

        Jieshan Bean, are you still working on this issue?

        Show
        Jimmy Xiang added a comment - Jieshan Bean , are you still working on this issue?
        Hide
        Jieshan Bean added a comment -

        Sorry, I don’t have enough time to finish it currently. Please feel free to take it over if you interest on it

        Show
        Jieshan Bean added a comment - Sorry, I don’t have enough time to finish it currently. Please feel free to take it over if you interest on it 
        Hide
        Jimmy Xiang added a comment -

        No problem. I will take a look when I get a chance.

        Show
        Jimmy Xiang added a comment - No problem. I will take a look when I get a chance.
        Hide
        Lars Hofhansl added a comment -

        I think this is an important improvement (for things like Phoenix for example James Taylor)

        Show
        Lars Hofhansl added a comment - I think this is an important improvement (for things like Phoenix for example James Taylor )
        Hide
        James Taylor added a comment -

        Yes, this is definitely important for Phoenix. If we could run without requiring any server-side jar installation, that would definitely remove one hurdle. It's not that big a deal for a dev environment, but as soon as it's production, getting a new jar installed on all region servers is like rolling a 1 ton rock up hill for a few miles.

        Show
        James Taylor added a comment - Yes, this is definitely important for Phoenix. If we could run without requiring any server-side jar installation, that would definitely remove one hurdle. It's not that big a deal for a dev environment, but as soon as it's production, getting a new jar installed on all region servers is like rolling a 1 ton rock up hill for a few miles.
        Hide
        Jimmy Xiang added a comment -

        I agree. I will take a look. I was thinking to reuse some logic in the coprocessor framework. The new class loader (most likely the existing one in coprocessor framework, with some refactory), will be used to load some non-exempt classes (not system classes) from a configurable location, which could be a local folder, or a HDFS folder. Ideally, this loader will be used only for get/scan calls in region server side (besides coprocessor related).

        Show
        Jimmy Xiang added a comment - I agree. I will take a look. I was thinking to reuse some logic in the coprocessor framework. The new class loader (most likely the existing one in coprocessor framework, with some refactory), will be used to load some non-exempt classes (not system classes) from a configurable location, which could be a local folder, or a HDFS folder. Ideally, this loader will be used only for get/scan calls in region server side (besides coprocessor related).
        Hide
        Jimmy Xiang added a comment -

        A patch was posted on RB: https://reviews.apache.org/r/10356/

        Show
        Jimmy Xiang added a comment - A patch was posted on RB: https://reviews.apache.org/r/10356/
        Hide
        Ted Yu added a comment -

        From https://builds.apache.org/job/PreCommit-HBASE-Build/5242/console , looks like there was compilation error (on hadoop 2.0).

        Show
        Ted Yu added a comment - From https://builds.apache.org/job/PreCommit-HBASE-Build/5242/console , looks like there was compilation error (on hadoop 2.0).
        Hide
        Jimmy Xiang added a comment -

        hbase-common must be installed so that hbase-client can be compiled. How does the pre-commit handle this?

        Show
        Jimmy Xiang added a comment - hbase-common must be installed so that hbase-client can be compiled. How does the pre-commit handle this?
        Hide
        Ted Yu added a comment -

        protobuf has been upgraded to 2.5.0

        Looks like patch needs update:

        1 out of 6 hunks FAILED – saving rejects to file hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java.rej

        Show
        Ted Yu added a comment - protobuf has been upgraded to 2.5.0 Looks like patch needs update: 1 out of 6 hunks FAILED – saving rejects to file hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java.rej
        Hide
        Jimmy Xiang added a comment -

        Rebased to trunk latest.

        Show
        Jimmy Xiang added a comment - Rebased to trunk latest.
        Hide
        Ted Yu added a comment -

        Applying 1936_v2.1.patch, I got:

        1 out of 6 hunks FAILED – saving rejects to file hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java.rej

        Please rebase.

        Show
        Ted Yu added a comment - Applying 1936_v2.1.patch, I got: 1 out of 6 hunks FAILED – saving rejects to file hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java.rej Please rebase.
        Hide
        Jimmy Xiang added a comment -

        Trunk moves fast. Ted Yu, could you please try this one? Thanks.

        Show
        Jimmy Xiang added a comment - Trunk moves fast. Ted Yu , could you please try this one? Thanks.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12578243/trunk-1936_v2.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 5 new or modified tests.

        -1 hadoop2.0. The patch failed to compile against the hadoop 2.0 profile.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5269//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12578243/trunk-1936_v2.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified tests. -1 hadoop2.0 . The patch failed to compile against the hadoop 2.0 profile. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5269//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        I was able to compile based on hadoop 1.0 and 2.0 without error.
        The following tests passed locally:

        mt -Dtest=TestGet,TestDynamicClassLoader

        Show
        Ted Yu added a comment - I was able to compile based on hadoop 1.0 and 2.0 without error. The following tests passed locally: mt -Dtest=TestGet,TestDynamicClassLoader
        Hide
        Jimmy Xiang added a comment -

        Attached v3. Moved Base64 from hbase-server to hbase-common. Changed TestGet to use our version of Base64.

        Show
        Jimmy Xiang added a comment - Attached v3. Moved Base64 from hbase-server to hbase-common. Changed TestGet to use our version of Base64.
        Hide
        Jimmy Xiang added a comment -

        Andrew Purtell, are you ok with v3? We can file a follow-up jira to consolidate/refactory the class loaders when we are ready to fix it.

        Show
        Jimmy Xiang added a comment - Andrew Purtell , are you ok with v3? We can file a follow-up jira to consolidate/refactory the class loaders when we are ready to fix it.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12578254/trunk-1936_v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12578254/trunk-1936_v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5271//console This message is automatically generated.
        Hide
        Andrew Purtell added a comment -

        +1 on v3 patch and notion of followon jira

        Show
        Andrew Purtell added a comment - +1 on v3 patch and notion of followon jira
        Hide
        Ted Yu added a comment -

        @Jimmy:
        Can you fill out description and release notes ?

        Thanks

        Show
        Ted Yu added a comment - @Jimmy: Can you fill out description and release notes ? Thanks
        Hide
        Jimmy Xiang added a comment -

        Integrated into trunk and 0.95. Thanks Andy and Ted for the review.

        Show
        Jimmy Xiang added a comment - Integrated into trunk and 0.95. Thanks Andy and Ted for the review.
        Hide
        Lars Hofhansl added a comment -

        This would be a great improvement for 0.94 (thinking Phoenix). Thoughts? Risk seems moderate.

        Show
        Lars Hofhansl added a comment - This would be a great improvement for 0.94 (thinking Phoenix). Thoughts? Risk seems moderate.
        Hide
        Jimmy Xiang added a comment -

        I am working on a backport patch for 0.94 now. Let me re-open it.

        Show
        Jimmy Xiang added a comment - I am working on a backport patch for 0.94 now. Let me re-open it.
        Hide
        Jimmy Xiang added a comment -

        Will post a patch for 0.94 soon.

        Show
        Jimmy Xiang added a comment - Will post a patch for 0.94 soon.
        Hide
        Jimmy Xiang added a comment -

        Attached is the patch backported to 0.94. Due to Filter class changes (because of protobuf), the patch is a little different (trunk version, the class loader is used in ProtobufUtil; this version, it is used in Get/Scan and a couple filters). However the class loader remains the same.

        Show
        Jimmy Xiang added a comment - Attached is the patch backported to 0.94. Due to Filter class changes (because of protobuf), the patch is a little different (trunk version, the class loader is used in ProtobufUtil; this version, it is used in Get/Scan and a couple filters). However the class loader remains the same.
        Hide
        Lars Hofhansl added a comment -

        Looks good to me. I'll do some more testing too.
        James Taylor Do you guys want to have a look?

        Show
        Lars Hofhansl added a comment - Looks good to me. I'll do some more testing too. James Taylor Do you guys want to have a look?
        Hide
        Ted Yu added a comment -

        Minor comments for 0.94 patch :

        +    } catch (InstantiationException e) {
        +      throw new RuntimeException("Failed deserialize.", e);
        +    } catch (IllegalAccessException e) {
        +      throw new RuntimeException("Failed deserialize.", e);
        

        Can we have better error message above ?

        +   * Dynamic class loader to load filter/comparators
        +   */
        +  private final static ClassLoader CLASS_LOADER;
        

        What if class loader for other purpose is introduced in the future ? Should the above variable name be more specific ?

        Show
        Ted Yu added a comment - Minor comments for 0.94 patch : + } catch (InstantiationException e) { + throw new RuntimeException( "Failed deserialize." , e); + } catch (IllegalAccessException e) { + throw new RuntimeException( "Failed deserialize." , e); Can we have better error message above ? + * Dynamic class loader to load filter/comparators + */ + private final static ClassLoader CLASS_LOADER; What if class loader for other purpose is introduced in the future ? Should the above variable name be more specific ?
        Hide
        Jimmy Xiang added a comment -

        Ted Yu, the error message is from the original readFilelds method of the filters. How do you want the error message changed?
        As to the variable name, we can change if a new one is introduced in the future? I agree it is better to be specific, but not sure what use cases we will have in the future.

        Show
        Jimmy Xiang added a comment - Ted Yu , the error message is from the original readFilelds method of the filters. How do you want the error message changed? As to the variable name, we can change if a new one is introduced in the future? I agree it is better to be specific, but not sure what use cases we will have in the future.
        Hide
        Ted Yu added a comment -

        For error messages, how about the following ?

        +    } catch (InstantiationException e) {
        +      throw new RuntimeException("Couldn't instantiate " + className, e);
        +    } catch (IllegalAccessException e) {
        +      throw new RuntimeException("No access to " + className, e);
        

        I am fine with keeping class loader variable name.

        Show
        Ted Yu added a comment - For error messages, how about the following ? + } catch (InstantiationException e) { + throw new RuntimeException( "Couldn't instantiate " + className, e); + } catch (IllegalAccessException e) { + throw new RuntimeException( "No access to " + className, e); I am fine with keeping class loader variable name.
        Hide
        Jimmy Xiang added a comment -

        Cool, will change the error messages as suggested. Thanks.

        Show
        Jimmy Xiang added a comment - Cool, will change the error messages as suggested. Thanks.
        Hide
        Hudson added a comment -

        Integrated in hbase-0.95-on-hadoop2 #66 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/66/)
        HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467094)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
        • /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
        • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Base64.java
        • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java
        • /hbase/branches/0.95/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        • /hbase/branches/0.95/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java
        • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Base64.java
        • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        Show
        Hudson added a comment - Integrated in hbase-0.95-on-hadoop2 #66 (See https://builds.apache.org/job/hbase-0.95-on-hadoop2/66/ ) HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467094) Result = FAILURE jxiang : Files : /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Base64.java /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java /hbase/branches/0.95/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java /hbase/branches/0.95/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Base64.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #4054 (See https://builds.apache.org/job/HBase-TRUNK/4054/)
        HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467092)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
        • /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Base64.java
        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java
        • /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        • /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Base64.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #4054 (See https://builds.apache.org/job/HBase-TRUNK/4054/ ) HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467092) Result = FAILURE jxiang : Files : /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Base64.java /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Base64.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        Hide
        Hudson added a comment -

        Integrated in hbase-0.95 #141 (See https://builds.apache.org/job/hbase-0.95/141/)
        HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467094)

        Result = SUCCESS
        jxiang :
        Files :

        • /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
        • /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
        • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Base64.java
        • /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java
        • /hbase/branches/0.95/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        • /hbase/branches/0.95/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java
        • /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Base64.java
        • /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        Show
        Hudson added a comment - Integrated in hbase-0.95 #141 (See https://builds.apache.org/job/hbase-0.95/141/ ) HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467094) Result = SUCCESS jxiang : Files : /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java /hbase/branches/0.95/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Base64.java /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java /hbase/branches/0.95/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java /hbase/branches/0.95/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Base64.java /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #494 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/494/)
        HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467092)

        Result = FAILURE
        jxiang :
        Files :

        • /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
        • /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Base64.java
        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java
        • /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        • /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Base64.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #494 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/494/ ) HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467092) Result = FAILURE jxiang : Files : /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestGet.java /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Base64.java /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Base64.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestBase64.java
        Hide
        Enis Soztutar added a comment -

        This looks cool. Can we please document this feature in the book?

        Show
        Enis Soztutar added a comment - This looks cool. Can we please document this feature in the book?
        Hide
        Jimmy Xiang added a comment -

        Sure. Will do that. Probably after the 0.94 version is verified by James Taylor and committed.

        Show
        Jimmy Xiang added a comment - Sure. Will do that. Probably after the 0.94 version is verified by James Taylor and committed.
        Hide
        James Taylor added a comment -

        Thanks for the back port, Jimmy Xiang. I'm thinking Phoenix will primary leverage this to make it easier for folks to get up and running. Currently, we require the phoenix jar to manually be copied to every region server and placed on the HBase classpath. Instead, if I understand this enhancement correctly, at install time we can:

        • copy the phoenix-<major>.<minor version>.jar into HDFS
        • create/replace an HDFS symlink of 'phoenix-latest.jar' at a known location to point to the new jar
        • use this path as the jar path for both coprocessors and filters (THIS PART WASN'T POSSIBLE BEFORE)
        • do a rolling restart of the cluster to start using the new jar
          We'd still need the rolling restart, because we'd need the classes from the old version unloaded and the classes from the new version loaded.

        Does this sound accurate? Thanks

        Show
        James Taylor added a comment - Thanks for the back port, Jimmy Xiang . I'm thinking Phoenix will primary leverage this to make it easier for folks to get up and running. Currently, we require the phoenix jar to manually be copied to every region server and placed on the HBase classpath. Instead, if I understand this enhancement correctly, at install time we can: copy the phoenix-<major>.<minor version>.jar into HDFS create/replace an HDFS symlink of 'phoenix-latest.jar' at a known location to point to the new jar use this path as the jar path for both coprocessors and filters (THIS PART WASN'T POSSIBLE BEFORE) do a rolling restart of the cluster to start using the new jar We'd still need the rolling restart, because we'd need the classes from the old version unloaded and the classes from the new version loaded. Does this sound accurate? Thanks
        Hide
        Jimmy Xiang added a comment -

        Yes, that's right. The reason for the rolling restart is to unload the old version. If you just add some new filters to the existing jar file even the jar file name is the same, you don't need rolling restart.

        It will be helpful if we can unload an old version class. However, this means we need to track where the class is loaded, and when the jar file is changed, to avoid performance issue. We also need to come up a way to unload a class in this scenario, which is not trivial.

        Show
        Jimmy Xiang added a comment - Yes, that's right. The reason for the rolling restart is to unload the old version. If you just add some new filters to the existing jar file even the jar file name is the same, you don't need rolling restart. It will be helpful if we can unload an old version class. However, this means we need to track where the class is loaded, and when the jar file is changed, to avoid performance issue. We also need to come up a way to unload a class in this scenario, which is not trivial.
        Hide
        Andrew Purtell added a comment -

        As soon as we start talking about unloading, we might as well move to an OSGi runtime, otherwise we will make all of the mistakes that went into development of same in new ways here, so let's avoid that until really necessary.

        Show
        Andrew Purtell added a comment - As soon as we start talking about unloading, we might as well move to an OSGi runtime, otherwise we will make all of the mistakes that went into development of same in new ways here, so let's avoid that until really necessary.
        Hide
        Lars Hofhansl added a comment -

        Agreed. Class unloading scares me. Servlet containers had that forever, and they all recommend disabling that feature in production (although that might have changed, has been a while since I worked with Servlets last).

        As far as this patch goes, are we fine committing to 0.94? +1 from me.
        We can discuss more options in further jiras.

        Show
        Lars Hofhansl added a comment - Agreed. Class unloading scares me. Servlet containers had that forever, and they all recommend disabling that feature in production (although that might have changed, has been a while since I worked with Servlets last). As far as this patch goes, are we fine committing to 0.94? +1 from me. We can discuss more options in further jiras.
        Hide
        Jimmy Xiang added a comment -

        Cool. I already committed the patch for trunk/0.95. For the 0.94 version, I will commit it tomorrow if no objection.

        Show
        Jimmy Xiang added a comment - Cool. I already committed the patch for trunk/0.95. For the 0.94 version, I will commit it tomorrow if no objection.
        Hide
        Andrew Purtell added a comment -

        +1 for 0.94

        Show
        Andrew Purtell added a comment - +1 for 0.94
        Hide
        Jimmy Xiang added a comment -

        Integrated into 0.94 as well. Thanks all for the nice review and discussion.

        Show
        Jimmy Xiang added a comment - Integrated into 0.94 as well. Thanks all for the nice review and discussion.
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #960 (See https://builds.apache.org/job/HBase-0.94/960/)
        HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services - ADDENDUM (Revision 1467793)
        HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467789)

        Result = SUCCESS
        jxiang :
        Files :

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java

        jxiang :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Get.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Scan.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #960 (See https://builds.apache.org/job/HBase-0.94/960/ ) HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services - ADDENDUM (Revision 1467793) HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467789) Result = SUCCESS jxiang : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java jxiang : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Get.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Scan.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #137 (See https://builds.apache.org/job/HBase-0.94-security/137/)
        HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services - ADDENDUM (Revision 1467793)
        HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467789)

        Result = SUCCESS
        jxiang :
        Files :

        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java

        jxiang :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Get.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Scan.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java
        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java
        • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #137 (See https://builds.apache.org/job/HBase-0.94-security/137/ ) HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services - ADDENDUM (Revision 1467793) HBASE-1936 ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services (Revision 1467789) Result = SUCCESS jxiang : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java jxiang : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Get.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Scan.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/Classes.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/DynamicClassLoader.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestGet.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestDynamicClassLoader.java
        Hide
        James Taylor added a comment -

        We're looking into leveraging this new feature to ease the installation of Phoenix (https://github.com/forcedotcom/phoenix). Currently we require that the phoenix jar be copied into the HBase lib dir of every region server, followed by a restart. For some background, Phoenix uses both coprocessors and custom filters. These are just the tip of the iceberg, so to speak. There's a ton of shared/foundational phoenix code being used by these coprocessors and filters - our type system, expression evaluation, schema interpretation, throttling code, memory management, etc. So when we say we'd like to upgrade our coprocessor and custom filters to a new version, that means all the foundational classes under it have changed as well.

        If we use this new feature, we're not sure we're easing the burden on our users, since users will still need to:
        1) update the hbase-sites.xml on each region server to set the hbase.dynamics.jar.dir path of the jar
        2) copy the phoenix jar to hdfs
        3) make a sym link to the new phoenix jar
        4) get a rolling restart to be done on the cluster

        My fear would be that (1) would be error prone, and for (2) & (3) the user wouldn't have the necessary perms. And (4), we'll probably just have to live with, but in a utopia, we could just have the new jar be used for new coprocessor/filter invocations.

        My question: how close can we come to automating all of this to the point where we could have a phoenix install script that looks like this:

        hbase install phoenix-1.2.jar

        Is HBASE-8400 a prerequisite? Any other missing pieces? We'd be happy to be a guinea pig/test case for how to solve this problem from a real application/platform standpoint.

        Thanks!

        Show
        James Taylor added a comment - We're looking into leveraging this new feature to ease the installation of Phoenix ( https://github.com/forcedotcom/phoenix ). Currently we require that the phoenix jar be copied into the HBase lib dir of every region server, followed by a restart. For some background, Phoenix uses both coprocessors and custom filters. These are just the tip of the iceberg, so to speak. There's a ton of shared/foundational phoenix code being used by these coprocessors and filters - our type system, expression evaluation, schema interpretation, throttling code, memory management, etc. So when we say we'd like to upgrade our coprocessor and custom filters to a new version, that means all the foundational classes under it have changed as well. If we use this new feature, we're not sure we're easing the burden on our users, since users will still need to: 1) update the hbase-sites.xml on each region server to set the hbase.dynamics.jar.dir path of the jar 2) copy the phoenix jar to hdfs 3) make a sym link to the new phoenix jar 4) get a rolling restart to be done on the cluster My fear would be that (1) would be error prone, and for (2) & (3) the user wouldn't have the necessary perms. And (4), we'll probably just have to live with, but in a utopia, we could just have the new jar be used for new coprocessor/filter invocations. My question: how close can we come to automating all of this to the point where we could have a phoenix install script that looks like this: hbase install phoenix-1.2.jar Is HBASE-8400 a prerequisite? Any other missing pieces? We'd be happy to be a guinea pig/test case for how to solve this problem from a real application/platform standpoint. Thanks!
        Hide
        stack added a comment -

        Jimmy Xiang Hey Jimmy, what is this:

        +  private static final String WRITABLE_GET =
        +    "AgD//////////wAAAAEBD3Rlc3QuTW9ja0ZpbHRlcgEAAAAAAAAAAH//////////AQAAAAAAAAAA";
        

        It seems to be a class named test.MockFilter.

        When Get goes to read into an instance of this class, it is failing with an AbstractMethodException in 0.94 when it looks like you were expecting a fail on can't find class test.MockFilter...

        See https://builds.apache.org/job/HBase-0.94/1040/testReport/org.apache.hadoop.hbase.client/TestGet/testDynamicFilter/

        It has been happening over the last few builds both in 0.94 and in security.

        I'm not sure what changed and it is hard to track (it was failing locally regardless of what version of 0.94 I checked out and now it is passing again – as though it were a library issue or something stuck in my repo).

        Let me add catching this exception for now so tests pass again...hbsae-8881

        Show
        stack added a comment - Jimmy Xiang Hey Jimmy, what is this: + private static final String WRITABLE_GET = + "AgD //////////wAAAAEBD3Rlc3QuTW9ja0ZpbHRlcgEAAAAAAAAAAH//////////AQAAAAAAAAAA" ; It seems to be a class named test.MockFilter. When Get goes to read into an instance of this class, it is failing with an AbstractMethodException in 0.94 when it looks like you were expecting a fail on can't find class test.MockFilter... See https://builds.apache.org/job/HBase-0.94/1040/testReport/org.apache.hadoop.hbase.client/TestGet/testDynamicFilter/ It has been happening over the last few builds both in 0.94 and in security. I'm not sure what changed and it is hard to track (it was failing locally regardless of what version of 0.94 I checked out and now it is passing again – as though it were a library issue or something stuck in my repo). Let me add catching this exception for now so tests pass again...hbsae-8881
        Hide
        stack added a comment -

        See over in HBASE-8881. I catch the new AbstractMethodException but then it fails later when we try load again later in the test method. What is this serialized Get Jimmy Xiang? What does its contained filter implement. Looking at commits I do not see anything that recently changed Filter classes making them abstract. What you reckin? Thanks.

        Show
        stack added a comment - See over in HBASE-8881 . I catch the new AbstractMethodException but then it fails later when we try load again later in the test method. What is this serialized Get Jimmy Xiang ? What does its contained filter implement. Looking at commits I do not see anything that recently changed Filter classes making them abstract. What you reckin? Thanks.
        Hide
        Jimmy Xiang added a comment -

        Stack, yes, it is a jar file with a test class. At first, the test expects a fail on can't find the class. Then we put the jar in place so it can find the class dynamically.

        Not sure what has changed recently. I will take a look.

        Show
        Jimmy Xiang added a comment - Stack , yes, it is a jar file with a test class. At first, the test expects a fail on can't find the class. Then we put the jar in place so it can find the class dynamically. Not sure what has changed recently. I will take a look.
        Hide
        kiran added a comment -

        Is this issue related (http://article.gmane.org/gmane.comp.java.hadoop.hbase.user/37652/match=0.94.7) to HBASE-1936 ? If so, how can the classes loaded from hdfs read the configurations and their values ?

        Show
        kiran added a comment - Is this issue related ( http://article.gmane.org/gmane.comp.java.hadoop.hbase.user/37652/match=0.94.7 ) to HBASE-1936 ? If so, how can the classes loaded from hdfs read the configurations and their values ?
        Hide
        Jimmy Xiang added a comment -

        Most likely it is not related. Have you change the code here a little to pin point which one is NULL?

        props.put("zk.connect", env.getConfiguration().get("a.zk.connect"));
        
        Show
        Jimmy Xiang added a comment - Most likely it is not related. Have you change the code here a little to pin point which one is NULL? props.put("zk.connect", env.getConfiguration().get("a.zk.connect"));
        Hide
        kiran added a comment -

        Actually, I did not change anything except the variable name 'a.zk.connect'. The real variable name is 'kafka.zk.connect'. I don't get why the configuration defined in 'hbase-site.xml' is not being read by the coprocessor. When I get time, I will test this with 0.94.6 and check if 0.94.7 is the real cause for error.

        Show
        kiran added a comment - Actually, I did not change anything except the variable name 'a.zk.connect'. The real variable name is 'kafka.zk.connect'. I don't get why the configuration defined in 'hbase-site.xml' is not being read by the coprocessor. When I get time, I will test this with 0.94.6 and check if 0.94.7 is the real cause for error.
        Hide
        kiran added a comment -

        I tested same code on HBase 0.94.6 and it ran successfully. Looks like the issue is related to 0.94.7 but I am not sure if HBASE-1936 is the reason.

        Show
        kiran added a comment - I tested same code on HBase 0.94.6 and it ran successfully. Looks like the issue is related to 0.94.7 but I am not sure if HBASE-1936 is the reason.
        Hide
        Lars Hofhansl added a comment -

        Hmm... So you're saying the configuration is not loaded when the coprocessors are loaded via HDFS?

        Show
        Lars Hofhansl added a comment - Hmm... So you're saying the configuration is not loaded when the coprocessors are loaded via HDFS?
        Hide
        kiran added a comment -

        Yes. I observed it with 0.94.7 but not with 0.94.6

        Show
        kiran added a comment - Yes. I observed it with 0.94.7 but not with 0.94.6
        Hide
        Jimmy Xiang added a comment -

        Can we create a jira to track this issue? I can do that if you don't mind. We can create a test case to confirm the issue.

        Show
        Jimmy Xiang added a comment - Can we create a jira to track this issue? I can do that if you don't mind. We can create a test case to confirm the issue.
        Hide
        kiran added a comment -

        Please go ahead. I just wanted to check with you before creating another issue. Thank you.

        Show
        kiran added a comment - Please go ahead. I just wanted to check with you before creating another issue. Thank you.
        Hide
        Jimmy Xiang added a comment -

        HBASE-9532. I will take a look. Thanks for reporting it, Kiran.

        Show
        Jimmy Xiang added a comment - HBASE-9532 . I will take a look. Thanks for reporting it, Kiran .

          People

          • Assignee:
            Jimmy Xiang
            Reporter:
            stack
          • Votes:
            2 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development