Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-13338

Hbase to use PowerPC supported Jruby version 1.7.20

Details

    • Task
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 1.0.0, 0.98.12
    • None
    • dependencies, shell
    • None
    • PowerPC64, PowerPC64LE

    Description

      Older versions of jffi (till 1.2.7) don't have native PPC64 & PPC64LE libraries. The latest released 1.2.8 version has PowerPC libraries and the jruby development version(1.7.20--SNAPSHOT) has been updated to make use of this version. Hbase still uses much older jruby 1.6.8 version which don't have the native libraries and this affects the Hbase shell in PowerPCs. So Hbase needs to be updated to make use of the upcoming Jruby release 1.7.20 to support PowerPC.

      Attachments

        1. Changes.patch
          1 kB
          Ayappan

        Issue Links

          Activity

            Hi Ayappan,

            Are you going to provide a patch for that? Testing might be a bit hard since JRuby is mostily used for the shell and there is no unit test for that.

            Also, any improvement on the JRuby side will not really have any impact on the HBase side since HBase mostly use the Java API even from the JRuby calls. What is the expectation of this upgrade?

            Thanks.

            jmspaggi Jean-Marc Spaggiari added a comment - Hi Ayappan, Are you going to provide a patch for that? Testing might be a bit hard since JRuby is mostily used for the shell and there is no unit test for that. Also, any improvement on the JRuby side will not really have any impact on the HBase side since HBase mostly use the Java API even from the JRuby calls. What is the expectation of this upgrade? Thanks.
            jmhsieh Jonathan Hsieh added a comment -

            I agree with jmspaggi here about testing being the concern here. Do you know if the old version you are using is compatible with the new?

            Ayappan, if you make the patch (hopefully just a straight foroward change to pom), can you make sure to run [1] and report on if manually verifies that things seem to work?

            [1] https://github.com/apache/hbase/blob/master/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java

            jmhsieh Jonathan Hsieh added a comment - I agree with jmspaggi here about testing being the concern here. Do you know if the old version you are using is compatible with the new? Ayappan , if you make the patch (hopefully just a straight foroward change to pom), can you make sure to run [1] and report on if manually verifies that things seem to work? [1] https://github.com/apache/hbase/blob/master/hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
            Ayappan Ayappan added a comment -

            "Are you going to provide a patch for that?" Yes. It will be a simple change in pom.xml file.

            "What is the expectation of this upgrade?"
            The Jruby-complete-1.6.8.jar don't have PowerPC native jffi libraries within it due to which we are not able to invoke Hbase shell. See the error below.

            1. bin/hbase shell
              2015-03-19 12:44:14,770 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
              java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: Could not locate stub library in jar file. Tried [jni/ppc64-Linux/libjffi-1.0.so, /jni/ppc64-Linux/libjffi-1.0.so]
              at com.kenai.jffi.Foreign$InValidInstanceHolder.getForeign(Foreign.java:90)
              at com.kenai.jffi.Foreign.getInstance(Foreign.java:95)
              at com.kenai.jffi.Library.openLibrary(Library.java:151)
              at com.kenai.jffi.Library.getCachedInstance(Library.java:125)

            The upcoming jruby version 1.7.20 [ they now call it as 1.7.20-SNAPSHOT ] will have the native PowerPC jffi libraries so that Hbase shell works in PowerPC.

            Ayappan Ayappan added a comment - "Are you going to provide a patch for that?" Yes. It will be a simple change in pom.xml file. "What is the expectation of this upgrade?" The Jruby-complete-1.6.8.jar don't have PowerPC native jffi libraries within it due to which we are not able to invoke Hbase shell. See the error below. bin/hbase shell 2015-03-19 12:44:14,770 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: Could not locate stub library in jar file. Tried [jni/ppc64-Linux/libjffi-1.0.so, /jni/ppc64-Linux/libjffi-1.0.so] at com.kenai.jffi.Foreign$InValidInstanceHolder.getForeign(Foreign.java:90) at com.kenai.jffi.Foreign.getInstance(Foreign.java:95) at com.kenai.jffi.Library.openLibrary(Library.java:151) at com.kenai.jffi.Library.getCachedInstance(Library.java:125) The upcoming jruby version 1.7.20 [ they now call it as 1.7.20-SNAPSHOT ] will have the native PowerPC jffi libraries so that Hbase shell works in PowerPC.
            Ayappan Ayappan added a comment -

            Yes, i will make sure to run the testcase [1].

            And what you refer by "if manually verifies that things seem to work" ?

            Ayappan Ayappan added a comment - Yes, i will make sure to run the testcase [1] . And what you refer by "if manually verifies that things seem to work" ?

            And what you refer by "if manually verifies that things seem to work" ?

            Jon means run the TestShell unit test locally and confirm that it passes for you. This can be done like so:

            mvn clean install -DskipITs -Dtest=TestShell
            

            Do this after you've made your POM changes. This will build HBase and, when in the hbase-shell module, run the TestShell unit test (and only this test). Post the results here. Of course this test fails with your proposed change, you'll want to dig in and try to fix the problem before we can proceed.

            apurtell Andrew Kyle Purtell added a comment - And what you refer by "if manually verifies that things seem to work" ? Jon means run the TestShell unit test locally and confirm that it passes for you. This can be done like so: mvn clean install -DskipITs -Dtest=TestShell Do this after you've made your POM changes. This will build HBase and, when in the hbase-shell module, run the TestShell unit test (and only this test). Post the results here. Of course this test fails with your proposed change, you'll want to dig in and try to fix the problem before we can proceed.
            stack Michael Stack added a comment -

            In past we have been reluctant to upgrade our jruby. See HBASE-7028. JRuby changed their licensing graciously so we could bundle it in an apache project. Would be worth checking it still compatible too before bringing it in.

            stack Michael Stack added a comment - In past we have been reluctant to upgrade our jruby. See HBASE-7028 . JRuby changed their licensing graciously so we could bundle it in an apache project. Would be worth checking it still compatible too before bringing it in.
            jmhsieh Jonathan Hsieh added a comment -

            +1 to what Andrew says.

            jmhsieh Jonathan Hsieh added a comment - +1 to what Andrew says.
            busbey Sean Busbey added a comment -

            Upgrading this won't be just a simple pom change. JRuby 1.7 changes the default Ruby version from 1.8 to 1.9, which is very incompatible. You should be able to tell JRuby to still run in Ruby 1.8 mode (ref). Please make sure to do this.

            I really want to update or Ruby version because Ruby 1.8 is super old, has be EOL by several Ruby communities, and finally starting to die off. I've been waiting for the release of JRuby 9000 to update us off of Ruby 1.8 (release 9.0.0.0 pre-1 just finally landed in january). It updates things to be Ruby 2.2 (and only 2.2 IIRC).

            busbey Sean Busbey added a comment - Upgrading this won't be just a simple pom change. JRuby 1.7 changes the default Ruby version from 1.8 to 1.9, which is very incompatible. You should be able to tell JRuby to still run in Ruby 1.8 mode ( ref ). Please make sure to do this. I really want to update or Ruby version because Ruby 1.8 is super old, has be EOL by several Ruby communities, and finally starting to die off. I've been waiting for the release of JRuby 9000 to update us off of Ruby 1.8 (release 9.0.0.0 pre-1 just finally landed in january). It updates things to be Ruby 2.2 (and only 2.2 IIRC).
            Ayappan Ayappan added a comment -

            Jruby-1_7 branch got all the needed stuffs to work properly in ppc64le as of today. I am able to build & install jruby-1.7.20-SNAPSHOT as well as hbase with the jruby-1.7.20-SNAPSHOT version.

            As expected, TestShell unit test fails with following message.
            .
            Running org.apache.hadoop.hbase.client.TestShell
            Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 15.613 sec <<< FAILURE! - in org.apache.hadoop.hbase.client.TestShell
            testRunShellTests(org.apache.hadoop.hbase.client.TestShell) Time elapsed: 1.433 sec <<< ERROR!
            org.jruby.embed.EvalFailedException: (LoadError) no such file to load – src/test/ruby/hbase/admin_test.rb
            at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:132)
            at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1341)
            at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1386)
            at org.apache.hadoop.hbase.client.TestShell.testRunShellTests(TestShell.java:36)
            Caused by: org.jruby.exceptions.RaiseException: (LoadError) no such file to load – src/test/ruby/hbase/admin_test.rb
            at org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1091)
            at RUBY.(root)(src/test/ruby/tests_runner.rb:68)
            at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)
            at RUBY.(root)(src/test/ruby/tests_runner.rb:57)

            Ayappan Ayappan added a comment - Jruby-1_7 branch got all the needed stuffs to work properly in ppc64le as of today. I am able to build & install jruby-1.7.20-SNAPSHOT as well as hbase with the jruby-1.7.20-SNAPSHOT version. As expected, TestShell unit test fails with following message. . Running org.apache.hadoop.hbase.client.TestShell Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 15.613 sec <<< FAILURE! - in org.apache.hadoop.hbase.client.TestShell testRunShellTests(org.apache.hadoop.hbase.client.TestShell) Time elapsed: 1.433 sec <<< ERROR! org.jruby.embed.EvalFailedException: (LoadError) no such file to load – src/test/ruby/hbase/admin_test.rb at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:132) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1341) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1386) at org.apache.hadoop.hbase.client.TestShell.testRunShellTests(TestShell.java:36) Caused by: org.jruby.exceptions.RaiseException: (LoadError) no such file to load – src/test/ruby/hbase/admin_test.rb at org.jruby.RubyKernel.load(org/jruby/RubyKernel.java:1091) at RUBY.(root)(src/test/ruby/tests_runner.rb:68) at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613) at RUBY.(root)(src/test/ruby/tests_runner.rb:57)
            busbey Sean Busbey added a comment -

            That issue likely means that you are loading JRuby in Ruby 1.9 mode. Please configure it to load in Ruby 1.8 mode.

            The issue specifically is the incompatible change done in 1.9 (one of many) that removed '.' from the load path used to resolve 'reqiure' statements. more info: 1 and 2

            busbey Sean Busbey added a comment - That issue likely means that you are loading JRuby in Ruby 1.9 mode. Please configure it to load in Ruby 1.8 mode. The issue specifically is the incompatible change done in 1.9 (one of many) that removed '.' from the load path used to resolve 'reqiure' statements. more info: 1 and 2
            Ayappan Ayappan added a comment -

            This is about the hbase shell issue.
            I configured the shell to be invoked in ruby 1.8 mode

            HBASE_OPTS="$HBASE_OPTS $HBASE_SHELL_OPTS"

            • CLASS="org.jruby.Main -X+O ${JRUBY_OPTS} ${HBASE_HOME}/bin/hirb.rb"
              + CLASS="org.jruby.Main --1.8 -X+O ${JRUBY_OPTS} ${HBASE_HOME}/bin/hirb.rb"
              elif [ "$COMMAND" = "hbck" ] ; then

            and got this error

            [ayappan@soe07-vm3 bin]$ ./hbase shell
            include_class is deprecated. Use java_import.
            include_class is deprecated. Use java_import.
            include_class is deprecated. Use java_import.
            NoMethodError: undefined method `getTerminal' for Java::Jline::Terminal:Module
            refresh_width at /home/ayappan/hbase/bin/../hbase-shell/src/main/ruby/shell/formatter.rb:33
            initialize at /home/ayappan/hbase/bin/../hbase-shell/src/main/ruby/shell/formatter.rb:46
            (root) at /home/ayappan/hbase/bin/../bin/hirb.rb:128

            It seems like the getTerminal method doesn't exist with the jline that ships with this version. The jline version is upated to 2.11.

            And also the jruby community suggest to go with 1.9 ruby mode since 1.8 has been EOL in the jruby world for years
            https://github.com/jruby/jruby/issues/2912

            Ayappan Ayappan added a comment - This is about the hbase shell issue. I configured the shell to be invoked in ruby 1.8 mode HBASE_OPTS="$HBASE_OPTS $HBASE_SHELL_OPTS" CLASS="org.jruby.Main -X+O ${JRUBY_OPTS} ${HBASE_HOME}/bin/hirb.rb" + CLASS="org.jruby.Main --1.8 -X+O ${JRUBY_OPTS} ${HBASE_HOME}/bin/hirb.rb" elif [ "$COMMAND" = "hbck" ] ; then and got this error [ayappan@soe07-vm3 bin] $ ./hbase shell include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. NoMethodError: undefined method `getTerminal' for Java::Jline::Terminal:Module refresh_width at /home/ayappan/hbase/bin/../hbase-shell/src/main/ruby/shell/formatter.rb:33 initialize at /home/ayappan/hbase/bin/../hbase-shell/src/main/ruby/shell/formatter.rb:46 (root) at /home/ayappan/hbase/bin/../bin/hirb.rb:128 It seems like the getTerminal method doesn't exist with the jline that ships with this version. The jline version is upated to 2.11. And also the jruby community suggest to go with 1.9 ruby mode since 1.8 has been EOL in the jruby world for years https://github.com/jruby/jruby/issues/2912
            busbey Sean Busbey added a comment -

            unfortunately, we're stuck with Ruby 1.8 until HBase 2.0. Can we override the version of jline?

            busbey Sean Busbey added a comment - unfortunately, we're stuck with Ruby 1.8 until HBase 2.0. Can we override the version of jline?
            Ayappan Ayappan added a comment -

            I don't understand what you mean by "override the version of jline". But if i add the older jline 1.0 to the hbase classpath, i am seeing other issues.

            Ayappan Ayappan added a comment - I don't understand what you mean by "override the version of jline". But if i add the older jline 1.0 to the hbase classpath, i am seeing other issues.
            busbey Sean Busbey added a comment -

            I mean tell jruby to use the older jline.

            What kind of failures are we talking about? Could you put up a patch so I can see things locally?

            busbey Sean Busbey added a comment - I mean tell jruby to use the older jline. What kind of failures are we talking about? Could you put up a patch so I can see things locally?
            Ayappan Ayappan added a comment -

            I attached a changes patch here. And i also i added the below line in the hbase/target/cached_classpath.txt file before invoking hbase shell
            /home/ayappan/.m2/repository/jline/jline/1.0/jline-1.0.jar

            Ayappan Ayappan added a comment - I attached a changes patch here. And i also i added the below line in the hbase/target/cached_classpath.txt file before invoking hbase shell /home/ayappan/.m2/repository/jline/jline/1.0/jline-1.0.jar
            asanjar Amir F Sanjar added a comment -

            any update on acceptance of this patch?

            asanjar Amir F Sanjar added a comment - any update on acceptance of this patch?
            busbey Sean Busbey added a comment -

            folding this into HBASE-16196 as discussed there.

            busbey Sean Busbey added a comment - folding this into HBASE-16196 as discussed there.

            People

              Unassigned Unassigned
              Ayappan Ayappan
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: