Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: hbase
    • Labels:
      None

      Description

      After upgrading HBase to 1.1.9 in BIGTOP-2740, the HMaster and RegionServer processes look good on ppc64le. This is a significant improvement, as the previous version (1.1.3) did not work at all.

      However, hbase shell still fails like this with 1.1.9:

      $ hbase shell
      OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
      include_class is deprecated. Use java_import.
      include_class is deprecated. Use java_import.
      include_class is deprecated. Use java_import.
      NoMethodError: undefined method `getTerminal' for Java::Jline::Terminal:Module
        refresh_width at /usr/lib/hbase/lib/ruby/shell/formatter.rb:34
           initialize at /usr/lib/hbase/lib/ruby/shell/formatter.rb:48
               (root) at /usr/lib/hbase/bin/hirb.rb:116
      

      This stems from an incompatibility in the JLine version that ships with the updated JRuby lib (which is required for ppc64le). See HBASE-13338 and jruby issue for details.

      You can work around this by patching formatter.rb:

      --- /usr/lib/hbase/lib/ruby/shell/formatter.rb.old  2017-04-17 23:28:49.339383104 +0000
      +++ /usr/lib/hbase/lib/ruby/shell/formatter.rb    2017-04-17 23:30:46.749288423 +0000
      @@ -31,8 +31,7 @@
             def refresh_width()
               require 'readline'
               if $stdout.tty?
      -          @max_width = Java::jline.Terminal.getTerminal().getTerminalWidth()
      +          @max_width = 0
      

      It's not ideal, but it's a simple enough workaround for anyone needing hbase shell on ppc64le.

      I'm opening this jira to determine how best to apply this workaround, or where to document this as a limitation of HBase 1.1.x on ppc64le.

        Issue Links

          Activity

          Hide
          kwmonroe Kevin W Monroe added a comment -

          LGTM, committed to master.

          Show
          kwmonroe Kevin W Monroe added a comment - LGTM, committed to master.
          Hide
          evans_ye Evans Ye added a comment -

          Patch looks good. Do we have any concern here?

          Show
          evans_ye Evans Ye added a comment - Patch looks good. Do we have any concern here?
          Hide
          kwmonroe Kevin W Monroe added a comment - - edited

          Olaf Flebbe looks great! My local build threw the same package errors as seen in the ci job. I think they can be safely ignored – they're definitely unrelated to your patch:

          https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hbase,OS=ubuntu-16.04-ppc64le/48/console

          On to the test:

          $ uname -a
          Linux juju-8d1ef7-4 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:05:15 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
          
          $ dpkg -l | grep hbase
          ii  hbase                                 1.1.9-1                                    all          HBase is the Hadoop database. Use it when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware.
          ii  hbase-master                          1.1.9-1                                    all          The Hadoop HBase master Server.
          ii  hbase-regionserver                    1.1.9-1                                    all          The Hadoop HBase RegionServer server.
          
          $ hbase shell
          include_class is deprecated. Use java_import.
          include_class is deprecated. Use java_import.
          include_class is deprecated. Use java_import.
          HBase Shell; enter 'help<RETURN>' for list of supported commands.
          Type "exit<RETURN>" to leave the HBase Shell
          Version 1.1.9, rUnknown, Wed Apr 26 20:24:43 UTC 2017
          
          /usr/lib/hbase/lib/ruby/irb/hirb.rb:22: Use RbConfig instead of obsolete and deprecated Config.
          hbase(main):001:0> version
          1.1.9, rUnknown, Wed Apr 26 20:24:43 UTC 2017
          
          hbase(main):002:0> create 'foo', 'f1'
          0 row(s) in 2.8520 seconds
          
          => Hbase::Table - foo
          hbase(main):003:0> list 'foo'
          TABLE
          foo
          1 row(s) in 0.0360 seconds
          
          => ["foo"]
          hbase(main):004:0>
          

          I should note that I only have access to ppc64le ubuntu machines, so I was only able to test the deb. That said, the rpm spec file change looks simple enough. I'm +1 to merge for bigtop-1.2.1.

          Show
          kwmonroe Kevin W Monroe added a comment - - edited Olaf Flebbe looks great! My local build threw the same package errors as seen in the ci job. I think they can be safely ignored – they're definitely unrelated to your patch: https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hbase,OS=ubuntu-16.04-ppc64le/48/console On to the test: $ uname -a Linux juju-8d1ef7-4 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:05:15 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux $ dpkg -l | grep hbase ii hbase 1.1.9-1 all HBase is the Hadoop database. Use it when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware. ii hbase-master 1.1.9-1 all The Hadoop HBase master Server. ii hbase-regionserver 1.1.9-1 all The Hadoop HBase RegionServer server. $ hbase shell include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 1.1.9, rUnknown, Wed Apr 26 20:24:43 UTC 2017 /usr/lib/hbase/lib/ruby/irb/hirb.rb:22: Use RbConfig instead of obsolete and deprecated Config. hbase(main):001:0> version 1.1.9, rUnknown, Wed Apr 26 20:24:43 UTC 2017 hbase(main):002:0> create 'foo', 'f1' 0 row(s) in 2.8520 seconds => Hbase::Table - foo hbase(main):003:0> list 'foo' TABLE foo 1 row(s) in 0.0360 seconds => ["foo"] hbase(main):004:0> I should note that I only have access to ppc64le ubuntu machines, so I was only able to test the deb. That said, the rpm spec file change looks simple enough. I'm +1 to merge for bigtop-1.2.1.
          Hide
          oflebbe Olaf Flebbe added a comment -

          Untested. Please verify Kevin W Monroe .

          Show
          oflebbe Olaf Flebbe added a comment - Untested. Please verify Kevin W Monroe .
          Hide
          oflebbe Olaf Flebbe added a comment -

          Will commit BIGTOP-2740 and propose a patch for this one.

          Show
          oflebbe Olaf Flebbe added a comment - Will commit BIGTOP-2740 and propose a patch for this one.
          Hide
          rvs Roman Shaposhnik added a comment -

          Agreed. Can somebody please turn this into a patch/PR?

          Show
          rvs Roman Shaposhnik added a comment - Agreed. Can somebody please turn this into a patch/PR?
          Hide
          evans_ye Evans Ye added a comment -

          Looks promising. +1.

          Show
          evans_ye Evans Ye added a comment - Looks promising. +1.
          Hide
          kwmonroe Kevin W Monroe added a comment -

          Olaf Flebbe, that's way better! Patched like this, which is just a minor tweak to max_width instead of maxwidth:

          --- /tmp/rbog   2017-04-18 20:21:38.909178083 +0000
          +++ /usr/lib/hbase/lib/ruby/shell/formatter.rb  2017-04-18 21:51:45.352710125 +0000
          @@ -31,7 +31,11 @@
                 def refresh_width()
                   require 'readline'
                   if $stdout.tty?
          -          @max_width = Java::jline.Terminal.getTerminal().getTerminalWidth()
          +          if Java::jline.Terminal.respond_to?(:getTerminal)
          +            @max_width = Java::jline.Terminal.getTerminal.getTerminalWidth
          +          else
          +            @max_width = Java::jline.TerminalFactory.get.getWidth
          +          end
                   else
                     @max_width = 0
                   end
          

          Results on ppc with a simple smoke:

          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ uname -a
          Linux juju-8d1ef7-4 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:05:15 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
          
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ hbase version
          OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
          2017-04-18 22:16:41,529 INFO  [main] util.VersionInfo: HBase 1.1.9
          2017-04-18 22:16:41,529 INFO  [main] util.VersionInfo: Source code repository file:///home/ubuntu/bigtop.release/bigtop-1.2.0/output/hbase/hbase-1.1.9 revision=Unknown
          2017-04-18 22:16:41,529 INFO  [main] util.VersionInfo: Compiled by ubuntu on Fri Apr 14 18:10:52 UTC 2017
          2017-04-18 22:16:41,529 INFO  [main] util.VersionInfo: From source with checksum 84e3be2483b47031d3a8c4bad05ff89a
          
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ tbl="bigtop4life"
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "create '$tbl', 'cf'" > hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "list '$tbl'" >> hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "put '$tbl', 'row1', 'cf:a', 'value1'" >> hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "put '$tbl', 'row2', 'cf:b', 'value2'" >> hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "put '$tbl', 'row3', 'cf:c', 'value3'" >> hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "put '$tbl', 'row4', 'cf:d', 'value4'" >> hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "scan '$tbl'" >> hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "get '$tbl', 'row1'" >> hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "disable '$tbl'" >> hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "enable '$tbl'" >> hbase_table.txt
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "exit;" >> hbase_table.txt
          
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ hbase shell ./hbase_table.txt
          OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
          include_class is deprecated. Use java_import.
          include_class is deprecated. Use java_import.
          include_class is deprecated. Use java_import.
          0 row(s) in 2.6260 seconds
          
          TABLE
          bigtop4life
          1 row(s) in 0.0210 seconds
          
          0 row(s) in 0.2140 seconds
          
          0 row(s) in 0.0090 seconds
          
          0 row(s) in 0.0070 seconds
          
          0 row(s) in 0.0070 seconds
          
          ROW                                                   COLUMN+CELL
           row1                                                 column=cf:a, timestamp=1492552738259, value=value1
           row2                                                 column=cf:b, timestamp=1492552738283, value=value2
           row3                                                 column=cf:c, timestamp=1492552738292, value=value3
           row4                                                 column=cf:d, timestamp=1492552738300, value=value4
          4 row(s) in 0.0620 seconds
          
          COLUMN                                                CELL
           cf:a                                                 timestamp=1492552738259, value=value1
          1 row(s) in 0.0230 seconds
          
          0 row(s) in 2.2810 seconds
          
          0 row(s) in 1.2810 seconds
          
          ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "describe '$tbl'" | hbase shell
          OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
          include_class is deprecated. Use java_import.
          include_class is deprecated. Use java_import.
          include_class is deprecated. Use java_import.
          HBase Shell; enter 'help<RETURN>' for list of supported commands.
          Type "exit<RETURN>" to leave the HBase Shell
          Version 1.1.9, rUnknown, Fri Apr 14 18:10:52 UTC 2017
          
          /usr/lib/hbase/lib/ruby/irb/hirb.rb:22: Use RbConfig instead of obsolete and deprecated Config.
          describe 'bigtop4life'
          Table bigtop4life is ENABLED
          bigtop4life
          COLUMN FAMILIES DESCRIPTION
          {NAME => 'cf', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACH
          E => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
          1 row(s) in 0.5250 seconds
          
          Show
          kwmonroe Kevin W Monroe added a comment - Olaf Flebbe , that's way better! Patched like this, which is just a minor tweak to max_width instead of maxwidth : --- /tmp/rbog 2017-04-18 20:21:38.909178083 +0000 +++ /usr/lib/hbase/lib/ruby/shell/formatter.rb 2017-04-18 21:51:45.352710125 +0000 @@ -31,7 +31,11 @@ def refresh_width() require 'readline' if $stdout.tty? - @max_width = Java::jline.Terminal.getTerminal().getTerminalWidth() + if Java::jline.Terminal.respond_to?(:getTerminal) + @max_width = Java::jline.Terminal.getTerminal.getTerminalWidth + else + @max_width = Java::jline.TerminalFactory.get.getWidth + end else @max_width = 0 end Results on ppc with a simple smoke: ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ uname -a Linux juju-8d1ef7-4 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:05:15 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ hbase version OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release 2017-04-18 22:16:41,529 INFO [main] util.VersionInfo: HBase 1.1.9 2017-04-18 22:16:41,529 INFO [main] util.VersionInfo: Source code repository file:///home/ubuntu/bigtop.release/bigtop-1.2.0/output/hbase/hbase-1.1.9 revision=Unknown 2017-04-18 22:16:41,529 INFO [main] util.VersionInfo: Compiled by ubuntu on Fri Apr 14 18:10:52 UTC 2017 2017-04-18 22:16:41,529 INFO [main] util.VersionInfo: From source with checksum 84e3be2483b47031d3a8c4bad05ff89a ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ tbl="bigtop4life" ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "create '$tbl', 'cf'" > hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "list '$tbl'" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "put '$tbl', 'row1', 'cf:a', 'value1'" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "put '$tbl', 'row2', 'cf:b', 'value2'" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "put '$tbl', 'row3', 'cf:c', 'value3'" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "put '$tbl', 'row4', 'cf:d', 'value4'" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "scan '$tbl'" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "get '$tbl', 'row1'" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "disable '$tbl'" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "enable '$tbl'" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "exit;" >> hbase_table.txt ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ hbase shell ./hbase_table.txt OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. 0 row(s) in 2.6260 seconds TABLE bigtop4life 1 row(s) in 0.0210 seconds 0 row(s) in 0.2140 seconds 0 row(s) in 0.0090 seconds 0 row(s) in 0.0070 seconds 0 row(s) in 0.0070 seconds ROW COLUMN+CELL row1 column=cf:a, timestamp=1492552738259, value=value1 row2 column=cf:b, timestamp=1492552738283, value=value2 row3 column=cf:c, timestamp=1492552738292, value=value3 row4 column=cf:d, timestamp=1492552738300, value=value4 4 row(s) in 0.0620 seconds COLUMN CELL cf:a timestamp=1492552738259, value=value1 1 row(s) in 0.0230 seconds 0 row(s) in 2.2810 seconds 0 row(s) in 1.2810 seconds ubuntu@juju-8d1ef7-4:~/bigtop.release/bigtop-1.2.0$ echo "describe '$tbl'" | hbase shell OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 1.1.9, rUnknown, Fri Apr 14 18:10:52 UTC 2017 /usr/lib/hbase/lib/ruby/irb/hirb.rb:22: Use RbConfig instead of obsolete and deprecated Config. describe 'bigtop4life' Table bigtop4life is ENABLED bigtop4life COLUMN FAMILIES DESCRIPTION {NAME => 'cf', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACH E => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 1 row(s) in 0.5250 seconds
          Hide
          oflebbe Olaf Flebbe added a comment - - edited

          May I propose this instead,

          if Java::jline.Terminal.respond_to?(:getTerminal) 
             @maxwidth = Java::jline.Terminal.getTerminal.getTerminalWidth
          else
             @maxwidth = Java::jline.TerminalFactory.get.getWidth
          end
          
          Show
          oflebbe Olaf Flebbe added a comment - - edited May I propose this instead, if Java::jline.Terminal.respond_to?(:getTerminal) @maxwidth = Java::jline.Terminal.getTerminal.getTerminalWidth else @maxwidth = Java::jline.TerminalFactory.get.getWidth end
          Hide
          kwmonroe Kevin W Monroe added a comment -

          With the workaround mentioned above, hbase shell 1.1.9 on ppc64le looks good:

          $ hbase shell
          OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
          include_class is deprecated. Use java_import.
          include_class is deprecated. Use java_import.
          include_class is deprecated. Use java_import.
          HBase Shell; enter 'help<RETURN>' for list of supported commands.
          Type "exit<RETURN>" to leave the HBase Shell
          Version 1.1.9, rUnknown, Fri Apr 14 18:10:52 UTC 2017
          
          /usr/lib/hbase/lib/ruby/irb/hirb.rb:22: Use RbConfig instead of obsolete and deprecated Config.
          hbase(main):001:0>
          
          Show
          kwmonroe Kevin W Monroe added a comment - With the workaround mentioned above, hbase shell 1.1.9 on ppc64le looks good: $ hbase shell OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. include_class is deprecated. Use java_import. HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 1.1.9, rUnknown, Fri Apr 14 18:10:52 UTC 2017 /usr/lib/hbase/lib/ruby/irb/hirb.rb:22: Use RbConfig instead of obsolete and deprecated Config. hbase(main):001:0>

            People

            • Assignee:
              oflebbe Olaf Flebbe
              Reporter:
              kwmonroe Kevin W Monroe
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development