HBase
  1. HBase
  2. HBASE-282

[hbase] provide multiple language bindings for HBase

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      There have been a number of requests for multiple language bindings for HBase. While there is now a REST interface, this may not be suited for high-volume applications. A couple of suggested approaches have been proposed:

      • Provide a Thrift based API (very fast socket based but some of the languages are not well supported)
      • Provide a JSON based API over sockets. (faster than REST, but probably slower than Thrift)

      Others?

      1. hbase-thrift.patch
        310 kB
        David Simpson
      2. libthrift-r746.jar
        44 kB
        David Simpson
      3. Hbase.thrift.txt
        14 kB
        David Simpson
      4. hbase-thrift.patch
        310 kB
        David Simpson

        Issue Links

          Activity

          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-Nightly #364 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/364/ )
          Hide
          stack added a comment -

          Committed. Resolving (Thrift IDL facilitiates, IMV, 'multiple language bindings'). Thanks for the patch Dave.

          Show
          stack added a comment - Committed. Resolving (Thrift IDL facilitiates, IMV, 'multiple language bindings'). Thanks for the patch Dave.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12373003/hbase-thrift.patch
          against trunk revision r611385.

          @author +1. The patch does not contain any @author tags.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new compiler warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1556/testReport/
          Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1556/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1556/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1556/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12373003/hbase-thrift.patch against trunk revision r611385. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1556/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1556/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1556/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1556/console This message is automatically generated.
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-Nightly #363 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/363/ )
          Hide
          David Simpson added a comment -

          re-uploading

          Show
          David Simpson added a comment - re-uploading
          Hide
          David Simpson added a comment -

          re-uploading to apply patch

          Show
          David Simpson added a comment - re-uploading to apply patch
          Hide
          stack added a comment -

          +1 on patch.

          I committed the libthrift.jar so that your patch will build on hudson when you want to submit it there.

          Show
          stack added a comment - +1 on patch. I committed the libthrift.jar so that your patch will build on hudson when you want to submit it there.
          Hide
          David Simpson added a comment -

          Hbase Thrift API

          Show
          David Simpson added a comment - Hbase Thrift API
          Hide
          David Simpson added a comment - - edited

          I attached a patch file and a thrift Java runtime jar file that implements a Thrift service for Hbase. The patch includes the Hbase.thrift interface defintion, a ThriftServer class which can be launched from './bin/hbase thrift', and some example clients which are in the src/examples/thrift directory. I'm also attaching the Hbase.thrift file directly so it's easy to view for comments.

          Here's what's included in the package.html:

          • Package org.apache.hadoop.hbase.thrift Description

          Provides an HBase Thrift service. This directory contains a Thrift interface
          definition file for an Hbase RPC service and a Java server implementation.

          • What is Thrift?

          "Thrift is a software framework for scalable cross-language services
          development. It combines a powerful software stack with a code generation
          engine to build services that work efficiently and seamlessly between C++,
          Java, Python, PHP, and Ruby. Thrift was developed at Facebook, and we are now
          releasing it as open source." For additional information, see
          http://developers.facebook.com/thrift/. Facebook has announced their intent to
          migrate Thrift into Apache Incubator.

          • Description

          The Hbase API is defined in the file Hbase.thrift. A server-side
          implementation of the API is in ThriftServer. The generated interfaces, types,
          and RPC utility files are checked into SVN under the
          org.apache.hadoop.hbase.thrift.generated directory.

          The files were generated by running the commands:

          thrift -strict -java Hbase.thrift
          mv gen-java/org/apache/hadoop/hbase/thrift/generated .
          rm -rf gen-java

          The 'thrift' binary is the Thrift compiler, and it is distributed as a part of
          the Thrift package. Additionally, specific language runtime libraries are a
          part of the Thrift package. A version of the Java runtime is checked into SVN
          under the hbase/lib directory.

          The version of Thrift used to generate the Java files is revision 746 from the
          SVN repository.

          The ThriftServer is run like:

          ./bin/hbase thrift [-h|--help] [-p|--port PORT]

          The default port is 9090.


          Show
          David Simpson added a comment - - edited I attached a patch file and a thrift Java runtime jar file that implements a Thrift service for Hbase. The patch includes the Hbase.thrift interface defintion, a ThriftServer class which can be launched from './bin/hbase thrift', and some example clients which are in the src/examples/thrift directory. I'm also attaching the Hbase.thrift file directly so it's easy to view for comments. Here's what's included in the package.html: Package org.apache.hadoop.hbase.thrift Description Provides an HBase Thrift service. This directory contains a Thrift interface definition file for an Hbase RPC service and a Java server implementation. What is Thrift? "Thrift is a software framework for scalable cross-language services development. It combines a powerful software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, and Ruby. Thrift was developed at Facebook, and we are now releasing it as open source." For additional information, see http://developers.facebook.com/thrift/ . Facebook has announced their intent to migrate Thrift into Apache Incubator. Description The Hbase API is defined in the file Hbase.thrift. A server-side implementation of the API is in ThriftServer. The generated interfaces, types, and RPC utility files are checked into SVN under the org.apache.hadoop.hbase.thrift.generated directory. The files were generated by running the commands: thrift -strict -java Hbase.thrift mv gen-java/org/apache/hadoop/hbase/thrift/generated . rm -rf gen-java The 'thrift' binary is the Thrift compiler, and it is distributed as a part of the Thrift package. Additionally, specific language runtime libraries are a part of the Thrift package. A version of the Java runtime is checked into SVN under the hbase/lib directory. The version of Thrift used to generate the Java files is revision 746 from the SVN repository. The ThriftServer is run like: ./bin/hbase thrift [-h|--help] [-p|--port PORT] The default port is 9090.
          Hide
          David Simpson added a comment -

          Thrift Java runtime library, taken from the facebook SVN repo at revision 746

          Show
          David Simpson added a comment - Thrift Java runtime library, taken from the facebook SVN repo at revision 746
          Hide
          David Simpson added a comment -

          diff that adds the Hbase.thrift API file, a ThriftServer class, and some example clients

          Show
          David Simpson added a comment - diff that adds the Hbase.thrift API file, a ThriftServer class, and some example clients
          Hide
          David Simpson added a comment -

          I'm looking at implementing the server-side of the Thrift API. Let me know if there's already work in progress going on beyond the spec on the wiki. As a matter of introduction, I'm a colleague of Chad, Michael, and Jim over at Powerset and I've been working on various non-Hbase/Hadoop projects that use Thrift.

          Show
          David Simpson added a comment - I'm looking at implementing the server-side of the Thrift API. Let me know if there's already work in progress going on beyond the spec on the wiki. As a matter of introduction, I'm a colleague of Chad, Michael, and Jim over at Powerset and I've been working on various non-Hbase/Hadoop projects that use Thrift.
          Show
          stack added a comment - Related: http://wiki.apache.org/lucene-hadoop/Hbase/ThriftApi
          Hide
          Edward J. Yoon added a comment -

          +1 on both JSON, Thrift.

          Show
          Edward J. Yoon added a comment - +1 on both JSON, Thrift.
          Hide
          Bryan Duxbury added a comment -

          -1 on JSON over sockets. JSON is probably more lightweight than XML in REST, true, but we'd still need some basic transmission protocol, like REST. However, I'm not convinced that JSON would make that much of a difference in the long run, since JSON is a printable format like XML, which means we'd still need to encode the binary data, which I have to think is the slow part.

          +1 on Thrift. It seems to be the best option. If other languages haven't got Thrift libraries, the solution is to make one or use REST. As more time passes, Thrift will support more and more languages, so this will be less of a problem.

          I've created a draft Thrift spec on the HBase wiki at http://wiki.apache.org/lucene-hadoop/Hbase/ThriftApi?action=show. Feel free to make suggestions or changes.

          Show
          Bryan Duxbury added a comment - -1 on JSON over sockets. JSON is probably more lightweight than XML in REST, true, but we'd still need some basic transmission protocol, like REST. However, I'm not convinced that JSON would make that much of a difference in the long run, since JSON is a printable format like XML, which means we'd still need to encode the binary data, which I have to think is the slow part. +1 on Thrift. It seems to be the best option. If other languages haven't got Thrift libraries, the solution is to make one or use REST. As more time passes, Thrift will support more and more languages, so this will be less of a problem. I've created a draft Thrift spec on the HBase wiki at http://wiki.apache.org/lucene-hadoop/Hbase/ThriftApi?action=show . Feel free to make suggestions or changes.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jim Kellerman
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development