Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15638

Shade protobuf

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 2.0.0
    • Protobufs
    • None
    • Incompatible change
    • Hide
      Shade/relocate and include the protobuf we use internally. See protobuf chapter in the refguide for more on how we protobuf in hbase-.2.0.0 and going forward.

      See https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit# for how we arrived at this approach.

      See http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3C07850EDD-7230-431B-9AB0-C5C91B105EEC%40gmail.com%3E for discussion around merging this change and of how we might revert if an alternative to this awkward patch presents itself; e.g. an hadoop with CLASSPATH isolation (and means of dealing with Sparks use of protobuf 2.5.0, etc.)
      Show
      Shade/relocate and include the protobuf we use internally. See protobuf chapter in the refguide for more on how we protobuf in hbase-.2.0.0 and going forward. See https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit# for how we arrived at this approach. See http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3C07850EDD-7230-431B-9AB0-C5C91B105EEC%40gmail.com%3E for discussion around merging this change and of how we might revert if an alternative to this awkward patch presents itself; e.g. an hadoop with CLASSPATH isolation (and means of dealing with Sparks use of protobuf 2.5.0, etc.)

    Description

      We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs expect all buffers to be on-heap byte arrays. It does not have facility for dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact frustrates the off-heaping-of-the-write-path project as marshalling/unmarshalling of protobufs involves a copy on-heap first.

      So, we need to patch our protobuf so it supports off-heap ByteBuffers. To ensure we pick up the patched protobuf always, we need to relocate/shade our protobuf and adjust all protobuf references accordingly.

      Given as we have protobufs in our public facing API, Coprocessor Endpoints – which use protobuf Service to describe new API – a blind relocation/shading of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) in particular. For example, in the Table Interface, to invoke a method on a registered CPEP, we have:

      <T extends com.google.protobuf.Service,R> Map<byte[],R> coprocessorService(
      Class<T> service, byte[] startKey, byte[] endKey,                                             org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R> callable)
      throws com.google.protobuf.ServiceException, Throwable

      This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving our API as is so CPEPs continue to work on the new hbase.

      Attachments

        1. HBASE-15638.master.009.patch
          505 kB
          Michael Stack
        2. HBASE-15638.master.008.patch
          497 kB
          Michael Stack
        3. HBASE-15638.master.007.patch
          490 kB
          Michael Stack
        4. HBASE-15638.master.007.patch
          490 kB
          Michael Stack
        5. HBASE-15638.master.006.patch
          479 kB
          Michael Stack
        6. HBASE-15638.master.005.patch
          409 kB
          Michael Stack
        7. HBASE-15638.master.004.patch
          343 kB
          Michael Stack
        8. HBASE-15638.master.003 (1).patch
          330 kB
          Michael Stack
        9. HBASE-15638.master.003 (1).patch
          330 kB
          Michael Stack
        10. HBASE-15638.master.003 (1).patch
          330 kB
          Michael Stack
        11. HBASE-15638.master.003.patch
          330 kB
          Michael Stack
        12. HBASE-15638.master.003.patch
          330 kB
          Michael Stack
        13. HBASE-15638.master.002.patch
          3.23 MB
          Michael Stack
        14. HBASE-15638.master.001.patch
          3.21 MB
          Michael Stack
        15. as.far.as.server.patch
          186 kB
          Michael Stack
        16. 15638v2.patch
          1.38 MB
          Michael Stack

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stack Michael Stack
            stack Michael Stack
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment