Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-15638

Shade protobuf

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: Protobufs
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      Shade/relocate and include the protobuf we use internally. See protobuf chapter in the refguide for more on how we protobuf in hbase-.2.0.0 and going forward.

      See https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit# for how we arrived at this approach.

      See http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3C07850EDD-7230-431B-9AB0-C5C91B105EEC%40gmail.com%3E for discussion around merging this change and of how we might revert if an alternative to this awkward patch presents itself; e.g. an hadoop with CLASSPATH isolation (and means of dealing with Sparks use of protobuf 2.5.0, etc.)
      Show
      Shade/relocate and include the protobuf we use internally. See protobuf chapter in the refguide for more on how we protobuf in hbase-.2.0.0 and going forward. See https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit# for how we arrived at this approach. See http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3C07850EDD-7230-431B-9AB0-C5C91B105EEC%40gmail.com%3E for discussion around merging this change and of how we might revert if an alternative to this awkward patch presents itself; e.g. an hadoop with CLASSPATH isolation (and means of dealing with Sparks use of protobuf 2.5.0, etc.)

      Description

      We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs expect all buffers to be on-heap byte arrays. It does not have facility for dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact frustrates the off-heaping-of-the-write-path project as marshalling/unmarshalling of protobufs involves a copy on-heap first.

      So, we need to patch our protobuf so it supports off-heap ByteBuffers. To ensure we pick up the patched protobuf always, we need to relocate/shade our protobuf and adjust all protobuf references accordingly.

      Given as we have protobufs in our public facing API, Coprocessor Endpoints – which use protobuf Service to describe new API – a blind relocation/shading of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) in particular. For example, in the Table Interface, to invoke a method on a registered CPEP, we have:

      <T extends com.google.protobuf.Service,R> Map<byte[],R> coprocessorService(
      Class<T> service, byte[] startKey, byte[] endKey,                                             org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R> callable)
      throws com.google.protobuf.ServiceException, Throwable

      This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving our API as is so CPEPs continue to work on the new hbase.

        Attachments

        1. HBASE-15638.master.009.patch
          505 kB
          Michael Stack
        2. HBASE-15638.master.008.patch
          497 kB
          Michael Stack
        3. HBASE-15638.master.007.patch
          490 kB
          Michael Stack
        4. HBASE-15638.master.007.patch
          490 kB
          Michael Stack
        5. HBASE-15638.master.006.patch
          479 kB
          Michael Stack
        6. HBASE-15638.master.005.patch
          409 kB
          Michael Stack
        7. HBASE-15638.master.004.patch
          343 kB
          Michael Stack
        8. HBASE-15638.master.003 (1).patch
          330 kB
          Michael Stack
        9. HBASE-15638.master.003 (1).patch
          330 kB
          Michael Stack
        10. HBASE-15638.master.003 (1).patch
          330 kB
          Michael Stack
        11. HBASE-15638.master.003.patch
          330 kB
          Michael Stack
        12. HBASE-15638.master.003.patch
          330 kB
          Michael Stack
        13. HBASE-15638.master.002.patch
          3.23 MB
          Michael Stack
        14. HBASE-15638.master.001.patch
          3.21 MB
          Michael Stack
        15. 15638v2.patch
          1.38 MB
          Michael Stack
        16. as.far.as.server.patch
          186 kB
          Michael Stack

          Issue Links

            Activity

              People

              • Assignee:
                stack Michael Stack
                Reporter:
                stack Michael Stack
              • Votes:
                0 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: