Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-16567

Upgrade to protobuf-3.1.x

    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: Protobufs
    • Labels:
      None
    • Release Note:
      Core is now up on protobuf 3.1.0 (Coprocessor Endpoints and REST are still on protobuf 2.5.0).

      Description

      Move master branch on to protobuf3. See https://github.com/google/protobuf/releases We'd do it because pb3 saves some on byte copies can work with offheap buffers – needed for the off-heap write path project – though read-time is still a TODO (this means pb3 is not enough; we'll have to patch it – or patch pb2.5).

      HBASE-15638 has us first shading protobufs before upgrading. Let us list here issues just going to pb3 without shading if only for completeness sake; i.e. do we have to shade?

      • pb3 is by default wire compatible with pb2.
      • protoc3 run against our .protos works fine except pb3 breaks our HBaseZeroCopyLiteralByteString hack so this has to be removed (possibly recast using new pb3 types)
      • Starting up a cluster that is all pb3 seems to work fine.
      • A pb2 branch-1 can read and write against the pb3 master cluster.

      What will break if we just upgrade to pb3?

      • We should be able to write HDFS messages on our AsyncWAL using pb3; the pb2 HDFS should be able to read them (not tested). Or maybe not. See policy here: https://github.com/google/protobuf/issues/1852 which seems to indicate pb3s will not be able to write compatible pb2 Messages. TODO.
      • Core Coprocessor Endpoints such as AccessControl seem to just work (their protos will have been protoc3'd). I did simple test with a server from master branch up on pb3 and then going against it with a branch-1 client on pb2. I was able to add grants.
      • For non-core CPEPs where the protos are pb2 still, it might just work. To test. It would not be the end-of-the-world if they did not.

        Attachments

        1. HBASE-16567.master.001.patch
          12.11 MB
          Michael Stack
        2. 16567.patch
          9.33 MB
          Michael Stack

          Issue Links

            Activity

              People

              • Assignee:
                stack Michael Stack
                Reporter:
                stack Michael Stack
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: