[HBASE-15638] Shade protobuf

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Protobufs
• Hadoop Flags: Incompatible change

Release Note

Shade/relocate and include the protobuf we use internally. See protobuf chapter in the refguide for more on how we protobuf in hbase-.2.0.0 and going forward.

See https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit# for how we arrived at this approach.

See http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3C07850EDD-7230-431B-9AB0-C5C91B105EEC%40gmail.com%3E for discussion around merging this change and of how we might revert if an alternative to this awkward patch presents itself; e.g. an hadoop with CLASSPATH isolation (and means of dealing with Sparks use of protobuf 2.5.0, etc.)

• Assignee: Michael Stack
• Reporter: Michael Stack
• Created: 12/Apr/16 22:52
• Updated: 28/Apr/17 13:08
• Resolved: 04/Oct/16 05:34

Description

We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs expect all buffers to be on-heap byte arrays. It does not have facility for dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact frustrates the off-heaping-of-the-write-path project as marshalling/unmarshalling of protobufs involves a copy on-heap first.

So, we need to patch our protobuf so it supports off-heap ByteBuffers. To ensure we pick up the patched protobuf always, we need to relocate/shade our protobuf and adjust all protobuf references accordingly.

Given as we have protobufs in our public facing API, Coprocessor Endpoints – which use protobuf Service to describe new API – a blind relocation/shading of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) in particular. For example, in the Table Interface, to invoke a method on a registered CPEP, we have:

<T extends com.google.protobuf.Service,R> Map<byte[],R> coprocessorService(
Class<T> service, byte[] startKey, byte[] endKey,                                             org.apache.hadoop.hbase.client.coprocessor.Batch.Call<T,R> callable)
throws com.google.protobuf.ServiceException, Throwable

This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving our API as is so CPEPs continue to work on the new hbase.

Issue Links

• breaks: HBASE-17975 TokenUtil.obtainToken squashes remote exceptions
• depends upon: HBASE-15174 Client Public API should not have PB objects in 2.0
• is related to: HADOOP-13363 Upgrade protobuf from 2.5.0 to something newer
• relates to: HBASE-16769 Deprecate/remove PB references from MasterObserver and RegionServerObserver
• relates to: MESOS-1203 Shade protobuf dependency in Mesos Java library
• relates to: HBASE-16756 InterfaceAudience annotate our protobuf; distinguish internal; publish public
• relates to: HBASE-16757 Integrate functionality currently done up as Coprocessor Endpoints into core.
• relates to: HBASE-16567 Upgrade to protobuf-3.1.x
• links to: Protobuf Shading Project Plan
• links to: Review Board (master)

Sub-Tasks

• Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol: Resolved
• Figure how to deal with endpoints and shaded pb: Resolved
• Contain protobuf references: Resolved
• Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf: Resolved
• Add chapter for devs on how we do protobufs going forward: Resolved
• Bring back HBaseZeroCopyByteStringer stuff: Resolved
• Avoid ByteString.copyFrom usage wherever possible: Resolved
• Deprecate ByteString related methods in Bytes.java: Resolved