[CRUNCH-534] Protobuf Size Limit Exception

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: IO
• Assignee: Unassigned
• Reporter: Micah Whitacre
• Created: 02/Jul/15 02:21
• Updated: 14/Jul/15 18:41

Description

With HBase's switch to ProtocolBuffers there is now an imposed restriction on the size of data (64MB) which can be stored in protocol buffers and therefore Cells/Puts/etc.

Call to failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the size limit.

While a single cell being that big seems extreme this limit also counts towards Result objects so an entire row.

It is definitely an HBase problem[1] but Crunch also might have changes needed since we use protobuf serialization in our HBaseTypes.[2] So might need to make sure error isn't only on our side as they work through the issue or might need to look at MOB support[3].

[1] - https://issues.apache.org/jira/browse/HBASE-13825
[2] - https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HBaseTypes.java#L33
[3] - https://issues.apache.org/jira/browse/HBASE-11339

Issue Links

• relates to: HBASE-14076 ResultSerialization and MutationSerialization can throw InvalidProtocolBufferException when serializing a cell larger than 64MB