[HIVE-21146] Enforce TransactionBatch size=1 for blob stores

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Component/s: Streaming, Transactions
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Assignee: David Lavati
• Reporter: Eugene Koifman
• Created: 22/Jan/19 18:30
• Updated: 05/Dec/19 10:52
• Resolved: 05/Dec/19 10:52

Description

Streaming Ingest API supports a concept of TransactionBatch where N transactions can be opened at once and the data in all of them will be written to the same delta_x_y directory where each transaction in the batch can be committed/aborted independently.  The implementation relies on FSDataOutputStream.hflush() (called from OrcRecordUpdater}} which is available on HDFS but is often implemented as no-op in Blob store backed FileSystem objects.

Need to add a check to HiveStreamingConnection() constructor to raise an error if builder.transactionBatchSize > 1 and the target table/partitions are backed by something that doesn't support hflush().

Issue Links

• links to: GitHub Pull Request #797