[HADOOP-11708] CryptoOutputStream synchronization differences from DFSOutputStream break HBase

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.6.0
• Component/s: fs
• Assignee: Sean Busbey
• Reporter: Sean Busbey
• Created: 12/Mar/15 15:08
• Updated: 25/Jun/18 14:14

Description

For the write-ahead-log, HBase writes to DFS from a single thread and sends sync/flush/hflush from a configurable number of other threads (default 5).

FSDataOutputStream does not document anything about being thread safe, and it is not thread safe for concurrent writes.

However, DFSOutputStream is thread safe for concurrent writes + syncs. When it is the stream FSDataOutputStream wraps, the combination is threadsafe for 1 writer and multiple syncs (the exact behavior HBase relies on).

When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted between FSDataOutputStream and DFSOutputStream. It is proactively labeled as not thread safe, and this composition is not thread safe for any operations.

Issue Links

• breaks: HBASE-13221 HDFS Transparent Encryption breaks WAL writing in Hadoop 2.6.0
• is duplicated by: HDFS-7911 Buffer Overflow when running HBase on HDFS Encryption Zone
• is related to: HADOOP-15557 CryptoInputStream can't handle concurrent access; inconsistent with HDFS
• relates to: HADOOP-9361 Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
• requires: HDFS-7911 Buffer Overflow when running HBase on HDFS Encryption Zone

Sub-Tasks

• Make CryptoOutputStream behave like DFSOutputStream wrt synchronization: Closed