[HADOOP-14747] S3AInputStream to implement CanUnbuffer

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.1
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Target Version/s: 3.3.0
• Assignee: Sahil Takiar
• Reporter: Steve Loughran
• Created: 08/Aug/17 13:31
• Updated: 01/Apr/20 18:02
• Resolved: 13/Apr/19 01:15

Description

HBase relies on FileSystems implementing CanUnbuffer.unbuffer() to force input streams to free up remote connections (HBASE-9393). This works for HDFS, but not elsewhere.

S3A input stream can implement CanUnbuffer.unbuffer() by closing the input stream and relying on lazy seek to reopen it on demand.

Needs

• Contract specification of unbuffer. As in "who added a new feature to filesystems but forgot to mention what it should do?"
• Contract test for filesystems which declare their support.
• S3AInputStream to call closeStream() on a call to unbuffer().
• Test case

Issue Links

• is related to: HADOOP-14748 Wasb input streams to implement CanUnbuffer
• is related to: HADOOP-14872 CryptoInputStream should implement unbuffer
• is related to: HADOOP-12805 Annotate CanUnbuffer with @InterfaceAudience.Public
• relates to: HADOOP-14765 AdlFsInputStream should implement unbuffer
• relates to: HBASE-9393 Region Server fails to properly close socket resulting in many CLOSE_WAIT to Data Nodes
• relates to: HDFS-7694 FSDataInputStream should support "unbuffer"
• links to: GitHub Pull Request #690