[HDFS-6735] A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 2.7.0
• Component/s: hdfs-client
• Hadoop Flags: Reviewed
• Assignee: Lars Hofhansl
• Reporter: Liang Xie
• Created: 23/Jul/14 03:13
• Updated: 25/Jun/18 14:15
• Resolved: 03/Dec/14 05:01

Description

In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor patch against the first encourtered lock, around getFileLength, in deed, after reading code and testing, it shows still other locks we could improve.
In this jira, i'll make a patch against other locks, and a simple test case to show the issue and the improved result.

This is important for HBase application, since in current HFile read path, we issue all read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution is another story i had a plan to do, but probably will take more time than i expected)

Issue Links

• is related to: HADOOP-15557 CryptoInputStream can't handle concurrent access; inconsistent with HDFS
• relates to: HDFS-7494 Checking of closed in DFSInputStream#pread() should be protected by synchronization
• relates to: HDFS-7495 Remove updatePosition argument from DFSInputStream#getBlockAt()
• relates to: HADOOP-9361 Strictly define the expected behavior of filesystem APIs and write tests to verify compliance

Sub-Tasks

• try to optimize DFSInputStream.getFileLength(): Patch Available
• Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context: Closed