[HDFS-8555] Random read support on HDFS files using Indexed Namenode feature

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.5.2
• Component/s: hdfs-client, namenode

Environment

Linux

• Tags: randomreads
• Assignee: Afzal Saan
• Reporter: amit sehgal
• Due: 31/Dec/15
• Created: 08/Jun/15 04:56
• Updated: 06/Jan/17 22:06

Description

Currently Namenode does not provide support to do random reads. With so many tools built on top of HDFS solving the use case of Exploratory BI and providing SQL over HDFS. The need of hour is to reduce the number of blocks read for a Random read.
E.g. extracting say 10 lines worth of information out of 100GB files should be reading only those block which can potentially have those 10 lines.
This can be achieved by adding a tagging feature per block in name node, each block written to HDFS will have tags associated to it stored in index.
Namednode when access via the Indexing feature will use this index native to reduce the no. of block returned to the client.