[ORC-1458] reduce namenode getFileinfo rpc

• Type: Wish
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Java, Reader
• Assignee: Unassigned
• Reporter: Mingchen_Ma
• Created: 30/Jun/23 11:06
• Updated: 24/Jan/24 08:00

Description

In the ReaderImpl.java code, there is the following logic:

if (maxFileLength == Long. MAX_VALUE) { 
FileStatus fileStatus = fs.getFileStatus(path);          
size = fileStatus. getLen();          
modificationTime = fileStatus. getModificationTime(); 
}

The above logic is to obtain the length of the file so as to read the footer of orc. But because of this, when we read the orc file on hdfs, an open operation will cause an additional getFileinfo rpc operation by default (unless we set the file length through ReaderOptions.set before the orc open).
Because we have opened the file in ReaderImpl, can we optimize the rpc call of NN in the following way (in a high-load cluster, the pressure on the namenode can be significantly reduced):

if (maxFileLength == Long. MAX_VALUE) {           
size = (DFSInputStream)file.getWrappedStream.getFileLength();          
modificationTime = -1; 
}

 

Issue Links

• links to: GitHub Pull Request #1767