In createCarbonDataFileBlockMetaInfoMapping method, we get list of carbondata files in the segment, loop through all the carbon files and make a map of fileNameToMetaInfoMapping<path-string, BlockMetaInfo>
In that carbon files loop, if the file is of AbstractDFSCarbonFile type, we get the org.apache.hadoop.fs.FileStatus thrice for each file. And the method to get file status is an RPC call(fileSystem.getFileStatus(path)). It takes ~2ms in the cluster for each call. Thus, incur an overhead of ~6ms per file. So overall driver side query processing time has increased significantly when there are more carbon files. Hence caused TPC-DS queries performance degradation.
Have shown the methods/calls which get the file status for the carbon file in loop:
I think, currently we make RPC call to get the file status upon each invocation because file status may change over a period of time. And we shouldn't cache the file status in AbstractDFSCarbonFile.
In the current case, just before the loop of carbon files, we get the file status of all the carbon files in the segment with RPC call shown below. LocatedFileStatus is a child class of FileStatus. It has BlockLocation along with file status.
Intention of getting all the file status here is to create instance of BlockMetaInfo and maintain the map of fileNameToMetaInfoMapping.
So it is safe to avoid these unnecessary rpc calls to get file status again in getLocations(), getSize() and getPath() methods.