[IMPALA-10780] Iceberg in Impala should support combination storage

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Epic Color: ghx-label-10
• Assignee: Unassigned
• Reporter: Yong Yang
• Created: 08/Jul/21 11:35
• Updated: 09/Jul/21 10:58

Description

Currently, the filesystem of the metadata path is used to check the data file, that is blocking the following scenario:

1. metadata is on hdfs
2. data is on s3a or other object store.

 

Following code in FeIcebergTable.Utils fails this combination:

private static HdfsPartition.FileDescriptor getFileDescriptor(Path fileLoc,
Path tableLoc, ListMap<TNetworkAddress> hostIndex) throws IOException 

{   FileSystem fs = FileSystemUtil.getFileSystemForPath(tableLoc);     FileStatus fileStatus = fs.getFileStatus(fileLoc);    return getFileDescriptor(fs, tableLoc, fileStatus, hostIndex); }

 

Issue Links

• is a child of: IMPALA-10149 Support Apache Iceberg in Impala