[IMPALA-3577] Partitions on multiple filesystems breaks with S3_SKIP_INSERT_STAGING

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0
• Fix Version/s: Impala 2.6.0
• Component/s: Backend
• Labels: s3 supportability
• Target Version: Impala 2.6.0
• Assignee: Sailesh Mukil
• Reporter: Sailesh Mukil
• Created: 19/May/16 22:45
• Updated: 26/May/16 16:56
• Resolved: 26/May/16 16:56

Description

The HdfsTableSink usualy creates a HDFS connection to the filesystem that the base table resides in. However, if we create a partition in a FS different than that of the base table and set S3_SKIP_INSERT_STAGING to "true", the table sink will try to write to a different filesystem with the wrong filesystem connector.

Current workaround: Set S3_SKIP_INSERT_STAGING to "false".
Considering a table on HDFS with a partition on S3 (just an example), this works because the table sink stages the files to the FS that the base table resides in (i.e. HDFS) and only the coordinator, who is aware of multiple filesystems, moves it to the right FS (i.e. S3).