[IMPALA-9867] Add Support for Spilling to HDFS

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Epic Link: Spill to S3 Support
• Epic Color: ghx-label-3
• Assignee: Yida Wu
• Reporter: Yida Wu
• Created: 18/Jun/20 18:07
• Updated: 16/Jul/21 23:45
• Resolved: 16/Jul/21 23:45

Description

The task is to provide the basic capability of writing and reading from HDFS disk during "spilling to disk", as a sub task to fulfill "spilling to S3".

In theory, after the task is done, by changing the HDFS uri to S3 uri in HDFS connector should provide "spilling to S3" functionality.

Performance issues are to be considered in later tasks.

Issue Links

• causes: IMPALA-10529 Hit DCHECK in DiskIoMgr::AssignQueue in core-s3 build
• causes: IMPALA-10527 DiskIoMgrTest.WriteToRemotePartialFileSuccess failed in tsan build
• causes: IMPALA-10528 DiskIoMgrTest.WriteToRemoteDiffPagesSuccess failed in asan build
• causes: IMPALA-10530 DiskIoMgrTest.WriteToRemoteEvictLocal failed in asan build
• causes: IMPALA-10531 TmpFileMgrTest.TestCompressBufferManagementEncryptedRemoteUpload failed in exhaustive release build
• causes: IMPALA-10533 TestScratchDir.test_scratch_dirs_mix_local_and_remote_dir_spill_local_only seems flaky
• causes: IMPALA-10534 TestScratchDir.test_scratch_dirs_remote_spill_with_options seems flaky
• causes: IMPALA-10555 Hit DCHECK in TmpFileGroup::RecoverWriteError
• fixes: IMPALA-9894 TmpFile incorrectly uses default hdfs queue

Sub-Tasks

• Add Support for Spilling to HDFS path parsing: Closed