[FLINK-31956] Extend the CompiledPlan to read from/write to Flink's FileSystem

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.18.0
• Fix Version/s: 1.18.0
• Component/s: Table SQL / Client, Table SQL / Planner
• Labels: pull-request-available
• Assignee: Shuai Xu
• Reporter: Jane Chan
• Created: 27/Apr/23 09:12
• Updated: 28/Jun/23 07:14
• Resolved: 25/Jun/23 09:37

Description

At present, COMPILE/EXECUTE PLAN FOR '${plan.json}' only supports writing to/reading from a local file without the scheme. We propose to extend the support for Flink's FileSystem.

-- before
COMPILE PLAN FOR '/tmp/foo/bar.json' <insert_query>
EXECUTE PLAN FOR '/tmp/foo/bar.json' 

-- after
COMPILE PLAN FOR 'file:///tmp/foo/bar.json' <insert_query>
COMPILE PLAN FOR 'hdfs:///tmp/foo/bar.json' <insert_query>
COMPILE PLAN FOR 's3:///tmp/foo/bar.json' <insert_query>
COMPILE PLAN FOR 'oss:///tmp/foo/bar.json' <insert_query> 
EXECUTE PLAN FOR 'file:///tmp/foo/bar.json'
EXECUTE PLAN FOR 'hdfs:///tmp/foo/bar.json'
EXECUTE PLAN FOR 's3:///tmp/foo/bar.json'
EXECUTE PLAN FOR 'oss:///tmp/foo/bar.json' 

Issue Links

• Testing discovered: FLINK-32219 SQL client hangs when executing EXECUTE PLAN
• Testing discovered: FLINK-32374 ExecNodeGraphInternalPlan#writeToFile should support TRUNCATE_EXISTING for overwriting
• links to: GitHub Pull Request #22539
• links to: GitHub Pull Request #22818