[HUDI-6041] add `properties` to Hudi Spark Procedures

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: bootstrap, spark-sql
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: lvyanquan
• Created: 05/Apr/23 12:23
• Updated: 26/May/23 06:14

Description

We need to write extra properties to a HDFS file for Bootstrap Procedure and set `props_file_path`, which make it troublesome to call this procedure, like:

call run_bootstrap(table => 'test_hudi_table', table_type => 'COPY_ON_WRITE', 
bootstrap_path => 'hdfs://ns1/hive/warehouse/hudi.db/test_hudi_table', 
base_path => 'hdfs://ns1//tmp/hoodie/test_hudi_table', 
rowKey_field => 'id', partition_path_field => 'dt',
props_file_path => 'hdfs://ns1//tmp/tableProp.txt'); 

Or we can set those properties by session config, which means that we need to execute some `set` SQLs.

We can add a new parameter for procedure input named `properties`, add  collect key-value pairs for this input, like:

call run_bootstrap(table => 'test_hudi_table', table_type => 'COPY_ON_WRITE', 
bootstrap_path => 'hdfs://ns1/hive/warehouse/hudi.db/test_hudi_table', 
base_path => 'hdfs://ns1//tmp/hoodie/test_hudi_table', 
rowKey_field => 'id', partition_path_field => 'dt', 
properties => 'hoodie.datasource.write.hive_style_partitioning=true');  

So that we don't need to put another file to HDFS

Issue Links

• links to: GitHub Pull Request #8387