[KYLIN-3626] Allow customization for Parquet path

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Resolved
• Component/s: Job Engine, Metadata, Storage - Parquet
• Assignee: Chao Long
• Reporter: Shao Feng Shi
• Created: 12/Oct/18 01:31
• Updated: 29/Apr/20 07:36
• Resolved: 03/Sep/19 03:02

Description

Due to the difference of HDFS and object store like S3/blob store, Kylin needs to allow different path patterns for the files when putting to the underlying storage. 

 

For example, to HDFS, this pattern is good:

/kylin/metadata_name/cube_name/segment_name/file1.parquet

 

But, the above pattern is not good for S3, since S3 uses the first couple bytes of the path to do the hashing, so we'd better add salt ahead of the original pathname:

/salt/kylin/metadata_name/cube_name/segment_name/file1.parquet

or do a reverse order:

/segment_name/cube_name/metadata_name/kylin/file1.parquet

 

To get the flexibility, we should make this extensible.