[HUDI-2494] Fix usage of different key generators with metadata enabled

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: spark
• Labels: oct18_2021 pull-request-available sev:critical
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 27/Sep/21 15:18
• Updated: 04/Jan/22 00:09
• Resolved: 17/Oct/21 18:16

Description

With sync metadata patch, when metadata is enabled by default, some spark datasource tests failed which were using timestamp based key gen and custom key gen. Metadata table's records are getting picked up when we do 

 

spark.read.format(hudi).load(basePath + "/*/*")

 

For now, I have disabled metadata for these tests. 

testSparkPartitonByWithTimestampBasedKeyGenerator

testSparkPartitonByWithCustomKeyGenerator

 

I was looking at options to ignore certain path, but looks like there is none. 

 

 

Issue Links

• is related to: HUDI-2493 Verify removing glob pattern works w/ all key generators
• links to: GitHub Pull Request #3768