[HUDI-6445] Fix CI stability Jun 26, 2023 - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Test
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: tests-ci
Labels:
- pull-request-available

Epic Link:
CI Instability

Description

CI has been unstable for the past few weeks. we need to triage them and fix it.

UT-spark datasource module times out after 3 hours.

https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=17956&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc

Looks like top 10 tests were taking 30 to 40 secs and now its taking 40 to 50 secs or more and hence reaching 3 hours

2023-06-20T05:03:58.6566739Z 52.124 org.apache.hudi.functional.TestIncrementalReadWithFullTableScan testFailEarlyForIncrViewQue
ryForNonExistingFiles{HoodieTableType}[2]
2023-06-20T05:03:58.6567324Z 49.446 org.apache.hudi.functional.TestIncrementalReadWithFullTableScan testFailEarlyForIncrViewQue
ryForNonExistingFiles{HoodieTableType}[1]
2023-06-20T05:03:58.6568005Z 48.659 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testMORDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[1]
2023-06-20T05:03:58.6568471Z 47.799 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testMORDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[3]
2023-06-20T05:03:58.6569093Z 47.586 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testMORDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[2]
2023-06-20T05:03:58.6569503Z 41.208 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, 
String}[2]
2023-06-20T05:03:58.6570090Z 41.034 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, 
String}[4]
2023-06-20T05:03:58.6570501Z 40.225 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, 
String}[3]
2023-06-20T05:03:58.6571231Z 39.853 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testCOWDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[1]
2023-06-20T05:03:58.6574224Z 39.357 org.apache.hudi.functional.TestMORDataSource testCount{HoodieRecordType, HoodieRecordType, 
String}[1]
2023-06-20T05:03:58.6575261Z 38.995 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testCOWDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[3]
2023-06-20T05:03:58.6575765Z 38.846 org.apache.hudi.functional.cdc.TestCDCDataFrameSuite testCOWDataSourceWrite{HoodieCDCSupple
mentalLoggingMode}[2]
2023-06-20T05:03:58.6576470Z 35.404 org.apache.hudi.functional.TestMORDataSourceWithBucketIndex testCountWithBucketIndex

TestHoodieDeltaStreamer.testUpsertsMORContinuousMode

and testAsyncClusteringServiceWithCompaction

https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18111/logs/19

Raymond.

~~TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWritersForConflicts~~

~~https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/18080/logs/21~~

~~Siva~~

~~TestWriteMergeOnRead.testUpsert~~

~~https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/17993/logs/35~~

~~Sagar, Danny~~

~~TestWriteMergeOnReadWithCompact.testUpsert~~

~~TestWriteCopyOnWrite.testSubtaskFails~~