[HUDI-3054] Fix flaky TestHoodieClientMultiWriter. testHoodieClientBasicMultiWriter - ASF JIRA

Details

Type: Test
Status: Closed
Priority: Blocker
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.12.1
Component/s: Testing, tests-ci
Labels:
- pull-request-available

Story Points:
3
Epic Link:
CI Instability

Description

Ref: https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/4428/logs/21

2021-12-17T11:39:57.1645757Z [INFO] Running org.apache.hudi.client.TestHoodieClientMultiWriter
2021-12-17T11:39:57.3453991Z 339506 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit8865530218583640556/dataset/.hoodie/metadata
2021-12-17T11:39:57.3984328Z 339559 [dispatcher-event-loop-5] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 0 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:39:57.5278608Z 339689 [dispatcher-event-loop-2] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 1 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:39:57.9783107Z 340139 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit8865530218583640556/dataset/.hoodie/metadata
2021-12-17T11:39:57.9927490Z 340154 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit8865530218583640556/dataset/.hoodie/metadata
2021-12-17T11:40:10.1428665Z 352304 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:40:10.9930023Z 353149 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:40:11.4294603Z 353590 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit3262960667280061850/dataset/.hoodie/metadata
2021-12-17T11:40:11.4763085Z 353637 [dispatcher-event-loop-5] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 0 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:40:11.6014876Z 353762 [dispatcher-event-loop-2] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 1 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:40:12.0892513Z 354250 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit3262960667280061850/dataset/.hoodie/metadata
2021-12-17T11:40:12.1061317Z 354267 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit3262960667280061850/dataset/.hoodie/metadata
2021-12-17T11:40:23.1499732Z 365311 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:40:24.1626167Z 366323 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit294667857867877904/dataset/.hoodie/metadata
2021-12-17T11:40:24.1945944Z 366355 [dispatcher-event-loop-5] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 0 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:40:24.3084730Z 366469 [dispatcher-event-loop-2] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 1 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:40:24.7350862Z 366896 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit294667857867877904/dataset/.hoodie/metadata
2021-12-17T11:40:24.7482727Z 366909 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit294667857867877904/dataset/.hoodie/metadata
2021-12-17T11:40:43.1530857Z 385314 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:40:44.0641298Z 386225 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:40:44.5065216Z 386667 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit4425942033067835021/dataset/.hoodie/metadata
2021-12-17T11:40:44.5408012Z 386702 [dispatcher-event-loop-5] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 0 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:40:44.6779567Z 386839 [dispatcher-event-loop-1] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 1 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:40:45.1410815Z 387301 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit4425942033067835021/dataset/.hoodie/metadata
2021-12-17T11:40:45.1543539Z 387315 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit4425942033067835021/dataset/.hoodie/metadata
2021-12-17T11:41:03.1384862Z 405299 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:41:04.0150577Z 406176 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit5137952068799441319/dataset/.hoodie/metadata
2021-12-17T11:41:04.0514934Z 406212 [dispatcher-event-loop-5] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 0 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:41:04.1677826Z 406329 [dispatcher-event-loop-2] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 1 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:41:04.6164436Z 406777 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit5137952068799441319/dataset/.hoodie/metadata
2021-12-17T11:41:04.6343307Z 406791 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit5137952068799441319/dataset/.hoodie/metadata
2021-12-17T11:41:13.0144546Z 415165 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:41:13.8545883Z 416010 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:41:14.2624248Z 416423 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit1733672335771853967/dataset/.hoodie/metadata
2021-12-17T11:41:14.2957800Z 416457 [dispatcher-event-loop-5] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 0 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:41:14.3989543Z 416560 [dispatcher-event-loop-3] WARN  org.apache.spark.scheduler.TaskSetManager  - Stage 1 contains a task of very large size (101 KB). The maximum recommended task size is 100 KB.
2021-12-17T11:41:14.7647136Z 416926 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit1733672335771853967/dataset/.hoodie/metadata
2021-12-17T11:41:14.7772128Z 416938 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit1733672335771853967/dataset/.hoodie/metadata
2021-12-17T11:41:22.6703021Z 424828 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:41:23.5771003Z 425738 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit4023714709478834450/dataset/.hoodie/metadata
2021-12-17T11:41:24.3341519Z 426495 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit4023714709478834450/dataset/.hoodie/metadata
2021-12-17T11:41:24.3488610Z 426510 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit4023714709478834450/dataset/.hoodie/metadata
2021-12-17T11:41:45.2669117Z 447428 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:41:46.1117672Z 448273 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:41:46.5285885Z 448689 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit6437731807948505371/dataset/.hoodie/metadata
2021-12-17T11:41:47.2040819Z 449365 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit6437731807948505371/dataset/.hoodie/metadata
2021-12-17T11:41:47.2190197Z 449380 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path /tmp/junit6437731807948505371/dataset/.hoodie/metadata
2021-12-17T11:42:12.0807715Z 474241 [main] WARN  org.apache.hudi.testutils.HoodieClientTestHarness  - Closing file-system instance used in previous test-run
2021-12-17T11:42:12.8434589Z [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 135.656 s <<< FAILURE! - in org.apache.hudi.client.TestHoodieClientMultiWriter
2021-12-17T11:42:12.8435532Z [ERROR] org.apache.hudi.client.TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriter(HoodieTableType)[1]  Time elapsed: 13.652 s  <<< ERROR!
2021-12-17T11:42:12.8436481Z java.util.concurrent.ExecutionException: java.lang.RuntimeException: org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
2021-12-17T11:42:12.8437368Z 	at org.apache.hudi.client.TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriter(TestHoodieClientMultiWriter.java:129)
2021-12-17T11:42:12.8438208Z Caused by: java.lang.RuntimeException: org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
2021-12-17T11:42:12.8439052Z 	at org.apache.hudi.client.TestHoodieClientMultiWriter.lambda$testHoodieClientBasicMultiWriter$1(TestHoodieClientMultiWriter.java:125)
2021-12-17T11:42:12.8439840Z Caused by: org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
2021-12-17T11:42:12.8440785Z 	at org.apache.hudi.client.TestHoodieClientMultiWriter.createCommitWithUpserts(TestHoodieClientMultiWriter.java:432)
2021-12-17T11:42:12.8441570Z 	at org.apache.hudi.client.TestHoodieClientMultiWriter.lambda$testHoodieClientBasicMultiWriter$1(TestHoodieClientMultiWriter.java:122)
2021-12-17T11:42:12.8442342Z Caused by: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
2021-12-17T11:42:12.8443007Z 	at org.apache.hudi.client.TestHoodieClientMultiWriter.createCommitWithUpserts(TestHoodieClientMultiWriter.java:432)
2021-12-17T11:42:12.8443792Z 	at org.apache.hudi.client.TestHoodieClientMultiWriter.lambda$testHoodieClientBasicMultiWriter$1(TestHoodieClientMultiWriter.java:122)
2021-12-17T11:42:12.8444188Z