Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-3043

Test failure with TestHoodieDeltaStreamerWithMultiWriter.testUpsertsContinuousModeWithMultipleWriters

    XMLWordPrintableJSON

Details

    • Task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.11.0
    • 0.10.1
    • None

    Description

      CI has been very flaky with this test. 

      reference: https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_apis/build/builds/4431/logs/35

       

      Test fails after a timeout of 360 secs. This is what the test is doing. 

      Instantiate two concurrent deltastreamer jobs. But are expected to conflict and one of them should fail.

       

      2021-12-15T12:31:08.1443043Z 415268 [pool-46-thread-1] WARN  org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.
      2021-12-15T12:31:08.1479662Z 415272 [pool-46-thread-1] WARN  org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
      2021-12-15T12:31:08.1480755Z org.apache.hudi.exception.HoodieIOException: Could not load Hoodie properties from /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
      2021-12-15T12:31:08.1481566Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
      2021-12-15T12:31:08.1482274Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
      2021-12-15T12:31:08.1483350Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
      2021-12-15T12:31:08.1484084Z 	at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
      2021-12-15T12:31:08.1484912Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
      2021-12-15T12:31:08.1485842Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
      2021-12-15T12:31:08.1486735Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
      2021-12-15T12:31:08.1487473Z 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      2021-12-15T12:31:08.1488117Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      2021-12-15T12:31:08.1488795Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      2021-12-15T12:31:08.1489389Z 	at java.lang.Thread.run(Thread.java:748)
      2021-12-15T12:31:08.1490080Z Caused by: java.io.FileNotFoundException: File file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup does not exist
      2021-12-15T12:31:08.1490870Z 	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
      2021-12-15T12:31:08.1491603Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
      2021-12-15T12:31:08.1492301Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
      2021-12-15T12:31:08.1492988Z 	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
      2021-12-15T12:31:08.1493700Z 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
      2021-12-15T12:31:08.1494397Z 	at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
      2021-12-15T12:31:08.1495022Z 	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
      2021-12-15T12:31:08.1495676Z 	at org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
      2021-12-15T12:31:08.1496385Z 	at org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
      2021-12-15T12:31:08.1497081Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
      2021-12-15T12:31:08.1497619Z 	... 10 more
      2021-12-15T12:31:09.0533739Z 416177 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit1940536249034667747/source/.hoodie/metadata
      2021-12-15T12:31:09.0821375Z 416206 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit1940536249034667747/source/.hoodie/metadata
      2021-12-15T12:31:09.4001348Z 06:50  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:09.4012877Z 06:50  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:09.4794927Z 06:50  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:11.4361520Z 06:52  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:12.3883215Z 419512 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit6695759085666687592/source/.hoodie/metadata
      2021-12-15T12:31:12.4075450Z 419531 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit6695759085666687592/source/.hoodie/metadata
      2021-12-15T12:31:12.7802513Z 06:54  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:12.7847857Z 06:54  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:12.8628212Z 06:54  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:13.1489569Z 420273 [pool-46-thread-1] WARN  org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.
      2021-12-15T12:31:13.1502918Z 420273 [pool-46-thread-1] WARN  org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
      2021-12-15T12:31:13.1503992Z org.apache.hudi.exception.HoodieIOException: Could not load Hoodie properties from /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
      2021-12-15T12:31:13.1504809Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
      2021-12-15T12:31:13.1505526Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
      2021-12-15T12:31:13.1506253Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
      2021-12-15T12:31:13.1506983Z 	at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
      2021-12-15T12:31:13.1507927Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
      2021-12-15T12:31:13.1508853Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
      2021-12-15T12:31:13.1509879Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
      2021-12-15T12:31:13.1510645Z 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      2021-12-15T12:31:13.1511272Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      2021-12-15T12:31:13.1511948Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      2021-12-15T12:31:13.1512551Z 	at java.lang.Thread.run(Thread.java:748)
      2021-12-15T12:31:13.1513607Z Caused by: java.io.FileNotFoundException: File file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup does not exist
      2021-12-15T12:31:13.1514412Z 	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
      2021-12-15T12:31:13.1515141Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
      2021-12-15T12:31:13.1515860Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
      2021-12-15T12:31:13.1516555Z 	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
      2021-12-15T12:31:13.1518333Z 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
      2021-12-15T12:31:13.1519089Z 	at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
      2021-12-15T12:31:13.1519729Z 	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
      2021-12-15T12:31:13.1520387Z 	at org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
      2021-12-15T12:31:13.1521094Z 	at org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
      2021-12-15T12:31:13.1521791Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
      2021-12-15T12:31:13.1522342Z 	... 10 more
      2021-12-15T12:31:14.7650788Z 06:56  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:15.5436369Z 422667 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit4191731663987779053/source/.hoodie/metadata
      2021-12-15T12:31:15.5616963Z 422686 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit4191731663987779053/source/.hoodie/metadata
      2021-12-15T12:31:15.8443523Z 06:57  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:15.8454420Z 06:57  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:15.9309839Z 06:57  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:17.8366064Z 06:59  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:18.1515156Z 425275 [pool-46-thread-1] WARN  org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.
      2021-12-15T12:31:18.1516556Z 425275 [pool-46-thread-1] WARN  org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
      2021-12-15T12:31:18.1517464Z org.apache.hudi.exception.HoodieIOException: Could not load Hoodie properties from /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
      2021-12-15T12:31:18.1518265Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
      2021-12-15T12:31:18.1519344Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
      2021-12-15T12:31:18.1519949Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
      2021-12-15T12:31:18.1520520Z 	at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
      2021-12-15T12:31:18.1521158Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
      2021-12-15T12:31:18.1521875Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
      2021-12-15T12:31:18.1522917Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
      2021-12-15T12:31:18.1523479Z 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      2021-12-15T12:31:18.1523916Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      2021-12-15T12:31:18.1524404Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      2021-12-15T12:31:18.1524820Z 	at java.lang.Thread.run(Thread.java:748)
      2021-12-15T12:31:18.1525313Z Caused by: java.io.FileNotFoundException: File file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup does not exist
      2021-12-15T12:31:18.1525956Z 	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
      2021-12-15T12:31:18.1526494Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
      2021-12-15T12:31:18.1527013Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
      2021-12-15T12:31:18.1527495Z 	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
      2021-12-15T12:31:18.1528014Z 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
      2021-12-15T12:31:18.1528523Z 	at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
      2021-12-15T12:31:18.1528952Z 	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
      2021-12-15T12:31:18.1529418Z 	at org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
      2021-12-15T12:31:18.1530054Z 	at org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
      2021-12-15T12:31:18.1530560Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
      2021-12-15T12:31:18.1530912Z 	... 10 more
      2021-12-15T12:31:18.6080652Z 425732 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit8182236215041118309/source/.hoodie/metadata
      2021-12-15T12:31:18.6248321Z 425749 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit8182236215041118309/source/.hoodie/metadata
      2021-12-15T12:31:18.9049323Z 07:00  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:18.9086840Z 07:00  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:18.9773610Z 07:00  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:20.8238902Z 07:02  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:21.8356036Z 428959 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit7367394654999483124/source/.hoodie/metadata
      2021-12-15T12:31:21.8519931Z 428976 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit7367394654999483124/source/.hoodie/metadata
      2021-12-15T12:31:22.1387636Z 07:03  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:22.1442159Z 07:03  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:22.1973498Z 07:03  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:23.1526405Z 430276 [pool-46-thread-1] WARN  org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.
      2021-12-15T12:31:23.1527490Z 430276 [pool-46-thread-1] WARN  org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
      2021-12-15T12:31:23.1528146Z org.apache.hudi.exception.HoodieIOException: Could not load Hoodie properties from /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
      2021-12-15T12:31:23.1528881Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
      2021-12-15T12:31:23.1529399Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
      2021-12-15T12:31:23.1529934Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
      2021-12-15T12:31:23.1530477Z 	at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
      2021-12-15T12:31:23.1531100Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
      2021-12-15T12:31:23.1531842Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
      2021-12-15T12:31:23.1532547Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
      2021-12-15T12:31:23.1533092Z 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      2021-12-15T12:31:23.1533647Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      2021-12-15T12:31:23.1534137Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      2021-12-15T12:31:23.1534554Z 	at java.lang.Thread.run(Thread.java:748)
      2021-12-15T12:31:23.1535044Z Caused by: java.io.FileNotFoundException: File file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup does not exist
      2021-12-15T12:31:23.1535642Z 	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
      2021-12-15T12:31:23.1536186Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
      2021-12-15T12:31:23.1536689Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
      2021-12-15T12:31:23.1537183Z 	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
      2021-12-15T12:31:23.1537702Z 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
      2021-12-15T12:31:23.1538216Z 	at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
      2021-12-15T12:31:23.1538644Z 	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
      2021-12-15T12:31:23.1539108Z 	at org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
      2021-12-15T12:31:23.1539625Z 	at org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
      2021-12-15T12:31:23.1540118Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
      2021-12-15T12:31:23.1540484Z 	... 10 more
      2021-12-15T12:31:24.0393154Z 07:05  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:24.2768054Z 431401 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata
      2021-12-15T12:31:24.3011474Z 431423 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata
      2021-12-15T12:31:24.5931674Z 07:05  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:24.5982776Z 07:05  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:24.6715314Z 07:06  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:26.4723566Z 07:07  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:26.6296934Z 433752 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit2866010450327332610/source/.hoodie/metadata
      2021-12-15T12:31:26.6990277Z 433823 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit8519882895692188385/source/.hoodie/metadata
      2021-12-15T12:31:26.7154263Z 433839 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit8519882895692188385/source/.hoodie/metadata
      2021-12-15T12:31:27.0226015Z 07:08  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:27.0285268Z 07:08  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:27.0977866Z 07:08  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:28.1534844Z 435277 [pool-46-thread-1] WARN  org.apache.hudi.common.table.HoodieTableConfig  - Run `table recover-configs` if config update/delete failed midway. Falling back to backed up configs.
      2021-12-15T12:31:28.1536264Z 435277 [pool-46-thread-1] WARN  org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer  - Got error :
      2021-12-15T12:31:28.1537195Z org.apache.hudi.exception.HoodieIOException: Could not load Hoodie properties from /tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties
      2021-12-15T12:31:28.1537984Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:190)
      2021-12-15T12:31:28.1539028Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:114)
      2021-12-15T12:31:28.1539640Z 	at org.apache.hudi.common.table.HoodieTableMetaClient.<init>(HoodieTableMetaClient.java:73)
      2021-12-15T12:31:28.1540183Z 	at org.apache.hudi.common.table.HoodieTableMetaClient$Builder.build(HoodieTableMetaClient.java:614)
      2021-12-15T12:31:28.1540885Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.assertAtleastNDeltaCommitsAfterCommit(TestHoodieDeltaStreamer.java:315)
      2021-12-15T12:31:28.1541621Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamerWithMultiWriter.lambda$runJobsInParallel$0(TestHoodieDeltaStreamerWithMultiWriter.java:312)
      2021-12-15T12:31:28.1542337Z 	at org.apache.hudi.utilities.functional.TestHoodieDeltaStreamer$TestHelpers.lambda$waitTillCondition$0(TestHoodieDeltaStreamer.java:340)
      2021-12-15T12:31:28.1542884Z 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      2021-12-15T12:31:28.1543318Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      2021-12-15T12:31:28.1543805Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      2021-12-15T12:31:28.1544221Z 	at java.lang.Thread.run(Thread.java:748)
      2021-12-15T12:31:28.1544706Z Caused by: java.io.FileNotFoundException: File file:/tmp/junit781255057992811963/testtable_MERGE_ON_READ/.hoodie/hoodie.properties.backup does not exist
      2021-12-15T12:31:28.1545306Z 	at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:611)
      2021-12-15T12:31:28.1545840Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
      2021-12-15T12:31:28.1546345Z 	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
      2021-12-15T12:31:28.1547105Z 	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
      2021-12-15T12:31:28.1547635Z 	at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:142)
      2021-12-15T12:31:28.1548151Z 	at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:346)
      2021-12-15T12:31:28.1548578Z 	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:769)
      2021-12-15T12:31:28.1549239Z 	at org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:459)
      2021-12-15T12:31:28.1549777Z 	at org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:212)
      2021-12-15T12:31:28.1550273Z 	at org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:180)
      2021-12-15T12:31:28.1550655Z 	... 10 more
      2021-12-15T12:31:28.8646121Z 07:10  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:29.0315217Z 436154 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit6507215968345229941/source/.hoodie/metadata
      2021-12-15T12:31:29.0474647Z 436171 [main] WARN  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Metadata table was not found at path file:/tmp/junit6507215968345229941/source/.hoodie/metadata
      2021-12-15T12:31:29.9039470Z 07:10  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:29.9040844Z 07:10  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:29.9041727Z 07:10  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:31.2335559Z 07:12  WARN: Timeline-server-based markers are configured as the marker type but embedded timeline server is not enabled.  Falling back to direct markers.
      2021-12-15T12:31:32.1873178Z [ERROR] Tests run: 26, Failures: 0, Errors: 1, Skipped: 9, Time elapsed: 439.775 s <<< FAILURE! - in JUnit Vintage
      2021-12-15T12:31:32.1874589Z [ERROR] HoodieTableType).[2] MERGE_ON_READ(testUpsertsContinuousModeWithMultipleWriters  Time elapsed: 360.847 s  <<< ERROR!
      2021-12-15T12:31:32.1875409Z java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.TimeoutException
      2021-12-15T12:31:32.1876498Z Caused by: java.lang.RuntimeException: java.util.concurrent.TimeoutException
      2021-12-15T12:31:32.1877024Z Caused by: java.util.concurrent.TimeoutException
      2021-12-15T12:31:32.1877203Z 
      2021-12-15T12:31:32.5675718Z [INFO] 
      2021-12-15T12:31:32.5710112Z [INFO] Results:
      2021-12-15T12:31:32.5710772Z [INFO] 
      2021-12-15T12:31:32.5711020Z [ERROR] Errors: 
      2021-12-15T12:31:32.5711359Z [ERROR]   java.lang.RuntimeException: java.util.concurrent.TimeoutException
      2021-12-15T12:31:32.5711695Z [INFO] 
      2021-12-15T12:31:32.5712044Z [ERROR] Tests run: 26, Failures: 0, Errors: 1, Skipped: 9 

      Attachments

        1. Screen Shot 2021-12-17 at 3.07.39 PM.png
          148 kB
          sivabalan narayanan

        Issue Links

          Activity

            People

              shivnarayan sivabalan narayanan
              shivnarayan sivabalan narayanan
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: