[HUDI-1487] after HUDI-1376 merged unit test testCopyOnWriteStorage will failed random

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Labels: pull-request-available
• Assignee: liwei
• Reporter: liwei
• Created: 22/Dec/20 15:00
• Updated: 21/Jan/21 06:05
• Resolved: 21/Jan/21 06:05

Description

 

 

TestCOWDataSource.testCopyOnWriteStorage will failed random. Because  before the  incremental read, add a new upsert commit.

// pull the latest commit
val hoodieIncViewDF2 = spark.read.format("org.apache.hudi")
.option(DataSourceReadOptions.QUERY_TYPE_OPT_KEY, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
.option(DataSourceReadOptions.BEGIN_INSTANTTIME_OPT_KEY, commitInstantTime2)
.load(basePath)

the new commit is :

// Upsert based on the written table with Hudi metadata columns
val verificationRowKey = snapshotDF1.limit(1).select("_row_key").first.getString(0)

as verificationRowKey will contains in "uniqueKeyCnt", so will failed as : "expected: <65> but was: <66>"

 

 

https://travis-ci.com/github/apache/hudi/jobs/463879606

 

org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.executeRequest(RemoteHoodieTableFileSystemView.java:173)
at org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView.getLatestFileSlices(RemoteHoodieTableFileSystemView.java:275)
... 30 more
[WARN ] 2020-12-22 12:32:40,788 org.apache.hudi.testutils.HoodieClientTestHarness - Closing file-system instance used in previous test-run
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.352 s <<< FAILURE! - in org.apache.hudi.functional.TestCOWDataSource
[ERROR] org.apache.hudi.functional.TestCOWDataSource.testCopyOnWriteStorage Time elapsed: 15.275 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <65> but was: <66>
at org.apache.hudi.functional.TestCOWDataSource.testCopyOnWriteStorage(TestCOWDataSource.scala:160)
[INFO] Running org.apache.hudi.functional.TestDataSourceForBootstrap
[WARN ] 2020-12-22 12:32:43,641 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:47,818 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:50,921 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:56,169 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:56,793 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:32:57,388 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:33:05,191 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:33:10,221 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:33:17,985 org.apache.hudi.DefaultSource - Loading Base File Only View.
[WARN ] 2020-12-22 12:33:22,498 org.apache.hudi.DefaultSource - Loading Base File Only View.

Issue Links

• links to: GitHub Pull Request #2364