Blocker Description
Documenting all failure cases here :
Integration Tests:
1. ITTestHoodieSanity : Command ([/var/hoodie/ws/hudi-spark/run_hoodie_streaming_app.sh, --hive-sync, --table-path, hdfs://namenode/docker_hoodie_single_partition_key_cow_test, --hive-url, jdbc:hive2://hiveserver:10000, --table-type, COPY_ON_WRITE, --hive-table, docker_hoodie_single_partition_key_cow_test])
05:17:48.384 [main] ERROR HoodieJavaStreamingApp - Got error running app java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Expecting 100 records, Got 50 at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212] at HoodieJavaStreamingApp.run(HoodieJavaStreamingApp.java:193) ~[test-classes/:?] at HoodieJavaStreamingApp.main(HoodieJavaStreamingApp.java:126) [test-classes/:?] Caused by: java.lang.IllegalArgumentException: Expecting 100 records, Got 50 at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40) ~[hudi-spark-bundle_2.11-0.6.0-SNAPSHOT.jar:0.6.0-SNAPSHOT] at HoodieJavaStreamingApp.addInputAndValidateIngestion(HoodieJavaStreamingApp.java:352) ~[test-classes/:?] at HoodieJavaStreamingApp.lambda$run$1(HoodieJavaStreamingApp.java:186) ~[test-classes/:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
Comments: From the context, it looks like out of 100 records in 1st batch, only 50 records were updated but all 100 records were expected to be upddated.
2. ITTestHoodieSanity : Command ([/var/hoodie/ws/hudi-spark/run_hoodie_streaming_app.sh, --hive-sync, --t
able-path, hdfs://namenode/docker_hoodie_single_partition_key_mor_test, --hive-url, jdbc:hive2://hiveserver:10000, --t
able-type, MERGE_ON_READ, --hive-table, docker_hoodie_single_partition_key_mor_test]) expected to succeed. Exit (255)
==> expected: <0> but was: <255>
Instants :[[20200812012618__deltacommit__COMPLETED], [20200812012629__deltacommit__COMPLETED]] Instants :[[20200812012618__deltacommit__COMPLETED], [20200812012629__deltacommit__COMPLETED]] 01:29:35.754 [main] ERROR HoodieJavaStreamingApp - Got error running app java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Timedout waiting for 3 commits to appear in hdfs://namenode/docker_hoodie_single_partition_key_mor_test at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_212] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_212] at HoodieJavaStreamingApp.run(HoodieJavaStreamingApp.java:185) ~[test-classes/:?] at HoodieJavaStreamingApp.main(HoodieJavaStreamingApp.java:118) [test-classes/:?] Caused by: java.lang.IllegalStateException: Timedout waiting for 3 commits to appear in hdfs://namenode/docker_hoodie_ single_partition_key_mor_test at HoodieJavaStreamingApp.waitTillNCommits(HoodieJavaStreamingApp.java:261) ~[test-classes/:?] at HoodieJavaStreamingApp.addInputAndValidateIngestion(HoodieJavaStreamingApp.java:298) ~[test-classes/:?] at HoodieJavaStreamingApp.lambda$run$1(HoodieJavaStreamingApp.java:178) ~[test-classes/:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_212]
From the context: Compaction should have been scheduled and executed but that did not happen here.