[HUDI-289] Implement a test suite to support long running test for Hudi writing and querying end-end

• Type: Test
• Status: Closed
• Priority: Major
• Resolution: Done
• Component/s: tests-ci, Usability
• Labels: pull-request-available
• Epic Link: Integ Test suite infra
• Assignee: vinoyang
• Reporter: Vinoth Chandar
• Created: 03/Oct/19 04:51
• Updated: 28/Apr/22 13:13
• Resolved: 28/Apr/22 13:13

Description

We would need an equivalent of an end-end test which runs some workload for few hours atleast, triggers various actions like commit, deltacopmmit, rollback, compaction and ensures correctness of code before every release

P.S: Learn from all the CSS issues managing compaction..

The feature branch is here: https://github.com/apache/incubator-hudi/tree/hudi_test_suite_refactor

Issue Links

• links to: GitHub Pull Request #1100
• links to: GitHub Pull Request #1100

Sub-Tasks

• Rename module name from hudi-bench to hudi-test-suite: Closed
• Introduce DIstributedTestDataSource to generate test data: Closed
• Provide a basic implementation of test suite: Closed
• Provide an documentation to describe how to use test suite: Closed
• Normalize log print statement: Closed
• Fix conflicts of test suite basic implementation based on master branch and fix checkstyle issues: Closed
• Rename WorkflowDagGenerator and some class names in test package: Resolved
• Fix TestComplexKeyGenerator#testSingleValueKeyGenerator and testMultipleValueKeyGenerator NPE: Resolved
• Create a hudi CI github organizations to support hudi CI via azure pipeline: Closed
• Add hudi test suite documentation into the README file of the test suite module: Resolved
• Add java doc for hudi test suite general classes: Closed
• Add java doc for hudi test suite test classes: Closed
• Support Spark version upgrade: Closed
• Remove duplicated dependencies in the pom file of test suite module: Closed
• Use scala_version placeholder to replace scala version hard code e.g. 2.11: Closed
• Fix issues related to running the test suite in docker due to Hive 2.x: Resolved
• Augment the existing DAG with more complex use-cases seen at Uber: Closed
• Add ability to induce failures to catch issues: Resolved
• Allow test suite to change hudi write configs for any dag node: Closed
• Specify version information for each component separately: Closed
• Support long options for prepare_integration_suite: Closed
• Ability to test across different Hudi versions using Hudi test suite: Closed
• [Test Suite] Add option to configure different path selector: Resolved
• Generate valid timestamp and partition for data generator: Closed