[HIVE-26871] TestCrudCompactorOnTez is flaky after HIVE-26479

• Type: Test
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-2
• Component/s: Test
• Labels: pull-request-available
• Assignee: Sourabh Badhya
• Reporter: Sourabh Badhya
• Created: 19/Dec/22 04:58
• Updated: 21/Dec/22 07:51
• Resolved: 21/Dec/22 07:39

Description

The 3 tests in TestCrudCompactorOnTez which use the ProtoLoggingHook run at different times. Unfortunately, the 3 tests are run at the following times as described in the logs - 
Test 1 - 

INFO [main] compactor.TestCrudCompactorOnTez: Current time: 2022-12-15T23:57:44 

Test 2 - 

INFO [main] compactor.TestCrudCompactorOnTez: Current time: 2022-12-16T00:00:32 

Test 3 - 

INFO [main] compactor.TestCrudCompactorOnTez: Current time: 2022-12-16T00:04:12 

As we can see, the tests are run on 2 different dates. Therefore, HiveProtoLoggingHook generates a unique event logs for every unique date. This is the behaviour of HiveProtoLoggingHook.
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java#L296-L310

However the expectation from the test side, while generating the log readers is that there must be a single file in the log folder defined.
https://github.com/apache/hive/blob/master/ql/src/test/org/apache/hadoop/hive/ql/hooks/TestHiveProtoLoggingHook.java#L310 

Unfortunately, since there are 2 files which are generated (as mentioned in the logs as well), the following tests fail - 

INFO [main] hooks.TestHiveProtoLoggingHook: List of paths: 
INFO [main] hooks.TestHiveProtoLoggingHook: file:/home/jenkins/agent/workspace/internal-hive-flaky-check/itests/hive-unit/target/tmp/junit441259831997042392/junit3438435196942546140/date=2022-12-15
INFO [main] hooks.TestHiveProtoLoggingHook: file:/home/jenkins/agent/workspace/internal-hive-flaky-check/itests/hive-unit/target/tmp/junit441259831997042392/junit3438435196942546140/date=2022-12-16 

The solution is to make getTestReader() in TestHiveProtoLoggingHook more compatible with multiple event log file scenario and be able to generate multiple readers for all files present in the folder instead of fixating on a single file clause.

Issue Links

• is caused by: HIVE-26479 Add ability to set parameters for query-based compaction
• links to: GitHub Pull Request #3876