[IMPALA-9266] TestLogFragments.test_log_fragments fails due to missing log

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 3.4.0
• Component/s: Infrastructure
• Labels: broken-build
• Epic Color: ghx-label-1
• Assignee: Sheng Wang
• Reporter: Joe McDonnell
• Created: 27/Dec/19 23:19
• Updated: 22/Jan/20 20:01
• Resolved: 22/Jan/20 20:01

Description

TestLogFragments.test_log_fragments is failing due to missing a log entry:

/data/jenkins/workspace/impala-asf-master-core/repos/Impala/tests/observability/test_log_fragments.py:46: in test_log_fragments
    "] Analysis and authorization finished.")
common/impala_test_suite.py:1149: in assert_impalad_log_contains
    self.assert_log_contains("impalad", level, line_regex, expected_count)
common/impala_test_suite.py:1185: in assert_log_contains
    (expected_count, log_file_path, line_regex, found, line)
E   AssertionError: Expected 1 lines in file /data0/jenkins/workspace/impala-asf-master-core/repos/Impala/logs/ee_tests/impalad.impala-ec2-centos74-m5-4xlarge-ondemand-088c.vpc.cloudera.com.jenkins.log.INFO.20191227-001949.23945 matching regex 'ce41d657e70d6890:6f0f227d00000000] Analysis and authorization finished.', but found 0 lines. Last line was: 
E   Caught signal: SIGTERM. Daemon will exit.

This started happening after the "IMPALA-8974: Fixed a bug when create kudu managed table without HMS" commit went in. That commit adds a test that restarts Impala in a frontend test. The problem is that it runs start-impala-cluster.py without arguments, whereas bin/run-all-tests.sh runs start-impala-cluster.py specifying the --log_dir. This would put the log files in a different location (/tmp?).

https://github.com/apache/impala/blob/320f05852060c1027326ac20be7df340a7a5263f/fe/src/test/java/org/apache/impala/catalog/CreateKuduTableWithoutHMSTest.java#L98

https://github.com/apache/impala/blob/master/bin/run-all-tests.sh#L165-L167

In one run that hit this issue, there are two sets of impalad logs in the ee_test directory. One set starts at 06:40:22 and ends at 07:11:28. The second set starts at 09:45:25 and ends at 09:47:30. So, this is missing 2.5 hours of ee_test log files, which matches the theory.

This is also likely to impact other things like erasure coding or tests that run against the data cache.

GVO doesn't hit this because the job that runs frontend tests does not run end to end tests.

Issue Links

• is duplicated by: IMPALA-9268 CreateKuduTableWithoutHMSTest caused TestLogFragments failed
• relates to: IMPALA-8974 Create kudu table failed when catalogd connected to mysql directly instead of use HMS