[IMPALA-2587] High variance in cdh5.5.x-rhel7-exhaustive run times

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Not A Bug
• Affects Version/s: Impala 2.3.0
• Component/s: Infrastructure
• Labels: test test-infra
• Target Version: Product Backlog
• Assignee: Mostafa Mokhtar
• Reporter: Jim Apple
• Created: 23/Oct/15 16:15
• Updated: 02/Mar/18 00:48
• Resolved: 02/Mar/18 00:48

Description

http://sandbox.jenkins.cloudera.com/job/impala-cdh5.5.x-rhel7-exhaustive/23/

took almost a week to run, and eventually failed due to reasons that look related to that lenght of time:

01:49:52 [gw8] PASSED stress/test_ddl_stress.py::TestDdlStress::test_create_cache_many_tables[test_id: 0 | exec_option:

{'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0}

| table_format: text/none] Build timed out (after 10,000 minutes). Marking the build as failed.
20:24:38 Build was aborted
20:24:38 [CHECKSTYLE] Skipping publisher since build result is FAILURE
20:24:38 [FINDBUGS] Skipping publisher since build result is FAILURE
20:24:38 [TASKS] Skipping publisher since build result is FAILURE
20:24:38 Archiving artifacts
20:24:38
20:24:38 [gw11] node down: Not properly terminated
20:24:38 [gw11] FAILED stress/test_mini_stress.py::TestMiniStress::test_mini_stress[test_id: 4 | exec_option:

{'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0}

| table_format: seq/snap/block]
20:24:38 Replacing failed node gw11
20:24:39 INTERNALERROR> Traceback (most recent call last):
20:24:39 INTERNALERROR> File "/data/jenkins/workspace/impala-cdh5.5.x-rhel7-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/_pytest/main.py", line 82, in wrap_session
20:24:39 INTERNALERROR> config.hook.pytest_sessionstart(session=session)
20:24:39 INTERNALERROR> File "/data/jenkins/workspace/impala-cdh5.5.x-rhel7-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/pytest/core.py", line 521, in __call_
20:24:39 INTERNALERROR> return self._docall(self.methods, kwargs)
20:24:39 INTERNALERROR> File "/data/jenkins/workspace/impala-cdh5.5.x-rhel7-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/_pytest/core.py", line 528, in _docall
20:24:39 INTERNALERROR> firstresult=self.firstresult).execute()
20:24:39 INTERNALERROR> File "/data/jenkins/workspace/impala-cdh5.5.x-rhel7-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/_pytest/core.py", line 394, in execute
20:24:39 INTERNALERROR> res = method(*args)
20:24:39 INTERNALERROR> File "<remote exec>", line 32, in pytest_sessionstart
20:24:39 INTERNALERROR> File "<remote exec>", line 23, in sendevent
20:24:39 INTERNALERROR> File "/data/jenkins/workspace/impala-cdh5.5.x-rhel7-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/execnet/gateway_base.py", line 717, in send
20:24:39 INTERNALERROR> self.gateway._send(Message.CHANNEL_DATA, self.id, dumps_internal(item))
20:24:39 INTERNALERROR> File "/data/jenkins/workspace/impala-cdh5.5.x-rhel7-exhaustive/repos/Impala/infra/python/env/lib/python2.7/site-packages/execnet/gateway_base.py", line 990, in _send
20:24:39 INTERNALERROR> raise IOError("cannot send (already closed?)")
20:24:39 INTERNALERROR> IOError: cannot send (already closed?)
20:24:42 Recording test results
20:24:45 Email was triggered for: Failure
20:24:45 Sending email for trigger: Failure
20:24:45 Sending email to: [e-mail]
20:24:46 Finished: FAILURE

Note that http://sandbox.jenkins.cloudera.com/job/impala-cdh5.5.x-rhel7-exhaustive/22/ took "only" 9 hours to run, and ended green.

Issue Links

• relates to: IMPALA-2592 DataStreamSender::Channel::CloseInternal() does not close the channel on an error.