[IMPALA-2592] DataStreamSender::Channel::CloseInternal() does not close the channel on an error.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.5.0
• Component/s: Distributed Exec
• Labels: hang query-lifecycle usability
• Target Version: Impala 2.5.0
• Assignee: Sailesh Mukil
• Reporter: Sailesh Mukil
• Created: 23/Oct/15 18:25
• Updated: 02/Mar/16 06:13
• Resolved: 02/Mar/16 06:13

Description

The DataStreamSender::Channel::CloseInternal() does not close the channel on an error. This will cause the node on the other side of the channel to wait indefinitely causing a hang.

We observed this occurring due to a kerberos issue related to the the credential cache being outdated, but in theory this can occur for any error that causes Channel::CloseInternal() to fail without closing properly.

The query "hangs" in that it remains registered forever and never makes progress, but the impalads remain responsive and continue to work as expected for other queries.

There is no workaround at this time.

Issue Links

• is duplicated by: IMPALA-2246 Stress test hung cluster
• is related to: IMPALA-2246 Stress test hung cluster
• is related to: IMPALA-2587 High variance in cdh5.5.x-rhel7-exhaustive run times