[IMPALA-12267] DMLs/DDLs can hang as a result of catalogd restart

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.2.0
• Component/s: Backend
• Epic Color: ghx-label-5
• Assignee: Daniel Becker
• Reporter: Csaba Ringhofer
• Created: 06/Jul/23 16:19
• Updated: 06/Nov/23 14:22
• Resolved: 26/Jul/23 11:17

Description

IMPALA-5476 added handling for changes in catalogd service ID during the lifetime of a DML/DDL query, but the loop that waits for a new catalogd service ID can wait indefinitely in case the DML/DDL was handled by the previous catalogd and restart happened just after replying to the coordinator:
https://github.com/apache/impala/blob/d0fe4c604f72d41019832513ebf65cfe8f469953/be/src/service/impala-server.cc#L2204

Could reproduce this by adding a sleep just before taking catalog_version_lock_ and restarting catalogd while a DML/DDL was waiting there. As a result the query hanged - it could be finished only by restarting catalogd. Canncelling the query also wouldn't help, the thread that handles the query is stuck until catalogd restart.

Note that this issue may be present on all Impala versions that contain IMPALA-5476, but I haven't verified this and other changes could also affect the behavior (e.g. async DML support).

Issue Links

• is broken by: IMPALA-5476 Catalogd restart bring about metadata is out of sync
• is related to: IMPALA-10875 Transient stale catalog if catalogd is restarted more than once shortly
• relates to: IMPALA-12545 Use 'unique_database' in test_restart_services.py::TestRestart