[PHOENIX-5802] Connection leaks in UPSERT SELECT/DELETE paths due to MutatingParallelIteratorFactory iterator not being closed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.0.0, 4.15.0, 4.14.3
• Fix Version/s: 5.1.0, 4.16.0
• Labels: phoenix-hardening quality-improvement
• Assignee: Chinmay Kulkarni
• Reporter: Chinmay Kulkarni
• Created: 27/Mar/20 06:08
• Updated: 20/Apr/20 20:25
• Resolved: 28/Mar/20 05:08

Description

During an UPSERT SELECT query or client-side DELETE, we clone the existing Phoenix connection and call mutate which eventually calls either upsertSelect or deleteRows.

When getting iterators, if we encounter any exception, then on calling close, we attempt to cancel any queued work, while accumulating already-started task futures.
Later we attempt to block on getting the result (iterator) of already-started tasks and close each iterator.

Here however, if any of the upsert-select or client-side delete tasks had thrown an exception E, we wouldn't be able to close the underlying iterator (which in-turn would've closed the cloned connection). This manifests as these logs:

"Failed to execute task during cancel", <exception E stack trace>

This is leading to a connection leak. Because we use CQSI to create the Phoenix Connection here, it also gets accounted for during open Phoenix connection throttling, thus leaving the client application with less connections that can be opened.

Issue Links

• links to: GitHub Pull Request #746