[IMPALA-9362] Update sqlparse used by impala-shell from version 0.1.19 to latest

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 4.0.0
• Component/s: Clients
• Epic Color: ghx-label-9
• Assignee: David Knupp
• Reporter: David Knupp
• Created: 06/Feb/20 20:34
• Updated: 20/Jul/22 16:27
• Resolved: 17/Apr/20 05:50

Description

The fix for IMPALA-6337 involved correcting the way that sqlparse, an upstream 3rd party python library used by the impala-shell, parses queries that contain line breaks embedded inside of double quotes. Initially, Impala's internally bundled version of sqlparse (based on 0.1.19) was patched; meanwhile, a pull request to get the fix into an official release was submitted.

That pull-request was finally included in the 0.3.0 version of sqlparse. However, there were other changes to the library in the interim, in terms of API's and also in some of the parsing logic, that breaks the impala-shell in other ways, so simply migrating to the newer release is not straightforward.

We need to find and fix all the places that the newer sqlparse breaks the impala-shell, so that we can stop relying on sqlparse 0.1.19 (which, in some places, is not python 3 compatible).

Issue Links

• is duplicated by: IMPALA-9501 Upgrade sqlparse to a version that supports python 3.0
• is related to: IMPALA-3343 Impala-shell compatibility with python 3
• is related to: IMPALA-9501 Upgrade sqlparse to a version that supports python 3.0
• is required by: IMPALA-10051 impala-shell exits with ValueError with WITH clauses
• relates to: IMPALA-6337 Infinite loop in impala_shell.py