[IMPALA-6790] sqlparse needs to be upgraded in the Python environment

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.0
• Fix Version/s: Impala 3.0, Impala 2.13.0
• Component/s: Infrastructure
• Target Version: Impala 3.0, Impala 2.12.0
• Epic Color: ghx-label-5
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 02/Apr/18 18:30
• Updated: 19/Apr/18 21:18
• Resolved: 19/Apr/18 17:21

Description

bin/load-data.py uses sqlparse to read SQL files and split them into SQL statements. Recently, some remote cluster tests have seen errors during dataload due to sqlparse failing to split SQL statements appropriately. Specifically, it does not detect the end of a SQL statement and tries to run dozens of SQL statements together. Impala's parser rejects this. The SQL file is identical to the SQL file generated during our normal dataload, so clearly, something about this system or its environment breaks sqlparse.

sqlparse in our environment is 0.1.15, which is quite old. The latest sqlparse is 0.2.4. Running the tests with sqlparse 0.2.4 does not encounter the error. sqlparse needs to be upgraded.