[IMPALA-10051] impala-shell exits with ValueError with WITH clauses

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.0.0
• Component/s: Clients
• Epic Color: ghx-label-1
• Assignee: Tamas Mate
• Reporter: Tamas Mate
• Created: 05/Aug/20 21:32
• Updated: 20/Jul/22 16:27
• Resolved: 23/Sep/20 17:05

Description

Some strings can cause shlex to throw an exception in WITH clauses, for example in a regexp_replace. This should be handled more gracefully and correctly.

Working query (impala-shell forwards the query for analysis):

impala-shell.sh -q 'with select regexp_replace(column_name, "[a-zA-Z]", "+ ");'

While same query fails with ValueError when empty spaces are removed from the arguments of the regexp_replace:

tmate@tmate-box:~/Projects/Impala$ impala-shell.sh -q 'with select regexp_replace(column_name,"[a-zA-Z]","+ ");'
Starting Impala Shell with no authentication using Python 2.7.16
Warning: live_progress only applies to interactive shell sessions, and is being skipped for now.
Opened TCP connection to localhost:21000
Connected to localhost:21000
Server version: impalad version 4.0.0-SNAPSHOT DEBUG (build b29cb4ca82a4f05ea7dc0eadc330a64fbe685ef0)
Traceback (most recent call last):
  File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1973, in <module>
    impala_shell_main()
  File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1927, in impala_shell_main
    if execute_queries_non_interactive_mode(options, query_options):
  File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1731, in execute_queries_non_interactive_mode
    shell.execute_query_list(queries))
  File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1564, in execute_query_list
    if self.onecmd(q) is CmdStatus.ERROR:
  File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 675, in onecmd
    return func(arg)
  File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1276, in do_with
    tokens = shlex.split(strip_comments(query.lstrip()), posix=False)
  File "/home/tmate/Projects/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/shlex.py", line 279, in split
    return list(lex)
  File "/home/tmate/Projects/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/shlex.py", line 269, in next
    token = self.get_token()
  File "/home/tmate/Projects/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/shlex.py", line 96, in get_token
    raw = self.read_token()
  File "/home/tmate/Projects/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/shlex.py", line 172, in read_token
    raise ValueError, "No closing quotation"
ValueError: No closing quotation

Issue Links

• is related to: IMPALA-7939 Impala shell not displaying results for a CTE query.
• requires: IMPALA-9362 Update sqlparse used by impala-shell from version 0.1.19 to latest