[IMPALA-6223] Gracefully handle malformed 'with' queries in impala-shell

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Clients
• Labels: newbie
• Epic Color: ghx-label-2
• Assignee: Pooja Nilangekar
• Reporter: Bharath Vissapragada
• Created: 20/Nov/17 21:46
• Updated: 26/Mar/19 01:50
• Resolved: 09/Jul/18 20:04

Description

Impala shell can throw a lexer error if it encounters a malformed "with" query.

impala-shell.sh -q "with foo as (select bar from temp where temp.a='"
Starting Impala Shell without Kerberos authentication
Connected to localhost:21000
Server version: impalad version 2.11.0-SNAPSHOT DEBUG (build 0ee1765f38082bc5c10aa37b23cb8e57caa57d4e)
Traceback (most recent call last):
  File "/home/bharath/Impala/shell/impala_shell.py", line 1463, in <module>
    execute_queries_non_interactive_mode(options, query_options)
  File "/home/bharath/Impala/shell/impala_shell.py", line 1338, in execute_queries_non_interactive_mode
    shell.execute_query_list(queries)):
  File "/home/bharath/Impala/shell/impala_shell.py", line 1218, in execute_query_list
    if self.onecmd(q) is CmdStatus.ERROR:
  File "/home/bharath/Impala/shell/impala_shell.py", line 505, in onecmd
    return cmd.Cmd.onecmd(self, line)
  File "/usr/lib/python2.7/cmd.py", line 221, in onecmd
    return func(arg)
  File "/home/bharath/Impala/shell/impala_shell.py", line 1024, in do_with
    tokens = list(lexer)
  File "/usr/lib/python2.7/shlex.py", line 269, in next
    token = self.get_token()
  File "/usr/lib/python2.7/shlex.py", line 96, in get_token
    raw = self.read_token()
  File "/usr/lib/python2.7/shlex.py", line 172, in read_token
    raise ValueError, "No closing quotation"
ValueError: No closing quotation

This happens because we use shlex to parse the input query to determine if its a DML and it can throw if the input doesn't have balanced quotes.

def do_with(self, args):
    """Executes a query with a WITH clause, fetching all rows"""
    query = self.imp_client.create_beeswax_query("with %s" % args,
                                                 self.set_query_options)
    # Set posix=True and add "'" to escaped quotes
    # to deal with escaped quotes in string literals
    lexer = shlex.shlex(query.query.lstrip(), posix=True)
    lexer.escapedquotes += "'"
    # Because the WITH clause may precede DML or SELECT queries,
    # just checking the first token is insufficient.
    is_dml = False
    tokens = list(lexer)  <----

A simple shlex repro of that is as follows,

>>> lexer = shlex.shlex("with foo as (select bar from temp where temp.a='", posix=True);
>>> list(lexer)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/shlex.py", line 269, in next
    token = self.get_token()
  File "/usr/lib/python2.7/shlex.py", line 96, in get_token
    raw = self.read_token()
  File "/usr/lib/python2.7/shlex.py", line 172, in read_token
    raise ValueError, "No closing quotation"
ValueError: No closing quotation

Fix: Either catch the exception and handle it gracefully or have a better way to figure out the query type, using a SQL parser (more involved).

This query also repros it:

with v as (select 1)
select foo('\\'), ('bar
;

Issue Links

• is duplicated by: IMPALA-6224 impala-shell exits with error "No closing quotation" with certain syntactic errors