Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-10294

ParseException issue (Failed to recognize predicate 'user') [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark
    • Labels:
      None

      Description

      ParseException (Failed to recognize predicate 'user') occurred when we run some Hive query (e.g. BigBench case Q03).
      By the way, if we change all the 'user' to 'user2', the parsing could be passed.

      (1) Exception Details:

      FailedPredicateException(identifier,{useSQL11ReservedKeywordsForIdentifier()}?)
              at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10912)
              at org.apache.hadoop.hive.ql.parse.HiveParser.identifier(HiveParser.java:45479)
              at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2933)
              at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1367)
              at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1122)
              at org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:45475)
              at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41136)
              at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41043)
              at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40059)
              at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39929)
              at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5295)
              at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3731)
              at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1865)
              at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1512)
              at org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45556)
              at org.apache.hadoop.hive.ql.parse.HiveParser.singleFromStatement(HiveParser.java:40666)
              at org.apache.hadoop.hive.ql.parse.HiveParser.fromStatement(HiveParser.java:40389)
              at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40044)
              at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39929)
              at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.subQuerySource(HiveParser_FromClauseParser.java:5295)
              at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromSource(HiveParser_FromClauseParser.java:3731)
              at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.joinSource(HiveParser_FromClauseParser.java:1865)
              at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.fromClause(HiveParser_FromClauseParser.java:1512)
              at org.apache.hadoop.hive.ql.parse.HiveParser.fromClause(HiveParser.java:45556)
              at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:41157)
              at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:40876)
              at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40059)
              at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:39929)
              at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1574)
              at org.apache.hadoop.hive.ql.parse.HiveParser.explainStatement(HiveParser.java:1269)
              at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1070)
              at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
              at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:397)
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:309)
              at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1115)
              at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1163)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1052)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1042)
              at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
              at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
              at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
              at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
              at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
              at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
              at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
              at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
              at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      FAILED: ParseException line 8:27 Failed to recognize predicate 'user'. Failed rule: 'identifier' in selection target
      

      (2) Query Details (BigBench Q03):

      ADD FILE ${hiveconf:QUERY_DIR}/reducer_q3.py;
      set hive.exec.compress.output=false;
      
      DROP TABLE IF EXISTS ${hiveconf:RESULT_TABLE};
      CREATE TABLE ${hiveconf:RESULT_TABLE} (
        lastviewed_item BIGINT,
        purchased_item  BIGINT,
        cnt             BIGINT
      )
      ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
      STORED AS ${env:BIG_BENCH_hive_default_fileformat_result_table} LOCATION '${hiveconf:RESULT_DIR}';
      
      INSERT INTO TABLE ${hiveconf:RESULT_TABLE}
      SELECT lastviewed_item, purchased_item, count(*)
      FROM (
        FROM (
          SELECT
            wcs_user_sk       AS user,
            wcs_click_date_sk AS lastviewed_date,
            wcs_item_sk       AS lastviewed_item,
            wcs_sales_sk      AS lastviewed_sale
          FROM web_clickstreams w
          WHERE wcs_user_sk IS NOT NULL
          CLUSTER BY user
        ) q03_map_output
        REDUCE
        q03_map_output.user,
        q03_map_output.lastviewed_date,
        q03_map_output.lastviewed_item,
        q03_map_output.lastviewed_sale
        USING 'python reducer_q3.py ${hiveconf:q03_days_before_purchase}'
        AS (lastviewed_item BIGINT, purchased_item BIGINT)
      ) q03_nPath
      join item i on (i.i_item_sk = q03_nPath.lastviewed_item
        AND i.i_category_id IN (${hiveconf:q03_purchased_item_category_IN})
      )
      WHERE purchased_item IN ( ${hiveconf:q03_purchased_item_IN} )
      GROUP BY lastviewed_item, purchased_item,i_category
      ;
      

        Attachments

          Activity

            People

            • Assignee:
              sivashankar.c Sivashankar
              Reporter:
              xhao1 Xin Hao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: