Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4170

Incorrect double quoting of identifier in SQL generated by COMPUTE INCREMENTAL STATS.

    Details

      Description

      COMPUTE INCREMENTAL STATS fails to update the stats of new partitions if the name of a partition column is a reserved word. A syntax error is reported because a generated SQL statement incorrectly uses two sets of backticks for quoting the identifier.

      Repro:

      create table t (i int) partitioned by (`date` int);
      insert into t partition(`date`) values (1, 1);
      compute incremental stats t;
      insert into t partition(`date`) values (2, 2);
      [localhost:21000] > compute incremental stats t;
      Query: compute incremental stats t
      WARNINGS: AnalysisException: Syntax error in line 1:
      ...te` FROM default.t WHERE (``date``=2) GROUP BY `date`
                                   ^
      Encountered: EMPTY IDENTIFIER
      Expected: CASE, CAST, EXISTS, FALSE, IF, INTERVAL, NOT, NULL, SELECT, TRUNCATE, TRUE, VALUES, WITH, IDENTIFIER
      
      CAUSED BY: Exception: Syntax error
      

      The problem is in HdfsPartition.getConjunctSql() where the double quoting is pretty obvious:

        public String getConjunctSql() {
          List<String> partitionCols = Lists.newArrayList();
          for (int i = 0; i < getTable().getNumClusteringCols(); ++i) {
            partitionCols.add(ToSqlUtils.getIdentSql(getTable().getColumns().get(i).getName()));
          }
      
          List<String> conjuncts = Lists.newArrayList();
          for (int i = 0; i < partitionCols.size(); ++i) {
            LiteralExpr expr = getPartitionValues().get(i);
            String sql = expr.toSql();
            if (expr instanceof NullLiteral || sql.isEmpty()) {
              conjuncts.add(ToSqlUtils.getIdentSql(partitionCols.get(i))
                  + " IS NULL");
            } else {
              conjuncts.add(ToSqlUtils.getIdentSql(partitionCols.get(i))
                  + "=" + sql);
            }
          }
          return "(" + Joiner.on(" AND " ).join(conjuncts) + ")";
        }
      

        Activity

        Hide
        alex.behm Alexander Behm added a comment -

        commit 3aa43516252fe4e75176ab62cb6ce1f04938a96e
        Author: Alex Behm <alex.behm@cloudera.com>
        Date: Tue Sep 20 10:19:29 2016 -0700

        IMPALA-4170: Fix identifier quoting in COMPUTE INCREMENTAL STATS.

        The SQL statements generated from COMPUTE INCREMENTAL STATS
        did not properly quote identifiers when incrementally updating
        the stats for newly added partitions.

        Our existing tests did not catch this case because the code paths
        for doing the initial stats computation and the incremental stats
        computation are different, in particular, the code for generating
        the SQL statements.

        Change-Id: I63adcc45dc964ce769107bf4139fc4566937bb96
        Reviewed-on: http://gerrit.cloudera.org:8080/4479
        Reviewed-by: Henry Robinson <henry@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        alex.behm Alexander Behm added a comment - commit 3aa43516252fe4e75176ab62cb6ce1f04938a96e Author: Alex Behm <alex.behm@cloudera.com> Date: Tue Sep 20 10:19:29 2016 -0700 IMPALA-4170 : Fix identifier quoting in COMPUTE INCREMENTAL STATS. The SQL statements generated from COMPUTE INCREMENTAL STATS did not properly quote identifiers when incrementally updating the stats for newly added partitions. Our existing tests did not catch this case because the code paths for doing the initial stats computation and the incremental stats computation are different, in particular, the code for generating the SQL statements. Change-Id: I63adcc45dc964ce769107bf4139fc4566937bb96 Reviewed-on: http://gerrit.cloudera.org:8080/4479 Reviewed-by: Henry Robinson <henry@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development