Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6454

CTAS into Kudu fails with mixed-case partition and/pr primary key column names

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0
    • Impala 3.0, Impala 2.12.0
    • Frontend

    Description

      CTAS into Kudu mail fail if the primary key and/or partition column names are not specified in lower case.

      The observed error is:

      ERROR: AnalysisException: null
      CAUSED BY: IllegalStateException: null
      

      Reproduction:

      CREATE TABLE ctas_test
      PRIMARY KEY (PK1, PK2)
      PARTITION BY HASH (PK1) PARTITIONS 10
      STORED AS KUDU
      AS
      SELECT 1 PK1, 2 PK2, 3 C1, 4 C2
      

      IMPALA Log:

      ===========
      I0122 13:39:15.829866 7557 Frontend.java:927] Compiled query.
      I0122 13:39:15.830080 7557 jni-util.cc:196] org.apache.impala.common.AnalysisException
      at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:433)
      at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:369)
      at org.apache.impala.service.Frontend.analyzeStmt(Frontend.java:899)
      at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1048)
      at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)
      Caused by: java.lang.IllegalStateException
      at com.google.common.base.Preconditions.checkState(Preconditions.java:129)
      at org.apache.impala.analysis.InsertStmt.prepareExpressions(InsertStmt.java:770)
      at org.apache.impala.analysis.InsertStmt.analyze(InsertStmt.java:383)
      at org.apache.impala.analysis.CreateTableAsSelectStmt.analyze(CreateTableAsSelectStmt.java:212)
      at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:388)
      ... 4 more
      I0122 13:39:15.845522 7557 status.cc:122] AnalysisException: null
      CAUSED BY: IllegalStateException: null
      @ 0x83e979 impala::Status::Status()
      @ 0xb71ec9 impala::JniUtil::GetJniExceptionMsg()
      @ 0xa7cc0b impala::Frontend::GetExecRequest()
      @ 0xa97368 impala::ImpalaServer::ExecuteInternal()
      @ 0xa9e408 impala::ImpalaServer::Execute()
      @ 0xadd27e impala::ImpalaServer::query()
      @ 0xd6ee05 beeswax::BeeswaxServiceProcessor::process_query()
      @ 0xd72104 beeswax::BeeswaxServiceProcessor::dispatchCall()
      @ 0x80e69c apache::thrift::TDispatchProcessor::process()
      @ 0x1b5666b apache::thrift::server::TThreadPoolServer::Task::run()
      @ 0x1b3df89 apache::thrift::concurrency::ThreadManager::Worker::run()
      @ 0x9d4ca9 impala::ThriftThread::RunRunnable()
      @ 0x9d5a82 boost::detail::function::void_function_obj_invoker0<>::invoke()
      @ 0xbd4722 impala::Thread::SuperviseThread()
      @ 0xbd4e84 boost::detail::thread_data<>::run()
      @ 0xe6113a (unknown)
      @ 0x7fd1efc19dc5 start_thread
      @ 0x7fd1ef946ced __clone
      I0122 13:39:15.845556 7557 impala-server.cc:989] UnregisterQuery(): query_id=42439790ebd1f31f:772bd00b00000000
      I0122 13:39:15.845562 7557 impala-server.cc:1075] Cancel(): query_id=42439790ebd1f31f:772bd00b00000000
      

      Rootcause: In the below call graph in the function KuduTable.isPrimaryKeyColumn() primaryKeyColumnNames_ does not have a matching case which causes
      primaryKeyExprs_ to be empty and hence we hit an Exception. Changing the case fixes the problem.
      Problem: In the InsertStmt.prepareExpressions() , primaryKeyExprs_ is empty, which is causing Preconditions.checkState(!primaryKeyExprs_.isEmpty()); to fail and as a result
      Exception is generated.
      Call graph
      -----------------
      AnalysisContext.analyze(0
      --->CreateTableAsSelectStmt.analyze()
      ------->InsertStmt.analyze
      ----------->InsertStmt.prepareExpressions
      -------------->KuduTable.isPrimaryKeyColumn

      Attachments

        Activity

          pranay_singh Pranay Singh added a comment -

          The issue has been fixed by the below PATCH

          IMPALA-6454: CTAS into Kudu fails with mixed-case partition or primary key column names.

          CTAS into Kudu fails if the primary key and/or the partition column names are not
          specified in lower case.The problem is that we pass in the primary key column names
          directly from the parser instead we should be passing the post-analysis ColumnDefs
          as primary keys. So it is fixed by making changes to createCtasTarget()
          in KuduTable class to take a list of ColumnDef class associated with the primary keys
          from getPrimaryKeyColumnDefs() in CreateTableStmt class.

          ColumnDef class has column names stored in lower case that are used by createCtasTarget()
          to populate primaryKeyColumnNames_ in KuduTable class that resolves the issue.

          Testing
          -------
          Verified against the newly added test case that reproduces the issue without the fix.

          Change-Id: Ica1c8ec1544339e9e80733a7a0c78594e0a727d2
          Reviewed-on: http://gerrit.cloudera.org:8080/9147
          Reviewed-by: Alex Behm <alex.behm@cloudera.com>
          Tested-by: Impala Public Jenkins

          pranay_singh Pranay Singh added a comment - The issue has been fixed by the below PATCH IMPALA-6454 : CTAS into Kudu fails with mixed-case partition or primary key column names. CTAS into Kudu fails if the primary key and/or the partition column names are not specified in lower case.The problem is that we pass in the primary key column names directly from the parser instead we should be passing the post-analysis ColumnDefs as primary keys. So it is fixed by making changes to createCtasTarget() in KuduTable class to take a list of ColumnDef class associated with the primary keys from getPrimaryKeyColumnDefs() in CreateTableStmt class. ColumnDef class has column names stored in lower case that are used by createCtasTarget() to populate primaryKeyColumnNames_ in KuduTable class that resolves the issue. Testing ------- Verified against the newly added test case that reproduces the issue without the fix. Change-Id: Ica1c8ec1544339e9e80733a7a0c78594e0a727d2 Reviewed-on: http://gerrit.cloudera.org:8080/9147 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Impala Public Jenkins
          laszlog Laszlo Gaal added a comment -

          pranay_singh, could you please set the FixVersion field, and also  AffectVersion (if it's known)? Thanks a lot!

          laszlog Laszlo Gaal added a comment - pranay_singh , could you please set the FixVersion field, and also  AffectVersion (if it's known)? Thanks a lot!
          alex.behm Alexander Behm added a comment -

          pranay_singh, can you please set the FixVersion and AffectedVersions (if known)?

          alex.behm Alexander Behm added a comment - pranay_singh , can you please set the FixVersion and AffectedVersions (if known)?

          People

            pranay_singh Pranay Singh
            alex.behm Alexander Behm
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: