Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4248

Refresh needed to pick up new Avro schema modified from inside impala

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.6.0
    • Fix Version/s: None
    • Component/s: Frontend

      Description

      Avro schema from within Impala can be changed by doing:

      ALTER TABLE avro_table SET TBLPROPERTIES('avro.schema.url'='hdfs:///tmp/evolved.avsc');
      ALTER TABLE avro_table ADD COLUMNS(c3 string);
      

      However, if the order of the statements is reversed, Impala also needs a refresh:

      ALTER TABLE avro_table ADD COLUMNS(c3 string);
      ALTER TABLE avro_table SET TBLPROPERTIES('avro.schema.url'='hdfs:///tmp/evolved.avsc');
      compute stats avro_table;
      ---->ERROR: AnalysisException: Cannot COMPUTE STATS on Avro table 'avro_table' because its column definitions do not match those in the Avro schema.
      Missing Avro-schema column corresponding to column definition 'c3' of type 'string' at position '1'.
      Please re-create the table with column definitions, e.g., using the result of 'SHOW CREATE TABLE'
      
      refresh avro_table;
      compute stats avro_table;
      +-----------------------------------------+
      | summary                                 |
      +-----------------------------------------+
      | Updated 1 partition(s) and 3 column(s). |
      +-----------------------------------------+
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                balazsj_impala_220b Balazs Jeszenszky
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: