Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4248

Refresh needed to pick up new Avro schema modified from inside impala

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • Impala 2.6.0
    • None
    • Frontend

    Description

      Avro schema from within Impala can be changed by doing:

      ALTER TABLE avro_table SET TBLPROPERTIES('avro.schema.url'='hdfs:///tmp/evolved.avsc');
      ALTER TABLE avro_table ADD COLUMNS(c3 string);
      

      However, if the order of the statements is reversed, Impala also needs a refresh:

      ALTER TABLE avro_table ADD COLUMNS(c3 string);
      ALTER TABLE avro_table SET TBLPROPERTIES('avro.schema.url'='hdfs:///tmp/evolved.avsc');
      compute stats avro_table;
      ---->ERROR: AnalysisException: Cannot COMPUTE STATS on Avro table 'avro_table' because its column definitions do not match those in the Avro schema.
      Missing Avro-schema column corresponding to column definition 'c3' of type 'string' at position '1'.
      Please re-create the table with column definitions, e.g., using the result of 'SHOW CREATE TABLE'
      
      refresh avro_table;
      compute stats avro_table;
      +-----------------------------------------+
      | summary                                 |
      +-----------------------------------------+
      | Updated 1 partition(s) and 3 column(s). |
      +-----------------------------------------+
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            balazsj_impala_220b Balazs Jeszenszky

            Dates

              Created:
              Updated:

              Slack

                Issue deployment