Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8406

Failed REFRESH can partially modify table without bumping version number

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 3.2.0
    • Impala 4.0.0
    • Catalog
    • None
    • ghx-label-1

    Description

      Currently, various incremental operations in the catalogd modify Table objects in place, including REFRESH, which modifies each partition. In this case, if one partition fails to refresh (eg due to incorrect partitions or some other file access problem), other partitions can still be modified, either because they were modified first (in a non-parallel operation) or modified in parallel (for REFRESH).

      In this case, the REFRESH operation will throw an Exception back to the user, but in fact it has modified the catalog entry. The version number, however, is not bumped, which breaks some invariants of the catalog that an object doesn't change without changing version numbers.

      This also produces some unexpected behavior such as:

      • SHOW FILES IN t;
      • REFRESH t; – gets a failure
      • SHOW FILES in t; – see the same result as originally
      • ALTER TABLE t SET UNCACHED; – bumps the version number due to unrelated operation
      • SHOW FILES IN t; – the set of files has changed due to the earlier partially-complete REFRESH

      Attachments

        Activity

          People

            stigahuang Quanlong Huang
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: