Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5955

Use the totalSize Hive table property instead of rawDataSize

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • Impala 2.11.0
    • Catalog, Frontend
    • None

    Description

      IMPALA-2373 changed COMPUTE STATS to also populate the 'rawDataSize' table property for the purpose of row count extrapolation. However, we should use 'totalSize' instead of 'rawDataSize' instead. Based on searching Google and looking at the Hive code it looks like the 'rawDataSize' roughly corresponds to the estimated in-memory size of a table (without encoding and compression), whereas the 'totalSize' property is used to represent the on-disk size.

      I confirmed in the SparkSQL code that it prefers the 'totalSize' property for query planning. Also, SparkSQL's ANALYZE TABLE populates the 'totalSize'. We should try to be as compatible as possible with Hive/SparkSQL to avoid hard-to-debug stats inconsistencies.

      Attachments

        Activity

          People

            alex.behm Alexander Behm
            alex.behm Alexander Behm
            Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: