Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3959

Update Partition Statistics in Metastore Layer

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 0.13.0
    • Metastore, Statistics
    • None

    Description

      When partitions are created using queries ("insert overwrite" and "insert into") then the StatsTask updates all stats. However, when partitions are added directly through metadata-only partitions (either CLI or direct calls to Thrift Metastore) no stats are populated even if hive.stats.reliable is set to true. This puts us in a situation where we can't decide if stats are truly reliable or not.

      We propose that the "fast stats" (numFiles and totalSize) which don't require a scan of the data should always be populated and be completely reliable. For now we are still excluding rowCount and rawDataSize because that will make these operations very expensive. Currently they are quick metadata-only ops.

      Attachments

        1. HIVE-3959.patch.1
          55 kB
          Gang Tim Liu
        2. HIVE-3959.patch.2
          55 kB
          Gang Tim Liu
        3. HIVE-3959.patch.11.txt
          740 kB
          Gang Tim Liu
        4. HIVE-3959.patch.12.txt
          745 kB
          Gang Tim Liu
        5. HIVE-3959.1.patch
          913 kB
          Ashutosh Chauhan
        6. HIVE-3959.2.patch
          913 kB
          Ashutosh Chauhan
        7. HIVE-3959.3.patch
          959 kB
          Ashutosh Chauhan
        8. HIVE-3959.4.patch
          919 kB
          Ashutosh Chauhan
        9. HIVE-3959.3.patch
          917 kB
          Ashutosh Chauhan
        10. HIVE-3959.4.patch
          914 kB
          Ashutosh Chauhan
        11. HIVE-3959.5.patch
          909 kB
          Ashutosh Chauhan
        12. HIVE-3959.6.patch
          915 kB
          Ashutosh Chauhan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ashutoshc Ashutosh Chauhan Assign to me
            bmandhani Bhushan Mandhani
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment