Hive
  1. Hive
  2. HIVE-3959

Update Partition Statistics in Metastore Layer

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: Metastore, Statistics
    • Labels:
      None

      Description

      When partitions are created using queries ("insert overwrite" and "insert into") then the StatsTask updates all stats. However, when partitions are added directly through metadata-only partitions (either CLI or direct calls to Thrift Metastore) no stats are populated even if hive.stats.reliable is set to true. This puts us in a situation where we can't decide if stats are truly reliable or not.

      We propose that the "fast stats" (numFiles and totalSize) which don't require a scan of the data should always be populated and be completely reliable. For now we are still excluding rowCount and rawDataSize because that will make these operations very expensive. Currently they are quick metadata-only ops.

      1. HIVE-3959.6.patch
        915 kB
        Ashutosh Chauhan
      2. HIVE-3959.5.patch
        909 kB
        Ashutosh Chauhan
      3. HIVE-3959.4.patch
        914 kB
        Ashutosh Chauhan
      4. HIVE-3959.3.patch
        917 kB
        Ashutosh Chauhan
      5. HIVE-3959.4.patch
        919 kB
        Ashutosh Chauhan
      6. HIVE-3959.3.patch
        959 kB
        Ashutosh Chauhan
      7. HIVE-3959.2.patch
        913 kB
        Ashutosh Chauhan
      8. HIVE-3959.1.patch
        913 kB
        Ashutosh Chauhan
      9. HIVE-3959.patch.12.txt
        745 kB
        Gang Tim Liu
      10. HIVE-3959.patch.11.txt
        740 kB
        Gang Tim Liu
      11. HIVE-3959.patch.2
        55 kB
        Gang Tim Liu
      12. HIVE-3959.patch.1
        55 kB
        Gang Tim Liu

        Issue Links

          Activity

          Bhushan Mandhani created issue -
          Gang Tim Liu made changes -
          Field Original Value New Value
          Assignee Bhushan Mandhani [ bmandhani ] Gang Tim Liu [ gangtimliu ]
          Gang Tim Liu made changes -
          Attachment HIVE-3959.patch.1 [ 12576832 ]
          Gang Tim Liu made changes -
          Attachment HIVE-3959.patch.2 [ 12576835 ]
          Gang Tim Liu made changes -
          Attachment HIVE-3959.patch.2.nohcat [ 12576837 ]
          Gang Tim Liu made changes -
          Link This issue depends upon HIVE-4456 [ HIVE-4456 ]
          Gang Tim Liu made changes -
          Attachment HIVE-3959.patch.2.nohcat [ 12576837 ]
          Gang Tim Liu made changes -
          Attachment HIVE-3959.patch.9.txt [ 12581427 ]
          Gang Tim Liu made changes -
          Attachment HIVE-3959.patch.11.txt [ 12581591 ]
          Gang Tim Liu made changes -
          Attachment HIVE-3959.patch.9.txt [ 12581427 ]
          Gang Tim Liu made changes -
          Attachment HIVE-3959.patch.12.txt [ 12581709 ]
          Gang Tim Liu made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Gang Tim Liu made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Gang Tim Liu made changes -
          Assignee Gang Tim Liu [ gangtimliu ] Dilip Joseph [ dilipjoseph ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3959.1.patch [ 12609940 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Dilip Joseph [ dilipjoseph ] Ashutosh Chauhan [ ashutoshc ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3959.2.patch [ 12610103 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3959.3.patch [ 12610139 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3959.4.patch [ 12610142 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3959.3.patch [ 12611419 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3959.4.patch [ 12611422 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3959.5.patch [ 12611520 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3959.6.patch [ 12611523 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Ashutosh Chauhan
              Reporter:
              Bhushan Mandhani
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development