Hive
  1. Hive
  2. HIVE-3777

add a property in the partition to figure out if stats are accurate

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      Currently, stats task tries to update the statistics in the table/partition
      being updated after the table/partition is loaded. In case of a failure to
      update these stats (due to the any reason), the operation either succeeds
      (writing inaccurate stats) or fails depending on whether hive.stats.reliable
      is set to true. This can be bad for applications who do not always care about
      reliable stats, since the query may have taken a long time to execute and then
      fail eventually.

      Another property should be added to the partition: areStatsAccurate. If hive.stats.reliable is
      set to false, and stats could not be computed correctly, the operation would
      still succeed, update the stats, but set areStatsAccurate to false.
      If the application cares about accurate stats, it can be obtained in the
      background.

      1. HIVE-3777.patch
        13 kB
        Ashutosh Chauhan
      2. HIVE-3777.5.patch
        552 kB
        Ashutosh Chauhan
      3. HIVE-3777.4.patch
        552 kB
        Ashutosh Chauhan
      4. HIVE-3777.3.patch
        552 kB
        Ashutosh Chauhan
      5. HIVE-3777.2.patch
        22 kB
        Ashutosh Chauhan
      6. HIVE-3777.2.patch
        541 kB
        Ashutosh Chauhan

        Issue Links

          Activity

          Namit Jain created issue -
          Namit Jain made changes -
          Field Original Value New Value
          Summary add hive.stats.accurate in the partition add a property in the partition to figure out if stats are accurate
          Description Currently, stats task tries to update the statistics in the table/partition
          being updated after the table/partition is loaded. In case of a failure to
          update these stats (due to the any reason), the operation either succeeds
          (writing inaccurate stats) or fails depending on whether hive.stats.reliable
          is set to true. This can be bad for applications who do not always care about
          reliable stats, since the query may have taken a long time to execute and then
          fail eventually.

          Another option should be added: hive.accurate.stats. If hive.stats.reliable is
          set to false, and stats could not be computed correctly, the operation would
          still succeed, update the stats, but set hive.accurate.stats to false.
          If the application cares about accurate stats, it can be obtained in the
          background.
          Currently, stats task tries to update the statistics in the table/partition
          being updated after the table/partition is loaded. In case of a failure to
          update these stats (due to the any reason), the operation either succeeds
          (writing inaccurate stats) or fails depending on whether hive.stats.reliable
          is set to true. This can be bad for applications who do not always care about
          reliable stats, since the query may have taken a long time to execute and then
          fail eventually.

          Another property should be added to the partition: areStatsAccurate. If hive.stats.reliable is
          set to false, and stats could not be computed correctly, the operation would
          still succeed, update the stats, but set areStatsAccurate to false.
          If the application cares about accurate stats, it can be obtained in the
          background.
          Gang Tim Liu made changes -
          Assignee Gang Tim Liu [ gangtimliu ]
          Gang Tim Liu made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Gang Tim Liu made changes -
          Assignee Gang Tim Liu [ gangtimliu ] Dilip Joseph [ dilipjoseph ]
          Gang Tim Liu made changes -
          Link This issue is depended upon by HIVE-4150 [ HIVE-4150 ]
          Dilip Joseph made changes -
          Assignee Dilip Joseph [ dilipjoseph ] Ashutosh Chauhan [ ashutoshc ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3777.patch [ 12610825 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3777.2.patch [ 12611644 ]
          Ashutosh Chauhan made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Affects Version/s 0.13.0 [ 12324986 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3777.2.patch [ 12612397 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3777.3.patch [ 12612486 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3777.4.patch [ 12612677 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ashutosh Chauhan made changes -
          Attachment HIVE-3777.5.patch [ 12612797 ]
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.13.0 [ 12324986 ]
          Resolution Fixed [ 1 ]
          Lefty Leverenz made changes -
          Link This issue relates to HIVE-1653 [ HIVE-1653 ]
          Brock Noland made changes -
          Link This issue is related to HIVE-8648 [ HIVE-8648 ]

            People

            • Assignee:
              Ashutosh Chauhan
              Reporter:
              Namit Jain
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development