Hive
  1. Hive
  2. HIVE-6298

Add config flag to turn off fetching partition stats

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None
    1. HIVE-6298.2.patch
      4 kB
      Gunther Hagleitner
    2. HIVE-6298.1.patch
      4 kB
      Gunther Hagleitner

      Issue Links

        Activity

        Hide
        Gunther Hagleitner added a comment -

        For debugging purposes it's helpful to be able to suppress loading of the stats for all partitions of a table when the stats annotation is being used.

        Show
        Gunther Hagleitner added a comment - For debugging purposes it's helpful to be able to suppress loading of the stats for all partitions of a table when the stats annotation is being used.
        Hide
        Prasanth Jayachandran added a comment -

        LGTM.

        Show
        Prasanth Jayachandran added a comment - LGTM.
        Hide
        Lefty Leverenz added a comment -

        Please document hive.stats.fetch.partition.stats in hive-default.xml.template, then I'll add it to the wiki for release 0.13.

        If you want to be thorough, you can also make these changes to hive-default.xml.template:

        • change the default value of hive.stats.key.prefix.max.length from 200 to 150
        • document 8 config properties that have been added since release 0.12:
          • hive.stats.max.variable.length
          • hive.stats.list.num.entries
          • hive.stats.map.num.entries
          • hive.stats.map.parallelism
          • hive.stats.fetch.column.stats
          • hive.stats.avg.row.size
          • hive.stats.join.factor
          • hive.stats.deserialization.factor

        These changes were found by comparing files in branch 0.12 & trunk (as of January 13th). I've learned my lesson: from now on I'll check stats patches for new configs.

        Show
        Lefty Leverenz added a comment - Please document hive.stats.fetch.partition.stats in hive-default.xml.template, then I'll add it to the wiki for release 0.13. If you want to be thorough, you can also make these changes to hive-default.xml.template: change the default value of hive.stats.key.prefix.max.length from 200 to 150 document 8 config properties that have been added since release 0.12: hive.stats.max.variable.length hive.stats.list.num.entries hive.stats.map.num.entries hive.stats.map.parallelism hive.stats.fetch.column.stats hive.stats.avg.row.size hive.stats.join.factor hive.stats.deserialization.factor These changes were found by comparing files in branch 0.12 & trunk (as of January 13th). I've learned my lesson: from now on I'll check stats patches for new configs.
        Hide
        Prasanth Jayachandran added a comment -

        My apologies. Its my mistake. I should have added these to hive-default.xml.template. All these were added as part of HIVE-5369 and its subtasks. I created another subtask to fix them here HIVE-6300. Will post a patch soon. I will document "hive.stats.fetch.partition.stats" as well in the new JIRA.

        Show
        Prasanth Jayachandran added a comment - My apologies. Its my mistake. I should have added these to hive-default.xml.template. All these were added as part of HIVE-5369 and its subtasks. I created another subtask to fix them here HIVE-6300 . Will post a patch soon. I will document "hive.stats.fetch.partition.stats" as well in the new JIRA.
        Hide
        Gunther Hagleitner added a comment -

        Thanks for taking up adding this particular flag as well, Prasanth Jayachandran. Lefty Leverenz thanks for the reminder. I wasn't sure that this particular flag needs to be documented (mostly for developers when debugging), but I think you're right. It should get added.

        Show
        Gunther Hagleitner added a comment - Thanks for taking up adding this particular flag as well, Prasanth Jayachandran . Lefty Leverenz thanks for the reminder. I wasn't sure that this particular flag needs to be documented (mostly for developers when debugging), but I think you're right. It should get added.
        Hide
        Sergey Shelukhin added a comment -

        lgtm

        Show
        Sergey Shelukhin added a comment - lgtm
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12624956/HIVE-6298.1.patch

        ERROR: -1 due to 13 failed/errored test(s), 4963 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name
        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
        org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1008/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1008/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 13 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12624956

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12624956/HIVE-6298.1.patch ERROR: -1 due to 13 failed/errored test(s), 4963 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_import_exported_table org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_load_hdfs_file_with_space_in_the_name org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_file_with_header_footer_negative Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1008/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1008/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed This message is automatically generated. ATTACHMENT ID: 12624956
        Hide
        Sergey Shelukhin added a comment -

        +1

        Show
        Sergey Shelukhin added a comment - +1
        Hide
        Gunther Hagleitner added a comment -

        rebased to trunk

        Show
        Gunther Hagleitner added a comment - rebased to trunk
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12626145/HIVE-6298.2.patch

        ERROR: -1 due to 2 failed/errored test(s), 4980 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6
        org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1125/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1125/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 2 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12626145

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12626145/HIVE-6298.2.patch ERROR: -1 due to 2 failed/errored test(s), 4980 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1125/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1125/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed This message is automatically generated. ATTACHMENT ID: 12626145
        Hide
        Gunther Hagleitner added a comment -

        Committed to trunk. Thanks Sergey Shelukhin, Prasanth Jayachandran and Lefty Leverenz for the reviews!

        Show
        Gunther Hagleitner added a comment - Committed to trunk. Thanks Sergey Shelukhin , Prasanth Jayachandran and Lefty Leverenz for the reviews!
        Hide
        Lefty Leverenz added a comment -

        This needs a fix version.

        Show
        Lefty Leverenz added a comment - This needs a fix version.

          People

          • Assignee:
            Gunther Hagleitner
            Reporter:
            Gunther Hagleitner
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development