Hive
  1. Hive
  2. HIVE-1653

Ability to enforce correct stats

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      This is a follow-up for https://issues.apache.org/jira/browse/HIVE-1361.

      If one of the mappers/reducers cannot publish stats, it may lead to wrong aggregated stats.
      There should be a way to avoid this - at the least, a configuration variable which fails the
      task if stats cannot be published

        Issue Links

          Activity

          Hide
          Lefty Leverenz added a comment -

          This adds hive.stats.reliable to HiveConf.java and hive-default.xml.template (see patch in Facebook reviews, since it isn't attached here: https://reviews.facebook.net/D4119).

          Show
          Lefty Leverenz added a comment - This adds hive.stats.reliable to HiveConf.java and hive-default.xml.template (see patch in Facebook reviews, since it isn't attached here: https://reviews.facebook.net/D4119 ).
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-1653. Ability to enforce correct stats. (njain via kevinwilfong) (Revision 1366103)

          Result = ABORTED
          kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366103
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsPublisher.java
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_2.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_2.q
          • /hive/trunk/ql/src/test/queries/clientpositive/stats_aggregator_error_1.q
          • /hive/trunk/ql/src/test/queries/clientpositive/stats_publisher_error_1.q
          • /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_2.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_2.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/stats_publisher_error_1.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-1653 . Ability to enforce correct stats. (njain via kevinwilfong) (Revision 1366103) Result = ABORTED kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366103 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsPublisher.java /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_1.q /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_2.q /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_1.q /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_2.q /hive/trunk/ql/src/test/queries/clientpositive/stats_aggregator_error_1.q /hive/trunk/ql/src/test/queries/clientpositive/stats_publisher_error_1.q /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_1.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_2.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_1.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_2.q.out /hive/trunk/ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out /hive/trunk/ql/src/test/results/clientpositive/stats_publisher_error_1.q.out
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1569 (See https://builds.apache.org/job/Hive-trunk-h0.21/1569/)
          HIVE-1653. Ability to enforce correct stats. (njain via kevinwilfong) (Revision 1366103)

          Result = FAILURE
          kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366103
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsPublisher.java
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_2.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_2.q
          • /hive/trunk/ql/src/test/queries/clientpositive/stats_aggregator_error_1.q
          • /hive/trunk/ql/src/test/queries/clientpositive/stats_publisher_error_1.q
          • /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_2.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_2.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/stats_publisher_error_1.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1569 (See https://builds.apache.org/job/Hive-trunk-h0.21/1569/ ) HIVE-1653 . Ability to enforce correct stats. (njain via kevinwilfong) (Revision 1366103) Result = FAILURE kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366103 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsPublisher.java /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_1.q /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_2.q /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_1.q /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_2.q /hive/trunk/ql/src/test/queries/clientpositive/stats_aggregator_error_1.q /hive/trunk/ql/src/test/queries/clientpositive/stats_publisher_error_1.q /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_1.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_2.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_1.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_2.q.out /hive/trunk/ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out /hive/trunk/ql/src/test/results/clientpositive/stats_publisher_error_1.q.out
          Hide
          Kevin Wilfong added a comment -

          Committed, thanks Namit.

          Show
          Kevin Wilfong added a comment - Committed, thanks Namit.
          Hide
          Kevin Wilfong added a comment -

          +1

          Will run tests tonight, and commit tomorrow unless there are further comments.

          Show
          Kevin Wilfong added a comment - +1 Will run tests tonight, and commit tomorrow unless there are further comments.
          Hide
          Namit Jain added a comment -

          Answered

          Show
          Namit Jain added a comment - Answered
          Hide
          Shreepadma Venugopalan added a comment -

          @Namit: Added comments on phabricator. Thanks!

          Show
          Shreepadma Venugopalan added a comment - @Namit: Added comments on phabricator. Thanks!
          Hide
          Namit Jain added a comment -

          comments

          Show
          Namit Jain added a comment - comments
          Hide
          Carl Steinbach added a comment -

          @Namit: Added a comment on phabricator. Thanks.

          Show
          Carl Steinbach added a comment - @Namit: Added a comment on phabricator. Thanks.
          Hide
          Namit Jain added a comment -

          addressed comments, added tests

          Show
          Namit Jain added a comment - addressed comments, added tests
          Hide
          Namit Jain added a comment -

          will address comments

          Show
          Namit Jain added a comment - will address comments
          Hide
          Namit Jain added a comment -

          comments

          Show
          Namit Jain added a comment - comments
          Hide
          Kevin Wilfong added a comment -

          Commented on the diff.

          Show
          Kevin Wilfong added a comment - Commented on the diff.
          Show
          Namit Jain added a comment - https://reviews.facebook.net/D4119

            People

            • Assignee:
              Namit Jain
              Reporter:
              Namit Jain
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development