Hive
  1. Hive
  2. HIVE-1653

Ability to enforce correct stats

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      This is a follow-up for https://issues.apache.org/jira/browse/HIVE-1361.

      If one of the mappers/reducers cannot publish stats, it may lead to wrong aggregated stats.
      There should be a way to avoid this - at the least, a configuration variable which fails the
      task if stats cannot be published

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          8d 22h 39m 3 Carl Steinbach 24/Jul/12 07:37
          Open Open Patch Available Patch Available
          667d 9h 48m 4 Namit Jain 25/Jul/12 06:35
          Patch Available Patch Available Resolved Resolved
          1d 12h 6m 1 Kevin Wilfong 26/Jul/12 18:41
          Resolved Resolved Closed Closed
          168d 2h 12m 1 Ashutosh Chauhan 10/Jan/13 19:54
          Lefty Leverenz made changes -
          Link This issue is related to HIVE-3777 [ HIVE-3777 ]
          Hide
          Lefty Leverenz added a comment -

          This adds hive.stats.reliable to HiveConf.java and hive-default.xml.template (see patch in Facebook reviews, since it isn't attached here: https://reviews.facebook.net/D4119).

          Show
          Lefty Leverenz added a comment - This adds hive.stats.reliable to HiveConf.java and hive-default.xml.template (see patch in Facebook reviews, since it isn't attached here: https://reviews.facebook.net/D4119 ).
          Gavin made changes -
          Link This issue depends upon HIVE-1361 [ HIVE-1361 ]
          Gavin made changes -
          Link This issue depends on HIVE-1361 [ HIVE-1361 ]
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-1653. Ability to enforce correct stats. (njain via kevinwilfong) (Revision 1366103)

          Result = ABORTED
          kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366103
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsPublisher.java
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_2.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_2.q
          • /hive/trunk/ql/src/test/queries/clientpositive/stats_aggregator_error_1.q
          • /hive/trunk/ql/src/test/queries/clientpositive/stats_publisher_error_1.q
          • /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_2.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_2.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/stats_publisher_error_1.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-1653 . Ability to enforce correct stats. (njain via kevinwilfong) (Revision 1366103) Result = ABORTED kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366103 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsPublisher.java /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_1.q /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_2.q /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_1.q /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_2.q /hive/trunk/ql/src/test/queries/clientpositive/stats_aggregator_error_1.q /hive/trunk/ql/src/test/queries/clientpositive/stats_publisher_error_1.q /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_1.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_2.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_1.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_2.q.out /hive/trunk/ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out /hive/trunk/ql/src/test/results/clientpositive/stats_publisher_error_1.q.out
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1569 (See https://builds.apache.org/job/Hive-trunk-h0.21/1569/)
          HIVE-1653. Ability to enforce correct stats. (njain via kevinwilfong) (Revision 1366103)

          Result = FAILURE
          kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366103
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsPublisher.java
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_2.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_1.q
          • /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_2.q
          • /hive/trunk/ql/src/test/queries/clientpositive/stats_aggregator_error_1.q
          • /hive/trunk/ql/src/test/queries/clientpositive/stats_publisher_error_1.q
          • /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_2.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_2.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/stats_publisher_error_1.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1569 (See https://builds.apache.org/job/Hive-trunk-h0.21/1569/ ) HIVE-1653 . Ability to enforce correct stats. (njain via kevinwilfong) (Revision 1366103) Result = FAILURE kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1366103 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsAggregator.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/stats/DummyStatsPublisher.java /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_1.q /hive/trunk/ql/src/test/queries/clientnegative/stats_aggregator_error_2.q /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_1.q /hive/trunk/ql/src/test/queries/clientnegative/stats_publisher_error_2.q /hive/trunk/ql/src/test/queries/clientpositive/stats_aggregator_error_1.q /hive/trunk/ql/src/test/queries/clientpositive/stats_publisher_error_1.q /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_1.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_aggregator_error_2.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_1.q.out /hive/trunk/ql/src/test/results/clientnegative/stats_publisher_error_2.q.out /hive/trunk/ql/src/test/results/clientpositive/stats_aggregator_error_1.q.out /hive/trunk/ql/src/test/results/clientpositive/stats_publisher_error_1.q.out
          Kevin Wilfong made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.10.0 [ 12320745 ]
          Resolution Fixed [ 1 ]
          Hide
          Kevin Wilfong added a comment -

          Committed, thanks Namit.

          Show
          Kevin Wilfong added a comment - Committed, thanks Namit.
          Hide
          Kevin Wilfong added a comment -

          +1

          Will run tests tonight, and commit tomorrow unless there are further comments.

          Show
          Kevin Wilfong added a comment - +1 Will run tests tonight, and commit tomorrow unless there are further comments.
          Hide
          Namit Jain added a comment -

          Answered

          Show
          Namit Jain added a comment - Answered
          Hide
          Shreepadma Venugopalan added a comment -

          @Namit: Added comments on phabricator. Thanks!

          Show
          Shreepadma Venugopalan added a comment - @Namit: Added comments on phabricator. Thanks!
          Namit Jain made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Namit Jain added a comment -

          comments

          Show
          Namit Jain added a comment - comments
          Carl Steinbach made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Carl Steinbach added a comment -

          @Namit: Added a comment on phabricator. Thanks.

          Show
          Carl Steinbach added a comment - @Namit: Added a comment on phabricator. Thanks.
          Namit Jain made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Namit Jain added a comment -

          addressed comments, added tests

          Show
          Namit Jain added a comment - addressed comments, added tests
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          will address comments

          Show
          Namit Jain added a comment - will address comments
          Namit Jain made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Namit Jain added a comment -

          comments

          Show
          Namit Jain added a comment - comments
          Kevin Wilfong made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Kevin Wilfong added a comment -

          Commented on the diff.

          Show
          Kevin Wilfong added a comment - Commented on the diff.
          Namit Jain made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Show
          Namit Jain added a comment - https://reviews.facebook.net/D4119
          Namit Jain made changes -
          Assignee Ning Zhang [ nzhang ] Namit Jain [ namit ]
          Carl Steinbach made changes -
          Fix Version/s 0.7.0 [ 12315150 ]
          Namit Jain made changes -
          Link This issue is related to HIVE-1361 [ HIVE-1361 ]
          Namit Jain made changes -
          Field Original Value New Value
          Link This issue depends on HIVE-1361 [ HIVE-1361 ]
          Namit Jain created issue -

            People

            • Assignee:
              Namit Jain
              Reporter:
              Namit Jain
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development