Hive
  1. Hive
  2. HIVE-2925

Support non-MR fetching for simple queries with select/limit/filter operations only

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It's trivial but frequently asked by end-users. Currently, select queries with simple conditions or limit should run MR job which takes some time especially for big tables, making the people irritated.

      For that kind of simple queries, using fetch task would make them happy.

      1. ASF.LICENSE.NOT.GRANTED--HIVE-2925.D2607.1.patch
        21 kB
        Phabricator
      2. ASF.LICENSE.NOT.GRANTED--HIVE-2925.D2607.2.patch
        24 kB
        Phabricator
      3. ASF.LICENSE.NOT.GRANTED--HIVE-2925.D2607.3.patch
        24 kB
        Phabricator
      4. ASF.LICENSE.NOT.GRANTED--HIVE-2925.D2607.4.patch
        24 kB
        Phabricator
      5. HIVE-2925.1.patch.txt
        112 kB
        Navis
      6. HIVE-2925.2.patch.txt
        121 kB
        Navis
      7. HIVE-2925.3.patch.txt
        144 kB
        Navis

        Issue Links

          Activity

          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-2925 - Support non-MR fetching for simple queries with select/limit/filter operations only (Navis via namit) (Revision 1374389)

          Result = ABORTED
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374389
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out
          • /hive/trunk/hbase-handler/src/test/results/positive/hbase_pushdown.q.out
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecMapperContext.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GlobalLimitCtx.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PrunedPartitionList.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListSinkDesc.java
          • /hive/trunk/ql/src/test/queries/clientpositive/nonmr_fetch.q
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat2.q
          • /hive/trunk/ql/src/test/results/clientpositive/input.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input0.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_limit.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_part0.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_part3.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_part4.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_part8.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/nonmr_fetch.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat2.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/ppr_pushdown3.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/regex_col.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/source.q.out
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedListObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedMapObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedObjectInspectorFactory.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedStructObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedUnionObjectInspector.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2925 - Support non-MR fetching for simple queries with select/limit/filter operations only (Navis via namit) (Revision 1374389) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374389 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out /hive/trunk/hbase-handler/src/test/results/positive/hbase_pushdown.q.out /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecMapperContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GlobalLimitCtx.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PrunedPartitionList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListSinkDesc.java /hive/trunk/ql/src/test/queries/clientpositive/nonmr_fetch.q /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat2.q /hive/trunk/ql/src/test/results/clientpositive/input.q.out /hive/trunk/ql/src/test/results/clientpositive/input0.q.out /hive/trunk/ql/src/test/results/clientpositive/input_limit.q.out /hive/trunk/ql/src/test/results/clientpositive/input_part0.q.out /hive/trunk/ql/src/test/results/clientpositive/input_part3.q.out /hive/trunk/ql/src/test/results/clientpositive/input_part4.q.out /hive/trunk/ql/src/test/results/clientpositive/input_part8.q.out /hive/trunk/ql/src/test/results/clientpositive/nonmr_fetch.q.out /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat2.q.out /hive/trunk/ql/src/test/results/clientpositive/ppr_pushdown3.q.out /hive/trunk/ql/src/test/results/clientpositive/regex_col.q.out /hive/trunk/ql/src/test/results/clientpositive/source.q.out /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedListObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedMapObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedObjectInspectorFactory.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedStructObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedUnionObjectInspector.java
          Navis made changes -
          Link This issue duplicates HIVE-3045 [ HIVE-3045 ]
          Ashutosh Chauhan made changes -
          Fix Version/s 0.10.0 [ 12320745 ]
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1615 (See https://builds.apache.org/job/Hive-trunk-h0.21/1615/)
          HIVE-2925 - Support non-MR fetching for simple queries with select/limit/filter operations only (Navis via namit) (Revision 1374389)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374389
          Files :

          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out
          • /hive/trunk/hbase-handler/src/test/results/positive/hbase_pushdown.q.out
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecMapperContext.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GlobalLimitCtx.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PrunedPartitionList.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListSinkDesc.java
          • /hive/trunk/ql/src/test/queries/clientpositive/nonmr_fetch.q
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat2.q
          • /hive/trunk/ql/src/test/results/clientpositive/input.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input0.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_limit.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_part0.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_part3.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_part4.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/input_part8.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/nonmr_fetch.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat2.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/ppr_pushdown3.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/regex_col.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/source.q.out
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedListObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedMapObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedObjectInspectorFactory.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedStructObjectInspector.java
          • /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedUnionObjectInspector.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1615 (See https://builds.apache.org/job/Hive-trunk-h0.21/1615/ ) HIVE-2925 - Support non-MR fetching for simple queries with select/limit/filter operations only (Navis via namit) (Revision 1374389) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374389 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/hbase-handler/src/test/results/positive/hbase_ppd_key_range.q.out /hive/trunk/hbase-handler/src/test/results/positive/hbase_pushdown.q.out /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecMapperContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GlobalLimitCtx.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/PrunedPartitionList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ListSinkDesc.java /hive/trunk/ql/src/test/queries/clientpositive/nonmr_fetch.q /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat2.q /hive/trunk/ql/src/test/results/clientpositive/input.q.out /hive/trunk/ql/src/test/results/clientpositive/input0.q.out /hive/trunk/ql/src/test/results/clientpositive/input_limit.q.out /hive/trunk/ql/src/test/results/clientpositive/input_part0.q.out /hive/trunk/ql/src/test/results/clientpositive/input_part3.q.out /hive/trunk/ql/src/test/results/clientpositive/input_part4.q.out /hive/trunk/ql/src/test/results/clientpositive/input_part8.q.out /hive/trunk/ql/src/test/results/clientpositive/nonmr_fetch.q.out /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat2.q.out /hive/trunk/ql/src/test/results/clientpositive/ppr_pushdown3.q.out /hive/trunk/ql/src/test/results/clientpositive/regex_col.q.out /hive/trunk/ql/src/test/results/clientpositive/source.q.out /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedListObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedMapObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedObjectInspectorFactory.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedStructObjectInspector.java /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/DelegatedUnionObjectInspector.java
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Namit Jain added a comment -

          Committed. Thanks Navis

          Show
          Namit Jain added a comment - Committed. Thanks Navis
          Hide
          Namit Jain added a comment -

          looks good - will start testing

          Show
          Namit Jain added a comment - looks good - will start testing
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Navis added a comment -

          Rebased on trunk. Sorry for late reply.

          Show
          Navis added a comment - Rebased on trunk. Sorry for late reply.
          Navis made changes -
          Attachment HIVE-2925.3.patch.txt [ 12541300 ]
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          Navis, can you refresh ?

          Since, jira was down for some time, I could not take a look.
          Please refresh, I will try to get it in asap

          Show
          Namit Jain added a comment - Navis , can you refresh ? Since, jira was down for some time, I could not take a look. Please refresh, I will try to get it in asap
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Navis added a comment -

          Passed all tests

          Show
          Navis added a comment - Passed all tests
          Hide
          Namit Jain added a comment -

          To validate this, can you also add equivalent of partition_wife_fileformat3.q with "more" non-mr fetch.

          Show
          Namit Jain added a comment - To validate this, can you also add equivalent of partition_wife_fileformat3.q with "more" non-mr fetch.
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          One test: partition_wise_fileformat2.q still fails.
          Can you debug ?

          It has a select * query with different file formats for different partitions.

          Show
          Namit Jain added a comment - One test: partition_wise_fileformat2.q still fails. Can you debug ? It has a select * query with different file formats for different partitions.
          Hide
          Namit Jain added a comment -

          comments

          Show
          Namit Jain added a comment - comments
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Navis made changes -
          Attachment HIVE-2925.2.patch.txt [ 12539016 ]
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          Some tests failed -

          Some failures in TestHBaseCliDriver
          TesCliDriver (bucketcontext_2.q, bucketcontext_3.q,,,,)

          Can you take a look ?

          Show
          Namit Jain added a comment - Some tests failed - Some failures in TestHBaseCliDriver TesCliDriver (bucketcontext_2.q, bucketcontext_3.q,,,,) Can you take a look ?
          Navis made changes -
          Attachment HIVE-2925.1.patch.txt [ 12538878 ]
          Namit Jain made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Namit Jain added a comment -

          +1

          Show
          Namit Jain added a comment - +1
          Hide
          Navis added a comment -

          dummy assignment removal and some explanations.

          Show
          Navis added a comment - dummy assignment removal and some explanations.
          Hide
          Namit Jain added a comment -

          question ?

          Show
          Namit Jain added a comment - question ?
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          test changes

          Show
          Namit Jain added a comment - test changes
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          minor comments on phabricator

          Show
          Namit Jain added a comment - minor comments on phabricator
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          comments on phabricator

          Show
          Namit Jain added a comment - comments on phabricator
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Namit Jain added a comment -

          Copying a mail I sent to dev@ for review....

          Currently, hive does not launch map-reduce jobs for the following queries:

          select * from <T> where <condition on partition columns> (limit <n>)?

          This behavior is not configurable, and cannot be altered.

          HIVE-2925 wants to extend this behavior. The goal is not to spawn map-reduce jobs for the following queries:

          Select <expr> from <T> where <any condition> (limit <n>)?

          It is currently controlled by one parameter: hive.aggressive.fetch.task.conversion, based on which it is decided, whether to spawn
          map-reduce jobs or not for the queries of the above type. Note that this can be beneficial for certain types of queries, since it is
          avoiding the expensive step of spawning map-reduce. However, it can be pretty expensive for certain types of queries: selecting
          a very large number of rows, the query having a very selective filter (which is satisfied by a very number of rows, and therefore involves
          scanning a very large table) etc. The user does not have any control on this. Note that it cannot be done by hooks, since the pre-semantic
          hooks does not have enough information: type of the query, inputs etc. and it is too late to do anything in the post-semantic hook (the
          query plan has already been altered).

          I would like to propose the following configuration parameters to control this behavior.
          hive.fetch.task.conversion: true, false, auto

          If the value is true, then all queries with only selects and filters will be converted
          If the value is false, then no query will be converted
          If the value is auto (which should be the default behavior), there should be additional parameters to control the semantics.

          hive.fetch.task.auto.limit.threshold ---> integer value X1
          hive.fetch.task.auto.inputsize.threshold ---> integer value X2

          If either the query has a limit lower than X1, or the input size is smaller than X2, the queries containing only filters and selects will be converted to not use
          map-reudce jobs.

          Show
          Namit Jain added a comment - Copying a mail I sent to dev@ for review.... Currently, hive does not launch map-reduce jobs for the following queries: select * from <T> where <condition on partition columns> (limit <n>)? This behavior is not configurable, and cannot be altered. HIVE-2925 wants to extend this behavior. The goal is not to spawn map-reduce jobs for the following queries: Select <expr> from <T> where <any condition> (limit <n>)? It is currently controlled by one parameter: hive.aggressive.fetch.task.conversion, based on which it is decided, whether to spawn map-reduce jobs or not for the queries of the above type. Note that this can be beneficial for certain types of queries, since it is avoiding the expensive step of spawning map-reduce. However, it can be pretty expensive for certain types of queries: selecting a very large number of rows, the query having a very selective filter (which is satisfied by a very number of rows, and therefore involves scanning a very large table) etc. The user does not have any control on this. Note that it cannot be done by hooks, since the pre-semantic hooks does not have enough information: type of the query, inputs etc. and it is too late to do anything in the post-semantic hook (the query plan has already been altered). I would like to propose the following configuration parameters to control this behavior. hive.fetch.task.conversion: true, false, auto If the value is true, then all queries with only selects and filters will be converted If the value is false, then no query will be converted If the value is auto (which should be the default behavior), there should be additional parameters to control the semantics. hive.fetch.task.auto.limit.threshold ---> integer value X1 hive.fetch.task.auto.inputsize.threshold ---> integer value X2 If either the query has a limit lower than X1, or the input size is smaller than X2, the queries containing only filters and selects will be converted to not use map-reudce jobs.
          Hide
          Namit Jain added a comment -

          I was thinking about it more - what I said above cannot be performed in a custom hook.
          In a pre-semantic hook, you dont have enough information — the type of the query, stats etc.
          In a post-semantic hook, it is too late.

          Show
          Namit Jain added a comment - I was thinking about it more - what I said above cannot be performed in a custom hook. In a pre-semantic hook, you dont have enough information — the type of the query, stats etc. In a post-semantic hook, it is too late.
          Hide
          Namit Jain added a comment -

          It would be very difficult to deploy it this way.

          In general, I can think of the following:

          1. For queries with limits, this optimization should be enabled.
          2. Ideally, it would be good, if there is a threshold of the limit.
          3. For queries without limits, given the fact that we dont a cost based optimizer, it might be a good to
          have a threshold on the total input data.

          I mean, in general, non MR fetch might make sense for the following.
          1. select from a small table (where small is configurable)
          2. select from a big table is OK if there is a limit

          Note that, it is still possible to get a plan where this optimization might make not sense.

          For eg: select col1 from T where col2 = 10 limit 10;

          It is possible that there are very rows for which col2 is 10, so not having a MR job may really slow down this query.
          Solving that would be more difficult without more statistics.

          But, it may be a good idea to add more config parameters to tune the hive.aggresive.fetch.task.conversion appropriately.
          It can also be done in a follow-up patch, and is independent of this.

          Show
          Namit Jain added a comment - It would be very difficult to deploy it this way. In general, I can think of the following: 1. For queries with limits, this optimization should be enabled. 2. Ideally, it would be good, if there is a threshold of the limit. 3. For queries without limits, given the fact that we dont a cost based optimizer, it might be a good to have a threshold on the total input data. I mean, in general, non MR fetch might make sense for the following. 1. select from a small table (where small is configurable) 2. select from a big table is OK if there is a limit Note that, it is still possible to get a plan where this optimization might make not sense. For eg: select col1 from T where col2 = 10 limit 10; It is possible that there are very rows for which col2 is 10, so not having a MR job may really slow down this query. Solving that would be more difficult without more statistics. But, it may be a good idea to add more config parameters to tune the hive.aggresive.fetch.task.conversion appropriately. It can also be done in a follow-up patch, and is independent of this.
          Hide
          Namit Jain added a comment -

          Can you break it into 2 separate jiras –

          reduce inputs for limits should be an optimization step

          and

          Support non-MR fetching for simple queries with select/limit/filter operations only

          I think, moving the limit stuff to an optimizer step is an excellent idea, but it might be easier to review/checkin independently

          Show
          Namit Jain added a comment - Can you break it into 2 separate jiras – reduce inputs for limits should be an optimization step and Support non-MR fetching for simple queries with select/limit/filter operations only I think, moving the limit stuff to an optimizer step is an excellent idea, but it might be easier to review/checkin independently
          Hide
          Namit Jain added a comment -

          Can you refresh ?

          The patch is not applying cleanly anymore.

          Show
          Namit Jain added a comment - Can you refresh ? The patch is not applying cleanly anymore.
          Hide
          Namit Jain added a comment -

          Sorry to have missed this.
          Will take a look now.

          Show
          Namit Jain added a comment - Sorry to have missed this. Will take a look now.
          Navis made changes -
          Link This issue duplicates HIVE-887 [ HIVE-887 ]
          Navis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.10.0 [ 12320745 ]
          Hide
          Phabricator added a comment -

          navis has commented on the revision "HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only".

          Ok, I'll try in next week.

          REVISION DETAIL
          https://reviews.facebook.net/D2607

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis has commented on the revision " HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only". Ok, I'll try in next week. REVISION DETAIL https://reviews.facebook.net/D2607 To: JIRA, navis Cc: njain
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only".

          Can you create a optimization step ? Most of the other comments are mute in that case,
          since they are very code specific, and this piece of code will be moved from SemanticAnalyzer to
          the optimizer

          REVISION DETAIL
          https://reviews.facebook.net/D2607

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only". Can you create a optimization step ? Most of the other comments are mute in that case, since they are very code specific, and this piece of code will be moved from SemanticAnalyzer to the optimizer REVISION DETAIL https://reviews.facebook.net/D2607 To: JIRA, navis Cc: njain
          Hide
          Phabricator added a comment -

          navis has commented on the revision "HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java:135 Just for the purpose of fetching final objects from operator tree, I thought this is much simpler and clearer than to use FileSinkOperator (via custom Record Writer?).
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6974 It is. But then tryConvertToFetchWork method should be split into two methods(for partitioned and not partitioned) or should have 5 arguments, seemed not good for reading. It can be changed to that if you insist.
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6964 Just for minimizing the difference. Is it better to be moved to qb?
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6976 Just for minimizing the difference, as above.

          And.. I noticed that the method should be renamed to isSelectStarOnly(). Not 'start'.
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7030 Just for minimizing the difference, too. I have many (unpublished) fetches and also have that much of conflicts _.
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7346 I agree. But I'm not sure that the code is valid, yet.

          REVISION DETAIL
          https://reviews.facebook.net/D2607

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis has commented on the revision " HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java:135 Just for the purpose of fetching final objects from operator tree, I thought this is much simpler and clearer than to use FileSinkOperator (via custom Record Writer?). ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6974 It is. But then tryConvertToFetchWork method should be split into two methods(for partitioned and not partitioned) or should have 5 arguments, seemed not good for reading. It can be changed to that if you insist. ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6964 Just for minimizing the difference. Is it better to be moved to qb? ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6976 Just for minimizing the difference, as above. And.. I noticed that the method should be renamed to isSelectStarOnly(). Not 'start'. ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7030 Just for minimizing the difference, too. I have many (unpublished) fetches and also have that much of conflicts _ . ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7346 I agree. But I'm not sure that the code is valid, yet. REVISION DETAIL https://reviews.facebook.net/D2607 To: JIRA, navis Cc: njain
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only".

          INLINE COMMENTS
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6964 Do you want to push the all these checks also in isSimpleSelectQuery ?
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6976 empty where clause should be checked in isSelectStarOnly ?
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6974 No need to initialize work if 1 of the below 2 conds is not true
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7030 I am missing something here: can the code be refactored so that this check happens only
          at the beginning ?

          I know, the code is not clean, but can you do some cleanup here ?
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7346 This is due to legacy bad code - now that you are cleaning it, it would be much simpler
          if all this was moved to a new optimizer step (including select star optimizations).

          That would simplify the code a lot, and also makes it easier to disable this optimization.
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java:135 Do you need this ? Can you use FileSinkOperator ?

          REVISION DETAIL
          https://reviews.facebook.net/D2607

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6964 Do you want to push the all these checks also in isSimpleSelectQuery ? ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6976 empty where clause should be checked in isSelectStarOnly ? ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:6974 No need to initialize work if 1 of the below 2 conds is not true ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7030 I am missing something here: can the code be refactored so that this check happens only at the beginning ? I know, the code is not clean, but can you do some cleanup here ? ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7346 This is due to legacy bad code - now that you are cleaning it, it would be much simpler if all this was moved to a new optimizer step (including select star optimizations). That would simplify the code a lot, and also makes it easier to disable this optimization. ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java:135 Do you need this ? Can you use FileSinkOperator ? REVISION DETAIL https://reviews.facebook.net/D2607 To: JIRA, navis Cc: njain
          Phabricator made changes -
          Attachment HIVE-2925.D2607.4.patch [ 12526704 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only".
          Reviewers: JIRA

          Rebased to trunk.. Phabricator confuses me in theses days.

          REVISION DETAIL
          https://reviews.facebook.net/D2607

          AFFECTED FILES
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
          ql/src/test/queries/clientpositive/nonmr_fetch.q
          ql/src/test/results/clientpositive/nonmr_fetch.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only". Reviewers: JIRA Rebased to trunk.. Phabricator confuses me in theses days. REVISION DETAIL https://reviews.facebook.net/D2607 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java ql/src/test/queries/clientpositive/nonmr_fetch.q ql/src/test/results/clientpositive/nonmr_fetch.q.out To: JIRA, navis Cc: njain
          Phabricator made changes -
          Attachment HIVE-2925.D2607.3.patch [ 12526703 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only".
          Reviewers: JIRA

          1. Fixed test results (passed all tests)

          REVISION DETAIL
          https://reviews.facebook.net/D2607

          AFFECTED FILES
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
          ql/src/test/queries/clientpositive/nonmr_fetch.q
          ql/src/test/results/clientpositive/nonmr_fetch.q.out

          To: JIRA, navis
          Cc: njain

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only". Reviewers: JIRA 1. Fixed test results (passed all tests) REVISION DETAIL https://reviews.facebook.net/D2607 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java ql/src/test/queries/clientpositive/nonmr_fetch.q ql/src/test/results/clientpositive/nonmr_fetch.q.out To: JIRA, navis Cc: njain
          Hide
          Anoop Sam John added a comment -

          Currently when we use the Hbase integration of Hive, querying data from HBase through Hive also will create MR job right? I think this feature can be useful there also. Ideally this might not need MR.

          Show
          Anoop Sam John added a comment - Currently when we use the Hbase integration of Hive, querying data from HBase through Hive also will create MR job right? I think this feature can be useful there also. Ideally this might not need MR.
          Phabricator made changes -
          Attachment HIVE-2925.D2607.2.patch [ 12521432 ]
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only".
          Reviewers: JIRA

          1. Fixed handling of predicates pushed-down to TS
          2. Added configuration for disabling aggresive fetch task conversion
          3. Fixed duplicated adding Partitions to ReadEntity set

          Seemed not yet complete.

          REVISION DETAIL
          https://reviews.facebook.net/D2607

          AFFECTED FILES
          common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
          ql/src/test/queries/clientpositive/nonmr_fetch.q
          ql/src/test/results/clientpositive/nonmr_fetch.q.out

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only". Reviewers: JIRA 1. Fixed handling of predicates pushed-down to TS 2. Added configuration for disabling aggresive fetch task conversion 3. Fixed duplicated adding Partitions to ReadEntity set Seemed not yet complete. REVISION DETAIL https://reviews.facebook.net/D2607 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java ql/src/test/queries/clientpositive/nonmr_fetch.q ql/src/test/results/clientpositive/nonmr_fetch.q.out
          Hide
          Phabricator added a comment -

          njain has commented on the revision "HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only".

          I will review this soon in detail.

          But, one comment I had is that this is a backward incompatible change.
          Can you add a new conf. parameter to control this ?
          The default behavior of this parameter can be true, but we definitely need to provide a
          knob to the end user to turn this off.

          REVISION DETAIL
          https://reviews.facebook.net/D2607

          Show
          Phabricator added a comment - njain has commented on the revision " HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only". I will review this soon in detail. But, one comment I had is that this is a backward incompatible change. Can you add a new conf. parameter to control this ? The default behavior of this parameter can be true, but we definitely need to provide a knob to the end user to turn this off. REVISION DETAIL https://reviews.facebook.net/D2607
          Phabricator made changes -
          Field Original Value New Value
          Attachment HIVE-2925.D2607.1.patch [ 12521275 ]
          Hide
          Phabricator added a comment -

          navis requested code review of "HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only".
          Reviewers: JIRA

          DPAL-1039 Support non-MR fetching for simple queries with select/limit/filter operations only

          It's trivial but frequently asked by end-users. Currently, select queries with simple conditions or limit should run MR job which takes some time especially for big tables, making the people irritated.

          For that kind of simple queries, using fetch task would make them happy.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D2607

          AFFECTED FILES
          ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java
          ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java
          ql/src/test/queries/clientpositive/nonmr_fetch.q
          ql/src/test/results/clientpositive/nonmr_fetch.q.out

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/5943/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - navis requested code review of " HIVE-2925 [jira] Support non-MR fetching for simple queries with select/limit/filter operations only". Reviewers: JIRA DPAL-1039 Support non-MR fetching for simple queries with select/limit/filter operations only It's trivial but frequently asked by end-users. Currently, select queries with simple conditions or limit should run MR job which takes some time especially for big tables, making the people irritated. For that kind of simple queries, using fetch task would make them happy. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2607 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/Driver.java ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/plan/FetchWork.java ql/src/test/queries/clientpositive/nonmr_fetch.q ql/src/test/results/clientpositive/nonmr_fetch.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/5943/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Navis created issue -

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development