Hive
  1. Hive
  2. HIVE-2385

Local Mode can be more aggressive if LIMIT optimization is on

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.

      1. HIVE-2385.1.patch
        6 kB
        Siying Dong
      2. HIVE-2385.2.patch
        6 kB
        Siying Dong

        Activity

        Hide
        Siying Dong added a comment -

        Further estimate input for LIMIT when deciding local mode. Also fix a bug (won't cause wrong result) of the LIMIT optimization.

        Show
        Siying Dong added a comment - Further estimate input for LIMIT when deciding local mode. Also fix a bug (won't cause wrong result) of the LIMIT optimization.
        Hide
        Siying Dong added a comment -

        I don't know why but I can't create review board using this patch.

        Show
        Siying Dong added a comment - I don't know why but I can't create review board using this patch.
        Hide
        Carl Steinbach added a comment -

        @Siying: I get a test failure in TestNegativeCliDriver.autlocal1. Can you please take a look? Thanks.

        Show
        Carl Steinbach added a comment - @Siying: I get a test failure in TestNegativeCliDriver.autlocal1. Can you please take a look? Thanks.
        Hide
        Siying Dong added a comment -

        Fix the bug and it passes autolocal1.q. I'm running the whole test suites now.

        Show
        Siying Dong added a comment - Fix the bug and it passes autolocal1.q. I'm running the whole test suites now.
        Hide
        Siying Dong added a comment -

        It passed all the tests.

        Show
        Siying Dong added a comment - It passed all the tests.
        Hide
        Siying Dong added a comment -

        @Carl, are you still seeing tests failing?

        Show
        Siying Dong added a comment - @Carl, are you still seeing tests failing?
        Hide
        Carl Steinbach added a comment -

        @Siying: autolocal1 now passes. +1. Will commit if full tests pass.

        Show
        Carl Steinbach added a comment - @Siying: autolocal1 now passes. +1. Will commit if full tests pass.
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Siying!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Siying!
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #918 (See https://builds.apache.org/job/Hive-trunk-h0.21/918/)
        HIVE-2385. Local Mode can be more aggressive if LIMIT optimization is on (Siying Dong via cws)

        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162202
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #918 (See https://builds.apache.org/job/Hive-trunk-h0.21/918/ ) HIVE-2385 . Local Mode can be more aggressive if LIMIT optimization is on (Siying Dong via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162202 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

          People

          • Assignee:
            Siying Dong
            Reporter:
            Siying Dong
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development