Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-2385

Local Mode can be more aggressive if LIMIT optimization is on

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Local mode now depends on total input data, but for LIMIT queries with no filtering, the data actually scanned can be much less and it's relatively predictable. We can place local mode more aggressively.

      1. HIVE-2385.2.patch
        6 kB
        Siying Dong
      2. HIVE-2385.1.patch
        6 kB
        Siying Dong

        Activity

        Hide
        sdong Siying Dong added a comment -

        Further estimate input for LIMIT when deciding local mode. Also fix a bug (won't cause wrong result) of the LIMIT optimization.

        Show
        sdong Siying Dong added a comment - Further estimate input for LIMIT when deciding local mode. Also fix a bug (won't cause wrong result) of the LIMIT optimization.
        Hide
        sdong Siying Dong added a comment -

        I don't know why but I can't create review board using this patch.

        Show
        sdong Siying Dong added a comment - I don't know why but I can't create review board using this patch.
        Hide
        cwsteinbach Carl Steinbach added a comment -

        @Siying: I get a test failure in TestNegativeCliDriver.autlocal1. Can you please take a look? Thanks.

        Show
        cwsteinbach Carl Steinbach added a comment - @Siying: I get a test failure in TestNegativeCliDriver.autlocal1. Can you please take a look? Thanks.
        Hide
        sdong Siying Dong added a comment -

        Fix the bug and it passes autolocal1.q. I'm running the whole test suites now.

        Show
        sdong Siying Dong added a comment - Fix the bug and it passes autolocal1.q. I'm running the whole test suites now.
        Hide
        sdong Siying Dong added a comment -

        It passed all the tests.

        Show
        sdong Siying Dong added a comment - It passed all the tests.
        Hide
        sdong Siying Dong added a comment -

        @Carl, are you still seeing tests failing?

        Show
        sdong Siying Dong added a comment - @Carl, are you still seeing tests failing?
        Hide
        cwsteinbach Carl Steinbach added a comment -

        @Siying: autolocal1 now passes. +1. Will commit if full tests pass.

        Show
        cwsteinbach Carl Steinbach added a comment - @Siying: autolocal1 now passes. +1. Will commit if full tests pass.
        Hide
        cwsteinbach Carl Steinbach added a comment -

        Committed to trunk. Thanks Siying!

        Show
        cwsteinbach Carl Steinbach added a comment - Committed to trunk. Thanks Siying!
        Hide
        hudson Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #918 (See https://builds.apache.org/job/Hive-trunk-h0.21/918/)
        HIVE-2385. Local Mode can be more aggressive if LIMIT optimization is on (Siying Dong via cws)

        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162202
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        Show
        hudson Hudson added a comment - Integrated in Hive-trunk-h0.21 #918 (See https://builds.apache.org/job/Hive-trunk-h0.21/918/ ) HIVE-2385 . Local Mode can be more aggressive if LIMIT optimization is on (Siying Dong via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1162202 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java

          People

          • Assignee:
            sdong Siying Dong
            Reporter:
            sdong Siying Dong
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development