Hadoop Common
  1. Hadoop Common
  2. HADOOP-4084

Add explain plan capabilities to Hive QL

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Introduced "EXPLAIN" plan for Hive.

      Description

      Adding explain plan for queries in hive.

      The current proposal is to support something like:

      EXPLAIN [EXTENDED]
      SELECT ....

      This will output the following:

      Abstract Syntax Tree:

      Number of Stages:

      Dependencies between Stages:

      Plan for each stage:

      If EXTENDED keyword is used then much more information will be emitted where as without that keyword only logical information will be emitted.

      e.g. In case of a group by query

      EXPLAIN
      SELECT T.c1, count(1) FROM T GROUP BY T.c1;

      The explain plan itself has two stages

      Stage1 and Stage2

      Stage1 will have the plan for generating the partial aggregates
      and Stage2 will have the plan for generating the complete aggregates.

      I also plan to convert the parse and semantic analysis tests so that they use this for finding differences in the plan instead of the programmatic plan dumps that we are using today (tests/queries/positive).

      1. patch-4084
        464 kB
        Ashish Thusoo
      2. patch-4084
        464 kB
        Ashish Thusoo
      3. patch-4084_v3
        544 kB
        Ashish Thusoo

        Activity

        Hide
        Ashish Thusoo added a comment -

        Uploaded patch for support to explain plan and some other minor fixes and cleanups.

        A note on the implementation of explain plan. This is implemented through a java annotation @explain (implemented in explain.java)

        This annotation can take two optional arguments

        @explain(displayName="name", normalExplain=false)

        by default displayName is "" and normalExplain is true

        displayName is the string used to display this class/return value of a method in the plan. normalExplain=false means that this class/return value of a method should only be displayed in case of an extended plan.

        Additionally there is an ExplainTask that does the actual explain and contains the explainWork that contains the AST string and the rootTasks that need to be explained.

        The general format of explain is

        PARSE TREE

        STAGE DEPENDENCIES

        STAGE PLAN:
        Plan for Stage 1

        Plan for Stage 2
        .
        .
        .

        Within the plan the parent child relationship is shown through indentation, the names of operators are displayed as specified in the @explain notation (if the displayName is "" then the name is not displayed in the plan).
        Additional each of the public functions of the class that are annotated with @explain is called and the explain is recursively called on non primitive and string values (if they are maps, lists or other classes). For primitive and string values
        we just print the value.

        In future I can make this xml instead of text blobs that I am doing now.

        The minor fixes/refactors include:
        1. -Doverwrite=true option for running tests with the purpose of capturing results, so now if you have to capture the results of TestCliDriver you can run
        ant -Dtestcase=TestCliDriver -Doverwrite=true clean-test test

        same is true for TestParse and TestParseNegative
        Additionally for all these tests you can run a specific query by using -Dqfile e.g.

        ant -Dtestcase=TestParse -Dqfile=input1.q -Doverwrite=true clean-test test

        would run the parse test on input1.q and capture its output in the source tree.

        2. Fixed some warnings related to generics

        3. Changed the location of velocity.log to be in the build directory for TestCliDriver (this was already in the build location for TestParse and TestParseNegative)

        4. Unified the function registries for UDF and UDAF and introduced the notion of displayName in FunctionInfo and FunctionRegistry which is used to show the function in the plan.

        Show
        Ashish Thusoo added a comment - Uploaded patch for support to explain plan and some other minor fixes and cleanups. A note on the implementation of explain plan. This is implemented through a java annotation @explain (implemented in explain.java) This annotation can take two optional arguments @explain(displayName="name", normalExplain=false) by default displayName is "" and normalExplain is true displayName is the string used to display this class/return value of a method in the plan. normalExplain=false means that this class/return value of a method should only be displayed in case of an extended plan. Additionally there is an ExplainTask that does the actual explain and contains the explainWork that contains the AST string and the rootTasks that need to be explained. The general format of explain is PARSE TREE STAGE DEPENDENCIES STAGE PLAN: Plan for Stage 1 Plan for Stage 2 . . . Within the plan the parent child relationship is shown through indentation, the names of operators are displayed as specified in the @explain notation (if the displayName is "" then the name is not displayed in the plan). Additional each of the public functions of the class that are annotated with @explain is called and the explain is recursively called on non primitive and string values (if they are maps, lists or other classes). For primitive and string values we just print the value. In future I can make this xml instead of text blobs that I am doing now. The minor fixes/refactors include: 1. -Doverwrite=true option for running tests with the purpose of capturing results, so now if you have to capture the results of TestCliDriver you can run ant -Dtestcase=TestCliDriver -Doverwrite=true clean-test test same is true for TestParse and TestParseNegative Additionally for all these tests you can run a specific query by using -Dqfile e.g. ant -Dtestcase=TestParse -Dqfile=input1.q -Doverwrite=true clean-test test would run the parse test on input1.q and capture its output in the source tree. 2. Fixed some warnings related to generics 3. Changed the location of velocity.log to be in the build directory for TestCliDriver (this was already in the build location for TestParse and TestParseNegative) 4. Unified the function registries for UDF and UDAF and introduced the notion of displayName in FunctionInfo and FunctionRegistry which is used to show the function in the plan.
        Hide
        Namit Jain added a comment -

        Overall, looks good - just some minor comments.

        LoadSemanticAnalyzer: line 32/33 - nitpick: remove commented import
        ExplainSemanticAnalyzer (line 47): not needed
        explain.java: no apache license header on top of file
        DDLWork.java: why display name only for create table ? why not for other DDLs
        MapRedTask: (line 91) remove explain()
        MoveTask: remove explain()

        Show
        Namit Jain added a comment - Overall, looks good - just some minor comments. LoadSemanticAnalyzer: line 32/33 - nitpick: remove commented import ExplainSemanticAnalyzer (line 47): not needed explain.java: no apache license header on top of file DDLWork.java: why display name only for create table ? why not for other DDLs MapRedTask: (line 91) remove explain() MoveTask: remove explain()
        Hide
        Ashish Thusoo added a comment -

        Incorporated Namit's comments.

        Show
        Ashish Thusoo added a comment - Incorporated Namit's comments.
        Hide
        Ashish Thusoo added a comment -

        Hey namit...

        I did not realize that you have not given a +1 on this yet. Can you take a look?

        Thanks,
        Ashish

        Show
        Ashish Thusoo added a comment - Hey namit... I did not realize that you have not given a +1 on this yet. Can you take a look? Thanks, Ashish
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12390212/patch-4084
        against trunk revision 696427.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 750 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to cause Findbugs to fail.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3287/testReport/
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3287/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3287/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390212/patch-4084 against trunk revision 696427. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 750 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs to fail. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3287/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3287/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3287/console This message is automatically generated.
        Hide
        Ashish Thusoo added a comment -

        I don;t know why this is not compiling. I checked the submitted patch and the references to UDAFRegistry have actually been removed from the sources. Is this a merge problem?

        The part of the patch file that does this is as follows:

        Index: src/contrib/hive/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
        ===================================================================
        — src/contrib/hive/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (revision 695984)
        +++ src/contrib/hive/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (working copy)
        @@ -134,7 +134,7 @@
        assert (expressionTree.getChildCount() != 0);
        assert (expressionTree.getChild(0).getType() == HiveParser.Identifier);
        String functionName = expressionTree.getChild(0).getText();

        • if (UDAFRegistry.getUDAF(functionName) != null) {
          + if (FunctionRegistry.getUDAF(functionName) != null) { aggregations.put(expressionTree.toStringTree(), expressionTree); return; }

          @@ -987,7 +987,7 @@
          for (Map.Entry<String, CommonTree> entry : aggregationTrees.entrySet())

          { CommonTree value = entry.getValue(); String aggName = value.getChild(0).getText(); - Class<? extends UDAF> aggClass = UDAFRegistry.getUDAF(aggName); + Class<? extends UDAF> aggClass = FunctionRegistry.getUDAF(aggName); assert (aggClass != null); ArrayList<exprNodeDesc> aggParameters = new ArrayList<exprNodeDesc>(); ArrayList<Class<?>> aggClasses = new ArrayList<Class<?>>(); @@ -1006,7 +1006,7 @@ aggClasses.add(paraExprInfo.getType().getPrimitiveClass()); }
        • if (null == UDAFRegistry.getUDAFMethod(aggName, aggClasses)) {
          + if (null == FunctionRegistry.getUDAFMethod(aggName, aggClasses)) {
          String reason = "Looking for UDAF \"" + aggName + "\" with parame
        Show
        Ashish Thusoo added a comment - I don;t know why this is not compiling. I checked the submitted patch and the references to UDAFRegistry have actually been removed from the sources. Is this a merge problem? The part of the patch file that does this is as follows: Index: src/contrib/hive/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java =================================================================== — src/contrib/hive/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (revision 695984) +++ src/contrib/hive/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (working copy) @@ -134,7 +134,7 @@ assert (expressionTree.getChildCount() != 0); assert (expressionTree.getChild(0).getType() == HiveParser.Identifier); String functionName = expressionTree.getChild(0).getText(); if (UDAFRegistry.getUDAF(functionName) != null) { + if (FunctionRegistry.getUDAF(functionName) != null) { aggregations.put(expressionTree.toStringTree(), expressionTree); return; } @@ -987,7 +987,7 @@ for (Map.Entry<String, CommonTree> entry : aggregationTrees.entrySet()) { CommonTree value = entry.getValue(); String aggName = value.getChild(0).getText(); - Class<? extends UDAF> aggClass = UDAFRegistry.getUDAF(aggName); + Class<? extends UDAF> aggClass = FunctionRegistry.getUDAF(aggName); assert (aggClass != null); ArrayList<exprNodeDesc> aggParameters = new ArrayList<exprNodeDesc>(); ArrayList<Class<?>> aggClasses = new ArrayList<Class<?>>(); @@ -1006,7 +1006,7 @@ aggClasses.add(paraExprInfo.getType().getPrimitiveClass()); } if (null == UDAFRegistry.getUDAFMethod(aggName, aggClasses)) { + if (null == FunctionRegistry.getUDAFMethod(aggName, aggClasses)) { String reason = "Looking for UDAF \"" + aggName + "\" with parame
        Hide
        Ashish Thusoo added a comment -

        looked at this with Dhrubha...

        seems like hadoopQA is not able to handle deleted files. UDAFRegistry.java and UDFRegistry.java have been deleted. But the patch command converts the deletes into empty files and does not actually remove the files, hence the failure. Is there a more permanent solution to this so that the patch command is able to delete files. Maybe we should upload the svn stat output as well to help in this.

        Thoughts?

        Show
        Ashish Thusoo added a comment - looked at this with Dhrubha... seems like hadoopQA is not able to handle deleted files. UDAFRegistry.java and UDFRegistry.java have been deleted. But the patch command converts the deletes into empty files and does not actually remove the files, hence the failure. Is there a more permanent solution to this so that the patch command is able to delete files. Maybe we should upload the svn stat output as well to help in this. Thoughts?
        Hide
        Namit Jain added a comment -

        +1

        Show
        Namit Jain added a comment - +1
        Hide
        Ashish Thusoo added a comment -

        Actually I found the problem.

        This patch conflicted with a checkin that was made by namit so that UDAFRegistry reference was added into SemanticAnalyzer and my patch which was generated from a previous version of the tree did not have those changes. Will fix and submit an new patch.

        Show
        Ashish Thusoo added a comment - Actually I found the problem. This patch conflicted with a checkin that was made by namit so that UDAFRegistry reference was added into SemanticAnalyzer and my patch which was generated from a previous version of the tree did not have those changes. Will fix and submit an new patch.
        Hide
        Ashish Thusoo added a comment -

        New patch after resolving merge conflicts.

        Show
        Ashish Thusoo added a comment - New patch after resolving merge conflicts.
        Hide
        Ashish Thusoo added a comment -

        the previous patch had merge conflicts as it was generated from a previous version of trunk and trunk has moved since...

        Show
        Ashish Thusoo added a comment - the previous patch had merge conflicts as it was generated from a previous version of trunk and trunk has moved since...
        Hide
        Ashish Thusoo added a comment -

        New patch with resolved merge conflicts.

        Show
        Ashish Thusoo added a comment - New patch with resolved merge conflicts.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12390313/patch-4084_v3
        against trunk revision 696551.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 750 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3297/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3297/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3297/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3297/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12390313/patch-4084_v3 against trunk revision 696551. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 750 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3297/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3297/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3297/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3297/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        I just committed this. Thanks Ashish!

        Show
        dhruba borthakur added a comment - I just committed this. Thanks Ashish!
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #611 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/ )

          People

          • Assignee:
            Ashish Thusoo
            Reporter:
            Ashish Thusoo
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development