Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Diagnosability
    • Labels:
      None

      Description

      In various situations it would have been useful to me to glance at the operator plan before we break it into tasks and apply join, total order sort, etc optimizations.

      I've added this as an options to explain. "Explain logical <QUERY>" will output the full operator tree (not the stage plans, tasks, AST etc).

      Again, I don't think this has to even be documented for users, but might be useful to developers.

      1. HIVE-4812.3.patch
        46 kB
        Gunther Hagleitner
      2. HIVE-4812.2.patch
        46 kB
        Gunther Hagleitner
      3. HIVE-4812.1.patch
        42 kB
        Gunther Hagleitner

        Issue Links

          Activity

          Hide
          Gunther Hagleitner added a comment -

          Another one that's not really Tez specific. This applies to trunk as well as branch.

          Show
          Gunther Hagleitner added a comment - Another one that's not really Tez specific. This applies to trunk as well as branch.
          Hide
          Navis added a comment -

          Is it a logical plan?

          Show
          Navis added a comment - Is it a logical plan?
          Hide
          Gunther Hagleitner added a comment -

          I was going back and forth on this. I don't think it's that clear cut. The plan consists entirely of physical operators and can contain stuff like mapjoin if you're using hints. Not something I'd expect from a logical plan. Reduce sinks also don't easily map to algebraic operations. On the other hand you can't run the plan on MR as it stands.

          I think the name "physical" is appropriate, and I tend to use MR plan for the task plan. But if you feel strongly, I'd be happy to change to logical, or maybe something else - less specific (explain operators <QUERY>?)

          Thoughts?

          Show
          Gunther Hagleitner added a comment - I was going back and forth on this. I don't think it's that clear cut. The plan consists entirely of physical operators and can contain stuff like mapjoin if you're using hints. Not something I'd expect from a logical plan. Reduce sinks also don't easily map to algebraic operations. On the other hand you can't run the plan on MR as it stands. I think the name "physical" is appropriate, and I tend to use MR plan for the task plan. But if you feel strongly, I'd be happy to change to logical, or maybe something else - less specific (explain operators <QUERY>?) Thoughts?
          Hide
          Ashutosh Chauhan added a comment -

          Can you create a phabricator or RB entry for it, that helps in review.

          Show
          Ashutosh Chauhan added a comment - Can you create a phabricator or RB entry for it, that helps in review.
          Hide
          Gunther Hagleitner added a comment -

          Absolutely, review is here: https://reviews.facebook.net/D11589

          Show
          Gunther Hagleitner added a comment - Absolutely, review is here: https://reviews.facebook.net/D11589
          Hide
          Gunther Hagleitner added a comment -

          Patch .2 makes the explain methods non-static (unit tests showed problem with 'sticky' settings). It also only skips seen operators in the physical explain not in the regular one.

          Show
          Gunther Hagleitner added a comment - Patch .2 makes the explain methods non-static (unit tests showed problem with 'sticky' settings). It also only skips seen operators in the physical explain not in the regular one.
          Hide
          Navis added a comment -

          Hive uses term 'Logical Optimizer' which modifies operator tree and 'Physical Optimizer' which modifies tasks. I'm a little worry about more confusion would be introduced by new term 'physical operators'.

          Show
          Navis added a comment - Hive uses term 'Logical Optimizer' which modifies operator tree and 'Physical Optimizer' which modifies tasks. I'm a little worry about more confusion would be introduced by new term 'physical operators'.
          Hide
          Gunther Hagleitner added a comment -

          .3 renames it to logical. Navis: Does it look ok now?

          Show
          Gunther Hagleitner added a comment - .3 renames it to logical. Navis : Does it look ok now?
          Hide
          Gunther Hagleitner added a comment -

          Added dependency on HIVE-4811. Code is independent, but will only apply cleanly on top of it.

          Show
          Gunther Hagleitner added a comment - Added dependency on HIVE-4811 . Code is independent, but will only apply cleanly on top of it.
          Hide
          Navis added a comment -

          LGTM except some lint warnings. Could I fix that and run test?

          Show
          Navis added a comment - LGTM except some lint warnings. Could I fix that and run test?
          Hide
          Gunther Hagleitner added a comment -

          Navis: I'd be greatful if you did!

          Show
          Gunther Hagleitner added a comment - Navis : I'd be greatful if you did!
          Hide
          Navis added a comment -

          Good. Running test.

          Show
          Navis added a comment - Good. Running test.
          Hide
          Navis added a comment -

          Committed to trunk. Thanks Gunther!

          Show
          Navis added a comment - Committed to trunk. Thanks Gunther!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop1-ptest #48 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/48/)
          HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) (Revision 1501036)

          Result = SUCCESS
          navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501036
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
          • /hive/trunk/ql/src/test/queries/clientpositive/explain_logical.q
          • /hive/trunk/ql/src/test/results/clientpositive/explain_logical.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop1-ptest #48 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/48/ ) HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) (Revision 1501036) Result = SUCCESS navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501036 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java /hive/trunk/ql/src/test/queries/clientpositive/explain_logical.q /hive/trunk/ql/src/test/results/clientpositive/explain_logical.q.out
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #2188 (See https://builds.apache.org/job/Hive-trunk-h0.21/2188/)
          HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) (Revision 1501036)

          Result = ABORTED
          navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501036
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
          • /hive/trunk/ql/src/test/queries/clientpositive/explain_logical.q
          • /hive/trunk/ql/src/test/results/clientpositive/explain_logical.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #2188 (See https://builds.apache.org/job/Hive-trunk-h0.21/2188/ ) HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) (Revision 1501036) Result = ABORTED navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501036 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java /hive/trunk/ql/src/test/queries/clientpositive/explain_logical.q /hive/trunk/ql/src/test/results/clientpositive/explain_logical.q.out
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #280 (See https://builds.apache.org/job/Hive-trunk-hadoop2/280/)
          HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) (Revision 1501036)

          Result = ABORTED
          navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501036
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
          • /hive/trunk/ql/src/test/queries/clientpositive/explain_logical.q
          • /hive/trunk/ql/src/test/results/clientpositive/explain_logical.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #280 (See https://builds.apache.org/job/Hive-trunk-hadoop2/280/ ) HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) (Revision 1501036) Result = ABORTED navis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501036 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java /hive/trunk/ql/src/test/queries/clientpositive/explain_logical.q /hive/trunk/ql/src/test/results/clientpositive/explain_logical.q.out
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2-ptest #14 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/14/)
          HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) (navis: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501036)

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
          • /hive/trunk/ql/src/test/queries/clientpositive/explain_logical.q
          • /hive/trunk/ql/src/test/results/clientpositive/explain_logical.q.out
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #14 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/14/ ) HIVE-4812 : Logical explain plan (Gunther Hagleitner V via Navis) (navis: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501036 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Context.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java /hive/trunk/ql/src/test/queries/clientpositive/explain_logical.q /hive/trunk/ql/src/test/results/clientpositive/explain_logical.q.out
          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

            People

            • Assignee:
              Gunther Hagleitner
              Reporter:
              Gunther Hagleitner
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development