Hive
  1. Hive
  2. HIVE-3610

Add a command "Explain dependency ..."

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0
    • Component/s: Query Processor
    • Labels:
      None
    • Release Note:
      Hide
      This introduces a new command

      EXPLAIN DEPENDENCY <query>

      which outputs a JSON object containing the input tables and partitions the query will access without actually running the query.
      Show
      This introduces a new command EXPLAIN DEPENDENCY <query> which outputs a JSON object containing the input tables and partitions the query will access without actually running the query.

      Description

      Add a new command "EXPLAIN DEPENDENCY".
      Any query can be passed to EXPLAIN DEPENDENCY as with EXPLAIN (FORMATTED/EXTENDED). The output of this command will be JSON that provides the list of tables and partitions that the query depends on.

      One possible use case is to determine the set of tables/views that are used by a view, and the set of partitions that are used by a given query on that view. This will allow a view to be replicated from one Hive instance to another, since we can determine the set of objects that need to be replicated for replication of the view to be successful.

      Example output:
      {"input_tables":[

      {"tablename": "default@test_sambavi_v2", "tabletype": "EXTERNAL_TABLE"}

      ,

      {"tablename": "default@test_sambavi_v1", "tabletype": "TABLE"}

      ], "input partitions":["default@srcpart@ds=2008-04-08/hr=11","default@srcpart@ds=2008-04-08/hr=12"]}

      1. explain_dependency.2.patch
        15 kB
        Sambavi Muthukrishnan
      2. explain_dependency.1.patch
        15 kB
        Sambavi Muthukrishnan

        Activity

        Hide
        Sambavi Muthukrishnan added a comment -
        Show
        Sambavi Muthukrishnan added a comment - Diff is at https://reviews.facebook.net/D6159
        Hide
        Kevin Wilfong added a comment -

        A minor comment on the diff, otherwise looks good.

        Show
        Kevin Wilfong added a comment - A minor comment on the diff, otherwise looks good.
        Hide
        Sambavi Muthukrishnan added a comment -

        Fixed for comments, added new patch.

        Show
        Sambavi Muthukrishnan added a comment - Fixed for comments, added new patch.
        Hide
        Kevin Wilfong added a comment -

        +1 Thanks Sambavi

        Show
        Kevin Wilfong added a comment - +1 Thanks Sambavi
        Hide
        Namit Jain added a comment -

        Why do you need this ?
        Isn't the same information available through hooks today, or am I missing something ?

        Show
        Namit Jain added a comment - Why do you need this ? Isn't the same information available through hooks today, or am I missing something ?
        Hide
        Sambavi Muthukrishnan added a comment -

        Namit: the set of inputs should be available through hooks. But this seems like a generically useful feature - to be able to determine the dependencies for a given query, which are the sets of inputs that are processed, hence decided to make it a command that is available in HIVE.

        Show
        Sambavi Muthukrishnan added a comment - Namit: the set of inputs should be available through hooks. But this seems like a generically useful feature - to be able to determine the dependencies for a given query, which are the sets of inputs that are processed, hence decided to make it a command that is available in HIVE.
        Hide
        Namit Jain added a comment -

        OK, I dont have any objection.

        Kevin, feel free to commit this.

        Show
        Namit Jain added a comment - OK, I dont have any objection. Kevin, feel free to commit this.
        Hide
        Kevin Wilfong added a comment -

        Committed, thanks Sambavi.

        Show
        Kevin Wilfong added a comment - Committed, thanks Sambavi.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1764 (See https://builds.apache.org/job/Hive-trunk-h0.21/1764/)
        HIVE-3610. Add a command "Explain dependency ..." (Sambavi Muthukrishnan via kevinwilfong) (Revision 1403467)

        Result = FAILURE
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403467
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
        • /hive/trunk/ql/src/test/queries/clientpositive/explain_dependency.q
        • /hive/trunk/ql/src/test/results/clientpositive/explain_dependency.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1764 (See https://builds.apache.org/job/Hive-trunk-h0.21/1764/ ) HIVE-3610 . Add a command "Explain dependency ..." (Sambavi Muthukrishnan via kevinwilfong) (Revision 1403467) Result = FAILURE kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403467 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java /hive/trunk/ql/src/test/queries/clientpositive/explain_dependency.q /hive/trunk/ql/src/test/results/clientpositive/explain_dependency.q.out
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3610. Add a command "Explain dependency ..." (Sambavi Muthukrishnan via kevinwilfong) (Revision 1403467)

        Result = ABORTED
        kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403467
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java
        • /hive/trunk/ql/src/test/queries/clientpositive/explain_dependency.q
        • /hive/trunk/ql/src/test/results/clientpositive/explain_dependency.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3610 . Add a command "Explain dependency ..." (Sambavi Muthukrishnan via kevinwilfong) (Revision 1403467) Result = ABORTED kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403467 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ExplainWork.java /hive/trunk/ql/src/test/queries/clientpositive/explain_dependency.q /hive/trunk/ql/src/test/results/clientpositive/explain_dependency.q.out
        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          People

          • Assignee:
            Sambavi Muthukrishnan
            Reporter:
            Sambavi Muthukrishnan
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development