Hive
  1. Hive
  2. HIVE-2727

add a testcase for partitioned view on union and base tables have index

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. ASF.LICENSE.NOT.GRANTED--HIVE-2727.D1323.1.patch
      210 kB
      Phabricator
    2. ASF.LICENSE.NOT.GRANTED--HIVE-2727.D1323.2.patch
      256 kB
      Phabricator
    3. ASF.LICENSE.NOT.GRANTED--HIVE-2727.D1323.3.patch
      254 kB
      Phabricator
    4. ASF.LICENSE.NOT.GRANTED--HIVE-2727.D1323.4.patch
      89 kB
      Phabricator
    5. HIVE-2727.1.patch
      210 kB
      He Yongqiang

      Activity

      Show
      He Yongqiang added a comment - https://reviews.facebook.net/D1323
      Hide
      Phabricator added a comment -

      heyongqiang requested code review of "HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index".
      Reviewers: JIRA

      https://issues.apache.org/jira/browse/HIVE-2727

      TEST PLAN
      EMPTY

      REVISION DETAIL
      https://reviews.facebook.net/D1323

      AFFECTED FILES
      ql/src/test/results/clientpositive/union_view.q.out
      ql/src/test/queries/clientpositive/union_view.q
      ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java
      ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java
      ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java

      MANAGE HERALD DIFFERENTIAL RULES
      https://reviews.facebook.net/herald/view/differential/

      WHY DID I GET THIS EMAIL?
      https://reviews.facebook.net/herald/transcript/2751/

      Tip: use the X-Herald-Rules header to filter Herald messages in your client.

      Show
      Phabricator added a comment - heyongqiang requested code review of " HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2727 TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1323 AFFECTED FILES ql/src/test/results/clientpositive/union_view.q.out ql/src/test/queries/clientpositive/union_view.q ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/2751/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
      Hide
      Phabricator added a comment -

      njain has commented on the revision "HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index".

      INLINE COMMENTS
      ql/src/test/queries/clientpositive/union_view.q:27 The code changes look OK.

      But, can you change the tests ?

      Create 2 partitions in each base table.
      Change the explain to Explain Extended for all the queries, to verify that we are only looking at the relevant partitions of the index.

      Can you unset all the hooks for this (printhook, lineagehook etc.)
      Also remove the merge (both map-only and map-reduce only).

      It will make the test output much smaller, and easier to validate

      REVISION DETAIL
      https://reviews.facebook.net/D1323

      Show
      Phabricator added a comment - njain has commented on the revision " HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index". INLINE COMMENTS ql/src/test/queries/clientpositive/union_view.q:27 The code changes look OK. But, can you change the tests ? Create 2 partitions in each base table. Change the explain to Explain Extended for all the queries, to verify that we are only looking at the relevant partitions of the index. Can you unset all the hooks for this (printhook, lineagehook etc.) Also remove the merge (both map-only and map-reduce only). It will make the test output much smaller, and easier to validate REVISION DETAIL https://reviews.facebook.net/D1323
      Hide
      Phabricator added a comment -

      heyongqiang updated the revision "HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index".
      Reviewers: JIRA

      refresh to address namit's comments.

      REVISION DETAIL
      https://reviews.facebook.net/D1323

      AFFECTED FILES
      ql/src/test/results/clientpositive/union_view.q.out
      ql/src/test/queries/clientpositive/union_view.q
      ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java
      ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java
      ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java

      Show
      Phabricator added a comment - heyongqiang updated the revision " HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index". Reviewers: JIRA refresh to address namit's comments. REVISION DETAIL https://reviews.facebook.net/D1323 AFFECTED FILES ql/src/test/results/clientpositive/union_view.q.out ql/src/test/queries/clientpositive/union_view.q ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
      Hide
      Phabricator added a comment -

      njain has commented on the revision "HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index".

      INLINE COMMENTS
      ql/src/test/results/clientpositive/union_view.q.out:2126 I am not sure - is this correct.

      Why are we doing a index scan for this table ?

      I mean src_union_3.

      We are only looking for index for src_union_1, which is correct.

      Let us talk offline

      REVISION DETAIL
      https://reviews.facebook.net/D1323

      Show
      Phabricator added a comment - njain has commented on the revision " HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index". INLINE COMMENTS ql/src/test/results/clientpositive/union_view.q.out:2126 I am not sure - is this correct. Why are we doing a index scan for this table ? I mean src_union_3. We are only looking for index for src_union_1, which is correct. Let us talk offline REVISION DETAIL https://reviews.facebook.net/D1323
      Hide
      Phabricator added a comment -

      njain has commented on the revision "HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index".

      INLINE COMMENTS
      ql/src/test/results/clientpositive/union_view.q.out:308 This needs to be masked

      or ignored from the diff comparison.

      This will always fail

      Can you check for all lastModifiedTime occurences ?
      ql/src/test/queries/clientpositive/union_view.q:34 needs a order by to make it deterministic
      ql/src/test/queries/clientpositive/union_view.q:35 same as above
      ql/src/test/queries/clientpositive/union_view.q:38 same as above (line 39 also)
      ql/src/test/queries/clientpositive/union_view.q:59 make this and next one ordered
      ql/src/test/queries/clientpositive/union_view.q:84 basically, there are multiple partitions for ds=2 and ds=3.

      Can you make this deterministic ?

      REVISION DETAIL
      https://reviews.facebook.net/D1323

      Show
      Phabricator added a comment - njain has commented on the revision " HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index". INLINE COMMENTS ql/src/test/results/clientpositive/union_view.q.out:308 This needs to be masked or ignored from the diff comparison. This will always fail Can you check for all lastModifiedTime occurences ? ql/src/test/queries/clientpositive/union_view.q:34 needs a order by to make it deterministic ql/src/test/queries/clientpositive/union_view.q:35 same as above ql/src/test/queries/clientpositive/union_view.q:38 same as above (line 39 also) ql/src/test/queries/clientpositive/union_view.q:59 make this and next one ordered ql/src/test/queries/clientpositive/union_view.q:84 basically, there are multiple partitions for ds=2 and ds=3. Can you make this deterministic ? REVISION DETAIL https://reviews.facebook.net/D1323
      Hide
      Phabricator added a comment -

      heyongqiang updated the revision "HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index".
      Reviewers: JIRA

      address namit's comments

      REVISION DETAIL
      https://reviews.facebook.net/D1323

      AFFECTED FILES
      ql/src/test/results/clientpositive/union_view.q.out
      ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
      ql/src/test/queries/clientpositive/union_view.q
      ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java
      ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java
      ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java

      Show
      Phabricator added a comment - heyongqiang updated the revision " HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index". Reviewers: JIRA address namit's comments REVISION DETAIL https://reviews.facebook.net/D1323 AFFECTED FILES ql/src/test/results/clientpositive/union_view.q.out ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ql/src/test/queries/clientpositive/union_view.q ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
      Hide
      Phabricator added a comment -

      heyongqiang updated the revision "HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index".
      Reviewers: JIRA

      replace explain extended with explain

      REVISION DETAIL
      https://reviews.facebook.net/D1323

      AFFECTED FILES
      ql/src/test/results/clientpositive/union_view.q.out
      ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
      ql/src/test/queries/clientpositive/union_view.q
      ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java
      ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java
      ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java

      Show
      Phabricator added a comment - heyongqiang updated the revision " HIVE-2727 [jira] add a testcase for partitioned view on union and base tables have index". Reviewers: JIRA replace explain extended with explain REVISION DETAIL https://reviews.facebook.net/D1323 AFFECTED FILES ql/src/test/results/clientpositive/union_view.q.out ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java ql/src/test/queries/clientpositive/union_view.q ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
      Hide
      Namit Jain added a comment -

      Committed. Thanks Yongqiang

      Show
      Namit Jain added a comment - Committed. Thanks Yongqiang
      Hide
      Hudson added a comment -

      Integrated in Hive-trunk-h0.21 #1220 (See https://builds.apache.org/job/Hive-trunk-h0.21/1220/)
      HIVE-2727 add a testcase for partitioned view on union and base tables have
      index (He Yongqiang via namit)

      namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235864
      Files :

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java
      • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
      • /hive/trunk/ql/src/test/queries/clientpositive/union_view.q
      • /hive/trunk/ql/src/test/results/clientpositive/union_view.q.out
      Show
      Hudson added a comment - Integrated in Hive-trunk-h0.21 #1220 (See https://builds.apache.org/job/Hive-trunk-h0.21/1220/ ) HIVE-2727 add a testcase for partitioned view on union and base tables have index (He Yongqiang via namit) namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235864 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java /hive/trunk/ql/src/test/queries/clientpositive/union_view.q /hive/trunk/ql/src/test/results/clientpositive/union_view.q.out
      Hide
      Ashutosh Chauhan added a comment -

      This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

      Show
      Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
      Hide
      Hudson added a comment -

      Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
      HIVE-2727 add a testcase for partitioned view on union and base tables have
      index (He Yongqiang via namit) (Revision 1235864)

      Result = ABORTED
      namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235864
      Files :

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java
      • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
      • /hive/trunk/ql/src/test/queries/clientpositive/union_view.q
      • /hive/trunk/ql/src/test/results/clientpositive/union_view.q.out
      Show
      Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2727 add a testcase for partitioned view on union and base tables have index (He Yongqiang via namit) (Revision 1235864) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235864 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/IndexMetadataChangeTask.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java /hive/trunk/ql/src/test/queries/clientpositive/union_view.q /hive/trunk/ql/src/test/results/clientpositive/union_view.q.out

        People

        • Assignee:
          He Yongqiang
          Reporter:
          He Yongqiang
        • Votes:
          0 Vote for this issue
          Watchers:
          1 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development