Hive
  1. Hive
  2. HIVE-2086

Add test coverage for external table data loss issue

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: Metastore
    • Labels:
      None
    • Environment:

      Amazon elastics mapreduce cluster

    • Hadoop Flags:
      Reviewed
    • Tags:
      data loss external table create like

      Description

      Data loss when using "create external table like" statement.

      1) Set up an external table S, point to location L. Populate data in S.
      2) Create another external table T, using statement like this:
      create external table T like S location L
      Make sure table T point to the same location as the original table S.
      3) Query table T, see the same set of data in S.
      4) drop table T.
      5) Query table S will return nothing, and location L is deleted.

      1. HIVE-2086.3.patch
        14 kB
        Jonathan Natkins
      2. HIVE-2086.2.patch
        17 kB
        Jonathan Natkins
      3. HIVE-2086.1.patch
        18 kB
        Jonathan Natkins
      4. create_like.q.out
        14 kB
        Ning Zhang

        Issue Links

          Activity

          Hide
          Edward Capriolo added a comment -

          Dropping an external table should not delete data. Are you saying that 'create table like' does not preserver the external property?

          Show
          Edward Capriolo added a comment - Dropping an external table should not delete data. Are you saying that 'create table like' does not preserver the external property?
          Hide
          Q Long added a comment -

          It seems that "create external table like" does not preserve the external property.

          Note that both the original table S and the new table T are external, and data loss will only occur when creating T using statement
          "create external table T like S location L". No data loss if T with full table definitions (i.e, does not use like statement)

          Show
          Q Long added a comment - It seems that "create external table like" does not preserve the external property. Note that both the original table S and the new table T are external, and data loss will only occur when creating T using statement "create external table T like S location L". No data loss if T with full table definitions (i.e, does not use like statement)
          Show
          Jonathan Natkins added a comment - https://reviews.apache.org/r/604/
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/604/
          -----------------------------------------------------------

          Review request for hive.

          Summary
          -------

          Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

          This addresses bug HIVE-2086.
          https://issues.apache.org/jira/browse/HIVE-2086

          Diffs


          build-common.xml 9f21a69
          data/files/ext_test/test.dat PRE-CREATION
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c
          ql/src/test/queries/clientpositive/create_like.q 2edde83
          ql/src/test/results/clientpositive/create_like.q.out 63a8939

          Diff: https://reviews.apache.org/r/604/diff

          Testing
          -------

          Automated tests added

          Thanks,

          Jonathan

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- Review request for hive. Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs build-common.xml 9f21a69 data/files/ext_test/test.dat PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c ql/src/test/queries/clientpositive/create_like.q 2edde83 ql/src/test/results/clientpositive/create_like.q.out 63a8939 Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/604/#review507
          -----------------------------------------------------------

          ql/src/test/queries/clientpositive/create_like.q
          <https://reviews.apache.org/r/604/#comment1031>

          the $var is cool!

          • Ning

          On 2011-04-15 01:34:56, Jonathan Natkins wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/604/

          -----------------------------------------------------------

          (Updated 2011-04-15 01:34:56)

          Review request for hive.

          Summary

          -------

          Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

          This addresses bug HIVE-2086.

          https://issues.apache.org/jira/browse/HIVE-2086

          Diffs

          -----

          build-common.xml 9f21a69

          data/files/ext_test/test.dat PRE-CREATION

          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c

          ql/src/test/queries/clientpositive/create_like.q 2edde83

          ql/src/test/results/clientpositive/create_like.q.out 63a8939

          Diff: https://reviews.apache.org/r/604/diff

          Testing

          -------

          Automated tests added

          Thanks,

          Jonathan

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/#review507 ----------------------------------------------------------- ql/src/test/queries/clientpositive/create_like.q < https://reviews.apache.org/r/604/#comment1031 > the $var is cool! Ning On 2011-04-15 01:34:56, Jonathan Natkins wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- (Updated 2011-04-15 01:34:56) Review request for hive. Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs ----- build-common.xml 9f21a69 data/files/ext_test/test.dat PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c ql/src/test/queries/clientpositive/create_like.q 2edde83 ql/src/test/results/clientpositive/create_like.q.out 63a8939 Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/604/#review508
          -----------------------------------------------------------

          Ship it!

          • Ning

          On 2011-04-15 01:34:56, Jonathan Natkins wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/604/

          -----------------------------------------------------------

          (Updated 2011-04-15 01:34:56)

          Review request for hive.

          Summary

          -------

          Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

          This addresses bug HIVE-2086.

          https://issues.apache.org/jira/browse/HIVE-2086

          Diffs

          -----

          build-common.xml 9f21a69

          data/files/ext_test/test.dat PRE-CREATION

          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c

          ql/src/test/queries/clientpositive/create_like.q 2edde83

          ql/src/test/results/clientpositive/create_like.q.out 63a8939

          Diff: https://reviews.apache.org/r/604/diff

          Testing

          -------

          Automated tests added

          Thanks,

          Jonathan

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/#review508 ----------------------------------------------------------- Ship it! Ning On 2011-04-15 01:34:56, Jonathan Natkins wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- (Updated 2011-04-15 01:34:56) Review request for hive. Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs ----- build-common.xml 9f21a69 data/files/ext_test/test.dat PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c ql/src/test/queries/clientpositive/create_like.q 2edde83 ql/src/test/results/clientpositive/create_like.q.out 63a8939 Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan
          Hide
          Ning Zhang added a comment -

          +1. Will commit if tests pass.

          Show
          Ning Zhang added a comment - +1. Will commit if tests pass.
          Hide
          Ning Zhang added a comment -

          Jonathan, there are a few diffs in the unit tests (1 in TestHBaseCliDriver and 6 in TestCliDriver). Can you take a look?

          Show
          Ning Zhang added a comment - Jonathan, there are a few diffs in the unit tests (1 in TestHBaseCliDriver and 6 in TestCliDriver). Can you take a look?
          Hide
          Jonathan Natkins added a comment -

          Sorry about that, I've updated the diffs. This time, there shouldn't be any failures

          Show
          Jonathan Natkins added a comment - Sorry about that, I've updated the diffs. This time, there shouldn't be any failures
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/604/
          -----------------------------------------------------------

          (Updated 2011-04-27 16:00:50.150750)

          Review request for hive.

          Changes
          -------

          Fixing test diffs

          Summary
          -------

          Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

          This addresses bug HIVE-2086.
          https://issues.apache.org/jira/browse/HIVE-2086

          Diffs (updated)


          build-common.xml 00c3680
          ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 22fde53
          ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 06a0447
          ql/src/test/queries/clientpositive/create_like.q 2edde83
          ql/src/test/results/clientpositive/create_like.q.out 63a8939

          Diff: https://reviews.apache.org/r/604/diff

          Testing
          -------

          Automated tests added

          Thanks,

          Jonathan

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- (Updated 2011-04-27 16:00:50.150750) Review request for hive. Changes ------- Fixing test diffs Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs (updated) build-common.xml 00c3680 ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 22fde53 ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 06a0447 ql/src/test/queries/clientpositive/create_like.q 2edde83 ql/src/test/results/clientpositive/create_like.q.out 63a8939 Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan
          Hide
          Ning Zhang added a comment -

          +1

          Show
          Ning Zhang added a comment - +1
          Hide
          Ning Zhang added a comment -

          @Johnathan, there are some tests failing. In particular the newly added test also failed (the output attached). Can you take a look and see if it is caused by newly committed patches (I've tested on the lasted trunk).

          Show
          Ning Zhang added a comment - @Johnathan, there are some tests failing. In particular the newly added test also failed (the output attached). Can you take a look and see if it is caused by newly committed patches (I've tested on the lasted trunk).
          Hide
          Vaibhav Aggarwal added a comment -

          Has this patch been committed or is anyone still working on this particular patch?

          Show
          Vaibhav Aggarwal added a comment - Has this patch been committed or is anyone still working on this particular patch?
          Hide
          Jonathan Natkins added a comment -

          It looks like this was actually just fixed today, as a part of HIVE-1218. However, that fix didn't add any regression tests for this bug, so I'm going to prepare an updated patch once I finish running the test suite.

          Show
          Jonathan Natkins added a comment - It looks like this was actually just fixed today, as a part of HIVE-1218 . However, that fix didn't add any regression tests for this bug, so I'm going to prepare an updated patch once I finish running the test suite.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/604/
          -----------------------------------------------------------

          (Updated 2011-07-19 16:50:07.742470)

          Review request for hive.

          Changes
          -------

          The bug was fixed as a part of HIVE-1218, and this diff adds some regression tests for this particular issue.

          Summary
          -------

          Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

          This addresses bug HIVE-2086.
          https://issues.apache.org/jira/browse/HIVE-2086

          Diffs (updated)


          build-common.xml b6807fa
          data/files/ext_test/test.dat PRE-CREATION
          ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 6d742e5
          ql/src/test/queries/clientpositive/create_like.q 3f8e58d
          ql/src/test/results/clientpositive/create_like.q.out 118235e

          Diff: https://reviews.apache.org/r/604/diff

          Testing
          -------

          Automated tests added

          Thanks,

          Jonathan

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- (Updated 2011-07-19 16:50:07.742470) Review request for hive. Changes ------- The bug was fixed as a part of HIVE-1218 , and this diff adds some regression tests for this particular issue. Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs (updated) build-common.xml b6807fa data/files/ext_test/test.dat PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 6d742e5 ql/src/test/queries/clientpositive/create_like.q 3f8e58d ql/src/test/results/clientpositive/create_like.q.out 118235e Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan
          Hide
          Jonathan Natkins added a comment -

          Just regression tests

          Show
          Jonathan Natkins added a comment - Just regression tests
          Hide
          Carl Steinbach added a comment -

          +1. Will commit if tests pass.

          Show
          Carl Steinbach added a comment - +1. Will commit if tests pass.
          Hide
          Carl Steinbach added a comment -

          Committed to trunk. Thanks Natty!

          Show
          Carl Steinbach added a comment - Committed to trunk. Thanks Natty!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #841 (See https://builds.apache.org/job/Hive-trunk-h0.21/841/)
          HIVE-2086. Add test coverage for external table data loss issue (Jonathan Natkins via cws)

          cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1149331
          Files :

          • /hive/trunk/data/files/ext_test
          • /hive/trunk/ql/src/test/queries/clientpositive/create_like.q
          • /hive/trunk/ql/src/test/results/clientpositive/create_like.q.out
          • /hive/trunk/data/files/ext_test/test.dat
          • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
          • /hive/trunk/build-common.xml
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #841 (See https://builds.apache.org/job/Hive-trunk-h0.21/841/ ) HIVE-2086 . Add test coverage for external table data loss issue (Jonathan Natkins via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1149331 Files : /hive/trunk/data/files/ext_test /hive/trunk/ql/src/test/queries/clientpositive/create_like.q /hive/trunk/ql/src/test/results/clientpositive/create_like.q.out /hive/trunk/data/files/ext_test/test.dat /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java /hive/trunk/build-common.xml

            People

            • Assignee:
              Jonathan Natkins
              Reporter:
              Q Long
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development