Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-2086

Add test coverage for external table data loss issue

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.7.0
    • 0.8.0
    • Metastore
    • None
    • Amazon elastics mapreduce cluster

    • Reviewed
    • data loss external table create like

    Description

      Data loss when using "create external table like" statement.

      1) Set up an external table S, point to location L. Populate data in S.
      2) Create another external table T, using statement like this:
      create external table T like S location L
      Make sure table T point to the same location as the original table S.
      3) Query table T, see the same set of data in S.
      4) drop table T.
      5) Query table S will return nothing, and location L is deleted.

      Attachments

        1. HIVE-2086.3.patch
          14 kB
          Jonathan Natkins
        2. HIVE-2086.2.patch
          17 kB
          Jonathan Natkins
        3. HIVE-2086.1.patch
          18 kB
          Jonathan Natkins
        4. create_like.q.out
          14 kB
          Ning Zhang

        Issue Links

          Activity

            hudson Hudson added a comment -

            Integrated in Hive-trunk-h0.21 #841 (See https://builds.apache.org/job/Hive-trunk-h0.21/841/)
            HIVE-2086. Add test coverage for external table data loss issue (Jonathan Natkins via cws)

            cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1149331
            Files :

            • /hive/trunk/data/files/ext_test
            • /hive/trunk/ql/src/test/queries/clientpositive/create_like.q
            • /hive/trunk/ql/src/test/results/clientpositive/create_like.q.out
            • /hive/trunk/data/files/ext_test/test.dat
            • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java
            • /hive/trunk/build-common.xml
            hudson Hudson added a comment - Integrated in Hive-trunk-h0.21 #841 (See https://builds.apache.org/job/Hive-trunk-h0.21/841/ ) HIVE-2086 . Add test coverage for external table data loss issue (Jonathan Natkins via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1149331 Files : /hive/trunk/data/files/ext_test /hive/trunk/ql/src/test/queries/clientpositive/create_like.q /hive/trunk/ql/src/test/results/clientpositive/create_like.q.out /hive/trunk/data/files/ext_test/test.dat /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java /hive/trunk/build-common.xml
            cwsteinbach Carl Steinbach added a comment -

            Committed to trunk. Thanks Natty!

            cwsteinbach Carl Steinbach added a comment - Committed to trunk. Thanks Natty!
            cwsteinbach Carl Steinbach added a comment -

            +1. Will commit if tests pass.

            cwsteinbach Carl Steinbach added a comment - +1. Will commit if tests pass.

            Just regression tests

            natty Jonathan Natkins added a comment - Just regression tests

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/604/
            -----------------------------------------------------------

            (Updated 2011-07-19 16:50:07.742470)

            Review request for hive.

            Changes
            -------

            The bug was fixed as a part of HIVE-1218, and this diff adds some regression tests for this particular issue.

            Summary
            -------

            Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

            This addresses bug HIVE-2086.
            https://issues.apache.org/jira/browse/HIVE-2086

            Diffs (updated)


            build-common.xml b6807fa
            data/files/ext_test/test.dat PRE-CREATION
            ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 6d742e5
            ql/src/test/queries/clientpositive/create_like.q 3f8e58d
            ql/src/test/results/clientpositive/create_like.q.out 118235e

            Diff: https://reviews.apache.org/r/604/diff

            Testing
            -------

            Automated tests added

            Thanks,

            Jonathan

            jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- (Updated 2011-07-19 16:50:07.742470) Review request for hive. Changes ------- The bug was fixed as a part of HIVE-1218 , and this diff adds some regression tests for this particular issue. Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs (updated) build-common.xml b6807fa data/files/ext_test/test.dat PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 6d742e5 ql/src/test/queries/clientpositive/create_like.q 3f8e58d ql/src/test/results/clientpositive/create_like.q.out 118235e Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan

            It looks like this was actually just fixed today, as a part of HIVE-1218. However, that fix didn't add any regression tests for this bug, so I'm going to prepare an updated patch once I finish running the test suite.

            natty Jonathan Natkins added a comment - It looks like this was actually just fixed today, as a part of HIVE-1218 . However, that fix didn't add any regression tests for this bug, so I'm going to prepare an updated patch once I finish running the test suite.

            Has this patch been committed or is anyone still working on this particular patch?

            vaggarw Vaibhav Aggarwal added a comment - Has this patch been committed or is anyone still working on this particular patch?
            nzhang Ning Zhang added a comment -

            @Johnathan, there are some tests failing. In particular the newly added test also failed (the output attached). Can you take a look and see if it is caused by newly committed patches (I've tested on the lasted trunk).

            nzhang Ning Zhang added a comment - @Johnathan, there are some tests failing. In particular the newly added test also failed (the output attached). Can you take a look and see if it is caused by newly committed patches (I've tested on the lasted trunk).
            nzhang Ning Zhang added a comment -

            +1

            nzhang Ning Zhang added a comment - +1

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/604/
            -----------------------------------------------------------

            (Updated 2011-04-27 16:00:50.150750)

            Review request for hive.

            Changes
            -------

            Fixing test diffs

            Summary
            -------

            Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

            This addresses bug HIVE-2086.
            https://issues.apache.org/jira/browse/HIVE-2086

            Diffs (updated)


            build-common.xml 00c3680
            ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 22fde53
            ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 06a0447
            ql/src/test/queries/clientpositive/create_like.q 2edde83
            ql/src/test/results/clientpositive/create_like.q.out 63a8939

            Diff: https://reviews.apache.org/r/604/diff

            Testing
            -------

            Automated tests added

            Thanks,

            Jonathan

            jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- (Updated 2011-04-27 16:00:50.150750) Review request for hive. Changes ------- Fixing test diffs Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs (updated) build-common.xml 00c3680 ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 22fde53 ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 06a0447 ql/src/test/queries/clientpositive/create_like.q 2edde83 ql/src/test/results/clientpositive/create_like.q.out 63a8939 Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan

            Sorry about that, I've updated the diffs. This time, there shouldn't be any failures

            natty Jonathan Natkins added a comment - Sorry about that, I've updated the diffs. This time, there shouldn't be any failures
            nzhang Ning Zhang added a comment -

            Jonathan, there are a few diffs in the unit tests (1 in TestHBaseCliDriver and 6 in TestCliDriver). Can you take a look?

            nzhang Ning Zhang added a comment - Jonathan, there are a few diffs in the unit tests (1 in TestHBaseCliDriver and 6 in TestCliDriver). Can you take a look?
            nzhang Ning Zhang added a comment -

            +1. Will commit if tests pass.

            nzhang Ning Zhang added a comment - +1. Will commit if tests pass.

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/604/#review508
            -----------------------------------------------------------

            Ship it!

            • Ning

            On 2011-04-15 01:34:56, Jonathan Natkins wrote:

            -----------------------------------------------------------

            This is an automatically generated e-mail. To reply, visit:

            https://reviews.apache.org/r/604/

            -----------------------------------------------------------

            (Updated 2011-04-15 01:34:56)

            Review request for hive.

            Summary

            -------

            Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

            This addresses bug HIVE-2086.

            https://issues.apache.org/jira/browse/HIVE-2086

            Diffs

            -----

            build-common.xml 9f21a69

            data/files/ext_test/test.dat PRE-CREATION

            ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c

            ql/src/test/queries/clientpositive/create_like.q 2edde83

            ql/src/test/results/clientpositive/create_like.q.out 63a8939

            Diff: https://reviews.apache.org/r/604/diff

            Testing

            -------

            Automated tests added

            Thanks,

            Jonathan

            jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/#review508 ----------------------------------------------------------- Ship it! Ning On 2011-04-15 01:34:56, Jonathan Natkins wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- (Updated 2011-04-15 01:34:56) Review request for hive. Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs ----- build-common.xml 9f21a69 data/files/ext_test/test.dat PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c ql/src/test/queries/clientpositive/create_like.q 2edde83 ql/src/test/results/clientpositive/create_like.q.out 63a8939 Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/604/#review507
            -----------------------------------------------------------

            ql/src/test/queries/clientpositive/create_like.q
            <https://reviews.apache.org/r/604/#comment1031>

            the $var is cool!

            • Ning

            On 2011-04-15 01:34:56, Jonathan Natkins wrote:

            -----------------------------------------------------------

            This is an automatically generated e-mail. To reply, visit:

            https://reviews.apache.org/r/604/

            -----------------------------------------------------------

            (Updated 2011-04-15 01:34:56)

            Review request for hive.

            Summary

            -------

            Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

            This addresses bug HIVE-2086.

            https://issues.apache.org/jira/browse/HIVE-2086

            Diffs

            -----

            build-common.xml 9f21a69

            data/files/ext_test/test.dat PRE-CREATION

            ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c

            ql/src/test/queries/clientpositive/create_like.q 2edde83

            ql/src/test/results/clientpositive/create_like.q.out 63a8939

            Diff: https://reviews.apache.org/r/604/diff

            Testing

            -------

            Automated tests added

            Thanks,

            Jonathan

            jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/#review507 ----------------------------------------------------------- ql/src/test/queries/clientpositive/create_like.q < https://reviews.apache.org/r/604/#comment1031 > the $var is cool! Ning On 2011-04-15 01:34:56, Jonathan Natkins wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- (Updated 2011-04-15 01:34:56) Review request for hive. Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs ----- build-common.xml 9f21a69 data/files/ext_test/test.dat PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c ql/src/test/queries/clientpositive/create_like.q 2edde83 ql/src/test/results/clientpositive/create_like.q.out 63a8939 Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan

            -----------------------------------------------------------
            This is an automatically generated e-mail. To reply, visit:
            https://reviews.apache.org/r/604/
            -----------------------------------------------------------

            Review request for hive.

            Summary
            -------

            Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external.

            This addresses bug HIVE-2086.
            https://issues.apache.org/jira/browse/HIVE-2086

            Diffs


            build-common.xml 9f21a69
            data/files/ext_test/test.dat PRE-CREATION
            ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c
            ql/src/test/queries/clientpositive/create_like.q 2edde83
            ql/src/test/results/clientpositive/create_like.q.out 63a8939

            Diff: https://reviews.apache.org/r/604/diff

            Testing
            -------

            Automated tests added

            Thanks,

            Jonathan

            jiraposter@reviews.apache.org jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ ----------------------------------------------------------- Review request for hive. Summary ------- Review request for HIVE-2086 . The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086 . https://issues.apache.org/jira/browse/HIVE-2086 Diffs build-common.xml 9f21a69 data/files/ext_test/test.dat PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 9d8919c ql/src/test/queries/clientpositive/create_like.q 2edde83 ql/src/test/results/clientpositive/create_like.q.out 63a8939 Diff: https://reviews.apache.org/r/604/diff Testing ------- Automated tests added Thanks, Jonathan
            natty Jonathan Natkins added a comment - https://reviews.apache.org/r/604/
            qlong Q Long added a comment -

            It seems that "create external table like" does not preserve the external property.

            Note that both the original table S and the new table T are external, and data loss will only occur when creating T using statement
            "create external table T like S location L". No data loss if T with full table definitions (i.e, does not use like statement)

            qlong Q Long added a comment - It seems that "create external table like" does not preserve the external property. Note that both the original table S and the new table T are external, and data loss will only occur when creating T using statement "create external table T like S location L". No data loss if T with full table definitions (i.e, does not use like statement)

            Dropping an external table should not delete data. Are you saying that 'create table like' does not preserver the external property?

            appodictic Edward Capriolo added a comment - Dropping an external table should not delete data. Are you saying that 'create table like' does not preserver the external property?

            People

              natty Jonathan Natkins
              qlong Q Long
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: