Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-3824

bug if different serdes are used for different partitions

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Consider the following testcase:

      create table tst5 (key string, value string) partitioned by (ds string) stored as rcfile;
      insert overwrite table tst5 partition (ds='1') select * from src;
      insert overwrite table tst5 partition (ds='2') select * from src;
      insert overwrite table tst5 partition (ds='3') select * from src;

      alter table tst5 stored as sequencefile;

      insert overwrite table tst5 partition (ds='4') select * from src;
      insert overwrite table tst5 partition (ds='5') select * from src;
      insert overwrite table tst5 partition (ds='6') select * from src;

      alter table tst5 set serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe';

      insert overwrite table tst5 partition (ds='7') select * from src;
      insert overwrite table tst5 partition (ds='8') select * from src;
      insert overwrite table tst5 partition (ds='9') select * from src;

      The following query works fine:

      select key + key, value from tst5 where ((ds = '4') or (ds = '1'));

      since both the partitions use ColumnarSerDe

      But the following query fails:

      select key + key, value from tst5 where ((ds = '4') or (ds = '1') or (ds='7'));

      since different serdes are used.

      1. hive.3824.1.patch
        19 kB
        Namit Jain
      2. hive.3824.3.patch
        19 kB
        Namit Jain
      3. hive.3824.4.patch
        20 kB
        Namit Jain

        Issue Links

          Activity

          Show
          namit Namit Jain added a comment - https://reviews.facebook.net/D7515
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          +1

          Show
          ashutoshc Ashutosh Chauhan added a comment - +1
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Namit, Feel free to commit this if tests pass.

          Show
          ashutoshc Ashutosh Chauhan added a comment - Namit, Feel free to commit this if tests pass.
          Hide
          namit Namit Jain added a comment -

          will refresh and run tests.

          Show
          namit Namit Jain added a comment - will refresh and run tests.
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Whats different in this patch and previous one?

          Show
          ashutoshc Ashutosh Chauhan added a comment - Whats different in this patch and previous one?
          Hide
          namit Namit Jain added a comment -

          Nothing, just the test output.
          Due to https://issues.apache.org/jira/browse/HIVE-3803, the test output changed.

          The tests passed.
          Ashutosh Chauhan, should I commit this ?

          Show
          namit Namit Jain added a comment - Nothing, just the test output. Due to https://issues.apache.org/jira/browse/HIVE-3803 , the test output changed. The tests passed. Ashutosh Chauhan , should I commit this ?
          Hide
          ashutoshc Ashutosh Chauhan added a comment -

          Cool. Please go ahead for commit.

          Show
          ashutoshc Ashutosh Chauhan added a comment - Cool. Please go ahead for commit.
          Hide
          namit Namit Jain added a comment -

          Committed. Thanks Ashutosh

          Show
          namit Namit Jain added a comment - Committed. Thanks Ashutosh
          Hide
          hudson Hudson added a comment -

          Integrated in hive-trunk-hadoop1 #12 (See https://builds.apache.org/job/hive-trunk-hadoop1/12/)
          HIVE-3824 bug if different serdes are used for different partitions
          (Namit Jain via Ashutosh and namit) (Revision 1432865)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Show
          hudson Hudson added a comment - Integrated in hive-trunk-hadoop1 #12 (See https://builds.apache.org/job/hive-trunk-hadoop1/12/ ) HIVE-3824 bug if different serdes are used for different partitions (Namit Jain via Ashutosh and namit) (Revision 1432865) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Hide
          hudson Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1911 (See https://builds.apache.org/job/Hive-trunk-h0.21/1911/)
          HIVE-3824 bug if different serdes are used for different partitions
          (Namit Jain via Ashutosh and namit) (Revision 1432865)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Show
          hudson Hudson added a comment - Integrated in Hive-trunk-h0.21 #1911 (See https://builds.apache.org/job/Hive-trunk-h0.21/1911/ ) HIVE-3824 bug if different serdes are used for different partitions (Namit Jain via Ashutosh and namit) (Revision 1432865) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Hide
          hudson Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #64 (See https://builds.apache.org/job/Hive-trunk-hadoop2/64/)
          HIVE-3824 bug if different serdes are used for different partitions
          (Namit Jain via Ashutosh and namit) (Revision 1432865)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Show
          hudson Hudson added a comment - Integrated in Hive-trunk-hadoop2 #64 (See https://builds.apache.org/job/Hive-trunk-hadoop2/64/ ) HIVE-3824 bug if different serdes are used for different partitions (Namit Jain via Ashutosh and namit) (Revision 1432865) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out

            People

            • Assignee:
              namit Namit Jain
              Reporter:
              namit Namit Jain
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development