Hive
  1. Hive
  2. HIVE-3824

bug if different serdes are used for different partitions

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Consider the following testcase:

      create table tst5 (key string, value string) partitioned by (ds string) stored as rcfile;
      insert overwrite table tst5 partition (ds='1') select * from src;
      insert overwrite table tst5 partition (ds='2') select * from src;
      insert overwrite table tst5 partition (ds='3') select * from src;

      alter table tst5 stored as sequencefile;

      insert overwrite table tst5 partition (ds='4') select * from src;
      insert overwrite table tst5 partition (ds='5') select * from src;
      insert overwrite table tst5 partition (ds='6') select * from src;

      alter table tst5 set serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe';

      insert overwrite table tst5 partition (ds='7') select * from src;
      insert overwrite table tst5 partition (ds='8') select * from src;
      insert overwrite table tst5 partition (ds='9') select * from src;

      The following query works fine:

      select key + key, value from tst5 where ((ds = '4') or (ds = '1'));

      since both the partitions use ColumnarSerDe

      But the following query fails:

      select key + key, value from tst5 where ((ds = '4') or (ds = '1') or (ds='7'));

      since different serdes are used.

      1. hive.3824.1.patch
        19 kB
        Namit Jain
      2. hive.3824.3.patch
        19 kB
        Namit Jain
      3. hive.3824.4.patch
        20 kB
        Namit Jain

        Issue Links

          Activity

          Show
          Namit Jain added a comment - https://reviews.facebook.net/D7515
          Hide
          Ashutosh Chauhan added a comment -

          +1

          Show
          Ashutosh Chauhan added a comment - +1
          Hide
          Ashutosh Chauhan added a comment -

          Namit, Feel free to commit this if tests pass.

          Show
          Ashutosh Chauhan added a comment - Namit, Feel free to commit this if tests pass.
          Hide
          Namit Jain added a comment -

          will refresh and run tests.

          Show
          Namit Jain added a comment - will refresh and run tests.
          Hide
          Ashutosh Chauhan added a comment -

          Whats different in this patch and previous one?

          Show
          Ashutosh Chauhan added a comment - Whats different in this patch and previous one?
          Hide
          Namit Jain added a comment -

          Nothing, just the test output.
          Due to https://issues.apache.org/jira/browse/HIVE-3803, the test output changed.

          The tests passed.
          Ashutosh Chauhan, should I commit this ?

          Show
          Namit Jain added a comment - Nothing, just the test output. Due to https://issues.apache.org/jira/browse/HIVE-3803 , the test output changed. The tests passed. Ashutosh Chauhan , should I commit this ?
          Hide
          Ashutosh Chauhan added a comment -

          Cool. Please go ahead for commit.

          Show
          Ashutosh Chauhan added a comment - Cool. Please go ahead for commit.
          Hide
          Namit Jain added a comment -

          Committed. Thanks Ashutosh

          Show
          Namit Jain added a comment - Committed. Thanks Ashutosh
          Hide
          Hudson added a comment -

          Integrated in hive-trunk-hadoop1 #12 (See https://builds.apache.org/job/hive-trunk-hadoop1/12/)
          HIVE-3824 bug if different serdes are used for different partitions
          (Namit Jain via Ashutosh and namit) (Revision 1432865)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Show
          Hudson added a comment - Integrated in hive-trunk-hadoop1 #12 (See https://builds.apache.org/job/hive-trunk-hadoop1/12/ ) HIVE-3824 bug if different serdes are used for different partitions (Namit Jain via Ashutosh and namit) (Revision 1432865) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1911 (See https://builds.apache.org/job/Hive-trunk-h0.21/1911/)
          HIVE-3824 bug if different serdes are used for different partitions
          (Namit Jain via Ashutosh and namit) (Revision 1432865)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1911 (See https://builds.apache.org/job/Hive-trunk-h0.21/1911/ ) HIVE-3824 bug if different serdes are used for different partitions (Namit Jain via Ashutosh and namit) (Revision 1432865) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #64 (See https://builds.apache.org/job/Hive-trunk-hadoop2/64/)
          HIVE-3824 bug if different serdes are used for different partitions
          (Namit Jain via Ashutosh and namit) (Revision 1432865)

          Result = FAILURE
          namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q
          • /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out
          • /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #64 (See https://builds.apache.org/job/Hive-trunk-hadoop2/64/ ) HIVE-3824 bug if different serdes are used for different partitions (Namit Jain via Ashutosh and namit) (Revision 1432865) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1432865 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat8.q /hive/trunk/ql/src/test/queries/clientpositive/partition_wise_fileformat9.q /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat8.q.out /hive/trunk/ql/src/test/results/clientpositive/partition_wise_fileformat9.q.out

            People

            • Assignee:
              Namit Jain
              Reporter:
              Namit Jain
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development