Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1409

File format information is retrieved from first partition

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.6.0
    • 0.6.0
    • Query Processor
    • None
    • Reviewed

    Description

      Currently, if no partitions match the partition predicate, the first partition is used to retrieve the file format. This can cause an problem if the table is set to use RCFile, but the first partition uses SequenceFile:

      java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer.()
      	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
      	at org.apache.hadoop.mapred.SequenceFileRecordReader.createKey(SequenceFileRecordReader.java:65)
      	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.createKey(CombineHiveRecordReader.java:76)
      	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.createKey(CombineHiveRecordReader.java:42)
      	at org.apache.hadoop.hive.shims.Hadoop20Shims$CombineFileRecordReader.createKey(Hadoop20Shims.java:212)
      	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.createKey(MapTask.java:167)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
      	at org.apache.hadoop.mapred.Child.main(Child.java:159)
      Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer.()
      	at java.lang.Class.getConstructor0(Class.java:2706)
      	at java.lang.Class.getDeclaredConstructor(Class.java:1985)
      	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
      	... 9 more
      
      

      The proposed change is to use the table's metadata in such cases.

      Attachments

        1. HIVE-1409.1.patch
          4 kB
          Paul Yang
        2. HIVE-1409.2.patch
          4 kB
          Paul Yang

        Activity

          namit Namit Jain added a comment -

          +1

          looks good - will commit if the tests pass

          namit Namit Jain added a comment - +1 looks good - will commit if the tests pass
          namit Namit Jain added a comment -

          After more review, please drop the table at the end of the test that you added.
          It can lead to non-deterministic results for other tests

          namit Namit Jain added a comment - After more review, please drop the table at the end of the test that you added. It can lead to non-deterministic results for other tests
          pauly Paul Yang added a comment -
          • Drop table added
          pauly Paul Yang added a comment - Drop table added
          namit Namit Jain added a comment -

          +1

          namit Namit Jain added a comment - +1
          namit Namit Jain added a comment -

          Committed. Thanks Paul

          namit Namit Jain added a comment - Committed. Thanks Paul

          People

            pauly Paul Yang
            pauly Paul Yang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: