Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1409

File format information is retrieved from first partition

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.6.0
    • 0.6.0
    • Query Processor
    • None
    • Reviewed

    Description

      Currently, if no partitions match the partition predicate, the first partition is used to retrieve the file format. This can cause an problem if the table is set to use RCFile, but the first partition uses SequenceFile:

      java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer.()
      	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
      	at org.apache.hadoop.mapred.SequenceFileRecordReader.createKey(SequenceFileRecordReader.java:65)
      	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.createKey(CombineHiveRecordReader.java:76)
      	at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.createKey(CombineHiveRecordReader.java:42)
      	at org.apache.hadoop.hive.shims.Hadoop20Shims$CombineFileRecordReader.createKey(Hadoop20Shims.java:212)
      	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.createKey(MapTask.java:167)
      	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
      	at org.apache.hadoop.mapred.Child.main(Child.java:159)
      Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.ql.io.RCFile$KeyBuffer.()
      	at java.lang.Class.getConstructor0(Class.java:2706)
      	at java.lang.Class.getDeclaredConstructor(Class.java:1985)
      	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
      	... 9 more
      
      

      The proposed change is to use the table's metadata in such cases.

      Attachments

        1. HIVE-1409.1.patch
          4 kB
          Paul Yang
        2. HIVE-1409.2.patch
          4 kB
          Paul Yang

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            pauly Paul Yang Assign to me
            pauly Paul Yang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment