Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-15004

Query for parquet tables failing with java.lang.IllegalArgumentException: FilterPredicate column: f's declared type (java.lang.Double) does not match the schema found in file metadata.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.2.1
    • Fix Version/s: 1.2.1
    • Component/s: Hive, HiveServer2
    • Labels:
      None

      Description

      Queries involving float data type, run against parquet tables failing

      hive> desc extended all100k;
      OK
      t                     tinyint
      si                    smallint
      i                     int
      b                     bigint
      f                     float
      d                     double
      s                     string
      dc                    decimal(38,18)
      bo                    boolean
      v                     varchar(25)
      c                     char(25)
      ts                    timestamp
      dt                    date

      Detailed Table Information  Table(tableName:all100k, dbName:default, owner:hrt_qa, createTime:1476765150, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:t, type:tinyint, comment:null), FieldSchema(name:si, type:smallint, comment:null), FieldSchema(name:i, type:int, comment:null), FieldSchema(name:b, type:bigint, comment:null), FieldSchema(name:f, type:float, comment:null), FieldSchema(name:d, type:double, comment:null), FieldSchema(name:s, type:string, comment:null), FieldSchema(name:dc, type:decimal(38,18), comment:null), FieldSchema(name:bo, type:boolean, comment:null), FieldSchema(name:v, type:varchar(25), comment:null), FieldSchema(name:c, type:char(25), comment:null), FieldSchema(name:ts, type:timestamp, comment:null), FieldSchema(name:dt, type:date, comment:null)], location:hdfs://ctr-e45-1475874954070-9012-01-000008.hwx.site:8020/apps/hive/warehouse/all100k, inputFormat:org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, parameters:

      {serialization.format=1}

      ), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{numFiles=1, transient_lastDdlTime=1476765184, COLUMN_STATS_ACCURATE={"COLUMN_STATS":

      {"t":"true","si":"true","i":"true","b":"true","f":"true","d":"true","s":"true","dc":"true","bo":"true","v":"true","c":"true","ts":"true"}

      ,"BASIC_STATS":"true"}, totalSize=6564143, numRows=100000, rawDataSize=1300000}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
      Time taken: 0.54 seconds, Fetched: 15 row(s)
      hive> select t from all100k
          >                     where t<>0 and s<>0 and b<>0 and (f<>0 or d<>0);
      OK
      SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
      SLF4J: Defaulting to no-operation (NOP) logger implementation
      SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
      Failed with exception java.io.IOException:java.lang.IllegalArgumentException: FilterPredicate column: f's declared type (java.lang.Double) does not match the schema found in file metadata. Column f is of type: FLOAT
      Valid types for this column are: [class java.lang.Float]
      Time taken: 0.919 seconds

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jdere Jason Dere
                Reporter:
                asreekumar Aswathy Chellammal Sreekumar
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: