1. Hive
  2. HIVE-3308

Mixing avro and snappy gives null values


    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:


      On default hive uses LazySimpleSerDe for output.
      When I now enable compression and "select count from avrotable" the output is a file with the .avro extension but this then will display null values since the file is in reality not an avro file but a file created by LazySimpleSerDe using compression so should be a .snappy file.
      This causes any job (exception select * from avrotable is that not truly a job) to show null values.
      If you use any serde other then avro you can temporarily fix this by setting "set hive.output.file.extension=.snappy" and it will correctly work again but this won't work on avro since it overwrites the hive.output.file.extension during initializing.

      When you dump the query result into a table with "create table bla as" you can rename the .avro file into .snappy and the "select from bla" will also magiacally work again.

      Input and Ouput serdes don't always match so when I use avro as an input format it should not set the hive.output.file.extension.
      Onces it's set all queries will use it and fail making the connection useless to reuse.

      1. HIVE-3308.patch1.txt
        1 kB
        Bennie Schut
      2. HIVE-3308.patch2.txt
        5 kB
        Bennie Schut

        Issue Links


          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Link This issue is duplicated by HIVE-4195 [ HIVE-4195 ]
          Navis made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.11.0 [ 12323587 ]
          Resolution Fixed [ 1 ]
          Ashutosh Chauhan made changes -
          Assignee Bennie Schut [ bennies ]
          Bennie Schut made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Bennie Schut made changes -
          Attachment HIVE-3308.patch2.txt [ 12566516 ]
          Bennie Schut made changes -
          Field Original Value New Value
          Attachment HIVE-3308.patch1.txt [ 12538157 ]
          Bennie Schut created issue -


            • Assignee:
              Bennie Schut
              Bennie Schut
            • Votes:
              1 Vote for this issue
              9 Start watching this issue


              • Created: