Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-1499

Ruby 2+ Writes Invalid avro files using the avro gem

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.5
    • Fix Version/s: 1.7.7
    • Component/s: ruby
    • Labels:

      Description

      The rubygem writes corrupted avro files under ruby 2.0.0 and ruby 2.1.1. It appears to work correctly under jruby-1.7.10 and ruby 1.9.3.

      Here is a reproducible:

      ```ruby
      require 'avro'

      data = [

      {"guid"=>"144045de-eb44-dd1b-d9af-6c8b5d41a96e", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617818, "updated_at"=>1398180288, "deleted_at"=>nil}

      ,

      {"guid"=>"51e06057-14d2-7527-81fa-b07dba0a263b", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"Student Loans R' Us", "created_at"=>1386178342, "updated_at"=>1398180286, "deleted_at"=>nil}

      ,

      {"guid"=>"b4d1d99f-4351-d0e7-221c-a3fae08716bc", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617026, "updated_at"=>1398180288, "deleted_at"=>nil}

      ,

      {"guid"=>"084638fa-a78d-bbdd-e075-7c9c957a9b46", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617138, "updated_at"=>1398180288, "deleted_at"=>nil}

      ,

      {"guid"=>"79287c76-4e8f-0a21-7569-a2bcdc2b2f4d", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617135, "updated_at"=>1398180288, "deleted_at"=>nil}

      ,

      {"guid"=>"3bcc26b2-7d3b-6c4d-cb27-4eb1574b3c20", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"Cayman Islands Bank", "created_at"=>1386902345, "updated_at"=>1398180288, "deleted_at"=>nil}

      ,

      {"guid"=>"75e1e56c-7611-4030-d002-afa2af70e5a1", "user_guid"=>"0cd41235-5c14-eae9-00ed-c6eb11dd9119", "name"=>"My Awesome Bank", "created_at"=>1390617427, "updated_at"=>1398180288, "deleted_at"=>nil}

      ,
      ]

      member_schema = <<-SCHEMA
      {"namespace": "md.data_logs",
      "type": "record",
      "name": "Member",
      "fields": [

      {"name": "guid", "type": "string"}

      ,

      {"name": "user_guid", "type": "string"}

      ,

      {"name": "name", "type": ["string","null"]}

      ,

      {"name": "created_at", "type":"long"}

      ,

      {"name": "updated_at", "type":"long"}

      ,

      {"name": "deleted_at", "type":["long","null"]}

      ]
      }
      SCHEMA
      filepath = "./members.avro"
      File.unlink(filepath) if File.exists?(filepath)

      Avro::DataFile.open(filepath, "w", member_schema) do |dw|
      data.each do |entry|
      dw << entry
      end
      end

      entries = []
      Avro::DataFile.open(filepath, "r") do |reader|
      reader.each do |entry|
      entries << entry
      end
      end

      puts "Here is the data I wrote into the file:"
      data.each

      {|e| p e }
      print "\n\n\n\n"

      puts "Here is the data I read from the file:"
      entries.each{|e| p e }

      ```

      Under ruby 2+ it fails with the message "undefined method 'unpack' for nil:NilClass (NoMethodError)". I have also tested that the rubygem can correctly read avro files written by the java client, but the java client fails to read files written by the ruby client, so the issue is definitely in how the rubygem is trying to write the binary file.

        Attachments

        1. AVRO-1499.patch
          0.5 kB
          Martin Kleppmann
        2. AVRO-1499-2.patch
          3 kB
          Willem van Bergen
        3. AVRO-1499-3.patch
          1 kB
          Martin Kleppmann

          Activity

            People

            • Assignee:
              martinkl Martin Kleppmann
              Reporter:
              hqmq Michael Ries
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: