[AVRO-1206] utf-8 serialisation problems

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.2
• Fix Version/s: 1.7.3
• Component/s: ruby

Environment

ruby-1.9.3p194, avro gem 1.7.2.

Release Note

exemple to redo the error :
require 'avro'
schema_json = <<-JSON
{
  "type": "record",
  "name": "Activities",
  "fields" : [
    {"name": "params", "type": ["string","null"]}
]
}
JSON

schema = Avro::Schema.parse(schema_json)
writer = Avro::IO::DatumWriter.new(schema)
fname = "test.avr"

puts "writing to #{fname}"
file = File.open(fname, 'wb')
dw = Avro::DataFile::Writer.new(file, writer, schema)

# don't work when reading !
dw << {"params" => "家"}
# don't work too
#dw << {"params" => "（"}
dw.close


puts "reading"
Avro::DataFile.open(fname, 'r') do |dr|
  dr.each {|a|
    puts a["params"]
  }
end

• Tags: utf-8 ruby avro
• Assignee: Unassigned
• Reporter: Tophe Vigny
• Created: 16/Nov/12 13:27
• Updated: 10/Dec/12 17:57
• Resolved: 28/Nov/12 22:35

Description

some serialized utf-8 characters like "家" cannot be read latter, avro break with
/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/io.rb:230:in `match_schemas': undefined method `type' for nil:NilClass (NoMethodError)
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/io.rb:288:in `read_data'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/io.rb:384:in `read_union'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/io.rb:317:in `read_data'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/io.rb:392:in `block in read_record'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/io.rb:390:in `each'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/io.rb:390:in `read_record'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/io.rb:318:in `read_data'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/io.rb:283:in `read'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/data_file.rb:223:in `block in each'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/data_file.rb:211:in `loop'
from /home/Tophe/.rvm/gems/ruby-1.9.3-p194/gems/avro-1.7.2/lib/avro/data_file.rb:211:in `each'
from avr_err_example.rb:42:in `block in <main>'

Issue Links

• duplicates: AVRO-1134 Ruby datafile serialization fails with UTF-8 characters
• is duplicated by: AVRO-1134 Ruby datafile serialization fails with UTF-8 characters