[AVRO-724] C implementation does not write datum values that are larger than the memory write buffer (currently 16K)

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.1
• Component/s: c
• Assignee: Unassigned
• Reporter: Jeremy Hinegardner
• Created: 03/Jan/11 05:07
• Updated: 13/Dec/11 17:45

Description

The current C implementation does not allow for datum values greater than 16K.

The avro_file_writer_append flushes blocks to disk over time, but does not deal with the single case of a single datum being larger than avro_file_writer_t.datum_buffer. This is noted in the source code:

datafile.c:294-313

int avro_file_writer_append(avro_file_writer_t w, avro_datum_t datum)
{
    int rval;                  
    if (!w || !datum) {        
        return EINVAL;         
    }
    rval = avro_write_data(w->datum_writer, w->writers_schema, datum);
    if (rval) {                
        check(rval, file_write_block(w));
        rval =
            avro_write_data(w->datum_writer, w->writers_schema, datum);
        if (rval) {            
            /* TODO: if the datum encoder larger than our buffer,
               just write a single large datum */
            return rval;       
        }
    }
    w->block_count++;          
    return 0;                  
}