Avro
  1. Avro
  2. AVRO-724

C implementation does not write datum values that are larger than the memory write buffer (currently 16K)

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.1
    • Fix Version/s: None
    • Component/s: c
    • Labels:
      None

      Description

      The current C implementation does not allow for datum values greater than 16K.

      The avro_file_writer_append flushes blocks to disk over time, but does not deal with the single case of a single datum being larger than avro_file_writer_t.datum_buffer. This is noted in the source code:

      datafile.c:294-313
      int avro_file_writer_append(avro_file_writer_t w, avro_datum_t datum)
      {
          int rval;                  
          if (!w || !datum) {        
              return EINVAL;         
          }
          rval = avro_write_data(w->datum_writer, w->writers_schema, datum);
          if (rval) {                
              check(rval, file_write_block(w));
              rval =
                  avro_write_data(w->datum_writer, w->writers_schema, datum);
              if (rval) {            
                  /* TODO: if the datum encoder larger than our buffer,
                     just write a single large datum */
                  return rval;       
              }
          }
          w->block_count++;          
          return 0;                  
      }
      

        Activity

        Jeremy Hinegardner created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Jeremy Hinegardner
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development