[AVRO-3183] Do Not Double Buffer Data in DataFileWriter

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.10.0
• Fix Version/s: 1.11.0
• Component/s: java
• Assignee: David Mollitor
• Reporter: David Mollitor
• Created: 04/Aug/21 17:15
• Updated: 06/Aug/21 13:45
• Resolved: 06/Aug/21 13:45

Description

DataFileWriter.java

  private void init(OutputStream outs) throws IOException {
    this.underlyingStream = outs;
    this.out = new BufferedFileOutputStream(outs);
    EncoderFactory efactory = new EncoderFactory();
    // binaryEncoder returns a buffered Encoder and is wrapping a BufferedFileOutputStream
    this.vout = efactory.binaryEncoder(out, null);
    dout.setSchema(schema);
    buffer = new NonCopyingByteArrayOutputStream(Math.min((int) (syncInterval * 1.25), Integer.MAX_VALUE / 2 - 1));
    // binaryEncoder returns a buffered Encoder and is wrapping a NonCopyingByteArrayOutputStream
    this.bufOut = efactory.binaryEncoder(buffer, null);
    if (this.codec == null) {
      this.codec = CodecFactory.nullCodec().createInstance();
    }
    this.isOpen = true;
  }

The FileWriter is double-buffering the output which just adds redundant overhead and truthfully the buffering offered by the object returned by binaryEncoder is a bit simplistic and does not do as good of a job as the buffering in BufferedFileOutputStream.

Remove this double buffering by using a 'direct' binaryEncoder