[AVRO-753] Java: Improve BinaryEncoder Performance - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Blocker
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.5.0
Component/s: java
Labels:
None

Hadoop Flags:

Incompatible change
Release Note:

Hide
The Encoder API has several resulting changes:
    * Construction and configuration is handled by EncoderFactory. All
      Constructors are hidden, and Encoder.init(OutputStream) is removed.
    * Some Encoders previously did not buffer output. Users must call
      Encoder.flush() to ensure output is written unless the EncoderFactory
      method used to construct an instance explicitly states that the Encoder
      does not buffer output.

Show
The Encoder API has several resulting changes:     * Construction and configuration is handled by EncoderFactory. All       Constructors are hidden, and Encoder.init(OutputStream) is removed.     * Some Encoders previously did not buffer output. Users must call       Encoder.flush() to ensure output is written unless the EncoderFactory       method used to construct an instance explicitly states that the Encoder       does not buffer output.

Description

BinaryEncoder has not had a performance improvement pass like BinaryDecoder did. It still mostly writes directly to the underlying OutputStream which is not optimal for performance. I like to use a rule that if you are writing to an OutputStream or reading from an InputStream in chunks smaller than 128 bytes, you have a performance problem.

Measurements indicate that optimizing BinaryEncoder yields a 2.5x to 6x performance improvement. The process is significantly simpler than BinaryDecoder because 'pushing' is easier than 'pulling' – and also because we do not need a 'direct' variant because BinaryEncoder already buffers sometimes.