[KAFKA-4745] KafkaLZ4BlockOutputStream.java incorrectly finishes the last frame

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.10.1.1
• Fix Version/s: 0.10.1.1
• Component/s: compression
• Assignee: Unassigned
• Reporter: Will Droste
• Created: 08/Feb/17 19:24
• Updated: 09/Mar/17 01:02
• Resolved: 08/Feb/17 19:41

Description

There is a scenario where by the delegated OutputStream does not call flush before close there will be missing data in the stream. The reason for this is the stream is actually marked close before it is actually flushed.

The end mark is written before the flush, also the writeEndMark was finishing the stream so its redundant in this context to mark it finished. In my fork the 'finished=true' was removed from the 'writeEndMark' method.

    @Override
    public void close() throws IOException {
        if (!finished) {
            writeEndMark();
            flush();
            finished = true;
        }
        if (out != null) {
            out.close();
            out = null;
        }
    }

should be

    @Override
    public void close() throws IOException {
        if (!finished) {
            // finish any pending data
            writeBlock();
            // write out the end mark
            writeEndMark();
            // mark the stream as finished
            finished = true;
        }
        if (out != null) {
            out.close();
            out = null;
        }
    }

Issue Links

• links to: GitHub Pull Request #2528