[IOTDB-1856] Improve raft log size check

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Core/Cluster
• Labels: distributed
• Assignee: Unassigned
• Reporter: Tian Jiang
• Created: 18/Oct/21 01:56
• Updated: 18/Oct/21 01:56

Description

To avoid log-append failures caused by oversized log entries, the size of each entry is checked to ensure that it will fit the flush buffer before it is appended to the local log. However, the current implementation has two drawbacks:

1. The size check falls in a synchronized block, which means concurrent requests cannot be checked in parallel, and other requests will be significantly blocked if the request is rather large.
2. The size check directly serializes an entry into a ByteBuffer and uses it to get the size of the entry, but the buffer is discarded immediately after the check, so both CPU computation and memory allocation are wasted during the serialization are wasted.

To resolve the two aforementioned problems, we need:

1. Move the size check out of the synchronized block.
2. Use a serialization-free way to get the size of an entry, or save the serialization result and reuse it during log persistency and log replication. If the second one is adopted, the serialization result should be freed in time to avoid a large memory footprint.