Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Implemented
-
None
-
None
Description
Add "-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true" to the ozone script to allow Netty unsafe memory allocation.
According to Netty doc.
When a new java.nio.ByteBuffer is allocated, its content is filled with zeroes. This "zeroing" consumes CPU cycles and memory bandwidth. Normally, the buffer is then immediately filled from some data source, so the zeroing did no good.
And Netty has an optimization to avoid this zero assignment, its enablement is controlled by a flag: PlatformDependent.USE_DIRECT_BUFFER_NO_CLEANER. When this is enabled netty does the following to allocate a DirectByteBuffer:
1. Call "Unsafe.allocateMemory" to allocate the direct memory. This returns an address of the allocated buffer.
2. Use reflection to call a private constructor "DirectByteBuffer(long addr, int cap)" to create the DirectByteBuffer objects. This avoids zeroing the created buffer, as opposed to the public path "DirectByteBuffer(int cap)".
However, the flag USE_DIRECT_BUFFER_NO_CLEANER is disabled by default in Java 9+ because the reflection in step #2 is disabled by default in Java 9+. This is controlled by a flag "io.netty.tryReflectionSetAccessible".
The result is today, the zeroing takes a large portion of Netty allocation when Ozone runs on Java 9+.
Setting the flag "-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true" helps Netty leverage the Unsafe allocation and avoid the zeroing cost.
Setting "io.netty.tryReflectionSetAccessible" is a common practice in other distributed systems, such as HBase, Spark... see: https://github.com/search?q=io.netty.tryReflectionSetAccessible&type=code
A good side-effect of using Netty unsafe allocation is that the used direct memory is exposed as a metric, see HDDS-9070.
Attachments
Attachments
Issue Links
- relates to
-
-
- Resolved
-
- links to