Details
-
Test
-
Status: Resolved
-
Trivial
-
Resolution: Fixed
-
3.2.1
-
None
-
-OS Version : CentOS Linux release 8.0.1905 (Core)
-Hadoop Version : hadoop-3.2.1
Description
I am testing the performance of erasure coding in Hadoop 3.2.1 version environment.
Apply ISA-L by referring to the manual (https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html)
▶ JOB LIST (Proceed in manual order)
- ISA-L library installed on all servers
ld ISA-L library : https://github.com/01org/isa-l - hadoop build source (hadoop-3.2.1-src.tar.gz / mvn package -Pdist,native -Dtar -Drequire.isal -Drequire.snappy)
- Deploy the built hadoop-3.2.1 folder to all servers
But In Test (File upload Time/Select Orc table in Hive)
There is no improvement in performance when tested.
Ask for advice on work
[Question 1]
Is there anything wrong with the installation? Are there any missing tasks?
[Question 2]
Why doesn't it speed up ? (File upload Time/Select Orc table in Hive)
[Question 3]
Whether to use ISA-L in Hadoop, How to check?
※ When I used the "hdfs ec -listCodecs" command, I expected to see ISA-L. But no
※ The warn log that occurs before applying ISA-L is It no longer occurs.
----------------------------------------------
WARN org.apache.hadoop.io.erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable
----------------------------------------------
▶Reference information
=======================================
-CPU INFO (namenode :2 / datanode : 5)
---------------------------------------------
namenode cpu : Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz
datanode cpu : Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
---------------------------------------------
-hadoop checknative (After source build)
---------------------------------------------
Native library checking:
hadoop: true /nds/hadoop-3.2.1/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
zstd : false
snappy: true /lib64/libsnappy.so.1
lz4: true revision:10301
bzip2: false
openssl: true /lib64/libcrypto.so
ISA-L: true /lib64/libisal.so.2
---------------------------------------------
-hdfs ec -listCodecs
Erasure Coding Codecs: Codec [Coder List]
RS [RS_NATIVE, RS_JAVA]
RS-LEGACY [RS-LEGACY_JAVA]
XOR [XOR_NATIVE, XOR_JAVA]
=======================================