[HDFS-11841] libc.so.6 detected double free on TestDFSIO write when erasure coding and enabled Intel ISA-L

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Component/s: erasure-coding, native

Environment

cluster: 3 nodes
os:(Red Hat 2.6.33.20, Red Hat 3.10.0-514.6.1.el7.x86_64, Ubuntu4.4.0-31-generic)
hadoop version: hadoop-3.0.0-alpha3
erasure coding: XOR-2-1-64k and enabled Intel ISA-L
TestDFSIO: hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar TestDFSIO -write -nrFiles 10000 -size 1MB

• Assignee: Unassigned
• Reporter: liaoyuxiangqin
• Created: 17/May/17 10:01
• Updated: 25/Jan/18 08:00
• Resolved: 25/Jan/18 08:00

Description

when i execute hadoop jar hadoop-mapreduce-client-jobclient-3.0.0-alpha3-SNAPSHOT-tests.jar TestDFSIO -write -nrFiles 10000 -size 1MB on above environment, glibc detected double free or corruption, detail information as follows:

{pane}

File System Counters
FILE: Number of bytes read=489391103089
FILE: Number of bytes written=42020064724
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=5751218285
HDFS: Number of bytes written=52456018688978
HDFS: Number of read operations=150165017
HDFS: Number of large read operations=0
HDFS: Number of write operations=150065006
Map-Reduce Framework
Map input records=10000
Map output records=50000
Map output bytes=707116
Map output materialized bytes=867116
Input split bytes=1288890
Combine input records=0
Combine output records=0
Reduce input groups=5
Reduce shuffle bytes=867116
Reduce input records=50000
Reduce output records=5
Spilled Records=100000
Shuffled Maps =10000
Failed Shuffles=0
Merged Map outputs=10000
GC time elapsed (ms)=34575
Total committed heap usage (bytes)=46386259689472
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=1148890
File Output Format Counters
Bytes Written=88
glibc detected /home/hadoop/jdk1.8.0_111/bin/java: double free or corruption (out): 0x00007fcd84a6c6c0
======= Backtrace: =========
/lib64/libc.so.6[0x3d1fe724a6]
/lib64/libc.so.6(cfree+0x6c)[0x3d1fe771bc]
[0x7fcdb138e8e6]
======= Memory map: ========
00400000-00401000 r-xp 00000000 08:01 3720754 /home/hadoop/jdk1.8.0_111/bin/java
00600000-00601000 rw-p 00000000 08:01 3720754 /home/hadoop/jdk1.8.0_111/bin/java
0204f000-04600000 rw-p 00000000 00:00 0 [heap]
649400000-6f2100000 rw-p 00000000 00:00 0
6f2100000-743180000 ---p 00000000 00:00 0
743180000-7c0000000 rw-p 00000000 00:00 0
7c0000000-7c03c0000 rw-p 00000000 00:00 0
7c03c0000-800000000 ---p 00000000 00:00 0
3d1fa00000-3d1fa21000 r-xp 00000000 08:01 8126508 /lib64/ld-2.14.1.so
3d1fc20000-3d1fc21000 r--p 00020000 08:01 8126508 /lib64/ld-2.14.1.so
3d1fc21000-3d1fc23000 rw-p 00021000 08:01 8126508 /lib64/ld-2.14.1.so
3d1fe00000-3d1ff83000 r-xp 00000000 08:01 8126562 /lib64/libc-2.14.1.so
3d1ff83000-3d20182000 ---p 00183000 08:01 8126562 /lib64/libc-2.14.1.so
/home/hadoop/jdk1.8.0_111/jre/lib/jsse.jar
7fcd9260a000-7fcd9260f000 r--s 00044000 08:01 3852117 /home/gy/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.0.0-alpha3-SNAPSHOT.jar
7fcd9260f000-7fcd92612000 r--s 0001f000 08:01 3852191 /home/gy/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-registry-3.0.0-alpha3-SNAPSHOT.jarAborted

Issue Links

• is duplicated by: HDFS-12606 When using native decoder, DFSStripedStream#close crashes JVM after being called multiple times.