[YARN-5546] NodeManager crashes due to SIGSEGV

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.6.0
• Component/s: nodemanager
• Assignee: Unassigned
• Reporter: Daniel Haviv
• Created: 21/Aug/16 13:21
• Updated: 31/Aug/16 11:10
• Resolved: 31/Aug/16 11:10

Description

NodeManager crash due to SIGSEGV.
hs_err includes the following java stack:
j org.fusesource.leveldbjni.internal.NativeDB$DBJNI.Put(JLorg/fusesource/leveldbjni/internal/NativeWriteOptions;Lorg/fusesource/leveldbjni/internal/NativeSlice;Lorg/fu
sesource/leveldbjni/internal/NativeSlice;)J+0
j org.fusesource.leveldbjni.internal.NativeDB.put(Lorg/fusesource/leveldbjni/internal/NativeWriteOptions;Lorg/fusesource/leveldbjni/internal/NativeSlice;Lorg/fusesourc
e/leveldbjni/internal/NativeSlice;)V+11
j org.fusesource.leveldbjni.internal.NativeDB.put(Lorg/fusesource/leveldbjni/internal/NativeWriteOptions;Lorg/fusesource/leveldbjni/internal/NativeBuffer;Lorg/fusesour
ce/leveldbjni/internal/NativeBuffer;)V+18
j org.fusesource.leveldbjni.internal.NativeDB.put(Lorg/fusesource/leveldbjni/internal/NativeWriteOptions;[B[B)V+36
j org.fusesource.leveldbjni.internal.JniDB.put([B[BLorg/iq80/leveldb/WriteOptions;)Lorg/iq80/leveldb/Snapshot;+28
j org.fusesource.leveldbjni.internal.JniDB.put([B[B)V+10
j org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.storeDeletionTask(ILorg/apache/hadoop/yarn/proto/YarnServerNodemanagerRecoveryProtos$De
letionServiceDeleteTaskProto;)V+32
j org.apache.hadoop.yarn.server.nodemanager.DeletionService.recordDeletionTaskInStateStore(Lorg/apache/hadoop/yarn/server/nodemanager/DeletionService$FileDeletionTask;
)V+245
j org.apache.hadoop.yarn.server.nodemanager.DeletionService.delete(Ljava/lang/String;Lorg/apache/hadoop/fs/Path;[Lorg/apache/hadoop/fs/Path;)V+44
j org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run()V+271
v ~StubRoutines::call_stub

and the culprit seems to be :

1. Problematic frame:
2. C [libleveldbjni-64-1-5625225739273738004.8+0x2aaac] leveldb::log::Writer::EmitPhysicalRecord(leveldb::log::RecordType, char const*, unsigned long)+0x7c