[IMPALA-947] Impala caused datanode storm

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.2.4
• Fix Version/s: Product Backlog
• Labels: impala
• Environment: CDH 4.6, Impala 1.2.4, all from parcels
• Assignee: Matthew Jacobs
• Reporter: Sergey
• Created: 15/Apr/14 10:41
• Updated: 20/Dec/15 00:05
• Resolved: 25/Feb/15 18:34

Description

Hi, this morning we got "datanode" storm.
Suddenly impala started to read blocks from many datanodes. And we didn't have any running queries. When we restarted the whole impala service, storm stopped.
Please see networking when storm started (screenshot).
Affected datanodes had these log entries:

Previous Next
Host	Log Level	Time	Source	Message
prod-node0101.kyc.megafon.ru	INFO	15 апрель 2014 10:21	clienttrace	
src: /10.66.49.226:50010, dest: /10.66.49.225:38969, bytes: 132096, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_722537315_1, offset: 0, srvID: DS-1895691478-10.66.49.226-50010-1392796222466, blockid: BP-2086241135-10.66.49.155-1363767213272:blk_8166683644010943415_23956636, duration: 1006760
 View Log File
prod-node0101.kyc.megafon.ru	INFO	15 апрель 2014 10:21	clienttrace	
src: 127.0.0.1, dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_FDS, blockid: 132388256993022462, srvID: DS-1895691478-10.66.49.226-50010-1392796222466, success: true
 View Log File
prod-node0107.kyc.megafon.ru	INFO	15 апрель 2014 10:21	clienttrace	
src: 127.0.0.1, dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_FDS, blockid: -6297200201966372620, srvID: DS-1792190177-10.66.49.232-50010-1392796222893, success: true
 View Log File
prod-node0118.kyc.megafon.ru	INFO	15 апрель 2014 10:21	clienttrace	
src: /10.66.49.243:50010, dest: /10.66.49.232:54808, bytes: 132096, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_453101187_1, offset: 0, srvID: DS-1075945273-10.66.49.243-50010-1392796222416, blockid: BP-2086241135-10.66.49.155-1363767213272:blk_-5476067019874567999_23956435, duration: 1184010
 View Log File
prod-node0118.kyc.megafon.ru	INFO	15 апрель 2014 10:21	clienttrace	
src: /10.66.49.243:50010, dest: /10.66.49.244:36042, bytes: 132096, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-1114160638_1, offset: 0, srvID: DS-1075945273-10.66.49.243-50010-1392796222416, blockid: BP-2086241135-10.66.49.155-1363767213272:blk_1086074988829525783_23956481, duration: 1181419
 View Log File
prod-node0118.kyc.megafon.ru	INFO	15 апрель 2014 10:21	clienttrace	
src: 127.0.0.1, dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_FDS, blockid: -7620517135406691819, srvID: DS-1075945273-10.66.49.243-50010-1392796222416, success: true
 View Log File
prod-node0120.kyc.megafon.ru	INFO	15 апрель 2014 10:21	clienttrace	
src: /10.66.49.245:50010, dest: /10.66.49.224:43178, bytes: 132096, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-1991936378_1, offset: 0, srvID: DS-1789713258-10.66.49.245-50010-1392796222375, blockid: BP-2086241135-10.66.49.155-1363767213272:blk_1107624047476372504_23956836, duration: 1029509
 View Log File
prod-node0120.kyc.megafon.ru	INFO	15 апрель 2014 10:21	clienttrace	
src: /10.66.49.245:50010, dest: /10.66.62.89:42025, bytes: 132096, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-1092616849_1, offset: 0, srvID: DS-1789713258-10.66.49.245-50010-1392796222375, blockid: BP-2086241135-10.66.49.155-1363767213272:blk_8764079704706642093_23956571, duration: 1583802
 View Log File
prod-node028.kyc.megafon.ru	INFO	15 апрель 2014 10:21	clienttrace	
src: /10.66.49.183:50010, dest: /10.66.62.59:43322, bytes: 132096, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_-70190780_1, offset: 0, srvID: DS-1141421786-10.66.49.183-50010-1363767221252, blockid: BP-2086241135-10.66.49.155-1363767213272:blk_6336672533820089443_23956530, duration: 1421004
 View Log File

View Log File:

10:21:26.601	INFO	org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace	
src: /10.66.49.226:50010, dest: /10.66.49.225:38969, bytes: 132096, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_722537315_1, offset: 0, srvID: DS-1895691478-10.66.49.226-50010-1392796222466, blockid: BP-2086241135-10.66.49.155-1363767213272:blk_8166683644010943415_23956636, duration: 1006760
10:21:26.606	INFO	org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace	
src: 127.0.0.1, dest: 127.0.0.1, op: REQUEST_SHORT_CIRCUIT_FDS, blockid: 132388256993022462, srvID: DS-1895691478-10.66.49.226-50010-1392796222466, success: true
10:21:26.607	INFO	org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace	
src: /10.66.49.226:50010, dest: /10.66.49.241:33859, bytes: 132096, op: HDFS_READ, cliID: DFSClient_NONMAPREDUCE_1845184176_1, offset: 0, srvID: DS-1895691478-10.66.49.226-50010-1392796222466, blockid: BP-2086241135-10.66.49.155-1363767213272:blk_2940094362365288428_23956714, duration: 1005634

"dest" on other host is impala process (ps -ef, nestat -alnpt)