[TAJO-406] PullServer occasionally causes could not find xxx in any of the configured local directories

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Fix Version/s: 0.8.0
• Component/s: Data Shuffle
• Assignee: Unassigned
• Reporter: Seungun Choe
• Created: 10/Dec/13 05:51
• Updated: 26/Feb/14 04:32
• Resolved: 26/Feb/14 04:32

Description

PullServer causes occasionally the following errors:

2013-12-10 14:45:43,344 INFO  pullserver.TajoPullServerService (TajoPullServerService.java:messageReceived(390)) - PullServer request param: repartitionType=h, sid=1, partitionId=39, taskIds=[12336_0,5791_0,6171_0,14315_0,6079_0,3862_0,6378_0,12672_0,5329_0,5702_0,12266_0,10184_0,14652_0,3676_0,5423_0,12529_0,14692_0,14758_0,5829_0,4813_0,12900_0,11795_0,6077_0,13092_0,6180_0,14620_0,5748_0,6008_0,6007_0,6309_0,5901_0,14584_0,12533_0,12933_0,6133_0,5385_0,14741_0,6411_0,10079_0,6090_0,12789_0,5709_0,13072_0,5970_0,14753_0,6373_0,5757_0,14395_0,14227_0,14694_0,11439_0,5060_0,12072_0,6247_0,14446_0,14206_0,5848_0,12473_0,5838_0,12654_0,14859_0,6319_0,6136_0,6156_0,13208_0,2896_0,6398_0,11248_0,14443_0,13246_0,5573_0,14686_0,6049_0,5465_0,5442_0,5802_0,14801_0,5839_0,5591_0,5778_0,5734_0,13102_0,5941_0,12883_0,6463_0,5825_0,13201_0,14150_0,11779_0,11271_0,14707_0,14790_0,5090_0,6310_0,14258_0,12468_0,5666_0,5721_0,6387_0,3928_0,5181_0,13088_0,6218_0,5998_0,13888_0,14240_0,12130_0,14441_0,11580_0,5973_0,3647_0,5868_0,6472_0,11269_0,12839_0,5953_0,12565_0,14773_0,12344_0,13527_0,11703_0,4666_0,14711_0,12127_0,2577_0,12116_0,4663_0,3100_0,5859_0,3246_0,5945_0,13242_0,5707_0,12395_0,6441_0]
2013-12-10 14:45:43,344 INFO  pullserver.TajoPullServerService (TajoPullServerService.java:messageReceived(396)) - PullServer baseDir: /data1/tmp/tajo-localdir,/data2/tmp/tajo-localdir,/data3/tmp/tajo-localdir,/data4/tmp/tajo-localdir,/data5/tmp/tajo-localdir,/data6/tmp/tajo-localdir/q_1386652478411_0002/output
2013-12-10 14:45:43,347 ERROR pullserver.TajoPullServerService (TajoPullServerService.java:exceptionCaught(536)) - PullServer error:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find q_1386652478411_0002/output/1/12266_0/output/39 in any of the configured local directories
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:445)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:164)
        at org.apache.tajo.pullserver.TajoPullServerService$PullServer.messageReceived(TajoPullServerService.java:423)