[PIG-3553] HadoopJobHistoryLoader fails to load job history on hadoop v 1.2

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.13.0
• Component/s: piggybank
• Assignee: Lohit Giri
• Reporter: Lohit Giri
• Created: 30/Oct/13 22:45
• Updated: 07/Jul/14 18:07
• Resolved: 10/Nov/13 06:21

Description

while following the example on pig wiki -

a = load '/mapred/history/done' using HadoopJobHistoryLoader() as (j:map[], m:map[], r:map[]);
b = foreach a generate (Chararray) j#'STATUS' as status, j#'PIG_SCRIPT_ID' as id, j#'USER' as user, j#'JOBNAME' as script_name, j#'JOBID' as job;
c = filter b by status != 'SUCCESS';
dump c;

when tried to parse /mapred/history logs, it fails to parse the job conf logs because in hadoop 1 the logs are stored as

job_201310301419_0001_conf.xml

rathar than as per the code says line number 203 in hadoopjobhistoryloader.java

Path parent = full.getParent();
String jobXml = jobDetails[0] + "" + jobDetails[1] + ""
+ jobDetails[2] + "" + jobDetails[3] + ""
+ jobDetails[4] + "_conf.xml";
Path p = new Path(parent, jobXml);

attaching the patch

Issue Links

• is related to: PIG-2894 [Piggybank] HadoopJobHistoryLoader for hadoop 0.20.205+