Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.2.0
-
None
-
None
-
None
-
Hadoop 0.18.3 on redhat, PIG svn from feb-01
Description
When using the DISTINCT function many of the map tasks are being killed because of failure to report for 600 seconds. It seems that PIG-646 should have addressed this but I'm still seeing many errors like this:
2009-02-21 11:41:53,916 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2009-02-21 11:41:57,727 WARN org.apache.pig.builtin.Distinct$Intermediate: No reporter object provided to UDF org.apache.pig.builtin.Distinct$Intermediate
2009-02-21 11:41:57,730 WARN org.apache.pig.builtin.Distinct$Intermediate: No reporter object provided to UDF org.apache.pig.builtin.Distinct$Intermediate
My query:
r0 = load 'domain-org/*' as (domain:chararray, org:chararray);
r3 = GROUP r0 BY org parallel 18;
r4 = FOREACH r3 {
r5 = r0.domain;
r6 = distinct r5;
GENERATE group as org, COUNT(r6) as domains;
}
store r4 into 'org-domain-count';
the source files are 21GB in total with some 800M lines, 60M distinct domains and 80K distinct orgs. Some orgs have 50M domains in them.