Pig
  1. Pig
  2. PIG-1241

Accumulator is turned on when a map is used with a non-accumulative UDF

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None

      Description

      Exception is thrown for a script like the following:

      register /homes/yinghe/owl/string.jar;
      a = load 'a.txt' as (id, url);
      b = group a by (id, url);
      c = foreach b generate COUNT(a), (CHARARRAY) string.URLPARSE(group.url)#'url';
      dump c;

      In this query, URLPARSE() is not accumulative, and it returns a map.

      The accumulator optimizer failed to check UDF in this case, and tries to run the job in accumulative mode. ClassCastException is thrown when trying to cast UDF into Accumulator interface.

        Activity

        Hide
        Ying He added a comment -

        patch to check UDF when it's with map operation

        Show
        Ying He added a comment - patch to check UDF when it's with map operation
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12436056/accum.patch
        against trunk revision 909921.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/212/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/212/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/212/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12436056/accum.patch against trunk revision 909921. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/212/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/212/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/212/console This message is automatically generated.
        Hide
        Olga Natkovich added a comment -

        Ying, I thought that accumlate is on by default but it looks like you are setting it on only if the property is present. Is this true?

        Show
        Olga Natkovich added a comment - Ying, I thought that accumlate is on by default but it looks like you are setting it on only if the property is present. Is this true?
        Hide
        Ying He added a comment -

        no, by default it is on.

        boolean isAccum = "true".equalsIgnoreCase(pc.getProperties().getProperty("opt.accumulator","true"));

        means if "opt.accumulator" is not present, the default value is "true"

        Show
        Ying He added a comment - no, by default it is on. boolean isAccum = "true".equalsIgnoreCase(pc.getProperties().getProperty("opt.accumulator","true")); means if "opt.accumulator" is not present, the default value is "true"
        Hide
        Olga Natkovich added a comment -

        patch committed to the trunk. Thanks, Ying

        Show
        Olga Natkovich added a comment - patch committed to the trunk. Thanks, Ying

          People

          • Assignee:
            Ying He
            Reporter:
            Ying He
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development