Pig
  1. Pig
  2. PIG-1603

dependency created by 'relation as scalar' not captured in graph

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      The LogicalOperator that has a ReadScalar udf has a dependency on the relation that is provides the input to scalar variables. But this is not captured in the graph representation, and as a result DependencyOrderWalker does not traverse the graph in the real dependency order.

      The testcase TestFRJoin2.testConcatenateJobForScalar3 fails as a result of this issue. (It has been commented out for now.)

      1. PIG-1603.2.patch
        71 kB
        Thejas M Nair
      2. PIG-1603.1.patch
        61 kB
        Thejas M Nair

        Activity

        Hide
        Thejas M Nair added a comment -

        PIG-1603.1.patch
        Patch adds a new Logical operator LOScalar . A LogicalOperator that is using a relation (say X) as scalar in its inner plan will have a LOScalar added as a predecessor which also have the relation X as a predecessor. Visitors have been changed to handle this new logical operator.
        The test case TestFRJoin2.testConcatenateJobForScalar3 has been enabled.

        Results of test-patch -
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 9 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] -1 release audit. The applied patch generated 463 release audit warnings (more than the trunk's current 454 warnings).

        Release audit warnings are all related to javadoc file diffs.

        Show
        Thejas M Nair added a comment - PIG-1603 .1.patch Patch adds a new Logical operator LOScalar . A LogicalOperator that is using a relation (say X) as scalar in its inner plan will have a LOScalar added as a predecessor which also have the relation X as a predecessor. Visitors have been changed to handle this new logical operator. The test case TestFRJoin2.testConcatenateJobForScalar3 has been enabled. Results of test-patch - [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 9 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 463 release audit warnings (more than the trunk's current 454 warnings). Release audit warnings are all related to javadoc file diffs.
        Hide
        Thejas M Nair added a comment -

        Daniel pointed out that changes are required in more Visitors. Attaching new patch with the changes. Changes have not yet been in PruneColumns, so it might not work with -Dpig.usenewlogicalplan=false .

        Show
        Thejas M Nair added a comment - Daniel pointed out that changes are required in more Visitors. Attaching new patch with the changes. Changes have not yet been in PruneColumns, so it might not work with -Dpig.usenewlogicalplan=false .
        Hide
        Thejas M Nair added a comment -

        This bug has been resolved in PIG-1605 .

        Show
        Thejas M Nair added a comment - This bug has been resolved in PIG-1605 .

          People

          • Assignee:
            Thejas M Nair
            Reporter:
            Thejas M Nair
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development