Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      PigPen has been found to be a usable feature. The underlying PIG functionality, ILLUSTRATE, however, has not been stable and complete. It also has unique access paths that are not shared by the mainstream PIG call paths, which makes it hard if not impossible to maintain as the PIG evolves along.

      The purpose of this work is to use the common access paths yet still follow the performance-sensitive criteria for ILLUSTRATE, complete support for as many logical operators as theoretically possible,
      plus algorithm polishes and bug fixes as necessary. Details can be found in http://wiki.apache.org/pig/PigIllustrate

      Pig-366 has been for the PigPen in general, and was used for the previous ILLUSTRATE work too. With this JIRA, the work on ILLUSTRATE will be separated.

      This JIRA also serves as an umbrella for existing issues in ILLUSTRATE. Specifically, PIG-502, PIG-903, PIG-1066 should all fold into this issue.

      1. illustrator_9.patch
        44 kB
        Yan Zhou
      2. illustrator_8.patch
        146 kB
        Yan Zhou
      3. illustrator_7.patch
        139 kB
        Yan Zhou
      4. illustrator_6.patch
        397 kB
        Yan Zhou
      5. illustrator_5.patch
        383 kB
        Yan Zhou
      6. illustrator_4.patch
        364 kB
        Yan Zhou
      7. illustrator_3.patch
        325 kB
        Yan Zhou
      8. illustrator_2.patch
        318 kB
        Yan Zhou
      9. illustrator_1.patch
        308 kB
        Yan Zhou

        Issue Links

          Activity

          Yan Zhou created issue -
          Hide
          Yan Zhou added a comment -

          illustrator_1 has the local Map/Reduce simulator and all illustrator-specific method in Physical Operators.

          Currently, LOAD, FILTER, UNION, SORT, COGROUP are working.

          JOIN is not quite working yet as there are more than necessary rows for the illustration purpose.

          LIMIT, FOREACH, DISTINCT, nested generate block in FOREACH, CROSS, script illustration are not working yet.

          The manual Hudson results are as follows:

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 21 new or modified tests.
          [exec]
          [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] -1 release audit. The applied patch generated 495 release audit warnings (more than the trunk's current 461 warnings).

          The release audit warnings are all on html files.

          The javadoc warnings seem to be unrelated to this patch.

          Show
          Yan Zhou added a comment - illustrator_1 has the local Map/Reduce simulator and all illustrator-specific method in Physical Operators. Currently, LOAD, FILTER, UNION, SORT, COGROUP are working. JOIN is not quite working yet as there are more than necessary rows for the illustration purpose. LIMIT, FOREACH, DISTINCT, nested generate block in FOREACH, CROSS, script illustration are not working yet. The manual Hudson results are as follows: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 21 new or modified tests. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 495 release audit warnings (more than the trunk's current 461 warnings). The release audit warnings are all on html files. The javadoc warnings seem to be unrelated to this patch.
          Yan Zhou made changes -
          Field Original Value New Value
          Attachment illustrator_1.patch [ 12459122 ]
          Hide
          Yan Zhou added a comment -

          illustrator_2.patch:

          Addition of JOIN input pruning, plus some bug fixes.

          Show
          Yan Zhou added a comment - illustrator_2.patch: Addition of JOIN input pruning, plus some bug fixes.
          Yan Zhou made changes -
          Attachment illustrator_2.patch [ 12459267 ]
          Hide
          Yan Zhou added a comment -

          Support for operators of logic subexpression, foreach, load, sort, plus bug fixes.

          Show
          Yan Zhou added a comment - Support for operators of logic subexpression, foreach, load, sort, plus bug fixes.
          Yan Zhou made changes -
          Attachment illustrator_3.patch [ 12459709 ]
          Hide
          Yan Zhou added a comment -

          LIMIT, DISTINCT, Aggregation, SPLIT support, plus bug fixes.

          Show
          Yan Zhou added a comment - LIMIT, DISTINCT, Aggregation, SPLIT support, plus bug fixes.
          Yan Zhou made changes -
          Attachment illustrator_4.patch [ 12464888 ]
          Hide
          Yan Zhou added a comment -

          augmentation for DISTINCT, script support, code cleanups;

          Show
          Yan Zhou added a comment - augmentation for DISTINCT, script support, code cleanups;
          Yan Zhou made changes -
          Attachment illustrator_5.patch [ 12465392 ]
          Hide
          Yan Zhou added a comment -

          code cleanups, scalar fix, multi-query fix.

          Show
          Yan Zhou added a comment - code cleanups, scalar fix, multi-query fix.
          Yan Zhou made changes -
          Attachment illustrator_6.patch [ 12466152 ]
          Hide
          Richard Ding added a comment -

          +1. The patch looks good. It fixed various issues about ILLUSTRATE command. It also added new feature that allows ILLUSTRATE of Pig Latin scripts:

          illustrate -script <script name>
          

          It still needs to remove the dependency on the old logical operators in a late date since those operators will be deprecated in the 0.9 release.

          Please commit once all tests pass.

          Show
          Richard Ding added a comment - +1. The patch looks good. It fixed various issues about ILLUSTRATE command. It also added new feature that allows ILLUSTRATE of Pig Latin scripts: illustrate -script <script name> It still needs to remove the dependency on the old logical operators in a late date since those operators will be deprecated in the 0.9 release. Please commit once all tests pass.
          Hide
          Yan Zhou added a comment -

          illustrator_6.patch committed to the trunk. It passes test-patch and test-core.

          Show
          Yan Zhou added a comment - illustrator_6.patch committed to the trunk. It passes test-patch and test-core.
          Hide
          Yan Zhou added a comment -

          Use of new logical operator, plus bug fixes.

          Show
          Yan Zhou added a comment - Use of new logical operator, plus bug fixes.
          Yan Zhou made changes -
          Attachment illustrator_7.patch [ 12466471 ]
          Hide
          Yan Zhou added a comment -

          some code cleanup.

          Show
          Yan Zhou added a comment - some code cleanup.
          Yan Zhou made changes -
          Attachment illustrator_8.patch [ 12466496 ]
          Hide
          Yan Zhou added a comment -

          test-patch and test-core pass cleanly.

          Show
          Yan Zhou added a comment - test-patch and test-core pass cleanly.
          Hide
          Richard Ding added a comment -

          +1

          Show
          Richard Ding added a comment - +1
          Hide
          Yan Zhou added a comment -

          illustrator_8 committed to the trunk.

          Show
          Yan Zhou added a comment - illustrator_8 committed to the trunk.
          Hide
          Yan Zhou added a comment -

          Join augmentation added, code cleanups.

          Show
          Yan Zhou added a comment - Join augmentation added, code cleanups.
          Yan Zhou made changes -
          Attachment illustrator_9.patch [ 12466903 ]
          Hide
          Richard Ding added a comment -

          +1

          Show
          Richard Ding added a comment - +1
          Hide
          Yan Zhou added a comment -

          illustrator_9 committed to the trunk.

          Show
          Yan Zhou added a comment - illustrator_9 committed to the trunk.
          Hide
          Olga Natkovich added a comment -

          major work is done here. We will track discovered bugs separately

          Show
          Olga Natkovich added a comment - major work is done here. We will track discovered bugs separately
          Olga Natkovich made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Olga Natkovich made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jeff Hammerbacher made changes -
          Link This issue is related to PIG-59 [ PIG-59 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          70d 7h 49m 1 Olga Natkovich 18/Jan/11 02:26
          Resolved Resolved Closed Closed
          197d 22h 8m 1 Olga Natkovich 04/Aug/11 01:34

            People

            • Assignee:
              Yan Zhou
              Reporter:
              Yan Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development