Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-1178

LogicalPlan and Optimizer are too complex and hard to work with

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.8.0
    • None
    • None
    • Reviewed

    Description

      The current implementation of the logical plan and the logical optimizer in Pig has proven to not be easily extensible. Developer feedback has indicated that adding new rules to the optimizer is quite burdensome. In addition, the logical plan has been an area of numerous bugs, many of which have been difficult to fix. Developers also feel that the logical plan is difficult to understand and maintain. The root cause for these issues is that a number of design decisions that were made as part of the 0.2 rewrite of the front end have now proven to be sub-optimal. The heart of this proposal is to revisit a number of those proposals and rebuild the logical plan with a simpler design that will make it much easier to maintain the logical plan as well as extend the logical optimizer.

      See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full details.

      Attachments

        1. lp.patch
          108 kB
          Ying He
        2. PIG_1178.patch
          110 kB
          Ying He
        3. expressions.patch
          36 kB
          Alan Gates
        4. expressions-2.patch
          37 kB
          Alan Gates
        5. lp.patch
          84 kB
          Ying He
        6. pig_1178.patch
          304 kB
          Ankit Modi
        7. pig_1178.patch
          435 kB
          Ankit Modi
        8. pig_1178_2.patch
          180 kB
          Ankit Modi
        9. pig_1178_3.patch
          342 kB
          Ankit Modi
        10. pig_1178_3.2.patch
          342 kB
          Daniel Dai
        11. pig_1178_3.3.patch
          342 kB
          Daniel Dai
        12. pig_1178_3.4.patch
          86 kB
          Daniel Dai
        13. PIG-1178-4.patch
          522 kB
          Daniel Dai
        14. PIG-1178-5.patch
          1.70 MB
          Daniel Dai
        15. PIG-1178-6.patch
          287 kB
          Daniel Dai
        16. PIG-1178-7.patch
          239 kB
          Daniel Dai
        17. PIG-1178-8.patch
          8 kB
          Daniel Dai
        18. PIG-1178-9.patch
          2 kB
          Daniel Dai
        19. PIG-1178-10.patch
          23 kB
          Daniel Dai
        20. PIG-1178-11.patch
          44 kB
          Daniel Dai

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. PIG-1319 New logical optimization rules

          • Major - Major loss of function.
          • Open

          Activity

            People

              daijy Daniel Dai
              gates Alan Gates
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: