1. Pig
  2. PIG-3898

Refactor PPNL for non-MR execution engine


    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch, 0.13.0
    • Component/s: None
    • Labels:


      Currently, PPNL assumes the MR plan, and thus, it's not compatible with non-MR execution engine. To support non-MR execution engines, I propose we changed initialPlanNotification() method as follows-

      public void initialPlanNotification(String scriptId, MROperPlan plan);
      public void initialPlanNotification(String scriptId, OperatorPlan<?> plan);

      Since MROperPlan and TezOperPlan are a subclass of OperatorPlan, this method can take both plans. In addition, if we add a new execution engine in the future, it won't break the interface again as long as we build the operator plan as a subclass of OperatorPlan.

      With this approach, applications such as Ambrose / Lipstick should be able to dynamically cast OperatorPlan to a concrete subclass depending on the ExecType.

      One disadvantage is that this isn't backward compatible with Pig 0.12 and older. But it only requires minor changes, and backward compatibility will be broken one time only.

      I also considered an alternative approach, for example, adding a new PPNL for Tez. But this approach has two problems.

      1. Pig registers PPNL via the Main function, and right now, only one PPNL can be registered. So having more than one PPNLs requires quite a few code changes in Main, ScriptState, and so on.
      2. Multiple PPNL interfaces mean multiple PPNL implementations. This results in more (duplicate) code in applications such as Ambrose / Lipstick.
      1. PIG-3898-1-tez.patch
        52 kB
        Cheolsoo Park
      2. PIG-3898-1-trunk.patch
        42 kB
        Cheolsoo Park


        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Cheolsoo Park made changes -
        Status In Progress [ 3 ] Resolved [ 5 ]
        Fix Version/s tez-branch [ 12324968 ]
        Resolution Fixed [ 1 ]
        Cheolsoo Park made changes -
        Status Open [ 1 ] In Progress [ 3 ]
        Cheolsoo Park made changes -
        Field Original Value New Value
        Attachment PIG-3898-1-trunk.patch [ 12640560 ]
        Attachment PIG-3898-1-tez.patch [ 12640561 ]
        Cheolsoo Park created issue -


          • Assignee:
            Cheolsoo Park
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            5 Start watching this issue


            • Created: