Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      In an effort to adapt Pig to work using Apache Tez (https://issues.apache.org/jira/browse/TEZ), I made some changes to allow for a cleaner ExecutionEngine abstraction than existed before. The changes are not that major as Pig was already relatively abstracted out between the frontend and backend. The changes in the attached commit are essentially the barebones changes – I tried to not change the structure of Pig's different components too much. I think it will be interesting to see in the future how we can refactor more areas of Pig to really honor this abstraction between the frontend and backend.

      Some of the changes was to reinstate an ExecutionEngine interface to tie together the front end and backend, and making the changes in Pig to delegate to the EE when necessary, and creating an MRExecutionEngine that implements this interface. Other work included changing ExecType to cycle through the ExecutionEngines on the classpath and select the appropriate one (this is done using Java ServiceLoader, exactly how MapReduce does for choosing the framework to use between local and distributed mode). Also I tried to make ScriptState, JobStats, and PigStats as abstract as possible in its current state. I think in the future some work will need to be done here to perhaps re-evaluate the usage of ScriptState and the responsibilities of the different statistics classes. I haven't touched the PPNL, but I think more abstraction is needed here, perhaps in a separate patch.

      1. execengine.patch
        62 kB
        Achal Soni
      2. mapreduce_execengine.patch
        130 kB
        Achal Soni
      3. stats_scriptstate.patch
        157 kB
        Achal Soni
      4. test_failures.txt
        16 kB
        Cheolsoo Park
      5. test_suite.patch
        77 kB
        Achal Soni
      6. updated-8-22-2013-exec-engine.patch
        364 kB
        Achal Soni
      7. updated-8-23-2013-exec-engine.patch
        394 kB
        Cheolsoo Park
      8. updated-8-27-2013-exec-engine.patch
        397 kB
        Cheolsoo Park
      9. updated-8-28-2013-exec-engine.patch
        398 kB
        Cheolsoo Park
      10. updated-8-29-2013-exec-engine.patch
        398 kB
        Cheolsoo Park

        Issue Links

          Activity

          Daniel Dai made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Cheolsoo Park made changes -
          Fix Version/s 0.13.0 [ 12324971 ]
          Fix Version/s 0.12.0 [ 12323380 ]
          Cheolsoo Park made changes -
          Link This issue relates to PIG-3464 [ PIG-3464 ]
          Rohini Palaniswamy made changes -
          Link This issue breaks PIG-3457 [ PIG-3457 ]
          Pradeep Gollakota made changes -
          Link This issue relates to PIG-3453 [ PIG-3453 ]
          Cheolsoo Park made changes -
          Link This issue is required by PIG-3446 [ PIG-3446 ]
          Cheolsoo Park made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.12 [ 12323380 ]
          Resolution Fixed [ 1 ]
          Cheolsoo Park made changes -
          Attachment updated-8-29-2013-exec-engine.patch [ 12600638 ]
          Cheolsoo Park made changes -
          Attachment updated-8-28-2013-exec-engine.patch [ 12600353 ]
          Cheolsoo Park made changes -
          Attachment updated-8-27-2013-exec-engine.patch [ 12600184 ]
          Cheolsoo Park made changes -
          Attachment updated-8-27-2013-exec-engine.patch [ 12600273 ]
          Cheolsoo Park made changes -
          Attachment updated-8-27-2013-exec-engine.patch [ 12600184 ]
          Cheolsoo Park made changes -
          Attachment updated-8-27-2013-exec-engine.patch [ 12600183 ]
          Cheolsoo Park made changes -
          Attachment updated-8-27-2013-exec-engine.patch [ 12600183 ]
          Cheolsoo Park made changes -
          Attachment updated-8-23-2013-exec-engine.patch [ 12599781 ]
          Achal Soni made changes -
          Attachment updated-8-22-2013-exec-engine.patch [ 12599540 ]
          Achal Soni made changes -
          Attachment updated-8-22-2013-exec-engine.patch [ 12599531 ]
          Achal Soni made changes -
          Attachment updated-8-22-2013-exec-engine.patch [ 12599531 ]
          Achal Soni made changes -
          Attachment updated-8-22-2013-exec-engine.patch [ 12599516 ]
          Achal Soni made changes -
          Attachment updated-8-22-2013-exec-engine.patch [ 12599516 ]
          Achal Soni made changes -
          Attachment finalpatch.patch [ 12599278 ]
          Achal Soni made changes -
          Attachment finalpatch.patch [ 12599515 ]
          Achal Soni made changes -
          Attachment finalpatch.patch [ 12599515 ]
          Cheolsoo Park made changes -
          Attachment test_failures.txt [ 12599438 ]
          Achal Soni made changes -
          Attachment finalpatch.patch [ 12599278 ]
          Cheolsoo Park made changes -
          Assignee Achal Soni [ achalsoni81 ]
          Achal Soni made changes -
          Attachment pluggable_execengine.patch [ 12597517 ]
          Achal Soni made changes -
          Attachment execengine.patch [ 12598029 ]
          Attachment test_suite.patch [ 12598030 ]
          Attachment stats_scriptstate.patch [ 12598031 ]
          Attachment mapreduce_execengine.patch [ 12598032 ]
          Achal Soni made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Achal Soni made changes -
          Field Original Value New Value
          Attachment pluggable_execengine.patch [ 12597517 ]
          Achal Soni created issue -

            People

            • Assignee:
              Achal Soni
              Reporter:
              Achal Soni
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development