Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      In an effort to adapt Pig to work using Apache Tez (https://issues.apache.org/jira/browse/TEZ), I made some changes to allow for a cleaner ExecutionEngine abstraction than existed before. The changes are not that major as Pig was already relatively abstracted out between the frontend and backend. The changes in the attached commit are essentially the barebones changes – I tried to not change the structure of Pig's different components too much. I think it will be interesting to see in the future how we can refactor more areas of Pig to really honor this abstraction between the frontend and backend.

      Some of the changes was to reinstate an ExecutionEngine interface to tie together the front end and backend, and making the changes in Pig to delegate to the EE when necessary, and creating an MRExecutionEngine that implements this interface. Other work included changing ExecType to cycle through the ExecutionEngines on the classpath and select the appropriate one (this is done using Java ServiceLoader, exactly how MapReduce does for choosing the framework to use between local and distributed mode). Also I tried to make ScriptState, JobStats, and PigStats as abstract as possible in its current state. I think in the future some work will need to be done here to perhaps re-evaluate the usage of ScriptState and the responsibilities of the different statistics classes. I haven't touched the PPNL, but I think more abstraction is needed here, perhaps in a separate patch.

      1. execengine.patch
        62 kB
        Achal Soni
      2. test_suite.patch
        77 kB
        Achal Soni
      3. stats_scriptstate.patch
        157 kB
        Achal Soni
      4. mapreduce_execengine.patch
        130 kB
        Achal Soni
      5. test_failures.txt
        16 kB
        Cheolsoo Park
      6. updated-8-22-2013-exec-engine.patch
        364 kB
        Achal Soni
      7. updated-8-23-2013-exec-engine.patch
        394 kB
        Cheolsoo Park
      8. updated-8-27-2013-exec-engine.patch
        397 kB
        Cheolsoo Park
      9. updated-8-28-2013-exec-engine.patch
        398 kB
        Cheolsoo Park
      10. updated-8-29-2013-exec-engine.patch
        398 kB
        Cheolsoo Park

        Issue Links

          Activity

            People

            • Assignee:
              Achal Soni
              Reporter:
              Achal Soni
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development