Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3419

Pluggable Execution Engine

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.12.0
    • 0.13.0
    • None
    • None
    • Patch Available

    Description

      In an effort to adapt Pig to work using Apache Tez (https://issues.apache.org/jira/browse/TEZ), I made some changes to allow for a cleaner ExecutionEngine abstraction than existed before. The changes are not that major as Pig was already relatively abstracted out between the frontend and backend. The changes in the attached commit are essentially the barebones changes – I tried to not change the structure of Pig's different components too much. I think it will be interesting to see in the future how we can refactor more areas of Pig to really honor this abstraction between the frontend and backend.

      Some of the changes was to reinstate an ExecutionEngine interface to tie together the front end and backend, and making the changes in Pig to delegate to the EE when necessary, and creating an MRExecutionEngine that implements this interface. Other work included changing ExecType to cycle through the ExecutionEngines on the classpath and select the appropriate one (this is done using Java ServiceLoader, exactly how MapReduce does for choosing the framework to use between local and distributed mode). Also I tried to make ScriptState, JobStats, and PigStats as abstract as possible in its current state. I think in the future some work will need to be done here to perhaps re-evaluate the usage of ScriptState and the responsibilities of the different statistics classes. I haven't touched the PPNL, but I think more abstraction is needed here, perhaps in a separate patch.

      Attachments

        1. execengine.patch
          62 kB
          Achal Soni
        2. mapreduce_execengine.patch
          130 kB
          Achal Soni
        3. stats_scriptstate.patch
          157 kB
          Achal Soni
        4. test_failures.txt
          16 kB
          Cheolsoo Park
        5. test_suite.patch
          77 kB
          Achal Soni
        6. updated-8-22-2013-exec-engine.patch
          364 kB
          Achal Soni
        7. updated-8-23-2013-exec-engine.patch
          394 kB
          Cheolsoo Park
        8. updated-8-27-2013-exec-engine.patch
          397 kB
          Cheolsoo Park
        9. updated-8-28-2013-exec-engine.patch
          398 kB
          Cheolsoo Park
        10. updated-8-29-2013-exec-engine.patch
          398 kB
          Cheolsoo Park

        Issue Links

          Activity

            People

              achalsoni81 Achal Soni
              achalsoni81 Achal Soni
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: