Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: spark-branch
    • Component/s: spark
    • Labels:
      None

      Description

      in current code, there are following class in mapreduce and tez package.
      mapreduce:
      MRCompiler
      MROperPlan MROpPlanVisitor
      MapReduceOper

      tez:
      TezCompiler
      TezOperPlan TezOpPlanVisitor
      TezOperator

      Following class needs to be added in spark package:
      SparkCompiler
      SparkOperPlan SparkOpPlanVisitor
      SparkOperator

      Current code need to be refactored because of above adding classes. Some unit tests like TestStoreInstances fail because of it.

      following are the error of unit test TestStoreInstances:
      Error Message

      num jobs expected:<1> but was:<0>
      Stacktrace

      junit.framework.AssertionFailedError: num jobs expected:<1> but was:<0>
      at org.apache.pig.test.TestStoreInstances.testBackendStoreCommunication(TestStoreInstances.java:122)

        Attachments

        1. How to change a physical plan to a spark plan.docx
          22 kB
          liyunzhang
        2. jenkins_PIG-4374_1_patch.png.png
          474 kB
          liyunzhang
        3. PIG-4374_1.patch
          75 kB
          liyunzhang
        4. PIG-4374_2.patch.1
          78 kB
          liyunzhang
        5. PIG-4374_3.patch.1
          76 kB
          liyunzhang
        6. PIG-4374_4.patch
          92 kB
          liyunzhang
        7. PIG-4374_4.patch_Jenkins_result.png
          364 kB
          liyunzhang
        8. PIG-4374_6_patch_jekins.png
          371 kB
          liyunzhang
        9. PIG-4374_6.patch
          76 kB
          liyunzhang
        10. PIG-4374_7.patch
          117 kB
          liyunzhang
        11. PIG-4374_8.patch.jenkins.png
          252 kB
          liyunzhang
        12. Pig-spark #76 [Jenkins .png
          261 kB
          liyunzhang

          Issue Links

            Activity

              People

              • Assignee:
                kellyzly liyunzhang
                Reporter:
                kellyzly liyunzhang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: