Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are a bunch of MR unit tests fail on tez branch.

      1. PIG-3965-2.patch
        35 kB
        Daniel Dai
      2. PIG-3965-1.patch
        42 kB
        Daniel Dai

        Activity

        Hide
        Daniel Dai added a comment -

        Attach a fix for those:
        TestMRExecutionEngine.testJobConfGeneration: caused by YARN-1748, upgrade build dependency to Hadoop 2.4.0 to solve the problem
        TestFRJoin2, TestParser, TestTmpFileCompression: caused by reusing properties in MiniCluster
        TestSecondarySortMR: Miss the code checking custom partitioner when doing SecondaryKeyOptimizer refactory
        TestSkewedJoin: fail on mr mode, reconfig log4j in test to solve the problem
        TestMRCompiler: update golden file
        TestTezCompiler: update golden file
        TestTezLauncher: fail on MR mode, hard code exectype to tez in test
        TestCombiner: use more memory after upgrade to Hadoop 2.4.0, set heap size in test

        Show
        Daniel Dai added a comment - Attach a fix for those: TestMRExecutionEngine.testJobConfGeneration: caused by YARN-1748 , upgrade build dependency to Hadoop 2.4.0 to solve the problem TestFRJoin2, TestParser, TestTmpFileCompression: caused by reusing properties in MiniCluster TestSecondarySortMR: Miss the code checking custom partitioner when doing SecondaryKeyOptimizer refactory TestSkewedJoin: fail on mr mode, reconfig log4j in test to solve the problem TestMRCompiler: update golden file TestTezCompiler: update golden file TestTezLauncher: fail on MR mode, hard code exectype to tez in test TestCombiner: use more memory after upgrade to Hadoop 2.4.0, set heap size in test
        Hide
        Rohini Palaniswamy added a comment -

        TestTezLauncher.java - MiniGenericCluster cluster = MiniGenericCluster.buildCluster(MiniGenericCluster.EXECTYPE_TEZ); instead of System.setProperty("test.exec.type", "tez");
        MRC18.gld - Is this expected? Don't think trunk has a failure in this test.
        TEZC-MQ-2.gld - This change can be reverted for now. Output is different with jdk7 and jdk6. That needs to be fixed in plan printing.

        Show
        Rohini Palaniswamy added a comment - TestTezLauncher.java - MiniGenericCluster cluster = MiniGenericCluster.buildCluster(MiniGenericCluster.EXECTYPE_TEZ); instead of System.setProperty("test.exec.type", "tez"); MRC18.gld - Is this expected? Don't think trunk has a failure in this test. TEZC-MQ-2.gld - This change can be reverted for now. Output is different with jdk7 and jdk6. That needs to be fixed in plan printing.
        Hide
        Daniel Dai added a comment -

        TestTezLauncher.java - MiniGenericCluster cluster = MiniGenericCluster.buildCluster(MiniGenericCluster.EXECTYPE_TEZ); instead of System.setProperty("test.exec.type", "tez");

        Sure, will change

        MRC18.gld - Is this expected? Don't think trunk has a failure in this test

        This is due to OperatorPlan does not define serialVersionUID, default serialVersionUID is susceptible to changes in the class. Here, there's some code change in OperatorPlan on tez, that makes the difference. Once I define OperatorPlan.serialVersionUI, both trunk and tez branch share the same MRC18.gld. I will open a ticket on trunk to add OperatorPlan.serialVersionUID

        TEZC-MQ-2.gld - This change can be reverted for now. Output is different with jdk7 and jdk6. That needs to be fixed in plan printing.

        Ok, let me revert this change

        Show
        Daniel Dai added a comment - TestTezLauncher.java - MiniGenericCluster cluster = MiniGenericCluster.buildCluster(MiniGenericCluster.EXECTYPE_TEZ); instead of System.setProperty("test.exec.type", "tez"); Sure, will change MRC18.gld - Is this expected? Don't think trunk has a failure in this test This is due to OperatorPlan does not define serialVersionUID, default serialVersionUID is susceptible to changes in the class. Here, there's some code change in OperatorPlan on tez, that makes the difference. Once I define OperatorPlan.serialVersionUI, both trunk and tez branch share the same MRC18.gld. I will open a ticket on trunk to add OperatorPlan.serialVersionUID TEZC-MQ-2.gld - This change can be reverted for now. Output is different with jdk7 and jdk6. That needs to be fixed in plan printing. Ok, let me revert this change
        Hide
        Daniel Dai added a comment -

        Opened PIG-3968 for MRC18.gld

        Show
        Daniel Dai added a comment - Opened PIG-3968 for MRC18.gld
        Hide
        Rohini Palaniswamy added a comment -

        +1

        Show
        Rohini Palaniswamy added a comment - +1
        Hide
        Daniel Dai added a comment -

        Patch committed to tez branch. Thanks Rohini for review!

        Show
        Daniel Dai added a comment - Patch committed to tez branch. Thanks Rohini for review!

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Daniel Dai
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development