Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: tez-branch
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None

      Description

      We need to add TezMiniCluster to write unit tests.

      1. PIG-3531-1.patch
        16 kB
        Cheolsoo Park
      2. PIG-3531-2.patch
        17 kB
        Cheolsoo Park
      3. PIG-3531-3.patch
        22 kB
        Cheolsoo Park
      4. PIG-3531-4.patch
        23 kB
        Cheolsoo Park

        Activity

        Hide
        rohini Rohini Palaniswamy added a comment -

        We just have to modify existing minicluster to copy tez libraries to hdfs if some system property is set. Don't need a new class.

        Show
        rohini Rohini Palaniswamy added a comment - We just have to modify existing minicluster to copy tez libraries to hdfs if some system property is set. Don't need a new class.
        Hide
        cheolsoo Cheolsoo Park added a comment -

        Rohini Palaniswamy, thank you for the comment. But I actually found it cleaner to have a separate class from MR MiniCluster. Please take a look at what I have and let me know what you think:
        https://reviews.apache.org/r/14801/

        Show
        cheolsoo Cheolsoo Park added a comment - Rohini Palaniswamy , thank you for the comment. But I actually found it cleaner to have a separate class from MR MiniCluster. Please take a look at what I have and let me know what you think: https://reviews.apache.org/r/14801/
        Hide
        cheolsoo Cheolsoo Park added a comment -

        While setting up a jenkins build, I learned that I need to set YARN_APPLICATION_CLASSPATH if Hadoop is not installed:

        m_mr_conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH, System.getProperty("java.class.path"));
        

        Uploading a new patch with the fix.

        Show
        cheolsoo Cheolsoo Park added a comment - While setting up a jenkins build, I learned that I need to set YARN_APPLICATION_CLASSPATH if Hadoop is not installed: m_mr_conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH, System .getProperty( "java.class.path" )); Uploading a new patch with the fix.
        Hide
        cheolsoo Cheolsoo Park added a comment -

        Incorporated Rohini's comments in a new patch.

        Show
        cheolsoo Cheolsoo Park added a comment - Incorporated Rohini's comments in a new patch.
        Hide
        cheolsoo Cheolsoo Park added a comment -

        I had a minor bug in ant script. I fixed it in a new patch as follows:

        from
        +    <condition property="test.exec.type" value="mr">
        +        <!-- By default, test.exec.type is mapreduce -->
        +        <not>
        +            <isset property="${exectype}"/> <--- THIS SHOULD BE exectype!
        +        </not>
        +    </condition>
        
        to
        +    <condition property="test.exec.type" value="${exectype}" else="tez">
        +        <!-- By default, test.exec.type is tez -->
        +        <isset property="exectype"/>
        +    </condition>
        

        I tested the patch on Ubuntu and Mac, and it works on both. I will commit it shortly.

        Show
        cheolsoo Cheolsoo Park added a comment - I had a minor bug in ant script. I fixed it in a new patch as follows: from + <condition property= "test.exec.type" value= "mr" > + <!-- By default , test.exec.type is mapreduce --> + <not> + <isset property= "${exectype}" /> <--- THIS SHOULD BE exectype! + </not> + </condition> to + <condition property= "test.exec.type" value= "${exectype}" else = "tez" > + <!-- By default , test.exec.type is tez --> + <isset property= "exectype" /> + </condition> I tested the patch on Ubuntu and Mac, and it works on both. I will commit it shortly.
        Hide
        cheolsoo Cheolsoo Park added a comment -

        Committed to tez branch.

        1. -Dexectype=[tez|mr] can be used to decide whether to run unit tests in tez or mr mode. By default, it's set to tez in tez branch.
        2. To run all the tez unit tests, please run ant clean test-tez.
        Show
        cheolsoo Cheolsoo Park added a comment - Committed to tez branch. -Dexectype=[tez|mr] can be used to decide whether to run unit tests in tez or mr mode. By default, it's set to tez in tez branch. To run all the tez unit tests, please run ant clean test-tez.

          People

          • Assignee:
            cheolsoo Cheolsoo Park
            Reporter:
            cheolsoo Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development