Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: tez-branch
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None

      Description

      We need to add TezMiniCluster to write unit tests.

      1. PIG-3531-1.patch
        16 kB
        Cheolsoo Park
      2. PIG-3531-2.patch
        17 kB
        Cheolsoo Park
      3. PIG-3531-3.patch
        22 kB
        Cheolsoo Park
      4. PIG-3531-4.patch
        23 kB
        Cheolsoo Park

        Activity

        Hide
        Rohini Palaniswamy added a comment -

        We just have to modify existing minicluster to copy tez libraries to hdfs if some system property is set. Don't need a new class.

        Show
        Rohini Palaniswamy added a comment - We just have to modify existing minicluster to copy tez libraries to hdfs if some system property is set. Don't need a new class.
        Hide
        Cheolsoo Park added a comment -

        Rohini Palaniswamy, thank you for the comment. But I actually found it cleaner to have a separate class from MR MiniCluster. Please take a look at what I have and let me know what you think:
        https://reviews.apache.org/r/14801/

        Show
        Cheolsoo Park added a comment - Rohini Palaniswamy , thank you for the comment. But I actually found it cleaner to have a separate class from MR MiniCluster. Please take a look at what I have and let me know what you think: https://reviews.apache.org/r/14801/
        Hide
        Cheolsoo Park added a comment -

        While setting up a jenkins build, I learned that I need to set YARN_APPLICATION_CLASSPATH if Hadoop is not installed:

        m_mr_conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH, System.getProperty("java.class.path"));
        

        Uploading a new patch with the fix.

        Show
        Cheolsoo Park added a comment - While setting up a jenkins build, I learned that I need to set YARN_APPLICATION_CLASSPATH if Hadoop is not installed: m_mr_conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH, System .getProperty( "java.class.path" )); Uploading a new patch with the fix.
        Hide
        Cheolsoo Park added a comment -

        Incorporated Rohini's comments in a new patch.

        Show
        Cheolsoo Park added a comment - Incorporated Rohini's comments in a new patch.
        Hide
        Cheolsoo Park added a comment -

        I had a minor bug in ant script. I fixed it in a new patch as follows:

        from
        +    <condition property="test.exec.type" value="mr">
        +        <!-- By default, test.exec.type is mapreduce -->
        +        <not>
        +            <isset property="${exectype}"/> <--- THIS SHOULD BE exectype!
        +        </not>
        +    </condition>
        
        to
        +    <condition property="test.exec.type" value="${exectype}" else="tez">
        +        <!-- By default, test.exec.type is tez -->
        +        <isset property="exectype"/>
        +    </condition>
        

        I tested the patch on Ubuntu and Mac, and it works on both. I will commit it shortly.

        Show
        Cheolsoo Park added a comment - I had a minor bug in ant script. I fixed it in a new patch as follows: from + <condition property= "test.exec.type" value= "mr" > + <!-- By default , test.exec.type is mapreduce --> + <not> + <isset property= "${exectype}" /> <--- THIS SHOULD BE exectype! + </not> + </condition> to + <condition property= "test.exec.type" value= "${exectype}" else = "tez" > + <!-- By default , test.exec.type is tez --> + <isset property= "exectype" /> + </condition> I tested the patch on Ubuntu and Mac, and it works on both. I will commit it shortly.
        Hide
        Cheolsoo Park added a comment -

        Committed to tez branch.

        1. -Dexectype=[tez|mr] can be used to decide whether to run unit tests in tez or mr mode. By default, it's set to tez in tez branch.
        2. To run all the tez unit tests, please run ant clean test-tez.
        Show
        Cheolsoo Park added a comment - Committed to tez branch. -Dexectype=[tez|mr] can be used to decide whether to run unit tests in tez or mr mode. By default, it's set to tez in tez branch. To run all the tez unit tests, please run ant clean test-tez.

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development