Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: tez-branch
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None

      Description

      We need to add TezMiniCluster to write unit tests.

      1. PIG-3531-1.patch
        16 kB
        Cheolsoo Park
      2. PIG-3531-2.patch
        17 kB
        Cheolsoo Park
      3. PIG-3531-3.patch
        22 kB
        Cheolsoo Park
      4. PIG-3531-4.patch
        23 kB
        Cheolsoo Park

        Activity

        Cheolsoo Park created issue -
        Hide
        Rohini Palaniswamy added a comment -

        We just have to modify existing minicluster to copy tez libraries to hdfs if some system property is set. Don't need a new class.

        Show
        Rohini Palaniswamy added a comment - We just have to modify existing minicluster to copy tez libraries to hdfs if some system property is set. Don't need a new class.
        Hide
        Cheolsoo Park added a comment -

        Rohini Palaniswamy, thank you for the comment. But I actually found it cleaner to have a separate class from MR MiniCluster. Please take a look at what I have and let me know what you think:
        https://reviews.apache.org/r/14801/

        Show
        Cheolsoo Park added a comment - Rohini Palaniswamy , thank you for the comment. But I actually found it cleaner to have a separate class from MR MiniCluster. Please take a look at what I have and let me know what you think: https://reviews.apache.org/r/14801/
        Cheolsoo Park made changes -
        Field Original Value New Value
        Status Open [ 1 ] Patch Available [ 10002 ]
        Cheolsoo Park made changes -
        Attachment PIG-3531-1.patch [ 12609567 ]
        Hide
        Cheolsoo Park added a comment -

        While setting up a jenkins build, I learned that I need to set YARN_APPLICATION_CLASSPATH if Hadoop is not installed:

        m_mr_conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH, System.getProperty("java.class.path"));
        

        Uploading a new patch with the fix.

        Show
        Cheolsoo Park added a comment - While setting up a jenkins build, I learned that I need to set YARN_APPLICATION_CLASSPATH if Hadoop is not installed: m_mr_conf.set(YarnConfiguration.YARN_APPLICATION_CLASSPATH, System .getProperty( "java.class.path" )); Uploading a new patch with the fix.
        Cheolsoo Park made changes -
        Attachment PIG-3531-2.patch [ 12609599 ]
        Hide
        Cheolsoo Park added a comment -

        Incorporated Rohini's comments in a new patch.

        Show
        Cheolsoo Park added a comment - Incorporated Rohini's comments in a new patch.
        Cheolsoo Park made changes -
        Attachment PIG-3531-3.patch [ 12609742 ]
        Hide
        Cheolsoo Park added a comment -

        I had a minor bug in ant script. I fixed it in a new patch as follows:

        from
        +    <condition property="test.exec.type" value="mr">
        +        <!-- By default, test.exec.type is mapreduce -->
        +        <not>
        +            <isset property="${exectype}"/> <--- THIS SHOULD BE exectype!
        +        </not>
        +    </condition>
        
        to
        +    <condition property="test.exec.type" value="${exectype}" else="tez">
        +        <!-- By default, test.exec.type is tez -->
        +        <isset property="exectype"/>
        +    </condition>
        

        I tested the patch on Ubuntu and Mac, and it works on both. I will commit it shortly.

        Show
        Cheolsoo Park added a comment - I had a minor bug in ant script. I fixed it in a new patch as follows: from + <condition property= "test.exec.type" value= "mr" > + <!-- By default , test.exec.type is mapreduce --> + <not> + <isset property= "${exectype}" /> <--- THIS SHOULD BE exectype! + </not> + </condition> to + <condition property= "test.exec.type" value= "${exectype}" else = "tez" > + <!-- By default , test.exec.type is tez --> + <isset property= "exectype" /> + </condition> I tested the patch on Ubuntu and Mac, and it works on both. I will commit it shortly.
        Cheolsoo Park made changes -
        Attachment PIG-3531-4.patch [ 12610318 ]
        Hide
        Cheolsoo Park added a comment -

        Committed to tez branch.

        1. -Dexectype=[tez|mr] can be used to decide whether to run unit tests in tez or mr mode. By default, it's set to tez in tez branch.
        2. To run all the tez unit tests, please run ant clean test-tez.
        Show
        Cheolsoo Park added a comment - Committed to tez branch. -Dexectype=[tez|mr] can be used to decide whether to run unit tests in tez or mr mode. By default, it's set to tez in tez branch. To run all the tez unit tests, please run ant clean test-tez.
        Cheolsoo Park made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        46m 42s 1 Cheolsoo Park 22/Oct/13 00:04
        Patch Available Patch Available Resolved Resolved
        3d 14h 59m 1 Cheolsoo Park 25/Oct/13 15:04
        Resolved Resolved Closed Closed
        391d 15h 53m 1 Daniel Dai 21/Nov/14 05:58

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development