Pig
  1. Pig
  2. PIG-3446 Umbrella jira for Pig on Tez
  3. PIG-3780

Tez mini cluster tests run for a very long time with TezSession reuse on

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: tez-branch
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None

      Description

      In the current tez branch, mini cluster unit tests are very slow. The reason is as follows:

      • TezSession reuse is by default on.
      • Each test case runs, and it waits for Tez AM to terminate.
      • After Tez AM times out (usually after several minutes), another test case runs.

      Two questions that I have are:

      1. Why doesn't TezSession reuse work in mini cluster?
      2. Why is TezSession reuse not disabled in some tests (e.g. TestAccumulator) where we explicitly set "tez.session.reuse" to false?

      As for #2, I realized that "tez.session.reuse" was never set in the properties object that is passed to PigServer. I am going to upload a patch that fixes this problem in this jira.

      As for #1, I don't have an answer yet. But I think we can fix this in a separate jira once we get Tez unit tests working again.

      1. PIG-3780-1.patch
        61 kB
        Cheolsoo Park
      2. PIG-3780-2.patch
        34 kB
        Cheolsoo Park

        Activity

        Hide
        Cheolsoo Park added a comment -

        In the attached patch, I am doing three things-

        1. Set "tez.session.reuse" to false in properties object and reuse its reference to start PigServer everywhere. I noticed that cluster.getProperties() returns a copy of object rather than a reference to it, so setting any properties without keeping the reference never takes effect.
        2. TestCustomerPartitioner was failing because "part-r-00000" was hardcoded in test cases. But the Tez output filename doesn't follow this convention. I changed the test to use FileStatus instead.
        3. TestTezCompiler was broken due to mismatch in gold files. This is probably because we haven't run unit tests for a while. I regenerated them to reflect recent changes.

        I have confirmed all the unit tests in test-tez pass.

        Show
        Cheolsoo Park added a comment - In the attached patch, I am doing three things- Set "tez.session.reuse" to false in properties object and reuse its reference to start PigServer everywhere. I noticed that cluster.getProperties() returns a copy of object rather than a reference to it, so setting any properties without keeping the reference never takes effect. TestCustomerPartitioner was failing because "part-r-00000" was hardcoded in test cases. But the Tez output filename doesn't follow this convention. I changed the test to use FileStatus instead. TestTezCompiler was broken due to mismatch in gold files. This is probably because we haven't run unit tests for a while. I regenerated them to reflect recent changes. I have confirmed all the unit tests in test-tez pass.
        Hide
        Cheolsoo Park added a comment -
        Show
        Cheolsoo Park added a comment - RB link- https://reviews.apache.org/r/18505/
        Hide
        Rohini Palaniswamy added a comment -

        Some issues with RB and the comments I gave yest wasn't published. Still not able to. So giving it here.

        1) PathFilter getDefaultPathFilter
        Just to know, is it the _SUCCESS files that you are trying to filter out? If so can the method be renamed to getSuccessMarkerPathFilter?
        Also can we add the method to org.apache.pig.test.Util in test package instead of this class?

        2) properties.setProperty(PigConfiguration.TEZ_SESSION_REUSE, "false");
        Why don't we set this in MiniCluster m_conf itself. Once we fix the original issue, then it will just be one place that needs to be fixed.

        3) TEZC11.gld
        Combiner has gone missing. We need to fix this instead of changing golden file. Can you leave this file as is?

        4) Other split related files
        These ones be left as is till PIG-3757 is fully done. The filter should not be there and the alias names also have gone missing. We knew we were breaking this.

        Show
        Rohini Palaniswamy added a comment - Some issues with RB and the comments I gave yest wasn't published. Still not able to. So giving it here. 1) PathFilter getDefaultPathFilter Just to know, is it the _SUCCESS files that you are trying to filter out? If so can the method be renamed to getSuccessMarkerPathFilter? Also can we add the method to org.apache.pig.test.Util in test package instead of this class? 2) properties.setProperty(PigConfiguration.TEZ_SESSION_REUSE, "false"); Why don't we set this in MiniCluster m_conf itself. Once we fix the original issue, then it will just be one place that needs to be fixed. 3) TEZC11.gld Combiner has gone missing. We need to fix this instead of changing golden file. Can you leave this file as is? 4) Other split related files These ones be left as is till PIG-3757 is fully done. The filter should not be there and the alias names also have gone missing. We knew we were breaking this.
        Hide
        Cheolsoo Park added a comment -

        Incorporate Rohini's comments.

        Show
        Cheolsoo Park added a comment - Incorporate Rohini's comments.
        Hide
        Rohini Palaniswamy added a comment -

        +1

        Show
        Rohini Palaniswamy added a comment - +1
        Hide
        Cheolsoo Park added a comment -

        Committed to tez branch. Thank you Rohini for the review!

        Show
        Cheolsoo Park added a comment - Committed to tez branch. Thank you Rohini for the review!

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development