Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: tez-branch
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None

      Description

      After committing PIG-3527 and PIG-3595, these e2e tests started failing.

      1. Checkin_3.diff
        25 kB
        Cheolsoo Park
      2. Join_1.diff
        14 kB
        Cheolsoo Park
      3. Operators_1.diff
        3 kB
        Cheolsoo Park
      4. Operators_5.diff
        23 kB
        Cheolsoo Park
      5. PIG-3599-1.patch
        1 kB
        Cheolsoo Park
      6. PIG-3599-2.patch
        13 kB
        Cheolsoo Park

        Activity

        Hide
        Cheolsoo Park added a comment -

        Attaching the diffs between Tez and MR runs. As can be seen, the differences seem due to non-deterministic nature of shuffle.

        Show
        Cheolsoo Park added a comment - Attaching the diffs between Tez and MR runs. As can be seen, the differences seem due to non-deterministic nature of shuffle.
        Hide
        Cheolsoo Park added a comment -

        Attached fixes Checkin_3, Operators_1 and 5.

        Join_1 is rather strange because it passes when running it independently (-Dtests.to.run="-t Join_1"), but it fails when running it with the other tests.

        Show
        Cheolsoo Park added a comment - Attached fixes Checkin_3, Operators_1 and 5. Join_1 is rather strange because it passes when running it independently (-Dtests.to.run="-t Join_1"), but it fails when running it with the other tests.
        Hide
        Cheolsoo Park added a comment -

        After debugging further, I learned that Join_1 was failing because of TezSession reuse. By tearing down TezSession after each test, All the e2e tests pass now.

        The new patch includes the following changes-

        • Makes the e2e tests more deterministic.
        • Makes reuse of TezSession configurable via a property pig.tez.session.reuse.
        • Sets this property to false in all the e2e tests except LoaderStorer_1. In this test, TezSession must be reused since a jar is registered as local resource by a DAG and it's reused by another DAG.

        I will commit this patch unless anyone objects.

        Show
        Cheolsoo Park added a comment - After debugging further, I learned that Join_1 was failing because of TezSession reuse. By tearing down TezSession after each test, All the e2e tests pass now. The new patch includes the following changes- Makes the e2e tests more deterministic. Makes reuse of TezSession configurable via a property pig.tez.session.reuse. Sets this property to false in all the e2e tests except LoaderStorer_1. In this test, TezSession must be reused since a jar is registered as local resource by a DAG and it's reused by another DAG. I will commit this patch unless anyone objects.
        Hide
        Cheolsoo Park added a comment -

        Committed to tez branch

        Show
        Cheolsoo Park added a comment - Committed to tez branch

          People

          • Assignee:
            Cheolsoo Park
            Reporter:
            Cheolsoo Park
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development