Pig
  1. Pig
  2. PIG-3446 Umbrella jira for Pig on Tez
  3. PIG-3626

Make combiners, custom partitioners and secondary key sort work for multiple outputs

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    1. PIG-3626-hotfix.patch
      0.8 kB
      Cheolsoo Park
    2. PIG-3626-2.patch
      195 kB
      Rohini Palaniswamy
    3. PIG-3626-1.patch
      195 kB
      Rohini Palaniswamy

      Activity

      Show
      Rohini Palaniswamy added a comment - https://reviews.apache.org/r/17191/
      Hide
      Rohini Palaniswamy added a comment -

      Committed to tez branch. Thanks Cheolsoo for the review

      Show
      Rohini Palaniswamy added a comment - Committed to tez branch. Thanks Cheolsoo for the review
      Hide
      Cheolsoo Park added a comment -

      Rohini Palaniswamy, I found a bug in your commit that breaks replicated join e2e tests in tez.conf-

      Tezcompiler.java#L755
      if (tezOp.getSplitOperatorKey() == null) {
        inputKeys.add(tezOp.getSplitOperatorKey().toString()); //--> NPE because getSplitOperatorKey() is null.
      } else {
        inputKeys.add(tezOp.getOperatorKey().toString());
      }
      

      I will commit the hotfix that I am attaching. Thanks!

      Show
      Cheolsoo Park added a comment - Rohini Palaniswamy , I found a bug in your commit that breaks replicated join e2e tests in tez.conf- Tezcompiler.java#L755 if (tezOp.getSplitOperatorKey() == null ) { inputKeys.add(tezOp.getSplitOperatorKey().toString()); //--> NPE because getSplitOperatorKey() is null . } else { inputKeys.add(tezOp.getOperatorKey().toString()); } I will commit the hotfix that I am attaching. Thanks!
      Hide
      Rohini Palaniswamy added a comment -

      Ah. Thanks Cheolsoo.

      Show
      Rohini Palaniswamy added a comment - Ah. Thanks Cheolsoo.
      Hide
      Cheolsoo Park added a comment -

      Rohini Palaniswamy, does TestCustomPartitioner pass? It always hangs for me.

      Show
      Cheolsoo Park added a comment - Rohini Palaniswamy , does TestCustomPartitioner pass? It always hangs for me.

        People

        • Assignee:
          Rohini Palaniswamy
          Reporter:
          Rohini Palaniswamy
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development