Pig
  1. Pig
  2. PIG-3446 Umbrella jira for Pig on Tez
  3. PIG-3626

Make combiners, custom partitioners and secondary key sort work for multiple outputs

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    1. PIG-3626-1.patch
      195 kB
      Rohini Palaniswamy
    2. PIG-3626-2.patch
      195 kB
      Rohini Palaniswamy
    3. PIG-3626-hotfix.patch
      0.8 kB
      Cheolsoo Park

      Activity

      Rohini Palaniswamy created issue -
      Rohini Palaniswamy made changes -
      Field Original Value New Value
      Component/s tez [ 12321016 ]
      Show
      Rohini Palaniswamy added a comment - https://reviews.apache.org/r/17191/
      Rohini Palaniswamy made changes -
      Attachment PIG-3626-1.patch [ 12624382 ]
      Rohini Palaniswamy made changes -
      Attachment PIG-3626-2.patch [ 12624565 ]
      Hide
      Rohini Palaniswamy added a comment -

      Committed to tez branch. Thanks Cheolsoo for the review

      Show
      Rohini Palaniswamy added a comment - Committed to tez branch. Thanks Cheolsoo for the review
      Rohini Palaniswamy made changes -
      Status Open [ 1 ] Resolved [ 5 ]
      Resolution Fixed [ 1 ]
      Hide
      Cheolsoo Park added a comment -

      Rohini Palaniswamy, I found a bug in your commit that breaks replicated join e2e tests in tez.conf-

      Tezcompiler.java#L755
      if (tezOp.getSplitOperatorKey() == null) {
        inputKeys.add(tezOp.getSplitOperatorKey().toString()); //--> NPE because getSplitOperatorKey() is null.
      } else {
        inputKeys.add(tezOp.getOperatorKey().toString());
      }
      

      I will commit the hotfix that I am attaching. Thanks!

      Show
      Cheolsoo Park added a comment - Rohini Palaniswamy , I found a bug in your commit that breaks replicated join e2e tests in tez.conf- Tezcompiler.java#L755 if (tezOp.getSplitOperatorKey() == null ) { inputKeys.add(tezOp.getSplitOperatorKey().toString()); //--> NPE because getSplitOperatorKey() is null . } else { inputKeys.add(tezOp.getOperatorKey().toString()); } I will commit the hotfix that I am attaching. Thanks!
      Cheolsoo Park made changes -
      Attachment PIG-3626-hotfix.patch [ 12625071 ]
      Hide
      Rohini Palaniswamy added a comment -

      Ah. Thanks Cheolsoo.

      Show
      Rohini Palaniswamy added a comment - Ah. Thanks Cheolsoo.
      Hide
      Cheolsoo Park added a comment -

      Rohini Palaniswamy, does TestCustomPartitioner pass? It always hangs for me.

      Show
      Cheolsoo Park added a comment - Rohini Palaniswamy , does TestCustomPartitioner pass? It always hangs for me.
      Daniel Dai made changes -
      Status Resolved [ 5 ] Closed [ 6 ]
      Transition Time In Source Status Execution Times Last Executer Last Execution Date
      Open Open Resolved Resolved
      38d 18h 44m 1 Rohini Palaniswamy 23/Jan/14 03:36
      Resolved Resolved Closed Closed
      302d 2h 22m 1 Daniel Dai 21/Nov/14 05:59

        People

        • Assignee:
          Rohini Palaniswamy
          Reporter:
          Rohini Palaniswamy
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development