Pig
  1. Pig
  2. PIG-3446 Umbrella jira for Pig on Tez
  3. PIG-3658

Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Tez provides way to cache objects with Vertex scope. We can use it to
      1) caching the replication join map constructed instead of reading from multiple broadcast inputs from one or more vertex and reconstructing it
      2) Cache the map in WeightedRangePartitioner. Thought the quantilefile is only one brodcast input from one vertex still it would be faster to use the local cache.

      1. PIG-3658-1.patch
        26 kB
        Rohini Palaniswamy

        Issue Links

          Activity

          Rohini Palaniswamy created issue -
          Rohini Palaniswamy made changes -
          Field Original Value New Value
          Component/s tez [ 12321016 ]
          Rohini Palaniswamy made changes -
          Link This issue requires TEZ-711 [ TEZ-711 ]
          Hide
          Rohini Palaniswamy added a comment -

          Patch caches for the other broadcast inputs - POPartitionRearrangeTez.java and SkewedPartitionerTez.java - as well.

          Show
          Rohini Palaniswamy added a comment - Patch caches for the other broadcast inputs - POPartitionRearrangeTez.java and SkewedPartitionerTez.java - as well.
          Rohini Palaniswamy made changes -
          Attachment PIG-3658-1.patch [ 12625328 ]
          Hide
          Cheolsoo Park added a comment -

          +1.

          Show
          Cheolsoo Park added a comment - +1.
          Hide
          Rohini Palaniswamy added a comment -

          Committed to tez-branch. Thanks Cheolsoo for the review.

          Show
          Rohini Palaniswamy added a comment - Committed to tez-branch. Thanks Cheolsoo for the review.
          Rohini Palaniswamy made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Daniel Dai made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          19d 3h 26m 1 Rohini Palaniswamy 27/Jan/14 20:50
          Resolved Resolved Closed Closed
          297d 9h 8m 1 Daniel Dai 21/Nov/14 05:58

            People

            • Assignee:
              Rohini Palaniswamy
              Reporter:
              Rohini Palaniswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development