Pig
  1. Pig
  2. PIG-3446 Umbrella jira for Pig on Tez
  3. PIG-3658

Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Tez provides way to cache objects with Vertex scope. We can use it to
      1) caching the replication join map constructed instead of reading from multiple broadcast inputs from one or more vertex and reconstructing it
      2) Cache the map in WeightedRangePartitioner. Thought the quantilefile is only one brodcast input from one vertex still it would be faster to use the local cache.

      1. PIG-3658-1.patch
        26 kB
        Rohini Palaniswamy

        Issue Links

          Activity

          Hide
          Rohini Palaniswamy added a comment -

          Patch caches for the other broadcast inputs - POPartitionRearrangeTez.java and SkewedPartitionerTez.java - as well.

          Show
          Rohini Palaniswamy added a comment - Patch caches for the other broadcast inputs - POPartitionRearrangeTez.java and SkewedPartitionerTez.java - as well.
          Hide
          Cheolsoo Park added a comment -

          +1.

          Show
          Cheolsoo Park added a comment - +1.
          Hide
          Rohini Palaniswamy added a comment -

          Committed to tez-branch. Thanks Cheolsoo for the review.

          Show
          Rohini Palaniswamy added a comment - Committed to tez-branch. Thanks Cheolsoo for the review.

            People

            • Assignee:
              Rohini Palaniswamy
              Reporter:
              Rohini Palaniswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development