Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3446 Umbrella jira for Pig on Tez
  3. PIG-3658

Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Tez provides way to cache objects with Vertex scope. We can use it to
      1) caching the replication join map constructed instead of reading from multiple broadcast inputs from one or more vertex and reconstructing it
      2) Cache the map in WeightedRangePartitioner. Thought the quantilefile is only one brodcast input from one vertex still it would be faster to use the local cache.

        Attachments

        1. PIG-3658-1.patch
          26 kB
          Rohini Palaniswamy

          Issue Links

            Activity

              People

              • Assignee:
                rohini Rohini Palaniswamy
                Reporter:
                rohini Rohini Palaniswamy
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: