Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-1648

Split combination may return too many block locations to map/reduce framework

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.8.0
    • 0.8.0
    • None
    • None

    Description

      For instance, if a small split has block locations h1, h2 and h3; another small split has h1, h3, h4. After combination, the composite split contains 4 block locations. If the number of component splits is big, then the number of block locations could be big too. In fact, the number of block locations serves as a hint to M/R as the best hosts this composite split should be run on so the list should contain a short list, say 5, of the hosts that contain the most data in this composite split.

      Attachments

        1. PIG-1648.patch
          5 kB
          Yan Zhou

        Activity

          People

            yanz Yan Zhou
            yanz Yan Zhou
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: