Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1996

Too many index lookups during range shuffle

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: Data Shuffle, Pull Server
    • Labels:
      None

      Description

      During range shuffle, there are a huge number of index lookups, thereby taking at least the half time of the total shuffle time. This is because it is difficult to know which data is stored on which machine, so each machine needs to ask to all machines including itself.

      Unfortunately, this problem is difficult to resolved using statistics. With an assumption of even data distribution, every machine is expected to have the data of the whole range.

      So, we need to investigate an alternative solution to reduce the frequent index lookups.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jihoonson Jihoon Son
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: