Uploaded image for project: 'Giraph'
  1. Giraph
  2. GIRAPH-1033

Remove zookeeper from input splits handling

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently we use zookeeper for handling input splits, by having each worker checking each split, and when a lot of splits are used this becomes very slow. We should have master coordinate input splits allocation instead, making the complexity proportional to #splits instead of #workers*#splits.

        Attachments

          Activity

            People

            • Assignee:
              majakabiljo Maja Kabiljo
              Reporter:
              majakabiljo Maja Kabiljo
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: