[TEZ-3207] Add support for fetching multiple partitions from the same source task to UnorderedKVInput

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Assignee: Ming Ma
• Reporter: Ming Ma
• Created: 08/Apr/16 22:35
• Updated: 22/Aug/17 00:03
• Resolved: 28/Apr/16 03:35

Description

The ordered grouped ShuffleScheduler can support fetching multiple partitions from the same source task. But for the unordered ShuffleManager, it only supports one partition per source task due to the following issue where identifier doesn't take partition id into account.

  public void addKnownInput(String hostName, int port,
      InputAttemptIdentifier srcAttemptIdentifier, int srcPhysicalIndex) {
    String identifier = InputHost.createIdentifier(hostName, port);
    InputHost host = knownSrcHosts.get(identifier);
    ....
  }

Issue Links

• is related to: TEZ-3245 Race condition between addKnowInput and clearAndGetOnepartition of InputHost