Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3207

Add support for fetching multiple partitions from the same source task to UnorderedKVInput

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9.0
    • None
    • None

    Description

      The ordered grouped ShuffleScheduler can support fetching multiple partitions from the same source task. But for the unordered ShuffleManager, it only supports one partition per source task due to the following issue where identifier doesn't take partition id into account.

        public void addKnownInput(String hostName, int port,
            InputAttemptIdentifier srcAttemptIdentifier, int srcPhysicalIndex) {
          String identifier = InputHost.createIdentifier(hostName, port);
          InputHost host = knownSrcHosts.get(identifier);
          ....
        }
      

      Attachments

        1. TEZ-3207-3.patch
          34 kB
          Ming Ma
        2. TEZ-3207-2.patch
          34 kB
          Ming Ma
        3. TEZ-3207.patch
          29 kB
          Ming Ma

        Issue Links

          Activity

            People

              mingma Ming Ma
              mingma Ming Ma
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: