Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6387

Serialize the recently added Task#encryptedSpillKey field at the end

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There was a recent addition of an encryptedSpillKey to the Task object. And when serialized, this field was written out somewhere in the middle. This caused deployments that do not use DistributedCache to push job jars before running the job to fail rolling upgrade.

      Although deploying via Distributed Cache is the recommended method, there might still be deployments that use the node local classpath to pick up the MR framework classes (eg. for efficiency purposes, since this does not require the jar being copied to hdfs and then to all the nodes)

      Ensuring that it is the last field written and read when the Task object is serialized would alleviate this issue.

        Attachments

          Activity

            People

            • Assignee:
              asuresh Arun Suresh
              Reporter:
              asuresh Arun Suresh
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: