Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6251

JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The JobClient is used to get job status information for running and completed jobs. Final state and history for a job is communicated from the application master to the job history server via a distributed file system - where the history is uploaded by the application master to the dfs and then scanned/loaded by the jobhistory server. While HDFS has strong consistency guarantees not all Hadoop DFS's do. When used in conjunction with a distributed file system which does not have this guarantee there will be cases where the history server may not see an uploaded file, resulting in the dreaded "no such job" and a null value for the RunningJob in the client.

        Attachments

        1. MAPREDUCE-6251.8.patch
          10 kB
          Craig Welch
        2. MAPREDUCE-6251.8.patch
          10 kB
          Craig Welch
        3. MAPREDUCE-6251.7.patch
          10 kB
          Craig Welch
        4. MAPREDUCE-6251.6.patch
          10 kB
          Craig Welch
        5. MAPREDUCE-6251.4.patch
          8 kB
          Craig Welch
        6. MAPREDUCE-6251.3.patch
          7 kB
          Craig Welch
        7. MAPREDUCE-6251.2.patch
          7 kB
          Craig Welch
        8. MAPREDUCE-6251.1.patch
          7 kB
          Craig Welch
        9. MAPREDUCE-6251.0.patch
          4 kB
          Craig Welch

          Issue Links

            Activity

              People

              • Assignee:
                cwelch Craig Welch
                Reporter:
                cwelch Craig Welch
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: