Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6258

add support to back up JHS files from application master



    • New Feature
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 2.4.1
    • None
    • applicationmaster


      In hadoop two, job history files are stored on HDFS with a default retention period of one week. In a cloud environment, these HDFS files are actually stored on the disks of ephemeral instances that could go away once the instances are terminated. Users may want to back up the job history files for issue investigation and performance analysis before and after the cluster is terminated.

      A centralized backup mechanism could have a scalability issue for big and busy Hadoop clusters where there are probably tens of thousands of jobs every day. As a result, it is preferred to have a distributed way to back up the job history files in this case. To achieve this goal, we could add a new feature to back up the job history files in Application master. More specifically, we could copy the job history files to a backup path when they are moved from the temporary staging directory to the intermediate_done path in application master. Since application masters could run on any slave nodes on a Hadoop cluster, we could achieve a better scalability by backing up the job history files in a distributed fashion.

      Please be aware, the backup path should be managed by the Hadoop users based on their needs. For example, some Hadoop users may copy the job history files to a cloud storage directly and keep them there forever. While some other users may want to store the job history files on local disks and clean them up from time to time.


        1. MAPREDUCE-6258.patch
          8 kB
          Jian Fang



            Unassigned Unassigned
            john.jian.fang Jian Fang
            0 Vote for this issue
            3 Start watching this issue