Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4119

[jobhistory] job history server can not support mutiple yarn.app.mapreduce.am.staging-dir directorys.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.1
    • Fix Version/s: None
    • Component/s: jobhistoryserver
    • Labels:
      None
    • Environment:

      suse

      Description

      jobclient can set the yarn.app.mapreduce.am.staging-dir value , but jobhistoryserver only read the yarn.app.mapreduce.am.staging-dir value from conf.
      so, if client set different yarn.app.mapreduce.am.staging-dir value, and the jobhistoryserver can not scan the directory.

        Activity

        Hide
        xieguiming added a comment -

        this is the job history server the the intermediateDoneDirPath dir logic.

        and the intermediateDoneDirPath value from the local conf. and the local conf maybe different with the real job yarn.app.mapreduce.am.staging-dir.

        JobHistoryUtils.java
        
          public static String getConfiguredHistoryIntermediateDoneDirPrefix(
              Configuration conf) {
            String doneDirPrefix = conf
                .get(JHAdminConfig.MR_HISTORY_INTERMEDIATE_DONE_DIR);
            if (doneDirPrefix == null) {
              doneDirPrefix = conf.get(MRJobConfig.MR_AM_STAGING_DIR,
                  MRJobConfig.DEFAULT_MR_AM_STAGING_DIR)
                  + "/history/done_intermediate";
            }
            return doneDirPrefix;
          }
        
        
        Show
        xieguiming added a comment - this is the job history server the the intermediateDoneDirPath dir logic. and the intermediateDoneDirPath value from the local conf. and the local conf maybe different with the real job yarn.app.mapreduce.am.staging-dir. JobHistoryUtils.java public static String getConfiguredHistoryIntermediateDoneDirPrefix( Configuration conf) { String doneDirPrefix = conf .get(JHAdminConfig.MR_HISTORY_INTERMEDIATE_DONE_DIR); if (doneDirPrefix == null ) { doneDirPrefix = conf.get(MRJobConfig.MR_AM_STAGING_DIR, MRJobConfig.DEFAULT_MR_AM_STAGING_DIR) + "/history/done_intermediate" ; } return doneDirPrefix; }

          People

          • Assignee:
            Unassigned
            Reporter:
            xieguiming
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development