Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4778

Fair scheduler event log is only written if directory exists on HDFS

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0, 2.0.2-alpha
    • Fix Version/s: 1.2.0, 2.0.3-alpha
    • Component/s: jobtracker, scheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The fair scheduler event log is supposed to be written to the local filesystem, at

      {hadoop.log.dir}

      /fairscheduler. The event log will not be written unless this directory exists on HDFS.

      1. MAPREDUCE-4778.patch
        5 kB
        Sandy Ryza
      2. MAPREDUCE-4778.branch1.patch
        4 kB
        Sandy Ryza

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1241 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1241/)
        MAPREDUCE-4778. Fair scheduler event log is only written if directory exists on HDFS. Contributed by Sandy Ryza. (Revision 1414729)

        Result = FAILURE
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414729
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1241 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1241/ ) MAPREDUCE-4778 . Fair scheduler event log is only written if directory exists on HDFS. Contributed by Sandy Ryza. (Revision 1414729) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414729 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #51 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/51/)
        MAPREDUCE-4778. Fair scheduler event log is only written if directory exists on HDFS. Contributed by Sandy Ryza. (Revision 1414729)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414729
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #51 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/51/ ) MAPREDUCE-4778 . Fair scheduler event log is only written if directory exists on HDFS. Contributed by Sandy Ryza. (Revision 1414729) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414729 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3068 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3068/)
        MAPREDUCE-4778. Fair scheduler event log is only written if directory exists on HDFS. Contributed by Sandy Ryza. (Revision 1414729)

        Result = SUCCESS
        tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414729
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3068 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3068/ ) MAPREDUCE-4778 . Fair scheduler event log is only written if directory exists on HDFS. Contributed by Sandy Ryza. (Revision 1414729) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414729 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerEventLog.java
        Hide
        Tom White added a comment -

        +1 I just committed this. Thanks, Sandy!

        Show
        Tom White added a comment - +1 I just committed this. Thanks, Sandy!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12552583/MAPREDUCE-4778.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2996//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2996//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12552583/MAPREDUCE-4778.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2996//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2996//console This message is automatically generated.
        Hide
        Wing Yew Poon added a comment -

        In FairSchedulerEventLog.java, we have:

          boolean init(Configuration conf, String jobtrackerHostname) {
            try {
              logDir = conf.get("mapred.fairscheduler.eventlog.location",
                  new File(System.getProperty("hadoop.log.dir")).getAbsolutePath()
                  + File.separator + "fairscheduler");
              Path logDirPath = new Path(logDir);
              FileSystem fs = logDirPath.getFileSystem(conf);
              if (!fs.exists(logDirPath)) {
                if (!fs.mkdirs(logDirPath)) {
                  throw new IOException(
                      "Mkdirs failed to create " + logDirPath.toString());
                }
              }
              String username = System.getProperty("user.name");
              logFile = String.format("%s%shadoop-%s-fairscheduler-%s.log",
                  logDir, File.separator, username, jobtrackerHostname);
              logDisabled = false;
              PatternLayout layout = new PatternLayout("%d{ISO8601}\t%m%n");
              appender = new DailyRollingFileAppender(layout, logFile, "'.'yyyy-MM-dd")\
        ;
              appender.activateOptions();
              LOG.info("Initialized fair scheduler event log, logging to " + logFile);
            } catch (IOException e) {
              LOG.error(
                  "Failed to initialize fair scheduler event log. Disabling it.", e);
              logDisabled = true;
            }
            return !(logDisabled);
          }
        

        The

              Path logDirPath = new Path(logDir);
              FileSystem fs = logDirPath.getFileSystem(conf);
              if (!fs.exists(logDirPath)) {
                if (!fs.mkdirs(logDirPath)) {
                  throw new IOException(
                      "Mkdirs failed to create " + logDirPath.toString());
                }
              }
        

        appears to be unnecessary.

        Show
        Wing Yew Poon added a comment - In FairSchedulerEventLog.java, we have: boolean init(Configuration conf, String jobtrackerHostname) { try { logDir = conf.get("mapred.fairscheduler.eventlog.location", new File(System.getProperty("hadoop.log.dir")).getAbsolutePath() + File.separator + "fairscheduler"); Path logDirPath = new Path(logDir); FileSystem fs = logDirPath.getFileSystem(conf); if (!fs.exists(logDirPath)) { if (!fs.mkdirs(logDirPath)) { throw new IOException( "Mkdirs failed to create " + logDirPath.toString()); } } String username = System.getProperty("user.name"); logFile = String.format("%s%shadoop-%s-fairscheduler-%s.log", logDir, File.separator, username, jobtrackerHostname); logDisabled = false; PatternLayout layout = new PatternLayout("%d{ISO8601}\t%m%n"); appender = new DailyRollingFileAppender(layout, logFile, "'.'yyyy-MM-dd")\ ; appender.activateOptions(); LOG.info("Initialized fair scheduler event log, logging to " + logFile); } catch (IOException e) { LOG.error( "Failed to initialize fair scheduler event log. Disabling it.", e); logDisabled = true; } return !(logDisabled); } The Path logDirPath = new Path(logDir); FileSystem fs = logDirPath.getFileSystem(conf); if (!fs.exists(logDirPath)) { if (!fs.mkdirs(logDirPath)) { throw new IOException( "Mkdirs failed to create " + logDirPath.toString()); } } appears to be unnecessary.

          People

          • Assignee:
            Sandy Ryza
            Reporter:
            Sandy Ryza
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development