Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2427

JT should ensure history directory is a directory

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.24.0
    • Component/s: jobtracker
    • Labels:
      None

      Description

      If the JT history directory doesn't exist or isn't a directory retired job files are renamed to a file called 'history' and eventually start overwriting each other. The JT should ensure 'history' exists and is a directory before performing the move.

      1. MAPREDUCE-2427.0.20.patch
        2 kB
        Devaraj K
      2. MAPREDUCE-2427.patch
        2 kB
        Devaraj K

        Activity

        E. Sammer created issue -
        Devaraj K made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-2427.0.20.patch [ 12479604 ]
        Attachment MAPREDUCE-2427.patch [ 12479605 ]
        Devaraj K made changes -
        Assignee Devaraj K [ devaraj.k ]
        Hide
        Devaraj K added a comment -

        When initializing throwing IOException if 'history' is present and it is not a directory.

        Show
        Devaraj K added a comment - When initializing throwing IOException if 'history' is present and it is not a directory.
        Devaraj K made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.20.4 [ 12316034 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Hide
        Devaraj K added a comment -

        When initializing job history throwing IOException if 'history' is present and it is not a directory.

        Show
        Devaraj K added a comment - When initializing job history throwing IOException if 'history' is present and it is not a directory.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479605/MAPREDUCE-2427.patch
        against trunk revision 1104687.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/265//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/265//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/265//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479605/MAPREDUCE-2427.patch against trunk revision 1104687. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/265//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/265//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/265//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Hi Devaraj. This patch seems to address the case where the "history" location is a file at startup. But, if I understand Sammer correctly, the problem here is when users do something like rm -Rf the history/ directory while the JT is running. Then, next time (and all future times) that a file is moving into the history dir, it ends up replacing this file.

        So, I think we need to change the place where the file gets renamed so that it checks/mkdirs the history/ directory first.

        Show
        Todd Lipcon added a comment - Hi Devaraj. This patch seems to address the case where the "history" location is a file at startup. But, if I understand Sammer correctly, the problem here is when users do something like rm -Rf the history/ directory while the JT is running. Then, next time (and all future times) that a file is moving into the history dir, it ends up replacing this file. So, I think we need to change the place where the file gets renamed so that it checks/mkdirs the history/ directory first.
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Devaraj K added a comment -

        Hi Todd,
        I don't see the code where it overwrites the history file.

        JobHistory.java
        
              FileSystem fs = logDir.getFileSystem(conf);
              if (!fs.exists(logDir)){
                if (!fs.mkdirs(logDir, new FsPermission(HISTORY_DIR_PERMISSION))) {
                  throw new IOException("Mkdirs failed to create " + logDir.toString());
                }
              }
              conf.set("hadoop.job.history.location", LOG_DIR);
        

        At the time initialization, it checks for the history if it doesn't present it will create history directory. It doesn't check existing history file is directory or not. If it is not a directory it will fail in the below line while creating a log file after job submission.

        JobHistory.java
        
           
           out = fs.create(logFile, 
                                    new FsPermission(HISTORY_FILE_PERMISSION),
                                    true, 
                                    defaultBufferSize, 
                                    fs.getDefaultReplication(), 
                                    jobHistoryBlockSize, null);
        

        With the above line of code, if the history directory doesn't exist it will create. If the existing history file is not a directory it fails saying "java.io.IOException: Mkdirs failed to create".

        Show
        Devaraj K added a comment - Hi Todd, I don't see the code where it overwrites the history file. JobHistory.java FileSystem fs = logDir.getFileSystem(conf); if (!fs.exists(logDir)){ if (!fs.mkdirs(logDir, new FsPermission(HISTORY_DIR_PERMISSION))) { throw new IOException( "Mkdirs failed to create " + logDir.toString()); } } conf.set( "hadoop.job.history.location" , LOG_DIR); At the time initialization, it checks for the history if it doesn't present it will create history directory. It doesn't check existing history file is directory or not. If it is not a directory it will fail in the below line while creating a log file after job submission. JobHistory.java out = fs.create(logFile, new FsPermission(HISTORY_FILE_PERMISSION), true , defaultBufferSize, fs.getDefaultReplication(), jobHistoryBlockSize, null ); With the above line of code, if the history directory doesn't exist it will create. If the existing history file is not a directory it fails saying "java.io.IOException: Mkdirs failed to create".
        Owen O'Malley made changes -
        Fix Version/s 0.20.4 [ 12316034 ]
        Arun C Murthy made changes -
        Fix Version/s 0.24.0 [ 12317654 ]
        Fix Version/s 0.23.0 [ 12315570 ]
        Devaraj K made changes -
        Assignee Devaraj K [ devaraj.k ]

          People

          • Assignee:
            Unassigned
            Reporter:
            E. Sammer
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development