Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-657

CompletedJobStatusStore hardcodes filesystem to hdfs

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      CompletedJobStatusStore was hardcored to persist to hdfs. This patch allows to persist to local fs. Just qualify mapred.job.tracker.persist.jobstatus.dir with file://

      Description

      Today, completedjobstatusstore stores only to hdfs. It should be configurable to write to local-fs too.

      1. MAPREDUCE-657-v1.2-branch-0.20.patch
        4 kB
        Amar Kamat
      2. MAPREDUCE-657-v1.2.patch
        10 kB
        Amar Kamat
      3. MAPREDUCE-657-v1.0.patch
        5 kB
        Amar Kamat

        Activity

        Amar Kamat created issue -
        Hide
        Amar Kamat added a comment -

        Attaching a early version of the patch. Added a testcase. In this patch the testcase is an end-to-end testcase. I have added a testcase in TestJobStatusPersistency. WIll file a jira to make TestJobStatusPersistency a unit test. Running test-patch.

        Show
        Amar Kamat added a comment - Attaching a early version of the patch. Added a testcase. In this patch the testcase is an end-to-end testcase. I have added a testcase in TestJobStatusPersistency. WIll file a jira to make TestJobStatusPersistency a unit test. Running test-patch.
        Amar Kamat made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-657-v1.0.patch [ 12411642 ]
        Hide
        Amar Kamat added a comment -

        Result of test-patch
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        The findbugs is because of the static clock variable set by JobTracker constructor. Running ant test now.

        Show
        Amar Kamat added a comment - Result of test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. The findbugs is because of the static clock variable set by JobTracker constructor. Running ant test now.
        Hide
        Amar Kamat added a comment -

        All ant test passed on my box except TestJobTrackerRestartWithLostTracker FAILED (timeout), TestKillSubProcesses FAILED, TestNodeRefresh FAILED (timeout), TestReduceFetch FAILED. All of these are known issues. On contrib the following testcase failed TestStreamingBadRecords FAILED (timeout), TestStreamingExitStatus FAILED, TestStreamingStderr FAILED (timeout), TestQueueCapacities FAILED (timeout). All of these are also known issues.

        Show
        Amar Kamat added a comment - All ant test passed on my box except TestJobTrackerRestartWithLostTracker FAILED (timeout), TestKillSubProcesses FAILED, TestNodeRefresh FAILED (timeout), TestReduceFetch FAILED. All of these are known issues. On contrib the following testcase failed TestStreamingBadRecords FAILED (timeout), TestStreamingExitStatus FAILED, TestStreamingStderr FAILED (timeout), TestQueueCapacities FAILED (timeout). All of these are also known issues.
        Hide
        Amar Kamat added a comment -

        Attaching a new patch that changes the call from jobtracker.getClock() to JobTracker.getClock() as getClock() is now a static api.

        Show
        Amar Kamat added a comment - Attaching a new patch that changes the call from jobtracker.getClock() to JobTracker.getClock() as getClock() is now a static api.
        Amar Kamat made changes -
        Attachment MAPREDUCE-657-v1.2.patch [ 12411900 ]
        Hide
        Amar Kamat added a comment -

        Result of test-patch
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        Amar Kamat added a comment - Result of test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Amar Kamat added a comment -

        Attaching a patch for branch 20.

        Show
        Amar Kamat added a comment - Attaching a patch for branch 20.
        Amar Kamat made changes -
        Attachment MAPREDUCE-657-v1.2-branch-0.20.patch [ 12411909 ]
        Hide
        Sharad Agarwal added a comment -

        I committed this to trunk and branch 0.20.

        Show
        Sharad Agarwal added a comment - I committed this to trunk and branch 0.20.
        Sharad Agarwal made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.20.1 [ 12314047 ]
        Resolution Fixed [ 1 ]
        Amar Kamat made changes -
        Release Note CompletedJobStatusStore was hardcored to persist to hdfs. This patch allows to persist to local fs.
        Amar Kamat made changes -
        Release Note CompletedJobStatusStore was hardcored to persist to hdfs. This patch allows to persist to local fs. CompletedJobStatusStore was hardcored to persist to hdfs. This patch allows to persist to local fs. Just qualify mapred.job.tracker.persist.jobstatus.dir with file://
        Amar Kamat made changes -
        Summary CompletedJobStatusStore hardcores filesystem to hdfs CompletedJobStatusStore hardcodes filesystem to hdfs
        Hide
        Ramya Sunil added a comment -

        The above fix does not seem to work. The job info files are not being removed from the local FS after "mapred.job.tracker.persist.jobstatus.hours" . Also "mapred.job.tracker.persist.jobstatus.hours" cannot be configured to a decimal value. Say if it has to be set to 15mins, then it is not possible to do so.

        Show
        Ramya Sunil added a comment - The above fix does not seem to work. The job info files are not being removed from the local FS after "mapred.job.tracker.persist.jobstatus.hours" . Also "mapred.job.tracker.persist.jobstatus.hours" cannot be configured to a decimal value. Say if it has to be set to 15mins, then it is not possible to do so.
        Hide
        Ramya Sunil added a comment -

        The job info files are not being removed from the local FS after "mapred.job.tracker.persist.jobstatus.hours"

        To be more clear, the job info files are not removed immediately after "mapred.job.tracker.persist.jobstatus.hours". Instead they get removed after some random time. So configuring the "mapred.job.tracker.persist.jobstatus.hours" is not behaving as expected.

        Show
        Ramya Sunil added a comment - The job info files are not being removed from the local FS after "mapred.job.tracker.persist.jobstatus.hours" To be more clear, the job info files are not removed immediately after "mapred.job.tracker.persist.jobstatus.hours". Instead they get removed after some random time. So configuring the "mapred.job.tracker.persist.jobstatus.hours" is not behaving as expected.
        Hide
        Iyappan Srinivasan added a comment -

        The job info files are not being removed from the hdfs also after "mapred.job.tracker.persist.jobstatus.hours"
        I set it to 1 hour and it got deleted after 1 hour 57 minutes.

        Show
        Iyappan Srinivasan added a comment - The job info files are not being removed from the hdfs also after "mapred.job.tracker.persist.jobstatus.hours" I set it to 1 hour and it got deleted after 1 hour 57 minutes.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #15 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/15/ )

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Amar Kamat
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development